[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237992#comment-15237992
]
Jonathan Eagles commented on TEZ-3212:
--------------------------------------
This patch is really just to start a discussion on what to do with really large
values. It is arguable that IFile should be able to read what was written.
MAX_ARRAY_LENGTH is due to the limitation where byte array of Integer.MAX_VALUE
cannot be allocated due to jvm limits
https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---------------------------------------------------------------------------
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
