[
https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15682025#comment-15682025
]
Rajesh Balamohan commented on TEZ-3212:
---------------------------------------
Even for a key/value size 1.3 GB, it would have thrown
{{NegativeArrayException}} in earlier case, as {{valBytes}} was trying to
allocate 2x the capacity.
With the patch, it would restrict the allocation to {{Integer.MAX_VALUE}}. In
such instances (i.e 2 GB), should {{valBytes}} allocation be restricted only to
current key/value length? (i.e {{Math.min(currentKeyLength,
MAX_ARRAY_LENGTH)}}. For e.g, in corner cases, it is possible that JVM was
able to allocate 1.4 GB instead of 2 GB and the min condition will help in such
instances.
It could throw OOM, if it is unable to allocate the space. But that is expected
behavior.
> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---------------------------------------------------------------------------
>
> Key: TEZ-3212
> URL: https://issues.apache.org/jira/browse/TEZ-3212
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was
> encountered with a job whose IFile record values (which can be of max size
> 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
