[
https://issues.apache.org/jira/browse/MAPREDUCE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750746#comment-17750746
]
ASF GitHub Bot commented on MAPREDUCE-7446:
-------------------------------------------
p-szucs opened a new pull request, #5919:
URL: https://github.com/apache/hadoop/pull/5919
(cherry picked from commit aae5527c9a8d01cfa337141249652ee8a32c8ab2)
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
We are using bit shifting to double the byte array in IFile's nextRawValue
method to store the byte values in it. With large dataset it can easily happen
that we shift the leftmost bit when we are calculating the size of the array,
which can lead to a negative number as the array size, causing the
NegativeArraySizeException.
It would be safer to expand the backing array with a 1.5x factor, and have a
check not to extend Integer's max value during that.
### How was this patch tested?
Tested on cluster.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> NegativeArraySizeException when running MR jobs with large data size
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-7446
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7446
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Peter Szucs
> Assignee: Peter Szucs
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We are using bit shifting to double the byte array in IFile's
> [nextRawValue|https://github.infra.cloudera.com/CDH/hadoop/blob/bef14a39c7616e3b9f437a6fb24fc7a55a676b57/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java#L437]
> method to store the byte values in it. With large dataset it can easily
> happen that we shift the leftmost bit when we are calculating the size of the
> array, which can lead to a negative number as the array size, causing the
> NegativeArraySizeException.
> It would be safer to expand the backing array with a 1.5x factor, and have a
> check not to extend Integer's max value during that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
