tomicooler commented on code in PR #5895:
URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276203099
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java:
##########
@@ -433,8 +434,11 @@ public boolean nextRawKey(DataInputBuffer key) throws
IOException {
}
public void nextRawValue(DataInputBuffer value) throws IOException {
+ long targetSizeLong = currentValueLength + (currentValueLength >> 1);
Review Comment:
This overflows like this. Example:
```
class HelloWorld {
public static void main(String[] args) {
int currentValueLength = Integer.MAX_VALUE >> 3;
final int ARRAY_MAX_SIZE = Integer.MAX_VALUE - 8;
for (int i = 0; i < 10; i++) {
long targetSizeLong = currentValueLength + (currentValueLength
>> 1);
int targetSize = (int) Math.min(targetSizeLong, ARRAY_MAX_SIZE);
System.out.println("targetSizeLong: " + targetSizeLong + "
targetSize: " + targetSize);
currentValueLength = targetSize;
}
}
}
```
```
targetSizeLong: 402653182 targetSize: 402653182
targetSizeLong: 603979773 targetSize: 603979773
targetSizeLong: 905969659 targetSize: 905969659
targetSizeLong: 1358954488 targetSize: 1358954488
targetSizeLong: 2038431732 targetSize: 2038431732
targetSizeLong: -1237319698 targetSize: -1237319698
targetSizeLong: -1855979547 targetSize: -1855979547
targetSizeLong: 1510997975 targetSize: 1510997975
targetSizeLong: -2028470334 targetSize: -2028470334
targetSizeLong: 1252261795 targetSize: 1252261795
```
Fix:
```
long targetSizeLong = currentValueLength + (long)(currentValueLength >> 1);
```
With the fix (casting the second part to `long`):
```
targetSizeLong: 402653182 targetSize: 402653182
targetSizeLong: 603979773 targetSize: 603979773
targetSizeLong: 905969659 targetSize: 905969659
targetSizeLong: 1358954488 targetSize: 1358954488
targetSizeLong: 2038431732 targetSize: 2038431732
targetSizeLong: 3057647598 targetSize: 2147483639
targetSizeLong: 3221225458 targetSize: 2147483639
targetSizeLong: 3221225458 targetSize: 2147483639
targetSizeLong: 3221225458 targetSize: 2147483639
targetSizeLong: 3221225458 targetSize: 2147483639
```
BTW allocating 2.14 GB at once, probably will fail anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
