[
https://issues.apache.org/jira/browse/HADOOP-18896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098120#comment-18098120
]
ASF GitHub Bot commented on HADOOP-18896:
-----------------------------------------
anshuksi282-ksolves opened a new pull request, #8622:
URL: https://github.com/apache/hadoop/pull/8622
## Description of PR
This PR fixes HADOOP-18896.
The buffer size in `FSOutputSummer` is computed as
`sum.getBytesPerChecksum() * BUFFER_NUM_CHUNKS`. When
`file.bytes-per-checksum` is configured to a very large value (e.g.
238609295), this multiplication overflows a signed 32-bit int and wraps around
to a negative number, causing a `NegativeArraySizeException` when allocating
the internal buffer array.
This PR adds a `Preconditions.checkArgument` check in the `FSOutputSummer`
constructor to validate that the computed buffer size is positive, failing fast
with a clear `IllegalArgumentException` instead of an obscure
`NegativeArraySizeException`.
This continues the work started in #6064 by @teamconfx, which went stale.
This PR additionally adds a unit test, as the original PR's `test4tests` check
had failed for not including one.
### How was this patch tested?
- Added a new test class `TestFSOutputSummer` in `hadoop-common` with
`testLargeBytesPerChecksumOverflow`, using a minimal concrete subclass of
`FSOutputSummer` to verify the constructor fails with
`IllegalArgumentException` (rather than `NegativeArraySizeException`) when
`bytesPerChecksum * BUFFER_NUM_CHUNKS` overflows.
- Verified the test fails with the original `NegativeArraySizeException`
when the fix is reverted, and passes with the fix applied.
- `mvn -pl hadoop-common-project/hadoop-common test
-Dtest=TestFSOutputSummer` passes.
Note: the JIRA's original reproduction steps reference an HDFS test
(`TestDecommissionWithStriped`), but since the buffer overflow bug lives
entirely in `hadoop-common`'s `FSOutputSummer` class, this PR adds a focused
unit test in `hadoop-common` instead, avoiding an unnecessary cross-module HDFS
test dependency.
### AI Tooling
Contains content generated by Claude (Anthropic). Used to help investigate
the root cause, draft the fix, and write/verify the accompanying unit test.
> NegativeArraySizeException thrown in FSOutputSummer.java given large
> file.bytes-per-checksum
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-18896
> URL: https://issues.apache.org/jira/browse/HADOOP-18896
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.3.6
> Reporter: rstest
> Priority: Critical
> Labels: pull-request-available
>
> Buffer size of FSOutputSummer equals to `file.bytes-per-checksum` times
> `BUFFER_NUM_CHUNKS`. A large `file.bytes-per-checksum` causes buffer size to
> overflow and crash with NegativeArraySizeException.
> To reproduce:
> 1. set `file.bytes-per-checksum` to 238609295
> 2. `mvn surefire:test
> -Dtest=org.apache.hadoop.hdfs.TestDecommissionWithStriped#testFileSmallerThanOneStripe`
> We created a PR that provides a fix which checks the buffer size is positive
> after multiplying `file.bytes-per-checksum` with `BUFFER_NUM_CHUNKS`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
