LuciferYang opened a new pull request, #8923: URL: https://github.com/apache/hadoop/pull/8923
### Description of PR JIRA: https://issues.apache.org/jira/browse/HADOOP-20060 When a `.crc` file exists but its header cannot be read (zero length, truncated or bad magic), `ChecksumFSInputChecker` logs a warning and reads the data without verification. `readVectored()` only hands the ranges to the data stream when `sums == null`, which is not the case here, so it reads checksum ranges from the broken file with `bytesPerSum` = 1 and fails with `EOFException: Read past End of File`. `hasCapability(VECTOREDIO_BUFFERS_SLICED)` also returns true. A zero-length `.crc` is what a writer leaves when it dies before closing, and Parquet 1.16 uses vectored reads by default. This patch makes `dataStreamToHandleVectorIO()` also delegate when checksums are not being verified (`!needChecksum()`), as its javadoc already describes. `ChecksumFileSystem#open` only builds this checker when checksum verification is on, so in practice the new branch is taken only when the `.crc` header could not be read. ### How was this patch tested? Added `TestLocalFSContractVectoredRead#testVectoredReadWithCorruptChecksumHeader` (4 bytes of garbage) and `#testVectoredReadWithEmptyChecksumFile` (zero-length `.crc`). Both read the file normally, then check that the stream does not report sliced vectored buffers and that `readVectored` returns the right data. Without the fix both fail on the capability check, and the vectored read fails with `EOFException`. ### For code changes: - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? ### AI Tooling - [x] The PR includes the phrase "Contains content generated by Claude Opus". - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html Contains content generated by Claude Opus. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
