etseidl opened a new pull request, #12369: URL: https://github.com/apache/arrow-rs/pull/12369
# Which issue does this PR close? - Closes #6255 - Part of #12325 # Rationale for this change See #12325 for the rationale. This is the file-read and metadata portion of the Parquet hardening audit. Separate serialized page traversal and range-filter arithmetic changes were split out due to size and will be submitted as a follow-up. # What changes are included in this PR? - Check `Bytes` read offsets and ranges before conversion or slicing. Bound larger `File` read reservations by the available file suffix, without an additional file-length query for requests of at most 64 KiB. - Reject negative data-page offsets, present dictionary-page offsets, and compressed column sizes when admitting file-derived metadata. Check visible encrypted-column stubs before retaining them and reconstructed metadata after decryption, without requiring otherwise unnecessary column keys. - Check footer-length addition in synchronous and both asynchronous metadata-fetch paths. - Check file row-count accumulation and row-group compressed-size totals during metadata serialization. - Reject present column-index `nan_counts` lists whose lengths differ from the page count. The public column metadata builder, `byte_range()` getter, and row-group `compressed_size()` getter are unchanged. The #6255 repair is at file-metadata admission, not a new invariant for caller-built metadata. File reservation checks are not a total memory limit. # Are these changes tested? The PR includes the existing audit regression tests for oversized `File`/`Bytes` ranges. Existing metadata, writer, column-index, and encryption tests exercise valid paths. On this standalone split snapshot, formatting, spelling, scoped all-feature Clippy, and the all-feature Parquet suite pass: 2,293 passed, 0 failed, 8 ignored. Private debug/release probes were rerun against this split for ordinary negative column ranges and 120 encrypted cases per profile (27 accepted, 93 rejected), including missing-key retention and decrypted replacements. Earlier private probes cover footer and aggregate arithmetic. The proposed shared #6255 fixture is deferred pending maintainer discussion; no new fixture, submodule update, or additional permanent test is included. Prior repeated paired benchmarks exercised these file-read and metadata changes as part of the combined patch, using a pinned unchanged baseline and matching toolchain/features/harness. Some cases measured slower amid substantial host noise. The standalone split has not been rebenchmarked; no equivalence or speed-improvement claim is made. No benchmark scaffolding is included. # Are there any significant user-facing changes? No new public API or changed public signature. Selected invalid read ranges and malformed metadata now return errors instead of panicking or wrapping. Valid-file and encrypted missing-key behavior are preserved. # AI usage disclosure AI tools assisted with the source audit, implementation, private validation harnesses, review, split, and this draft. The author is responsible for reviewing and supporting the submitted changes and evidence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
