etseidl opened a new pull request, #12369:
URL: https://github.com/apache/arrow-rs/pull/12369

   # Which issue does this PR close?
   
   - Closes #6255
   - Part of #12325
   
   # Rationale for this change
   
   See #12325 for the rationale.
   
   This is the file-read and metadata portion of the Parquet hardening audit. 
Separate serialized page traversal and range-filter arithmetic changes were 
split out due to size and will be submitted as a follow-up.
   
   # What changes are included in this PR?
   
   - Check `Bytes` read offsets and ranges before conversion or slicing. Bound 
larger `File` read reservations by the available file suffix, without an 
additional file-length query for requests of at most 64 KiB.
   - Reject negative data-page offsets, present dictionary-page offsets, and 
compressed column sizes when admitting file-derived metadata. Check visible 
encrypted-column stubs before retaining them and reconstructed metadata after 
decryption, without requiring otherwise unnecessary column keys.
   - Check footer-length addition in synchronous and both asynchronous 
metadata-fetch paths.
   - Check file row-count accumulation and row-group compressed-size totals 
during metadata serialization.
   - Reject present column-index `nan_counts` lists whose lengths differ from 
the page count.
   
   The public column metadata builder, `byte_range()` getter, and row-group 
`compressed_size()` getter are unchanged. The #6255 repair is at file-metadata 
admission, not a new invariant for caller-built metadata. File reservation 
checks are not a total memory limit.
   
   # Are these changes tested?
   
   The PR includes the existing audit regression tests for oversized 
`File`/`Bytes` ranges. Existing metadata, writer, column-index, and encryption 
tests exercise valid paths. On this standalone split snapshot, formatting, 
spelling, scoped all-feature Clippy, and the all-feature Parquet suite pass: 
2,293 passed, 0 failed, 8 ignored.
   
   Private debug/release probes were rerun against this split for ordinary 
negative column ranges and 120 encrypted cases per profile (27 accepted, 93 
rejected), including missing-key retention and decrypted replacements. Earlier 
private probes cover footer and aggregate arithmetic. The proposed shared #6255 
fixture is deferred pending maintainer discussion; no new fixture, submodule 
update, or additional permanent test is included.
   
   Prior repeated paired benchmarks exercised these file-read and metadata 
changes as part of the combined patch, using a pinned unchanged baseline and 
matching toolchain/features/harness. Some cases measured slower amid 
substantial host noise. The standalone split has not been rebenchmarked; no 
equivalence or speed-improvement claim is made. No benchmark scaffolding is 
included.
   
   # Are there any significant user-facing changes?
   
   No new public API or changed public signature. Selected invalid read ranges 
and malformed metadata now return errors instead of panicking or wrapping. 
Valid-file and encrypted missing-key behavior are preserved.
   
   # AI usage disclosure
   
   AI tools assisted with the source audit, implementation, private validation 
harnesses, review, split, and this draft. The author is responsible for 
reviewing and supporting the submitted changes and evidence.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to