adriangb opened a new pull request, #11235: URL: https://github.com/apache/arrow-rs/pull/11235
# Which issue does this PR close? - Closes https://github.com/apache/arrow-rs/issues/11233. # Rationale for this change `ParquetPushDecoder` released a pushed buffer only if its range was equal to a requested range. A caller that fetches the requested ranges with one larger request (a common pattern for object storage) kept all bytes until the end of the scan. The reproduction in #11233 shows `buffered_bytes()` increase by one row group for each row group. | After row group | Before this PR | With this PR | |---|---:|---:| | 0 | 195,128 | 0 | | 1 | 390,256 | 0 | | 2 | 585,384 | 0 | | 3 | 780,512 | 0 | # What changes are included in this PR? | Change | Where | |---|---| | `PushBuffers::release_ranges`: remove the buffered bytes in the given ranges, whatever the shape of the pushed buffers. A buffer that overlaps a range is trimmed or split into zero-copy slices. | `util/push_buffers.rs` | | When the decoder is done with a row group (it built the reader, or the row group produced no rows), release all column chunks of that row group, unless the queue reads the row group again. | `push_decoder/remaining.rs`, `push_decoder/reader_builder/mod.rs` | | Document this on `push_range`. | `push_decoder/mod.rs` | The existing exact-match release of requested ranges (`clear_ranges`) does not change. Bytes that are outside the column chunks of the row groups that the decoder reads (for example, a speculative push of the full file) stay buffered, as before. This is also the first step of https://github.com/apache/arrow-rs/issues/11234, which uses `release_ranges`. This PR is useful without the rest of that EPIC. # Are these changes tested? Yes. - `release_ranges_trims_and_splits_buffers`, `release_ranges_removes_exact_and_overlapping_buffers`: unit tests of `release_ranges`. - `test_decoder_releases_coalesced_push`, `test_decoder_releases_unrequested_bytes_of_row_group` (a pushed buffer that also covers a column that is not read), `test_decoder_releases_coalesced_push_with_predicate`: check that `buffered_bytes()` is 0 after each row group. - `test_decoder_clear_all_ranges` now expects the bytes of the first row group to be released after its reader is built. - `test_row_group_local_selections_allow_duplicate_row_groups` (not changed) covers the "reads the row group again" case: it fails if the decoder releases a row group that is queued again. I checked that the 3 new decoder tests and the changed test fail without the fix. # Are there any user-facing changes? `buffered_bytes()` decreases after each row group when the caller pushes buffers that are larger than the requested ranges. There is no API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
