adriangb opened a new pull request, #12190: URL: https://github.com/apache/arrow-rs/pull/12190
# Which issue does this PR close? - Related to https://github.com/apache/arrow-rs/issues/7186 **Stacked on https://github.com/apache/arrow-rs/pull/12184.** GitHub cannot stack a PR on another PR from a fork, so this PR targets `main`. Please review only the 5 commits after the `minify` commits: 1. refactor: extract the `minify` row selection decisions into helpers (no behavior change) 2. feat: add `compacted_size`, `row_sizes` and `split_by_size` 3. bench: compare `compacted_size` and `row_sizes` with `minify` 4. test: cover gaps that mutation testing found 5. perf: build list child row size prefix sums in one buffer # Rationale for this change Callers often need the size of an array after `minify` without paying for the copy. Examples are splitting a large batch into pieces under a byte limit before writing it, and knowing when an output builder has collected enough rows. `get_slice_memory_size` cannot answer this. It counts list children, list view children, dense union children, run-end encoded values, dictionary values and view data buffers in full. # What changes are included in this PR? Three functions in `arrow_select::minify`: - `compacted_size(&dyn Array) -> Result<usize, ArrowError>` returns the exact buffer bytes that `minify` produces, without copying. - `row_sizes(&dyn Array, &mut [u64]) -> Result<(), ArrowError>` adds the compacted size of each row into the slice. A caller can sum all columns of a batch into one slice. - `split_by_size(&[u64], max_size) -> Vec<Range<usize>>` splits rows greedily into ranges under `max_size`. A row larger than the limit gets its own range. `minify` and the size functions share the helpers that decide which rows and bytes are used, so they cannot disagree. `row_sizes` charges shared data (dictionary values, run-end encoded runs, shared dense union child rows, overlapping list views, shared view bytes) to every row that references it. So a split by bytes never underestimates. It can overestimate for long runs or for large dictionary values that many rows reference. The doc comment of `row_sizes` states the bound. # Are these changes tested? Yes. Unit tests check exact values. A seeded random test over every supported type checks that `compacted_size(a)` equals the buffer bytes of `minify(a)`, and that the sum of `row_sizes` over a range stays within the documented bound of `compacted_size` of that slice. Fuzzing, longer property runs and mutation testing were run outside the repository, and their findings are in the unit tests. # Are there any significant user-facing changes? New public API only: `compacted_size`, `row_sizes` and `split_by_size` in `arrow_select::minify`. # AI usage disclosure Claude Code wrote the implementation, tests and this description under my direction. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
