ryux1 opened a new pull request, #24319: URL: https://github.com/apache/datafusion/pull/24319
## Which issue does this PR close? - Closes #24310. ## Rationale for this change Record batch memory accounting runs on hot execution paths. The current implementation materializes ArrayData for every array and allocates a hash set even for small batches, adding measurable overhead to queries that frequently update memory reservations. ## What changes are included in this PR? - Traverse Arrow arrays directly and recursively count their backing buffers without materializing ArrayData. - Track the first 16 buffer identities inline, then promote to a hash set for wider batches or counters spanning many batches. - Preserve shared-buffer deduplication and full buffer-capacity accounting semantics. - Add parity coverage against the previous ArrayData traversal for primitive, binary/view, list/view, fixed-size, struct, union, dictionary, map, and run-end encoded layouts. - Add a focused Criterion benchmark for narrow through wide record batches. Criterion point estimates from cargo bench -p datafusion-common --bench record_batch_memory, measured in isolated target directories on the same machine: | Columns | main | this PR | Speedup | |---:|---:|---:|---:| | 1 | 79.327 ns | 10.412 ns | 7.62x | | 4 | 288.97 ns | 38.157 ns | 7.57x | | 16 | 1.1770 us | 185.09 ns | 6.36x | | 64 | 4.9819 us | 1.5537 us | 3.21x | ## Are these changes tested? Yes. - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - cargo test -p datafusion-common --lib (550 passed) - RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption - Focused Criterion comparison shown above ## Are there any user-facing changes? No API or behavior changes. This reduces CPU and allocation overhead in record batch memory accounting. AI assistance: OpenAI Codex assisted with implementation and test execution. I reviewed the change and its behavior end to end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
