patrickswedish opened a new pull request, #26402: URL: https://github.com/apache/datafusion/pull/26402
## Which issue does this PR close? - Closes #26381 ## Rationale for this change PartitionedTopKExec keeps whole Utf8View and BinaryView input buffers alive for the rows it retains. When admitted rows are gathered via take_record_batch or compacted via interleave_referenced, the view arrays clone buffer references to the source batches. Retaining a few rows from an 8192-row batch keeps that batch's whole string buffer resident in memory and charged to the memory pool. Calling crate::spill::gc_view_arrays on gathered and compacted batches compacts view buffers down to strictly the retained bytes. ## What changes are included in this PR? - RecordBatchStore::insert_rows: compact view arrays after take_record_batch with gc_view_arrays - RecordBatchStore::compact: compact interleaved chunks with gc_view_arrays - TopKHeap::maybe_compact: compact view arrays on compacted batch with gc_view_arrays - tests: add test_record_batch_store_compacts_view_arrays verifying buffer compaction below initial 12KB buffer ## What is the testing strategy for this PR? - Unit tests: cargo test -p datafusion-physical-plan --lib -- topk - Lints/Format: cargo clippy -p datafusion-physical-plan --lib -- -D warnings && rustfmt --check datafusion/physical-plan/src/topk/mod.rs - Integration tests: cargo test -p datafusion-physical-plan --lib -- topk ## Are there any user-facing changes? None. Internal physical-plan memory compaction preserving exact query results. --- *Diff minimization and pre-flight validation performed with [ZeroSlop™ CI](https://github.com/marketplace/actions/zeroslop-by-ribbsaeter-systems) — [Ribbsaeter Systems](https://ribbsaetersystems.com/).* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
