raghavyadav01 commented on code in PR #27484:
URL: https://github.com/apache/datafusion/pull/27484#discussion_r4593945123
##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -3052,6 +3054,11 @@ fn concat_build_batches(
metrics.build_mem_used.add(copy_size);
let batch = concat_batches(schema, batches.iter())?;
+ // #26606: a concatenated build batch can carry one data buffer per input
batch for
+ // Utf8View/BinaryView columns; a downstream join then retains and
re-counts every one,
+ // blowing up build memory. Compact over-fragmented view columns so the
retained size is
+ // both smaller and accurately accounted below.
+ let batch = gc_oversized_view_columns(batch);
Review Comment:
Should the `gc()` copy be reserved before it happens?
`estimate_concat_allocation` assumes the view buffers stay shared, so on a big
build side the copy is made first and only the `try_grow` below catches it.
##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -3070,6 +3077,62 @@ fn concat_build_batches(
Ok(batch)
}
+/// Minimum average rows-per-data-buffer below which a view column is
compacted by
+/// [`gc_oversized_view_columns`]. Concatenating the build side can leave a
`Utf8View`/
+/// `BinaryView` column with roughly one data buffer per input batch; a
downstream join then
+/// retains and re-counts every one (DataFusion #26606: build `output_bytes`
reaching hundreds
+/// of GB and the memory metric over-counting). Compacting once the average
drops below this
+/// many rows per buffer keeps the fully-inlined / already-compact cases on
the zero-copy path.
+const VIEW_GC_MIN_ROWS_PER_BUFFER: usize = 4;
+
+/// Returns `batch` with any over-fragmented `Utf8View`/`BinaryView` columns
rewritten into a
+/// single compact buffer via [`GenericByteViewArray::gc`]. The schema and row
count are
+/// unchanged; only such columns' backing buffers differ (dropping the shared
input buffers, so
+/// the retained size shrinks and is counted accurately). A batch with no
over-fragmented view
+/// column is returned untouched (no allocation).
+fn gc_oversized_view_columns(batch: RecordBatch) -> RecordBatch {
+ let num_rows = batch.num_rows();
+ let mut compacted = false;
+ let columns = batch
+ .columns()
+ .iter()
+ .map(|column| match gc_view_column(column, num_rows) {
+ Some(gc) => {
+ compacted = true;
+ gc
+ }
+ None => Arc::clone(column),
+ })
+ .collect::<Vec<_>>();
+ if !compacted {
+ return batch;
+ }
+ // Same schema and row count; only some view columns' buffers changed.
+ RecordBatch::try_new(batch.schema(), columns)
+ .expect("gc_oversized_view_columns preserves schema and row count")
Review Comment:
nit: could this be `?` instead of `expect`, since the function already
returns `Result`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]