JoeryH opened a new issue, #12305:
URL: https://github.com/apache/arrow-rs/issues/12305
**Describe the bug**
`BatchCoalescer`'s StringView/BinaryView path (`coalesce/byte_view.rs`)
allocates the blocks it copies strings into from a `BufferSource` whose size
doubles on every new block (8 KiB → 1 MiB) and is never reset.
`InProgressByteViewArray::finish()` resets the size counters but keeps
`buffer_source`, so once a coalescer has emitted a few batches, **every later
output batch gets at least one 1 MiB block**, no matter how few string bytes it
holds.
`get_array_memory_size()` reports buffer capacity, so a long-running
coalescer reports far more memory than it stores, and the per-row figure grows
the longer the coalescer runs. Most of the extra capacity is never touched, so
RSS stays much lower than what is accounted.
**To Reproduce**
Arrow 60.1.0 (same code in 59.x):
```rust
use std::sync::Arc;
use arrow::array::{Array, RecordBatch, StringViewArray, UInt32Array};
use arrow::compute::{take, BatchCoalescer};
use arrow::datatypes::{DataType, Field, Schema};
fn main() {
// One large source array whose strings live in shared data buffers
(like a
// decoded parquet batch); 10% of the strings are longer than 12 bytes.
let n = 1_000_000;
let strings: Vec<String> = (0..n)
.map(|i| if i % 10 == 0 { format!("long-string-{i:012}") } else {
format!("s{}", i % 1000) })
.collect();
let source = StringViewArray::from_iter_values(strings.iter());
let schema = Arc::new(Schema::new(vec![Field::new("s",
DataType::Utf8View, false)]));
// Like a hash repartitioner: push small takes of the source into one
coalescer,
// so each pushed array barely uses the buffers it points into and gets
gc'd (copied).
let mut coalescer = BatchCoalescer::new(schema.clone(), 8192);
let mut out = 0;
for start in (0..n).step_by(160) {
let idx = UInt32Array::from_iter_values(start as u32..(start +
160).min(n) as u32);
let col = take(&source, &idx, None).unwrap();
coalescer.push_batch(RecordBatch::try_new(schema.clone(),
vec![col]).unwrap()).unwrap();
while let Some(batch) = coalescer.next_completed_batch() {
let a =
batch.column(0).as_any().downcast_ref::<StringViewArray>().unwrap();
let stored: usize = a.data_buffers().iter().map(|b|
b.len()).sum();
let capacity: usize = a.data_buffers().iter().map(|b|
b.capacity()).sum();
if out < 3 || out % 30 == 0 {
println!("batch {out:3}: stored {stored:>8} B capacity
{capacity:>8} B get_array_memory_size {:>8} B",
a.get_array_memory_size());
}
out += 1;
}
}
}
```
Output:
```
batch 0: stored 19680 B capacity 24576 B get_array_memory_size
155760 B
batch 1: stored 19656 B capacity 32768 B get_array_memory_size
163952 B
batch 2: stored 19656 B capacity 65536 B get_array_memory_size
196720 B
batch 30: stored 19680 B capacity 1048576 B get_array_memory_size
1179760 B
batch 60: stored 19680 B capacity 1048576 B get_array_memory_size
1179760 B
batch 90: stored 19680 B capacity 1048576 B get_array_memory_size
1179760 B
```
Identical batches, 7.5x larger after the coalescer warms up. Data buffer
capacity is 53x what is stored.
**Impact in DataFusion**
`RepartitionExec` (hash) runs one coalescer per output partition and feeds
it `take`-ed slices of scan batches, which is exactly this pattern. Partitions
are fed concurrently, so they also end up with many partly filled 256 KiB–1 MiB
blocks. With a hash join building on a repartitioned parquet scan (several
Utf8View columns, 50 partitions), the build side's accounted bytes per row grew
with input size even though the stored string bytes grew linearly:
| input (same files, ×N) | stored string bytes | data buffer capacity |
accounted bytes/row |
|---|---|---|---|
| ×1 | 22 MB | 81 MB | 336 |
| ×2 | 45 MB | 430 MB | 637 |
| ×4 | 90 MB | 1.4 GB | 954 |
| ×8 | 180 MB | 3.6 GB | 1188 |
On real queries the memory pool peak was ~3.5x process RSS, which leads to
spills or `ResourcesExhausted` well before memory is actually short. Switching
the same query to `Utf8` dropped the build from 15.6 GB to 1.6 GB.
**Expected behavior**
An output batch's buffer capacity should be in proportion to the bytes it
holds, and should not depend on how many batches the coalescer has emitted
before.
**Possible fix**
I patched arrow-select locally and measured the ×8 case above (stock 3.94 GB
accounted):
1. Reset `buffer_source` in `InProgressByteViewArray::finish()`
(`self.buffer_source = BufferSource::new();`): **1.53 GB**.
2. Also `shrink_to_fit()` a block in `finish_current()` when it is less than
half full: **1.09 GB**.
3. Also size blocks by need with a floor (`max(min_size, 64 KiB)`) instead
of doubling: **0.76 GB**. Bytes per row stay flat (~230) as input grows,
against ~100 for `Utf8`. The rest is the 16-byte views.
(1) is a one-liner and removes the dependence on history. (2) costs one
realloc per under-filled block. (3) means more, smaller blocks per batch. Happy
to open a PR for whichever you prefer.
**Additional context**
Related but different: #7765, #11280, #11635, apache/datafusion#25712.
#11770 touches the same file for performance and does not change `BufferSource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]