cetra3 opened a new issue, #11148:
URL: https://github.com/apache/arrow-rs/issues/11148
### Describe the bug
When using ArrowWriter with Parquet 2.0, the default `DELTA_BYTE_ARRAY`
fallback encoder creates two `DeltaBitPackEncoder` instances that each
pre-allocate a 1MiB buffer, which quickly blows up heap space when you have a
lot of columns.
There is currently no configuration option to adjust this behaviour, so if
you have, say, 100 columns, this allocates 200MiB of heap up front, which is
less than ideal.
Alongside this: if you have a dictionary column, that doesn't overflow, the
fallback is never used, and you've allocated 200MiB for nothing.
### To Reproduce
Here's a test that will fail on `main` currently:
```rust
#[test]
fn unused_delta_fallback_does_not_preallocate() {
const COLUMNS: usize = 100;
let column: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "a"]));
let batch = RecordBatch::try_from_iter((0..COLUMNS).map(|i|
(format!("c{i}"), column.clone())))
.unwrap();
let props = WriterProperties::builder()
.set_writer_version(WriterVersion::PARQUET_2_0)
.set_encoding(Encoding::DELTA_BYTE_ARRAY)
.build();
let peak = peak_heap_bytes(|| {
let mut writer = ArrowWriter::try_new(Vec::new(), batch.schema(),
Some(props)).unwrap();
writer.write(&batch).unwrap();
});
assert!(peak < COLUMNS * 64 * 1024, "peak: {peak}");
}
```
You will get the following:
```
thread 'unused_delta_fallback_does_not_preallocate' panicked at
parquet/tests/arrow_writer/mod.rs:589:5:
peak: 210203256
test result: FAILED. 0 passed; 1 failed
```
### Expected behavior
Either:
* We don't have things allocate upfront
* We add a config option to adjust what this upfront allocation is
### Additional context
This was caught in some production code using
[thresher](https://github.com/cetra3/thresher) with jemalloc profiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
