Tpt opened a new issue, #11792:
URL: https://github.com/apache/arrow-rs/issues/11792
### Describe the bug
`ArrowWriter` only splits an incoming batch on `max_row_group_bytes` when
the in-progress row group already has buffered rows, because it extrapolates
from their average size. The first batch written to a row group is therefore
always accepted whole, and the row group is flushed afterwards.
When a single batch is larger than the limit, every row group ends up
holding exactly one batch, whatever its size. In practice this hits wide rows
(images, embeddings, documents), which is precisely where a byte limit matters
most: we see 128 MB row groups with a 16 MB limit, one per 128-row batch.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::{BinaryArray, RecordBatch};
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;
fn main() {
// 128 rows of 1 MB each
let values: Vec<Vec<u8>> = (0..128u64)
.map(|i| {
let mut v = vec![0u8; 1_000_000];
v[..8].copy_from_slice(&i.to_le_bytes());
v
})
.collect();
let batch = RecordBatch::try_from_iter([(
"b",
Arc::new(BinaryArray::from_iter_values(&values)) as _,
)])
.unwrap();
let props = WriterProperties::builder()
.set_max_row_group_bytes(Some(16_000_000))
.set_dictionary_enabled(false)
.build();
let mut writer = ArrowWriter::try_new(Vec::new(), batch.schema(),
Some(props)).unwrap();
for _ in 0..4 {
writer.write(&batch).unwrap();
}
for rg in writer.close().unwrap().row_groups() {
println!("{} rows, {} bytes", rg.num_rows(), rg.total_byte_size());
}
}
```
Output: four row groups of 128 rows and ~128 MB each.
The same happens through DataFusion with
`datafusion.execution.parquet.max_row_group_bytes` (and
`allow_single_file_parallelism = false`, which it requires): once
`execution.batch_size` rows exceed the limit, row groups are exactly
`batch_size` rows.
### Expected behavior
Row groups of roughly 16 MB (15 rows each here), as when batches are smaller
than the limit: with 100 KB rows the same writer correctly produces ~16 MB
groups of 159 rows.
### Additional context
The relevant check is `if let Some(max_bytes) = self.max_row_group_bytes &&
buffered_rows > 0` in `ArrowWriter::write`. Other projects have hit this
through parquet-rs and worked around it in their own writers, e.g.
https://github.com/apache/paimon-rust/pull/1072 and
https://github.com/duckdb/duckdb/pull/25060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
