sarankumar1325 opened a new pull request, #11796: URL: https://github.com/apache/arrow-rs/pull/11796
# Which issue does this PR close? - Closes #11792. # Rationale for this change `ArrowWriter::write` only splits a batch on `max_row_group_bytes` when the row group already has buffered rows, because it needs them to measure the average row size. The first batch written to an empty row group is always taken whole, so a batch larger than the limit produces a row group of exactly that batch. # What changes are included in this PR? When the row group is empty and a byte limit is set, `write` now estimates the row size from the batch's in-memory size (`ArrayData::get_slice_memory_size`, which only counts the sliced range) and splits off the rows that fit. Once those rows are buffered, the existing measured-average logic takes over for the rest of the row group, so the estimate only decides the size of the first write into each group. If the estimate is too high, the row group simply keeps filling on the next loop iteration. If it is too low, the group overshoots by at most one write, as it does today. If the slice size can't be computed, the current behaviour is kept. # Are these changes tested? Yes. `test_row_group_limit_bytes_splits_first_batch` writes 100-row batches that are each larger than the limit. On `main` it gets row groups of `[100, 100]`; with this change every group stays near the limit. The existing `arrow_writer` tests pass (106), and `cargo fmt` and `cargo clippy -p parquet --features arrow --lib --tests -D warnings` are clean. With the reproducer from the issue (128 rows of 1 MB, 16 MB limit, four writes), `main` gives four 128 MB row groups. With this change the groups hold 15 rows and about 15 MB each, the same as the issue's expected output. # Are there any user-facing changes? Row groups written with `max_row_group_bytes` now respect the limit when individual batches exceed it. No API changes. AI disclosure: I used an AI coding assistant to help write this change and its test. I reviewed the code, checked the new test fails on `main`, and ran the reproducer from the issue before and after the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
