subotac opened a new pull request, #10504:
URL: https://github.com/apache/arrow-rs/pull/10504
# Which issue does this PR close?
- Closes #10489.
# Rationale for this change
The byte-budget chunker estimates byte-array batches using their plain
encoded size. For `DELTA_BYTE_ARRAY`, this splits repeated oversized values
across pages and resets the prefix state, increasing the
output from one large value plus small deltas to one full value per page
# What changes are included in this PR?
Use the active byte-array encoder state when sizing a mini-batch. After an
oversized first delta value, subsequent prefix-compressed values can remain in
the page while their suffixes fit within one
additional page budget. Incompressible oversized values remain split
across pages
# Are these changes tested?
Yea New Arrow writer layout tests cover both repeated and distinct 64 KiB
values with a 16 KiB page limit. The focused layout suite, full `parquet`
suite, package/workspace clippy, rustdoc, rustfmt, and
diff checks pass
# Are there any user-facing changes?
Parquet files using `DELTA_BYTE_ARRAY` no longer lose prefix compression
solely because individual values exceed the configured page size limit There
are no public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
