mohitgurav20 opened a new pull request, #12148:
URL: https://github.com/apache/arrow-rs/pull/12148
# Which issue does this PR close?
- Closes #12105
<!-- NOTE: switch to "Relates to #12105" if the maintainer prefers the
breaking
CompressionCodec variant (Option B in the issue discussion) instead. -->
# Rationale for this change
#12084 pinned the reusable LZ4 frame encoder to a 256 KiB block size so a
reused
encoder's block size is not decided by the first buffer it ever writes.
Callers who
want to trade retained encoder buffers against compression ratio currently
have no
way to choose a different size. This lets callers configure the block size
while
keeping 256 KiB as the default, so existing output is unchanged.
# What changes are included in this PR?
- `IpcWriteOptions::try_with_lz4_block_size(Option<usize>)`, following the
existing
`try_with_compression_level` pattern: private field plus a validating
builder.
It accepts 64 KiB / 256 KiB / 1 MiB / 4 MiB (the sizes the LZ4 frame
header can
describe), requires metadata V5 like the other compression settings, and
rejects
a block size when ZSTD is selected.
- `IpcWriteContext` stores the configured size and drops its cached encoder
when
the size changes so the next compression call rebuilds it; the size is
synced
from the write options in `encode_record_batch_data`, the shared path for
record
batches and dictionary batches.
- `Lz4Compressor` remembers its block size so the existing mid-frame error
recovery
rebuilds the same configuration.
- 8 MiB is deliberately not supported: the frame header's block size field
is 3
bits with the codes 4 to 7, and `lz4_flex`'s `BlockSize::Max8MB` sets the
reserved bit, which its own decoder rejects.
# Are these changes tested?
- Frame header level: each supported size produces the expected block size
descriptor code (4/5/6/7) and roundtrips, an explicitly configured 256 KiB
is
byte identical to the untouched default, and switching sizes on a reused
context
still roundtrips (the encoder reset path).
- Options level: unsupported sizes (including 0 and 8 MiB), metadata V4, and
ZSTD return `InvalidArgumentError` instead of panicking or being silently
ignored.
- End to end: a batch with nulls and a dictionary column roundtrips through
`StreamWriter`/`StreamReader` at every supported size, and comparing the
encoded
outputs shows the configured size reaches the encoder while the default
output
stays byte identical.
# Are there any significant user-facing changes?
No - the API addition is purely additive and default behaviour, including the
encoded bytes, is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
