hzuo opened a new pull request, #5554:
URL: https://github.com/apache/arrow-rs/pull/5554
# Which issue does this PR close?
Closes #5553
# Rationale for this change
`FileWriter` and `StreamWriter` should ensure that the data is written with
appropriately alignment such that arrays can be used without copying to a
more-aligned buffer.
In particular, as of Rust 1.77.0 and LLVM 18, `i128` now has a 16-byte
alignment requirement, i.e. `std::mem::align_of::<i128> == 16`. So
`Decimal128Array`s must be aligned to a 16-byte boundary when it serialized
into an IPC buffer. The `pad_to_8` used everywhere in the IPC code causes it to
pad insufficiently.
(copied from issue)
# What changes are included in this PR?
- Make all the IPC code actually respect the `alignment` specified in
`IpcWriteOptions`
- Add new `enforce_zero_copy` option to `FileDecoder`, `read_record_batch`,
and `read_dictionary`, which enforces that the read is a zero-copy read.
Previously if the buffer was unaligned then the code does a memcpy to an
aligned buffer, which users of Arrow as a zero-copy format may not expect / may
want to enforce does not happen.
# Are there any user-facing changes?
Yes:
- `alignment` in `IpcWriteOptions::try_with` is now a u8 inside of a usize
(breaking but minor)
- New `enforce_zero_copy` argument on `read_record_batch` and
`read_dictionary` (breaking but minor)
- New option `with_enforce_zero_copy` to FileDecoder (non-breaking)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
