The Buffer struct / metadata need not be a multiple of 8 bytes necessarily
but you must write padding bytes when emitting the IPC protocol. So if your
validity bitmap is 2 bytes in-memory then you must write at least 6 more
bytes of padding on the wire.
On Fri, Apr 26, 2019, 3:48 PM Micah Kornfiel
Hi Neville,
Here is my understanding. Per the spec [1], 8 bytes of padding is
allowed/required but 64 bytes is recommended (Is "bits" in your e-mail is a
typo?). The main rationale is to allow SIMD instructions.
For actual record batches only padding to a multiple of 8-bytes are
required [2].
N
Hi Arrow developers,
I'm currently working on IPC in Rust, specifically reading Arrow files.
I've noticed that null buffers/bitmaps are always padded to 64 bits (from
pyarrow, not sure about others), while in Rust we pad to 8 bits.
1. Is this fine re. Rust per the spec?
I'm having issues with re
