azecevic000 opened a new pull request, #11694: URL: https://github.com/apache/arrow-rs/pull/11694
# Which issue does this PR close? N/A # Rationale for this change The IPC writer writes a validity bitmap for every array that has a validity buffer, also when the array has no nulls. If the array has no null buffer, `write_array_data` creates an all-ones bitmap of `ceil(len / 8)` bytes for it. Readers do not use it: the arrow-rs reader, like the C++ reader, ignores the validity buffer when `null_count` is 0. The format allows a 0-length buffer instead. [`FieldNode.null_count` in `Message.fbs`](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/format/Message.fbs#L39-L41): > Fields with null_count == 0 may choose not to write their physical validity bitmap out as a materialized buffer, instead setting the length of the bitmap buffer to 0. The [C++](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/ipc/writer.cc#L178-L187), [Go](https://github.com/apache/arrow-go/blob/a6e44ad7a6aaa4c02a7d277032af8eebd5cc2508/arrow/ipc/writer.go#L634-L638) and [JS](https://github.com/apache/arrow-js/blob/43c9547baf3766e79eb44d3fedcd0787f899d956/src/visitor/vectorassembler.ts#L95-L98) writers already do this. So arrow-rs already reads such buffers, in the integration tests and in the `arrow-testing` gold files, which C++ wrote. # What changes are included in this PR? - `write_array_data` writes a 0-length validity buffer when `null_count` is 0. This also covers sliced arrays whose null buffer has no nulls in the slice. Arrays with nulls are written as before. - V4 unions keep their validity buffer (#1793), which is now always 0-length, as in the C++ writer. - `test_decimal128_alignment8_is_unaligned` relied on the all-ones bitmap to move the decimal values to an 8-byte offset. It now uses a nullable column with a null, so it gets the expected misalignment error with both the old and the new writer. - The exact overage limits in the arrow-flight `verify_encoded_split` tests go down, for example from 120 to 88. # Are these changes tested? Yes. `test_write_empty_validity_buffer_without_nulls` encodes a batch with three columns: an array without a null buffer, a slice without nulls that keeps its null buffer, and an array with a null. It checks the validity buffer lengths in the encoded message (`[0, 0, 1]`) and the round trip. Without the change, the first two columns get 1-byte bitmaps and the test fails. # Are there any significant user-facing changes? The IPC output for arrays without nulls is smaller. The spec allows this, and the C++, Java, Go, JS, C# and nanoarrow readers accept it. Code that compares IPC output byte for byte, or expects exact sizes, will see a difference. There are no API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
