Mola-maker opened a new pull request, #12267:
URL: https://github.com/apache/arrow-rs/pull/12267

   - Closes #12248
   
   # Rationale for this change
   
   A `RunEndEncoded` array written with `MetadataVersion::V4` does not read 
back. `has_validity_bitmap` in `arrow-ipc/src/writer.rs` excluded only `Null` 
in the V4 branch, so the writer emitted a validity buffer for RunEndEncoded 
arrays (a 0-length one when `null_count` is 0). The reader never consumes a 
validity buffer for RunEndEncoded in any metadata version, so it takes that 
buffer as the run-ends buffer and every later buffer is off by one. The reader 
fails with `InvalidArgumentError("Need at least 8 bytes in buffers[0] in array 
of type Int32, but got 0")`.
   
   # What changes are included in this PR?
   
   - `has_validity_bitmap` now also excludes `DataType::RunEndEncoded(_, _)` in 
the V4 branch, matching the V5 branch, the reader, and the function's own doc 
comment (which already stated RunEndEncoded has no validity bitmap). The 
flatbuffers spec lists only Union's validity bitmap as a V4 to V5 difference, 
so no V4 reader expects one.
   - One regression test `test_roundtrip_stream_run_array_v4` in 
`arrow-ipc/src/reader.rs`, following the issue reproducer exactly (Int32 
run-ends `[2, 5]`, values `[10, 20]`, written with `MetadataVersion::V4`, read 
back and asserted equal).
   
   # Are these changes tested?
   
   Yes. The new test fails on current main with the exact error from the issue 
and passes with the fix. The full `arrow-ipc` suite (167 tests) and the 
downstream `arrow-array` and `arrow-data` suites (1068 tests) pass, and `cargo 
fmt` and `cargo clippy` are clean.
   
   # Are there any significant user-facing changes?
   
   Behavior fix only: RunEndEncoded arrays written with MetadataVersion::V4 now 
read back correctly. No public API change.
   
   # AI usage disclosure
   
   AI assistance was used to implement the fix and tests, following the 
approach described in #12248; the changes were verified with the crate's test 
suite, clippy and fmt.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to