Mola-maker opened a new pull request, #12267:
URL: https://github.com/apache/arrow-rs/pull/12267
- Closes #12248
# Rationale for this change
A `RunEndEncoded` array written with `MetadataVersion::V4` does not read
back. `has_validity_bitmap` in `arrow-ipc/src/writer.rs` excluded only `Null`
in the V4 branch, so the writer emitted a validity buffer for RunEndEncoded
arrays (a 0-length one when `null_count` is 0). The reader never consumes a
validity buffer for RunEndEncoded in any metadata version, so it takes that
buffer as the run-ends buffer and every later buffer is off by one. The reader
fails with `InvalidArgumentError("Need at least 8 bytes in buffers[0] in array
of type Int32, but got 0")`.
# What changes are included in this PR?
- `has_validity_bitmap` now also excludes `DataType::RunEndEncoded(_, _)` in
the V4 branch, matching the V5 branch, the reader, and the function's own doc
comment (which already stated RunEndEncoded has no validity bitmap). The
flatbuffers spec lists only Union's validity bitmap as a V4 to V5 difference,
so no V4 reader expects one.
- One regression test `test_roundtrip_stream_run_array_v4` in
`arrow-ipc/src/reader.rs`, following the issue reproducer exactly (Int32
run-ends `[2, 5]`, values `[10, 20]`, written with `MetadataVersion::V4`, read
back and asserted equal).
# Are these changes tested?
Yes. The new test fails on current main with the exact error from the issue
and passes with the fix. The full `arrow-ipc` suite (167 tests) and the
downstream `arrow-array` and `arrow-data` suites (1068 tests) pass, and `cargo
fmt` and `cargo clippy` are clean.
# Are there any significant user-facing changes?
Behavior fix only: RunEndEncoded arrays written with MetadataVersion::V4 now
read back correctly. No public API change.
# AI usage disclosure
AI assistance was used to implement the fix and tests, following the
approach described in #12248; the changes were verified with the crate's test
suite, clippy and fmt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]