adriangb opened a new issue, #12248:
URL: https://github.com/apache/arrow-rs/issues/12248
**Describe the bug**
An IPC stream or file that contains a `RunEndEncoded` array and is written
with `MetadataVersion::V4` does not read back. The reader fails with `Need at
least 8 bytes in buffers[0] in array of type Int32, but got 0`.
**To Reproduce**
Commit `df096d7cf042b645a7e35759ca865bedbd98ea92` (main). Dependencies:
`arrow-array`, `arrow-ipc` and `arrow-schema` from that commit.
```rust
use std::sync::Arc;
use arrow_array::types::Int32Type;
use arrow_array::{Array, Int32Array, RecordBatch, RunArray};
use arrow_ipc::MetadataVersion;
use arrow_ipc::reader::StreamReader;
use arrow_ipc::writer::{IpcWriteOptions, StreamWriter};
use arrow_schema::{Field, Schema};
fn main() {
let run_ends = Int32Array::from(vec![2, 5]);
let values = Int32Array::from(vec![10, 20]);
let array = RunArray::<Int32Type>::try_new(&run_ends, &values).unwrap();
let schema = Schema::new(vec![Field::new("a", array.data_type().clone(),
false)]);
let batch = RecordBatch::try_new(Arc::new(schema),
vec![Arc::new(array)]).unwrap();
let options = IpcWriteOptions::try_new(8, false,
MetadataVersion::V4).unwrap();
let mut bytes = vec![];
let mut writer =
StreamWriter::try_new_with_options(&mut bytes, &batch.schema(),
options).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
drop(writer);
let mut reader = StreamReader::try_new(bytes.as_slice(), None).unwrap();
let read = reader.next().unwrap();
println!("{read:?}");
assert_eq!(read.unwrap(), batch);
}
```
**Expected behavior**
The batch reads back equal to the input, or the writer returns an error for
`RunEndEncoded` with metadata V4.
**Actual behavior**
```
Err(InvalidArgumentError("Need at least 8 bytes in buffers[0] in array of
type Int32, but got 0"))
```
**Additional context**
`has_validity_bitmap` makes the writer emit a validity buffer for every type
except `Null` when the metadata version is below V5:
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L2261-L2270
The reader never reads a validity buffer for `RunEndEncoded`, in any
metadata version:
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/reader.rs#L143-L157
The reader therefore takes the extra buffer as the run ends buffer, and
every later buffer is off by one. With `DataType::RunEndEncoded(_, _)` added to
the V4 branch of `has_validity_bitmap`, the reproducer reads back the input.
The format adds run-end encoding in version 1.3, so rejecting it for V4 in the
writer is another option.
AI-assisted, verified by running the reproducer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]