adriangb opened a new issue, #12249:
URL: https://github.com/apache/arrow-rs/issues/12249
## Describe the bug
The IPC writer writes a `RunEndEncoded` array with `Utf8View` (or
`BinaryView`) values that is sliced to zero rows, but the reader cannot read it
back. The values must have at least one non-inlined string (longer than 12
bytes), so that they own a data buffer. The same happens when the view array is
nested inside the run values, for example `RunEndEncoded<Int32,
ListView<Utf8View>>` or `RunEndEncoded<Int32, List<Utf8View>>`.
## To Reproduce
Put this file in `arrow-ipc/tests/ree_repro.rs` and run `cargo test -p
arrow-ipc --test ree_repro -- --nocapture`.
```rust
use arrow_array::types::Int32Type;
use arrow_array::*;
use arrow_ipc::{reader::StreamReader, writer::StreamWriter};
use std::sync::Arc;
#[test]
fn repro() {
let values = StringViewArray::from(vec!["a long string that is not
inlined", "b"]);
let ree = RunArray::<Int32Type>::try_new(&Int32Array::from(vec![1, 2]),
&values).unwrap();
let batch = RecordBatch::try_from_iter([("a", Arc::new(ree) as
ArrayRef)])
.unwrap()
.slice(1, 0);
let mut buf = Vec::new();
let mut writer = StreamWriter::try_new(&mut buf,
&batch.schema()).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
drop(writer);
let read = StreamReader::try_new(buf.as_slice(),
None).unwrap().next().unwrap();
println!("{:?}", read.map(|b| b.num_rows()));
}
```
## Expected behavior
The reader returns a batch with 0 rows: `Ok(0)`.
Actual output:
```
Err(IpcError("Invalid variadic count 1 for Utf8View column, with 2 buffer(s)
left in the message"))
```
## Additional context
Version: `main` at
https://github.com/apache/arrow-rs/commit/df096d7cf042b645a7e35759ca865bedbd98ea92
Suspected cause: the buffers and the variadic buffer counts come from two
different arrays.
- For a run array, the writer writes the children of
`unslice_run_array(array_data)`:
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L2667-L2681
- For an empty run array, `into_zero_offset_run_array` returns
`ArrayData::new_empty`, whose `Utf8View` values have no data buffers:
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L1321-L1324
- `append_variadic_buffer_counts` walks the original `array_data`, so it
counts the data buffer of the original values:
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L1200
and
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L1277-L1293
A possible fix is to compute the variadic counts from the same unsliced run
array that the writer writes.
AI-assisted, verified by running the reproducer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]