adriangb opened a new issue, #12248:
URL: https://github.com/apache/arrow-rs/issues/12248

   **Describe the bug**
   
   An IPC stream or file that contains a `RunEndEncoded` array and is written 
with `MetadataVersion::V4` does not read back. The reader fails with `Need at 
least 8 bytes in buffers[0] in array of type Int32, but got 0`.
   
   **To Reproduce**
   
   Commit `df096d7cf042b645a7e35759ca865bedbd98ea92` (main). Dependencies: 
`arrow-array`, `arrow-ipc` and `arrow-schema` from that commit.
   
   ```rust
   use std::sync::Arc;
   
   use arrow_array::types::Int32Type;
   use arrow_array::{Array, Int32Array, RecordBatch, RunArray};
   use arrow_ipc::MetadataVersion;
   use arrow_ipc::reader::StreamReader;
   use arrow_ipc::writer::{IpcWriteOptions, StreamWriter};
   use arrow_schema::{Field, Schema};
   
   fn main() {
       let run_ends = Int32Array::from(vec![2, 5]);
       let values = Int32Array::from(vec![10, 20]);
       let array = RunArray::<Int32Type>::try_new(&run_ends, &values).unwrap();
       let schema = Schema::new(vec![Field::new("a", array.data_type().clone(), 
false)]);
       let batch = RecordBatch::try_new(Arc::new(schema), 
vec![Arc::new(array)]).unwrap();
   
       let options = IpcWriteOptions::try_new(8, false, 
MetadataVersion::V4).unwrap();
       let mut bytes = vec![];
       let mut writer =
           StreamWriter::try_new_with_options(&mut bytes, &batch.schema(), 
options).unwrap();
       writer.write(&batch).unwrap();
       writer.finish().unwrap();
       drop(writer);
   
       let mut reader = StreamReader::try_new(bytes.as_slice(), None).unwrap();
       let read = reader.next().unwrap();
       println!("{read:?}");
       assert_eq!(read.unwrap(), batch);
   }
   ```
   
   **Expected behavior**
   
   The batch reads back equal to the input, or the writer returns an error for 
`RunEndEncoded` with metadata V4.
   
   **Actual behavior**
   
   ```
   Err(InvalidArgumentError("Need at least 8 bytes in buffers[0] in array of 
type Int32, but got 0"))
   ```
   
   **Additional context**
   
   `has_validity_bitmap` makes the writer emit a validity buffer for every type 
except `Null` when the metadata version is below V5:
   
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/writer.rs#L2261-L2270
   
   The reader never reads a validity buffer for `RunEndEncoded`, in any 
metadata version:
   
https://github.com/apache/arrow-rs/blob/df096d7cf042b645a7e35759ca865bedbd98ea92/arrow-ipc/src/reader.rs#L143-L157
   
   The reader therefore takes the extra buffer as the run ends buffer, and 
every later buffer is off by one. With `DataType::RunEndEncoded(_, _)` added to 
the V4 branch of `has_validity_bitmap`, the reproducer reads back the input. 
The format adds run-end encoding in version 1.3, so rejecting it for V4 in the 
writer is another option.
   
   AI-assisted, verified by running the reproducer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to