Cintu07 opened a new issue, #11059:
URL: https://github.com/apache/arrow-rs/issues/11059
### Describe the bug
**Describe the bug**
`FixedSizeListArray::value_offset` works the offset out in usize and then
narrows it with `as i32`. Once a row starts past i32::MAX child values it
returns a negative number with no error, while `value(i)` on the same row is
still right because it never leaves usize.
arrow-avro's FixedSizeList encoder reads it as `self.list.value_offset(idx)
as usize` (writer/encoder.rs:2129), so the negative offset turns into an index
near u64::MAX and writing the batch panics.
**To Reproduce**
arrow-array, arrow-buffer, arrow-schema and arrow-avro 59.3.0, arrow-avro
with default-features = false.
```rust
use std::sync::Arc;
use arrow_array::{ArrayRef, BooleanArray, FixedSizeListArray, RecordBatch};
use arrow_avro::writer::AvroWriter;
use arrow_buffer::{BooleanBuffer, NullBuffer};
use arrow_schema::{DataType, Field, Schema};
fn main() {
let size: i32 = 1 << 30;
let values = BooleanArray::new(BooleanBuffer::new_unset(size as usize *
3), None);
let item = Arc::new(Field::new_list_field(DataType::Boolean, false));
let nulls = NullBuffer::from(vec![false, false, true]);
let list = FixedSizeListArray::try_new(item.clone(), size,
Arc::new(values), Some(nulls)).unwrap();
println!("row 2: value_offset = {}, actual offset = {}",
list.value_offset(2), 2 * size as usize);
let schema = Schema::new(vec![Field::new("v",
DataType::FixedSizeList(item, size), true)]);
let batch = RecordBatch::try_new(Arc::new(schema.clone()),
vec![Arc::new(list) as ArrayRef]).unwrap();
let mut writer = AvroWriter::new(std::io::sink(), schema).unwrap();
writer.write(&batch).unwrap();
}
```
```
row 2: value_offset = -2147483648, actual offset = 2147483648
thread 'main' panicked at
arrow-array-59.3.0/src/array/boolean_array.rs:245:9:
Trying to access an element at index 18446744071562067968 from a
BooleanArray of length 3221225472
```
Rows 0 and 1 are null so only row 2 gets encoded. The child is 384 MiB of
booleans, so it runs on a laptop.
**Expected behavior**
The batch writes. value_offset either returns the real offset or isn't there
to call.
**Additional context**
FixedSizeBinaryArray::value_offset had the same narrowing and was deprecated
in 59.0.0 in favour of `i * value_size()`. The FixedSizeList one could go the
same way, with the avro encoder using `idx * value_length() as usize` instead.
The only other non-test caller is take_value_indices_from_fixed_size_list,
which casts the result to u32, so it comes out right up to u32::MAX, where its
UInt32 indices run out anyway.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
