mbutrovich opened a new issue, #11832: URL: https://github.com/apache/arrow-rs/issues/11832
### Describe the bug `RecordBatch::try_new` checks a non-nullable field with `null_count()` ([record_batch.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-array/src/record_batch.rs#L347-L354)), which counts only the null buffer. It accepts a union, run-end encoded or dictionary column whose nulls are in its children or values. `StructArray::try_new` ([struct_array.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-array/src/array/struct_array.rs#L169-L178)), `ListArray::try_new` ([list_array.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-array/src/array/list_array.rs#L235-L241)) and `FixedSizeListArray::try_new` ([fixed_size_list_array.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-array/src/array/fixed_size_list_array.rs#L289-L302)) check logical nulls and reject the same columns in a non-nullable field. ### To Reproduce ```rust use std::sync::Arc; use arrow_array::types::{Int32Type, Int8Type}; use arrow_array::{ Array, ArrayRef, DictionaryArray, FixedSizeListArray, Int32Array, Int8Array, ListArray, RecordBatch, RunArray, StructArray, UnionArray, }; use arrow_buffer::OffsetBuffer; use arrow_schema::{DataType, Field, Fields, Schema, UnionFields}; // Prints the column's logical null count, then whether each constructor accepts it in a non-nullable field fn check(column: ArrayRef) { let field = Arc::new(Field::new("a", column.data_type().clone(), false)); let n = column.len(); let batch = RecordBatch::try_new(Arc::new(Schema::new(vec![field.clone()])), vec![column.clone()]); let strukt = StructArray::try_new(Fields::from(vec![field.clone()]), vec![column.clone()], None); let list = ListArray::try_new(field.clone(), OffsetBuffer::from_lengths([n]), column.clone(), None); let fixed = FixedSizeListArray::try_new(field, n as i32, column.clone(), None); println!("{} {} {} {} {}", column.logical_null_count(), batch.is_ok(), strukt.is_ok(), list.is_ok(), fixed.is_ok()); } fn main() { // A sparse union whose field "u" holds a null in row 1 let fields = UnionFields::try_new([0], [Field::new("u", DataType::Int32, true)]).unwrap(); let child = Arc::new(Int32Array::from(vec![Some(1), None])) as ArrayRef; check(Arc::new(UnionArray::try_new(fields, vec![0, 0].into(), None, vec![child]).unwrap())); // 1 true false false false // One run whose value is null let values = Int32Array::from(vec![None]); check(Arc::new(RunArray::<Int32Type>::try_new(&Int32Array::from(vec![2]), &values).unwrap())); // 2 true false false false // Valid keys that point to a null value let values = Arc::new(Int32Array::from(vec![None])); check(Arc::new(DictionaryArray::<Int8Type>::try_new(Int8Array::from(vec![0, 0]), values).unwrap())); // 2 true false false false } ``` ### Expected behavior An error, as the other three constructors return. ### Additional context This is not a regression. 58.4.0, 59.3.1 and 60.0.0 behave the same. It is why #11831 does not reject a null index into a non-nullable union or run-end encoded column of `take_record_batch` (#11824). Found with property-based tests that build arrays in every physical layout the Arrow format allows, on `main` at `01a991765`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
