mbutrovich opened a new issue, #11835: URL: https://github.com/apache/arrow-rs/issues/11835
### Describe the bug #10992 settled that a kernel returns an error when it would introduce a null that a union or run-end encoded array cannot hold. These three write the null, and the result passes `validate_full`: - `shift` fills vacated rows with `new_null_array` ([window.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/window.rs#L54-L79)). `ArrayData::new_null` puts the nulls in a union's non-nullable children ([data.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-data/src/data.rs#L821-L844)) and in a run-end encoded array's non-nullable values ([data.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-data/src/data.rs#L846-L882)). - `merge_n` with a `None` index appends a null run-end encoded value without checking the values field ([run.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-data/src/transform/run.rs#L41-L48)). The union case returns an error since #11209. - `union_extract` fills unselected rows with `new_null_array` when the target is a union or run-end encoded array ([union_extract.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/union_extract.rs#L163), [#L195](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/union_extract.rs#L195), [#L222](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/union_extract.rs#L222), [#L257](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/union_extract.rs#L257)). ### To Reproduce ```rust use std::sync::Arc; use arrow_array::types::Int32Type; use arrow_array::{make_array, Array, ArrayRef, Int32Array, RunArray, UnionArray}; use arrow_schema::{DataType, Field, UnionFields}; use arrow_select::merge::merge_n; use arrow_select::union_extract::union_extract; use arrow_select::window::shift; fn main() { // A sparse union whose only field is not nullable let fields = UnionFields::try_new([0], [Field::new("a", DataType::Int32, false)]).unwrap(); let union = UnionArray::try_new( fields, vec![0, 0].into(), None, vec![Arc::new(Int32Array::from(vec![1, 2])) as ArrayRef], ) .unwrap(); let out = shift(&union, 1).unwrap(); let out = out.as_any().downcast_ref::<UnionArray>().unwrap(); println!("{}", out.child(0).null_count()); // 1, in the non-nullable field "a" out.to_data().validate_full().unwrap(); // passes // A run-end encoded array whose values field is not nullable let run = RunArray::<Int32Type>::try_new(&Int32Array::from(vec![2]), &Int32Array::from(vec![7])).unwrap(); let ree_type = DataType::RunEndEncoded( Arc::new(Field::new("run_ends", DataType::Int32, false)), Arc::new(Field::new("values", DataType::Int32, false)), ); let ree = make_array(run.to_data().into_builder().data_type(ree_type).build().unwrap()); let out = shift(ree.as_ref(), 1).unwrap(); let out = out.as_any().downcast_ref::<RunArray<Int32Type>>().unwrap(); println!("{}", out.values().null_count()); // 1, in the non-nullable field "values" let out = merge_n(&[ree.as_ref()], &[Some(0), None]).unwrap(); let out = out.as_any().downcast_ref::<RunArray<Int32Type>>().unwrap(); println!("{}", out.values().null_count()); // 1, in the non-nullable field "values" // A dense union whose only row selects "a", so "b" is null in every row let outer_fields = UnionFields::try_new( [0, 1], [Field::new("a", DataType::Int32, true), Field::new("b", ree.data_type().clone(), false)], ) .unwrap(); let outer = UnionArray::try_new( outer_fields, vec![0].into(), Some(vec![0].into()), vec![Arc::new(Int32Array::from(vec![1])) as ArrayRef, ree.slice(0, 0)], ) .unwrap(); let out = union_extract(&outer, "b").unwrap(); let out = out.as_any().downcast_ref::<RunArray<Int32Type>>().unwrap(); println!("{}", out.values().null_count()); // 1, in the non-nullable field "values" // The union case of merge_n already returns an error println!("{:?}", merge_n(&[&union as &dyn Array], &[Some(0), None]).err()); // Some(InvalidArgumentError("Cannot extend a union with no nullable fields with nulls")) } ``` ### Expected behavior An error from the four calls that unwrap, like the last call's. ### Additional context With #11288, `ArrayData::new_null` `expect`s a nullable union child, so `shift` and `union_extract` would panic instead. Found with property-based tests that build arrays in every physical layout the Arrow format allows, on `main` at `01a991765`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
