mzabaluev opened a new issue, #10478: URL: https://github.com/apache/arrow-rs/issues/10478
### Describe the bug The [implementation](https://github.com/apache/arrow-rs/blob/005a2fd43f1b243ed43f4cfa356ebe8f0757939b/arrow-json/src/reader/run_end_array.rs#L51) for REE decoder forces the child field's decoder to accept nulls even though the Arrow schema for the values field may declare it non-nullable. As the REE encoding has no container-level null mask, this may smuggle null values into the values array that must not have them. Furthermore, even if the REE field itself is non-nullable, the flag passed down to e.g. `StringArrayDecoder` is ignored. The resulting nulls are not invalidated by a post-hoc check the way `StructArrayDecoder` does. ### To Reproduce Repro test case: ```rust #[test] fn test_read_run_end_encoded_nulls_in_non_nullable_values() { // A `RunArray` has no validity buffer of its own, so unlike a struct child // there is no parent-level mask that could make these nulls representable. // Declaring the enclosing field nullable does not change that. let ree_type = DataType::RunEndEncoded( Arc::new(Field::new("run_ends", DataType::Int32, false)), Arc::new(Field::new("values", DataType::Utf8, false)), ); let schema = Arc::new(Schema::new(vec![Field::new( "a", ree_type, true, )])); // An explicit null, and a missing field, should be both rejected. for buf in [ r#" {"a": "x"} {"a": null} {"a": "y"} "#, r#" {"a": "x"} {} {"a": "y"} "#, ] { let mut decoder = ReaderBuilder::new(schema.clone()).build_decoder().unwrap(); let res = decoder .decode(buf.as_bytes()) .and_then(|_| decoder.flush()); assert!(res.is_err()); } } ``` ### Expected behavior The schema violation should produce an error. ### Additional context Note that this is a breaking change, since the current implementation may tolerate nullable inputs, even if they produce non-compliant outputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
