Cintu07 opened a new issue, #11520:
URL: https://github.com/apache/arrow-rs/issues/11520
### Describe the bug
a dictionary whose keys are all valid but point at a null value has no
physical nulls, so RecordBatch::try_new accepts it under a non-nullable field.
ArrowWriter maps that field to a required column, so those rows get no
definition levels and are written as values. they read back as 0 for Int32
values and as an empty string for Utf8 values, with no error on write or read.
StructArray::try_new already rejects the same column as a non-nullable child
("Found unmasked nulls for non-nullable StructArray field"), so it only gets
through at the top level of a RecordBatch.
this is the dictionary side of #11277. #11278 handles REE by folding the run
values field nullability into the schema, but dictionary values carry no field,
so the schema cant tell.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::{Array, DictionaryArray, Int32Array, Int8Array,
RecordBatch, types::Int8Type};
use arrow_schema::{Field, Schema};
use bytes::Bytes;
use parquet::arrow::{ArrowWriter,
arrow_reader::ParquetRecordBatchReaderBuilder};
fn main() {
// every key is valid, the second dictionary value is null
let dict = DictionaryArray::<Int8Type>::new(
Int8Array::from(vec![0, 1, 0, 1]),
Arc::new(Int32Array::from(vec![Some(7), None])),
);
assert_eq!(dict.logical_null_count(), 2);
let schema = Arc::new(Schema::new(vec![Field::new("value",
dict.data_type().clone(), false)]));
// accepted: the physical null count is 0
let batch = RecordBatch::try_new(schema.clone(),
vec![Arc::new(dict)]).unwrap();
let mut file = Vec::new();
let mut writer = ArrowWriter::try_new(&mut file, schema, None).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
let read = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(file))
.unwrap().build().unwrap().next().unwrap().unwrap();
// keys [0, 1, 0, 1], values [7, 0]: rows 1 and 3 read back as 0, not
null
println!("{:?}", read.column(0));
}
```
### Expected behavior
the nulls survive, or the write fails. a required column cant hold them, so
the narrow fix is an error from the writer when a non-nullable leaf has logical
nulls. checking logical nulls in RecordBatch::try_new, like
StructArray::try_new does, would stop it earlier for every consumer, if that
cost is acceptable there.
### Additional context
reproduced on main at 0706d31. happy to send the pr for whichever you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
