hanke580 opened a new issue, #11735:
URL: https://github.com/apache/arrow-rs/issues/11735
### Describe the bug
With `with_utf8_view(true)`, the Avro reader returns a `StringViewArray`
with no null buffer. Every NULL of a nullable string field
(`["null","string"]`, in either branch order) is returned as a non-null `""`,
so `null_count` is 0 and the NULLs cannot be told apart from real empty
strings. No error is raised. Without the option the same file reads correctly
(`StringArray`, `null_count = 2`).
### To Reproduce
`Cargo.toml` dependencies: `arrow-array`, `arrow-schema` and `arrow-avro`
(60.0.0, or git main).
```rust
use std::io::Cursor;
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, RecordBatch, StringArray};
use arrow_avro::reader::ReaderBuilder;
use arrow_avro::writer::AvroWriter;
use arrow_schema::{DataType, Field, Schema};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Write an OCF file with one nullable string column (Avro type
["null","string"]).
let schema = Schema::new(vec![Field::new("s", DataType::Utf8, true)]);
let col: ArrayRef = Arc::new(StringArray::from(vec![Some("a"), None,
Some(""), None, Some("b")]));
let batch = RecordBatch::try_new(Arc::new(schema.clone()), vec![col])?;
let mut w = AvroWriter::new(Vec::<u8>::new(), schema)?;
w.write(&batch)?;
w.finish()?;
let bytes = w.into_inner();
for utf8_view in [false, true] {
let mut reader = ReaderBuilder::new()
.with_utf8_view(utf8_view)
.build(Cursor::new(bytes.clone()))?;
let out = reader.next().unwrap()?;
let c = out.column(0);
println!("with_utf8_view({utf8_view}): null_count={} {:?}",
c.null_count(), c);
}
Ok(())
}
```
Output on main (33d338dd) and on 60.0.0:
```
with_utf8_view(false): null_count=2 StringArray
[
"a",
null,
"",
null,
"b",
]
with_utf8_view(true): null_count=0 StringViewArray
[
"a",
"",
"",
"",
"b",
]
```
### Expected behavior
`with_utf8_view(true)` changes only the output type: a `StringViewArray`
`["a", null, "", null, "b"]` with `null_count = 2`, the same values as without
the option.
### Additional context
Tested: main 33d338ddf661b6e85364a9b0bd4d367c3468257d (2026-10-18), 60.0.0
and 59.3.1.
Likely cause: the `StringView` arm of `Decoder::flush` in
`arrow-avro/src/reader/record.rs` ([lines
1687-1701](https://github.com/apache/arrow-rs/blob/33d338ddf661b6e85364a9b0bd4d367c3468257d/arrow-avro/src/reader/record.rs#L1687-L1701))
builds a `StringArray` with the right null buffer, then maps it to a
`Vec<&str>` with NULL replaced by `""` and calls
`StringViewArray::from(values)`, which creates an array without nulls. Passing
the null buffer through (for example `StringViewArray::from(&array)` /
`from_iter` over `Option<&str>`, or building with `nulls`) should fix it. This
code dates from the PR that added the option (#7434).
Related but different: #8380 (the `read_with_utf8view` example failed with
an EOF error; fixed in #8381).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
