sdf-jkl opened a new issue, #11947:
URL: https://github.com/apache/arrow-rs/issues/11947
## Describe the bug
`cast_to_variant` converts a present struct field or map entry with a null
value into a missing object member. For example, a struct representing `{"a":
null}` becomes `{}`. This loses field presence and changes subsequent
missing-field checks.
Found while implementing Spark-compatible `to_variant_object` in
[datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant).
The output is a valid Variant, but it no longer represents all of the input
members.
## To Reproduce
Executed with `arrow` and `parquet-variant-compute` version `60.0.0`:
```rust
use std::sync::Arc;
use arrow::array::{ArrayRef, Int32Array, StructArray};
use arrow::datatypes::{DataType, Field};
use parquet_variant_compute::{cast_to_variant, variant_to_json};
fn main() {
let input = StructArray::new(
vec![Arc::new(Field::new("a", DataType::Int32, true))].into(),
vec![Arc::new(Int32Array::from(vec![Some(1), None]))],
None, // Both parent structs are present.
);
let output = ArrayRef::from(cast_to_variant(&input).unwrap());
let json = variant_to_json(&output).unwrap();
println!("{}", json.value(0));
println!("{}", json.value(1));
}
```
Actual output:
```json
{"a":1}
{}
```
Expected second row: `{"a":null}`. The parent struct is present in both rows.
Additional probes on 60.0.0, with identical results in strict and safe modes:
- A map entry `"a" -> NULL` is omitted too.
- A null list, map, or struct stored as an object member is omitted.
- Null values reached through dictionary, dense/sparse union, and run-end
encoded arrays are also omitted when used as object members.
- Null list elements are preserved in all five layouts: `List`, `LargeList`,
`FixedSizeList`, `ListView`, and `LargeListView`. A null nested container
inside a list remains a null element.
- Top-level null values remain Arrow nulls; that behavior should be retained.
## Expected behavior
Preserve every present struct field and map entry, encoding its null value
as Variant null. Apply this recursively, including null nested containers and
encoded child arrays. Keep null parent containers distinct from present
containers whose members are null.
Regression coverage should distinguish missing members, present-null
members, null parent containers, and null list elements. Do not change callers
that intentionally use missing-field semantics.
## Additional context
On current `main` at `fd2b5d156d`, both [struct
conversion](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant-compute/src/arrow_to_variant.rs#L593-L604)
and [map
conversion](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant-compute/src/arrow_to_variant.rs#L652-L662)
route members through `ObjectFieldBuilder`. Its [`append_null`
implementation](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant/src/builder/object.rs#L431-L438)
deliberately inserts nothing. The conversion path needs present-null semantics
instead.
The [Variant shredding
specification](https://github.com/apache/parquet-format/blob/e365c6f7c05eea69c5aa11f1ead6c3db261ca9f9/VariantShredding.md#L193-L196)
distinguishes absent object fields from present fields containing Variant
null. This report concerns loss of that distinction while constructing a
Variant from Arrow input, not an invalid encoded result: the observed outputs
pass full Variant validation.
Related closed issue [#8289](https://github.com/apache/arrow-rs/issues/8289)
concerns distinguishing an empty map from a null map. It does not cover
null-valued members of a present struct or map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
