sdf-jkl opened a new issue, #11942:
URL: https://github.com/apache/arrow-rs/issues/11942
## Describe the bug
`cast_to_variant` does not honor Variant extension annotations on nested
input fields. When an ordinary Arrow struct contains a Variant child, the
kernel converts the child's physical `metadata` and `value` fields into object
members instead of incorporating the existing Variant value.
For example, a struct with a Variant-valued field `a` containing `1` becomes
`{"a":{"metadata":"AQAA","value":"FAEAAAA="}}` instead of `{"a":1}`.
Found while implementing Spark-compatible `to_variant_object` in
[datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant).
This concerns constructing a Variant from an ordinary Arrow container
containing Variant children; an already encoded Variant object has a different
representation.
## To Reproduce
Reproduced with `arrow`, `parquet-variant`, and `parquet-variant-compute`
version `60.0.0`. The nested field retains its Variant extension annotation, so
this reproducer does not depend on DataFusion or metadata loss in SQL
constructors.
```rust
use std::sync::Arc;
use arrow::array::{ArrayRef, StructArray};
use parquet_variant::Variant;
use parquet_variant_compute::{VariantArrayBuilder, cast_to_variant,
variant_to_json};
fn main() {
let mut builder = VariantArrayBuilder::new(1);
builder.append_variant(Variant::Int32(1));
let child = builder.build();
// An ordinary struct with an explicitly annotated Variant child.
let field = Arc::new(child.field("a"));
let input = StructArray::new(vec![field].into(),
vec![ArrayRef::from(child)], None);
let output = ArrayRef::from(cast_to_variant(&input).unwrap());
let json = variant_to_json(&output).unwrap();
println!("{}", json.value(0));
assert_eq!(json.value(0), r#"{"a":1}"#);
}
```
The assertion fails with:
```text
actual: {"a":{"metadata":"AQAA","value":"FAEAAAA="}}
expected: {"a":1}
```
The same failure is reproducible for arrays. Replacing the `field` and
`input` construction above with:
```rust
let field = Arc::new(child.field("item"));
let input = arrow::array::ListArray::new(
field,
arrow::buffer::OffsetBuffer::new(vec![0_i32, 1].into()),
ArrayRef::from(child),
None,
);
```
produces `[{"metadata":"AQAA","value":"FAEAAAA="}]` instead of `[1]` (also
reproduced on 60.0.0).
Additional probes on 60.0.0 reproduce the same problem with explicitly
annotated children in `Map` values, `LargeList`, `FixedSizeList`, `ListView`,
`LargeListView`, dense/sparse `Union`, and `RunEndEncoded` values. Map values
become storage objects; the other list layouts match the result above; union
and run-end encoded values become storage objects instead of the logical scalar
`1`.
## Expected behavior
Recognize nested fields annotated with the Variant extension type and
incorporate their logical values when recursively converting containers or
decoding union/run-end encoded values. Unannotated structs with fields named
`metadata` and `value` should continue to be treated as ordinary objects.
## Additional context
Source inspection of `main` at `5746b443f7` shows that the [struct
conversion
builder](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/arrow_to_variant.rs#L567-L578)
passes each child's `DataType` and array to the recursive builder, omitting
the child's field metadata. The nested annotation is already available in the
enclosing struct's fields.
Spark 4.2 [explicitly tests casting an array of existing Variant
objects](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala#L924-L927)
and expects their logical contents to be preserved. Its [recursive conversion
implementation](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L117-L155)
handles Variant children explicitly through `appendVariant` for arrays, maps,
and structs. These Spark references were inspected, not executed locally.
Related: [#8319's
discussion](https://github.com/apache/arrow-rs/issues/8319#issuecomment-3285956937)
identified the ambiguity between Variant storage and ordinary structs. #9519
concerns the opposite direction, extracting into structs containing Variant
fields. #11937 concerns preserving UUID field metadata in the same casting API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
