peterxcli opened a new issue, #10802: URL: https://github.com/apache/arrow-rs/issues/10802
### Describe the bug The canonical `arrow.parquet.variant` extension permits its non-nullable `metadata` child to be dictionary-encoded or run-end-encoded. However, `VariantArray::try_new` calls a validator whose validation and row access accept only `Binary`, `LargeBinary`, and `BinaryView`. This affects arrow-rs 59.2.0 and current `main`: - Canonical extension definition: https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant - `VariantArray::try_new` documents Dictionary metadata as permissible, but rejects it: https://github.com/apache/arrow-rs/blob/59.2.0/parquet-variant-compute/src/variant_array.rs#L285-L320 - Byte access and validation only handle unencoded binary arrays: https://github.com/apache/arrow-rs/blob/59.2.0/parquet-variant-compute/src/variant_array.rs#L43-L83 - Current `main` has the same limitation: https://github.com/apache/arrow-rs/blob/cd7c6b83abd6605a83014b3d043930a592542510/parquet-variant-compute/src/variant_array.rs#L44-L83 This is specifically about encoding the Variant `metadata` child. It is not about shredded `typed_value` layouts or Dictionary/REE output from `variant_to_arrow` (#10013 / #10014). ### To Reproduce A conforming StructArray with this storage schema fails construction: ```text Struct< metadata: Dictionary<Int8, Binary> not null, value: Binary not null > ``` For example, using an empty Variant metadata dictionary and an Int8 Variant value: ```rust let metadata_values = Arc::new(BinaryArray::from(vec![ Some(&[0x01, 0x00, 0x00][..]), ])); let metadata = Arc::new( DictionaryArray::<Int8Type>::try_new( Int8Array::from(vec![0]), metadata_values, ) .unwrap(), ); let value = Arc::new(BinaryArray::from(vec![Some(&[12, 1][..])])); let array = StructArray::try_new( Fields::from(vec![ Field::new("metadata", metadata.data_type().clone(), false), Field::new("value", DataType::Binary, false), ]), vec![metadata, value], None, ) .unwrap(); VariantArray::try_new(&array).unwrap(); ``` The final call currently fails with: ```text VariantArray 'metadata' field must be Binary, LargeBinary, or BinaryView, got Dictionary(Int8, Binary) ``` Run-end-encoded metadata is rejected for the same reason. ### Expected behavior `VariantArray` should accept Dictionary- and RunEndEncoded metadata whose logical values are `Binary`, `LargeBinary`, or `BinaryView`. All paths that consume metadata bytes—including row access and compute paths such as `variant_get` and `unshred_variant`—must resolve the logical metadata value correctly; relaxing constructor validation alone would not be sufficient. Please add focused coverage for both Dictionary and RunEndEncoded metadata, including null/sliced arrays and at least one downstream compute path. ### Additional context Downstream consumers currently have to decode the `metadata` child eagerly before constructing `VariantArray`. This is a follow-up to the raw binary layout support in #8387 / #9610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
