peterxcli opened a new issue, #11068: URL: https://github.com/apache/arrow-rs/issues/11068
### Is your feature request related to a problem or challenge? Comet decodes dictionary-encoded Variant `value` and `typed_value` children before calling `VariantArray::try_new`. Such arrays can arise when a Parquet reader restores dictionary types from an embedded Arrow schema. #10802 / #10810 cover encoded **metadata** only. This is a compatibility feature request: the [canonical Variant extension](https://github.com/apache/arrow/blob/main/docs/source/format/CanonicalExtensions.rst#parquet-variant) explicitly permits dictionary/run-end encoding for `metadata`, but does not currently permit these encodings for `value` or `typed_value`. On arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec`, this reproducer rejects both inputs: ```rust use std::sync::Arc; use arrow::array::{Array, ArrayRef, BinaryArray, DictionaryArray, Int8Array, Int32Array, StructArray}; use arrow::datatypes::{Field, Int8Type}; use parquet_variant_compute::VariantArray; fn main() { let value: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::new( Int8Array::from(vec![0]), Arc::new(BinaryArray::from_vec(vec![&[12, 1]])), )); let typed: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::new( Int8Array::from(vec![0]), Arc::new(Int32Array::from(vec![1])), )); for (name, encoded) in [("value", value), ("typed_value", typed)] { let metadata: ArrayRef = Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])); let mut fields = vec![Arc::new(Field::new("metadata", metadata.data_type().clone(), false))]; let mut columns = vec![metadata]; if name == "typed_value" { let value: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>])); fields.push(Arc::new(Field::new("value", value.data_type().clone(), true))); columns.push(value); } fields.push(Arc::new(Field::new(name, encoded.data_type().clone(), true))); columns.push(encoded); let input = StructArray::new(fields.into(), columns, None); println!("{}", VariantArray::try_new(&input).unwrap_err()); } } ``` Errors: `Illegal shredded value type: Dictionary(Int8, Binary)` and `Illegal shredded value type: Dictionary(Int8, Int32)`. ### Describe the solution you'd like Support dictionary-encoded `value` and supported primitive `typed_value` inputs, including nested shredding states, through the shared Variant input path. Preserve logical nulls, parent nulls and sliced dictionary indices. If canonical storage remains strict, a supported normalization entry point would also let callers avoid maintaining their own recursive decoder. ### Describe alternatives you've considered Comet currently decodes dictionaries recursively. For Parquet scans it also suppresses advisory Arrow schema hints to match Spark's physical interpretation; that can avoid these arrays at the reader boundary. Rejecting noncanonical encoded inputs is another valid policy, but encoded-metadata support alone does not justify removing the existing compatibility branch. ### Additional context Tracked downstream in apache/datafusion-comet#5477; direct projection work is apache/datafusion-comet#5868. This concerns encoded inputs, distinct from dictionary output support in #10013 / #10014. Run-end-encoded `value`/`typed_value` is outside this request's immediate scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
