peterxcli opened a new issue, #11069: URL: https://github.com/apache/arrow-rs/issues/11069
### Describe the bug `unshred_variant` still panics when a present shredded object field has no entry in the row's metadata dictionary. `VariantArray::try_new` accepts the array, but the read-only metadata builder later returns an error that `ObjectBuilder::insert` unwraps. Reproduced on arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec`, after #9741. This is malformed input under the current [Variant metadata requirement](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#variant-metadata), which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data. ### To Reproduce ```rust use std::sync::Arc; use arrow::array::{Array, ArrayRef, BinaryArray, Int32Array, StructArray}; use arrow::datatypes::Field; use parquet_variant_compute::{VariantArray, unshred_variant}; fn structure(fields: Vec<(&str, ArrayRef, bool)>) -> StructArray { let schema = fields.iter().map(|(name, array, nullable)| Arc::new(Field::new(*name, array.data_type().clone(), *nullable))) .collect::<Vec<_>>(); StructArray::new(schema.into(), fields.into_iter().map(|(_, a, _)| a).collect(), None) } fn main() { let missing: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>])); let a: ArrayRef = Arc::new(structure(vec![ ("value", missing.clone(), true), ("typed_value", Arc::new(Int32Array::from(vec![1])), true), ])); let typed: ArrayRef = Arc::new(structure(vec![("a", a, false)])); let input = structure(vec![ ("metadata", Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])), false), ("value", missing, true), ("typed_value", typed, true), ]); let input = VariantArray::try_new(&input).unwrap(); let _ = unshred_variant(&input); // panics instead of returning Err } ``` Panic: `called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary")` at `parquet-variant/src/builder/object.rs:104`. ### Expected behavior Return `ArrowError` for a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix. ### Additional context Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently [extends metadata and remaps residual field IDs](https://github.com/apache/datafusion-comet/blob/6e556c944873554310aedd6a27f0d3e77ef17e71/native/core/src/parquet/cast_column/variant.rs#L586-L658) before unshredding in apache/datafusion-comet#5868. Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
