peterxcli opened a new issue, #5979: URL: https://github.com/apache/datafusion-comet/issues/5979
### What is the problem the feature request solves? #5868 accepts Spark-readable shredded objects whose field names are absent from the row's metadata dictionary. [`extend_shredded_metadata`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant.rs#L595-L668) adds those names and remaps residual field IDs before Arrow unshredding, adding a separate compatibility pass. apache/arrow-rs#11069 covers the upstream panic on this input. Returning an error fixes that bug but does not preserve Spark's successful reconstruction. The [Parquet specification](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#variant-metadata) requires these names in metadata, so permissive reading is an explicit compatibility feature. ### Describe the potential solution Provide equivalent permissive reconstruction in the native path, preferably through shared upstream support or a supported compatibility entry point. Keep Arrow's canonical validation contract explicit. Completion requires: - Reconstruct present shredded fields even when their names are absent from metadata. - Preserve residual values while assigning valid field IDs; cover nested objects and lists, empty keys, absent object fields, SQL NULL, and Variant null. - Keep these inputs on the native scan path. Replacing the repair with rejection or Spark fallback does not satisfy this issue. - Delete the separate metadata-extension/remapping pass only after a replacement produces Spark-compatible results through Comet. - Measure allocation and scan-time changes using the existing benchmarks. ### Additional context Parent: #5477. Separate panic-safety issue: apache/arrow-rs#11069. Existing regression: [`normalize_shredded_objects_extend_metadata_and_preserve_missing_fields`](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant/tests.rs#L360-L431). Retain the existing repair until equivalent native support is available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
