peterxcli opened a new issue, #5549: URL: https://github.com/apache/datafusion-comet/issues/5549
### Problem Once Variant identity exists in the native schema, ordinary Parquet can still return `metadata`, residual `value`, and optional `typed_value` in a physical Struct that is not Spark's required whole-value layout. Parent: #5546 Depends on: #5548 Related: #3983 ### Solution - Route only explicitly marked Variant Fields to a dedicated schema-adapter normalization path. - Construct Arrow-rs `VariantArray` by child name. - Call `unshred_variant` once at the native Parquet boundary. - Remove `typed_value` and rebuild exactly `[value, metadata]` as ordinary Binary children. - Preserve parent nulls and the original marked output Field. - Handle the canonical and legacy Spark object-key ordering required by supported Spark profiles. The JVM scan gate remains closed in this task. It must not add subfield pruning, predicate pushdown, shredded writing, or JVM-side unshredding. ### Tests - Canonical unshredded, fully shredded, and partially shredded values agree. - Inputs whose children are not in Spark output order are reordered. - Objects, arrays, scalars, JSON null, SQL NULL, nested residuals, and nullable parents round-trip. - `typed_value` never reaches the output. - The output Field marker and parent null bitmap are preserved. - #5474 is linked to removable legacy ordering compatibility code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
