peterxcli opened a new issue, #5435: URL: https://github.com/apache/datafusion-comet/issues/5435
### What is the problem the feature request solves? #5407 adds direct, top-level whole-value Variant projection for ordinary native Parquet scans. Spark also permits Variant nested inside structs, arrays, and string-keyed maps, and its Parquet Variant suite covers `struct<... variant>`, `array<variant>`, and `map<string,variant>` shapes ([Spark tests](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala#L200-L223)). Comet's current scan admission only special-cases a directly top-level Variant field ([CometScanRule](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L963-L985)), and native normalization runs only when the target Field itself is marked Variant ([cast_column.rs](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/parquet/cast_column.rs#L545-L553)). Nested Variant therefore remains a full scan fallback. ### Describe the potential solution - Recursively identify explicitly marked Variant leaf Fields while adapting ordinary Parquet batches. - Apply the existing whole-value dictionary decode, unshredding, Binary normalization, Spark object ordering, and `[value, metadata]` reordering exactly once at each Variant leaf. - Preserve every enclosing struct field, list/map offset, container and element null bitmap, field name/nullability, and extension marker. - Admit only Spark-supported logical shapes; keep Variant map keys and malformed/unmarked lookalike Structs rejected. Add Spark parity and vector-layout tests for nullable struct parents, array elements, and map values; shredded and unshredded input; SQL NULL and Variant JSON null; and fields before/after each nested Variant. Verify pruning an unread nested Variant remains compatible with #5377. ### Additional context #3983 continues to own broader shredded writer, subfield-pruning, and predicate-pushdown work. This issue covers full-value ordinary-Parquet projection only. Nested Variant expressions, C2R, shuffle/spill, Python, writes, and Iceberg remain separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
