peterxcli opened a new issue, #5519: URL: https://github.com/apache/datafusion-comet/issues/5519
## What is the problem the feature request solves? #5407 supports direct top-level `VariantType` projection through the ordinary native Parquet scan, but leaves Spark's `PushVariantIntoScan` output (`VariantStruct`) as an explicit Spark fallback. `spark.sql.variant.pushVariantIntoScan` defaults to `false` on Spark 4.0.x but `true` (as an internal conf) on Spark 4.1.0, 4.1.3, and 4.2.0. On those versions every query that projects a Variant column is rewritten before Comet's scan rule runs, so out of the box the native Variant projection path from #5407 is unreachable: users must set the internal conf to `false` to get a native scan. ## Describe the potential solution When the rewrite requests exactly one field with `RequestedVariantField.fullVariant` (path `$`, target type `VariantType`), the request is semantically identical to whole-value projection. Comet could: - recognize this one shape of marked `VariantStruct` in `CometScanRule` and keep the scan native; - emit the `struct<0: variant>` output layout Spark's rewritten expressions consume by ordinal, wrapping the normalized `[value, metadata]` Variant struct in the outer single-field struct with its per-field `VariantMetadata` preserved; and - map that layout across the FFI boundary and the JVM vector bridge. Multi-field, typed, and path-extracted requests remain future work under #5438. ## Additional context Follow-up to #5407 (see the review discussion there). Related: #4295, #5438. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
