andygrove commented on PR #5407: URL: https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5444201255
`spark.sql.variant.pushVariantIntoScan` defaults to `false` on Spark 4.0.x but `true` on 4.1.0, 4.1.3, and 4.2.0. On 4.1 I see `SELECT v FROM t` rewritten to `ReadSchema: struct<v:struct<0:variant>>` and the scan falls back, so nothing that projects `v` keeps a native scan out of the box. Every test here pins the conf to `false`, including the header in `variant.sql`, so CI doesn't cover the default, and 4.1 is our default profile. Could you add the required configuration to the description and the user docs, and note it next to the benchmark numbers? And is handling the pushed struct worth attempting here? `RequestedVariantField.fullVariant` uses path `$` with `targetType = VariantType`, so a whole-value projection under pushdown is exactly a one-field `VariantStruct`. If that's better as a follow-up, could you file an issue and link it so this doesn't sit unreachable by default on 4.1 and 4.2? Separately, the reason users will see is `Unsupported v of type VariantType (shredded; not supported by native scan)`. That word "shredded" points at `allowReadingShredded`, which isn't the lever. Can it name `pushVariantIntoScan` instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
