chenghuichen opened a new pull request, #7657: URL: https://github.com/apache/paimon/pull/7657
### Purpose Queries like `SELECT variant_get(v, '$.age', 'int') FROM T` on a shredded Variant column still read all sub-columns and reassemble the full binary Variant, leaving Paimon's `VariantRowType` / `clipVariantType` infrastructure unused. This PR adds `PushDownVariantExtract` (Spark 4 only), a Catalyst optimizer rule that replaces `VariantGet` with `GetStructField` and sets `variantProjections` on `PaimonScan`, so only the accessed `typed_value.*` Parquet sub-columns are read. The rule runs in the "User Provided Optimizers" batch (via `experimentalMethods.extraOptimizations`) to ensure it fires after `V2ScanRelationPushDown` has built the scan relation. ### Tests VariantTest.scala::VariantPushDownPlanTest (paimon-spark-4.0) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
