LuciferYang opened a new pull request, #11292: URL: https://github.com/apache/paimon/pull/11292
### Purpose Since #7913, the Spark 4.1 connector accepts `variant_get` extractions through `SupportsPushDownVariantExtractions`. On Spark 4.1 and 4.2, Spark then builds the scan without column pruning and without its metadata columns (SPARK-57499 fixes this in 4.3.0). `spark.sql.variant.pushVariantIntoScan` is on by default, so ordinary queries break: - With two variant columns, `SELECT variant_get(v1, '$.a', 'string') FROM t` fails at planning with `UnresolvedException`. - A metadata column next to an extraction fails with `INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND`, and so do UPDATE and DELETE with a variant condition under the default v2 write. - The scan reads every column. 4.2 never had the trait. This change removes it from 4.1 too, so Spark evaluates `variant_get` on the full value on every version, and shredded tables lose the leaf-only read on 4.1. `VariantPushDownUtils` is kept for enabling pushdown on 4.3, where a scan must accept all extractions in a call or none. ### Tests The 4.2 module now depends on the `paimon-spark-ut` test jar and runs the four shared Variant suites, as 4.1 does. Spark 4.2 adds a field to `DataSourceV2ScanRelation`, so two 5-field pattern matches in `VariantTestBase` now match the type instead. New `VariantTestBase` cases run the queries above: an extraction next to a second variant column, which also checks that the scan reads only the extracted column; an extraction next to a metadata column; and UPDATE and DELETE with a variant condition, with and without deletion vectors. They fail on 4.1 with the removed trait put back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
