LuciferYang opened a new pull request, #11292:
URL: https://github.com/apache/paimon/pull/11292

   ### Purpose
   
   Since #7913, the Spark 4.1 connector accepts `variant_get` extractions 
through `SupportsPushDownVariantExtractions`. On Spark 4.1 and 4.2, Spark then 
builds the scan without column pruning and without its metadata columns 
(SPARK-57499 fixes this in 4.3.0). `spark.sql.variant.pushVariantIntoScan` is 
on by default, so ordinary queries break:
   
   - With two variant columns, `SELECT variant_get(v1, '$.a', 'string') FROM t` 
fails at planning with `UnresolvedException`.
   - A metadata column next to an extraction fails with 
`INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND`, and so do UPDATE and DELETE with a 
variant condition under the default v2 write.
   - The scan reads every column.
   
   4.2 never had the trait. This change removes it from 4.1 too, so Spark 
evaluates `variant_get` on the full value on every version, and shredded tables 
lose the leaf-only read on 4.1. `VariantPushDownUtils` is kept for enabling 
pushdown on 4.3, where a scan must accept all extractions in a call or none.
   
   ### Tests
   
   The 4.2 module now depends on the `paimon-spark-ut` test jar and runs the 
four shared Variant suites, as 4.1 does. Spark 4.2 adds a field to 
`DataSourceV2ScanRelation`, so two 5-field pattern matches in `VariantTestBase` 
now match the type instead.
   
   New `VariantTestBase` cases run the queries above: an extraction next to a 
second variant column, which also checks that the scan reads only the extracted 
column; an extraction next to a metadata column; and UPDATE and DELETE with a 
variant condition, with and without deletion vectors. They fail on 4.1 with the 
removed trait put back.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to