taoran92 opened a new pull request, #11270:
URL: https://github.com/apache/paimon/pull/11270

   ### Purpose
   
   Fixes #11013.
   
   A Spark JOIN on a masked partition key can fail when dynamic partition 
pruning (DPP) is enabled. For example, two date partitions whose visible `dt` 
values are both `'***'` should both match a JOIN on `'***'`. Spark may instead 
push that value down as a physical partition filter, which Paimon rejects 
because it would be evaluated against raw partition values.
   
   ### Changes
   
   - For single-branch file-store tables with query authorization enabled, 
exclude masked partition keys from the runtime filter attributes advertised to 
Spark. For a table partitioned by `(region, dt)` with only `dt` masked, Spark 
can still prune on `region`.
   - Resolve masked columns without planning splits. Spark can then discover 
runtime filter attributes during logical optimization without creating a 
batch-read protection tag.
   
   ### Tests
   
   - All 25 cases in `SparkPartitionMetadataAuthTest` passed on both Spark 
3.5.9 and 4.1.4. The new masked-key JOIN test reproduced the original exception 
on the parent commit.
   - On Spark 3.5.8, the query from #11013 returned IDs `1` and `2` with AQE 
and DPP independently enabled or disabled. On Spark 3.5.9, the two-key JOIN 
returned ID `1` with DPP both on and off. With DPP off, the fact scan planned 
one file each from `CN` and `US`. With DPP on, `Equal(region, CN)` was the only 
runtime partition filter, and the scan planned only the `CN` file.
   - With batch-read protection enabled, discovering runtime filter attributes 
returned only the unmasked `region` and created no tag.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to