taoran92 opened a new pull request, #11270: URL: https://github.com/apache/paimon/pull/11270
### Purpose Fixes #11013. A Spark JOIN on a masked partition key can fail when dynamic partition pruning (DPP) is enabled. For example, two date partitions whose visible `dt` values are both `'***'` should both match a JOIN on `'***'`. Spark may instead push that value down as a physical partition filter, which Paimon rejects because it would be evaluated against raw partition values. ### Changes - For single-branch file-store tables with query authorization enabled, exclude masked partition keys from the runtime filter attributes advertised to Spark. For a table partitioned by `(region, dt)` with only `dt` masked, Spark can still prune on `region`. - Resolve masked columns without planning splits. Spark can then discover runtime filter attributes during logical optimization without creating a batch-read protection tag. ### Tests - All 25 cases in `SparkPartitionMetadataAuthTest` passed on both Spark 3.5.9 and 4.1.4. The new masked-key JOIN test reproduced the original exception on the parent commit. - On Spark 3.5.8, the query from #11013 returned IDs `1` and `2` with AQE and DPP independently enabled or disabled. On Spark 3.5.9, the two-key JOIN returned ID `1` with DPP both on and off. With DPP off, the fact scan planned one file each from `CN` and `US`. With DPP on, `Equal(region, CN)` was the only runtime partition filter, and the scan planned only the `CN` file. - With batch-read protection enabled, discovering runtime filter attributes returned only the unmasked `region` and created no tag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
