taoran92 opened a new pull request, #11014:
URL: https://github.com/apache/paimon/pull/11014

   ### Purpose
   
   Fixes #11013.
   
   Spark JOINs on masked partition keys fail when dynamic partition pruning 
(DPP) pushes visible join values into raw partition selection. For example, a 
key masked to `'***'` must be matched by the JOIN after masking, rather than 
used to select physical partitions.
   
   **1. Authorization-aware runtime filtering**
   
   Determine eligible output partition columns once from the initial authorized 
split plan, excluding columns masked in any split. Advertise only those columns 
to Spark and validate incoming runtime predicates against the same stable set.
   
   For unmasked partition keys, prune the cached authorized splits and rebuild 
input partitions, preserving authorization and fallback wrappers. Chain splits 
are filtered by their logical partition values and retained with all snapshot 
and delta dependencies intact. Scans without query authorization retain the 
existing re-planning path.
   
   **2. Keep scan metrics accurate after cached pruning**
   
   Filtering cached splits no longer invokes core scan planning, so the 
existing file-count gauges would otherwise retain their pre-DPP values. Capture 
the initial planned file count after static pruning, then update the remaining 
count and total runtime reduction after each runtime filter. The reported 
skipped count combines the original static count with that runtime reduction; 
repeated filters or metric reports do not accumulate it again. File counting 
handles wrappers and includes the historical files referenced by chain splits.
   
   Expose the adjusted counts through the existing scan gauges so Spark driver 
metrics and JMX both reflect pruning immediately. Each scan metric group owns 
its runtime counts, preventing an older query's updates from overwriting the 
gauges exported by a newer query on the same table.
   
   ### Tests
   
   Passed on Spark 3.5.9 / JDK 11:
   
   - `SparkPartitionMetadataAuthTest`: 52 cases covering masked and unmasked 
keys, authorization rules, primary-key, append, Data Evolution and chain 
tables, partial masking, grouped scans, and DPP/AQE combinations.
   - `QueryAuthRuntimeFilteringTest`: 9 cases covering stable capabilities, 
successive filters, wrapper preservation, chain dependencies, null values, and 
unsupported predicates.
   - `PaimonMetricTest`: 14 cases, including consistent Spark/JMX counts, 
repeated reporting, and isolation between scan groups.
   
   ```bash
   mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
     -DfailIfNoTests=false \
     -Dtest=SparkPartitionMetadataAuthTest \
     
-DwildcardSuites=org.apache.paimon.spark.read.QueryAuthRuntimeFilteringTest,org.apache.paimon.spark.sql.PaimonMetricTest
 \
     test
   ```
   
   The issue's SQL reproduction also returns IDs `1` and `2` on Spark 3.5.8 
with DPP and AQE independently enabled or disabled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to