taoran92 opened a new pull request, #11014:
URL: https://github.com/apache/paimon/pull/11014
### Purpose
Fixes #11013.
Spark JOINs on masked partition keys fail when dynamic partition pruning
(DPP) pushes visible join values into raw partition selection. For example, a
key masked to `'***'` must be matched by the JOIN after masking, rather than
used to select physical partitions.
**1. Authorization-aware runtime filtering**
Determine eligible output partition columns once from the initial authorized
split plan, excluding columns masked in any split. Advertise only those columns
to Spark and validate incoming runtime predicates against the same stable set.
For unmasked partition keys, prune the cached authorized splits and rebuild
input partitions, preserving authorization and fallback wrappers. Chain splits
are filtered by their logical partition values and retained with all snapshot
and delta dependencies intact. Scans without query authorization retain the
existing re-planning path.
**2. Keep scan metrics accurate after cached pruning**
Filtering cached splits no longer invokes core scan planning, so the
existing file-count gauges would otherwise retain their pre-DPP values. Capture
the initial planned file count after static pruning, then update the remaining
count and total runtime reduction after each runtime filter. The reported
skipped count combines the original static count with that runtime reduction;
repeated filters or metric reports do not accumulate it again. File counting
handles wrappers and includes the historical files referenced by chain splits.
Expose the adjusted counts through the existing scan gauges so Spark driver
metrics and JMX both reflect pruning immediately. Each scan metric group owns
its runtime counts, preventing an older query's updates from overwriting the
gauges exported by a newer query on the same table.
### Tests
Passed on Spark 3.5.9 / JDK 11:
- `SparkPartitionMetadataAuthTest`: 52 cases covering masked and unmasked
keys, authorization rules, primary-key, append, Data Evolution and chain
tables, partial masking, grouped scans, and DPP/AQE combinations.
- `QueryAuthRuntimeFilteringTest`: 9 cases covering stable capabilities,
successive filters, wrapper preservation, chain dependencies, null values, and
unsupported predicates.
- `PaimonMetricTest`: 14 cases, including consistent Spark/JMX counts,
repeated reporting, and isolation between scan groups.
```bash
mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
-DfailIfNoTests=false \
-Dtest=SparkPartitionMetadataAuthTest \
-DwildcardSuites=org.apache.paimon.spark.read.QueryAuthRuntimeFilteringTest,org.apache.paimon.spark.sql.PaimonMetricTest
\
test
```
The issue's SQL reproduction also returns IDs `1` and `2` on Spark 3.5.8
with DPP and AQE independently enabled or disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]