taoran92 opened a new issue, #11013:
URL: https://github.com/apache/paimon/issues/11013

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   2.2-SNAPSHOT, master at `f9629393b911634ab490a8cc955a17d7659e030e`.
   
   ### Compute Engine
   
   Spark 3.5.8.
   
   ### Minimal reproduce step
   
   A query-authorized table has a masked partition key. Joining on that key 
fails when Spark applies dynamic partition pruning (DPP).
   
   Use the existing `SparkPartitionMetadataAuthTest.PolicySparkCatalog` fixture 
from the `paimon-spark-ut` test classpath, with a fresh warehouse and these 
Spark configurations:
   
   ```properties
   
spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkPartitionMetadataAuthTest$PolicySparkCatalog
   spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-dpp-issue-warehouse
   
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
   ```
   
   The `test.*` table options below are specific to this fixture, not 
production configuration. They allow `id` and `dt`, apply the row filter `id < 
3`, and mask `dt` as `'***'`.
   
   ```sql
   CREATE TABLE spark_test.default.T (id INT, dt STRING)
   USING paimon PARTITIONED BY (dt)
   TBLPROPERTIES ('primary-key'='dt,id', 'bucket-key'='id', 'bucket'='1');
   
   INSERT INTO spark_test.default.T VALUES
     (1, '20250810'), (2, '20250811'), (3, '20250810');
   
   CREATE TABLE spark_test.default.D (dt STRING, k INT)
   USING paimon TBLPROPERTIES ('bucket'='-1');
   
   INSERT INTO spark_test.default.D VALUES ('***', 1), ('other', 0);
   
   ALTER TABLE spark_test.default.T SET TBLPROPERTIES (
     'query-auth.enabled'='true',
     'test.auth-mode'='constant',
     'test.allowed'='id,dt'
   );
   
   SET spark.sql.optimizer.dynamicPartitionPruning.enabled=true;
   
   SELECT l.id
   FROM spark_test.default.T l
   JOIN spark_test.default.D r ON l.dt = r.dt
   WHERE r.k = 1
   ORDER BY l.id;
   ```
   
   Leave AQE and other DPP settings at their defaults.
   
   ### What doesn't meet your expectations?
   
   Expected: return IDs `1` and `2`, since both authorized rows have the 
visible partition key `'***'`.
   
   Actual: the query fails with:
   
   ```text
   java.lang.UnsupportedOperationException: A partition filter cannot be 
enforced on masked partition key 'dt': engines push partition predicates past 
the reader, so it would be matched against the raw partition value.
   ```
   
   The same query returns IDs `1` and `2` with 
`spark.sql.optimizer.dynamicPartitionPruning.enabled=false`. With DPP enabled, 
the query fails whether AQE is enabled or disabled.
   
   ### Anything else?
   
   Root cause: Spark's runtime-filtering capability currently advertises 
partition columns without accounting for column masks. DPP derives predicates 
from visible join values, such as `dt IN ('***')`, but Paimon pushes them into 
raw partition selection. The core authorization check correctly rejects this 
mismatch: applying the predicate to raw partition values would incorrectly 
discard matching rows.
   
   Proposed approach: Exclude masked partition columns from runtime-filterable 
attributes and ignore runtime predicates that reference them. Preserve DPP for 
unmasked partition columns by filtering the already-authorized split plan, 
retaining authorization wrappers and chain-read dependencies. For unmasked 
partition keys, prune whole chain splits using their logical partition values, 
keeping each retained split's snapshot and delta files intact.
   
   Related: #10837.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to