taoran92 opened a new issue, #11013: URL: https://github.com/apache/paimon/issues/11013
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.2-SNAPSHOT, master at `f9629393b911634ab490a8cc955a17d7659e030e`. ### Compute Engine Spark 3.5.8. ### Minimal reproduce step A query-authorized table has a masked partition key. Joining on that key fails when Spark applies dynamic partition pruning (DPP). Use the existing `SparkPartitionMetadataAuthTest.PolicySparkCatalog` fixture from the `paimon-spark-ut` test classpath, with a fresh warehouse and these Spark configurations: ```properties spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkPartitionMetadataAuthTest$PolicySparkCatalog spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-dpp-issue-warehouse spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions ``` The `test.*` table options below are specific to this fixture, not production configuration. They allow `id` and `dt`, apply the row filter `id < 3`, and mask `dt` as `'***'`. ```sql CREATE TABLE spark_test.default.T (id INT, dt STRING) USING paimon PARTITIONED BY (dt) TBLPROPERTIES ('primary-key'='dt,id', 'bucket-key'='id', 'bucket'='1'); INSERT INTO spark_test.default.T VALUES (1, '20250810'), (2, '20250811'), (3, '20250810'); CREATE TABLE spark_test.default.D (dt STRING, k INT) USING paimon TBLPROPERTIES ('bucket'='-1'); INSERT INTO spark_test.default.D VALUES ('***', 1), ('other', 0); ALTER TABLE spark_test.default.T SET TBLPROPERTIES ( 'query-auth.enabled'='true', 'test.auth-mode'='constant', 'test.allowed'='id,dt' ); SET spark.sql.optimizer.dynamicPartitionPruning.enabled=true; SELECT l.id FROM spark_test.default.T l JOIN spark_test.default.D r ON l.dt = r.dt WHERE r.k = 1 ORDER BY l.id; ``` Leave AQE and other DPP settings at their defaults. ### What doesn't meet your expectations? Expected: return IDs `1` and `2`, since both authorized rows have the visible partition key `'***'`. Actual: the query fails with: ```text java.lang.UnsupportedOperationException: A partition filter cannot be enforced on masked partition key 'dt': engines push partition predicates past the reader, so it would be matched against the raw partition value. ``` The same query returns IDs `1` and `2` with `spark.sql.optimizer.dynamicPartitionPruning.enabled=false`. With DPP enabled, the query fails whether AQE is enabled or disabled. ### Anything else? Root cause: Spark's runtime-filtering capability currently advertises partition columns without accounting for column masks. DPP derives predicates from visible join values, such as `dt IN ('***')`, but Paimon pushes them into raw partition selection. The core authorization check correctly rejects this mismatch: applying the predicate to raw partition values would incorrectly discard matching rows. Proposed approach: Exclude masked partition columns from runtime-filterable attributes and ignore runtime predicates that reference them. Preserve DPP for unmasked partition columns by filtering the already-authorized split plan, retaining authorization wrappers and chain-read dependencies. For unmasked partition keys, prune whole chain splits using their logical partition values, keeping each retained split's snapshot and delta files intact. Related: #10837. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
