taoran92 opened a new issue, #10837: URL: https://github.com/apache/paimon/issues/10837
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. Related fixes are listed below; they do not cover partition metadata on chain or query-authorized splits. ### Paimon version `2.2-SNAPSHOT`, Apache Paimon `master` at `f35f2d84671a8c1429182d2c0f17ad8fd4f3efc2`. This revision includes [#10725](https://github.com/apache/paimon/pull/10725), which preserves query authorization requirements through fallback and chain scans and separates internal partition selection from user partition filters. ### Compute Engine Apache Spark 3.5.x. The focused reader-level reproduction used Spark 3.5.8 dependencies and JDK 11. ### Minimal reproduce step The chain-table case does not require query authorization, column masking, or a REST catalog. The SQL below is reduced from the setup in the existing `SparkChainTableMetadataTest`, which currently asserts that chain partition metadata queries fail. Start a Spark session with the Paimon Spark connector, its session extensions, and a filesystem catalog named `spark_test`. For example, use these session startup options with a fresh warehouse directory: ```text --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions --conf spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkCatalog --conf spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-issue-c-warehouse ``` Create a chain table with a snapshot and a later delta: ```sql CREATE DATABASE spark_test.issue_c; CREATE TABLE spark_test.issue_c.orders ( id INT, payload STRING, seq BIGINT, dt STRING ) USING paimon PARTITIONED BY (dt) TBLPROPERTIES ( 'primary-key' = 'dt,id', 'bucket-key' = 'id', 'bucket' = '1', 'sequence.field' = 'seq', 'chain-table.enabled' = 'true', 'partition.timestamp-pattern' = '$dt', 'partition.timestamp-formatter' = 'yyyyMMdd' ); CALL spark_test.sys.create_branch('issue_c.orders', 'snapshot'); CALL spark_test.sys.create_branch('issue_c.orders', 'delta'); ALTER TABLE spark_test.issue_c.orders SET TBLPROPERTIES ( 'scan.fallback-snapshot-branch' = 'snapshot', 'scan.fallback-delta-branch' = 'delta' ); ALTER TABLE spark_test.issue_c.`orders$branch_snapshot` SET TBLPROPERTIES ( 'scan.fallback-snapshot-branch' = 'snapshot', 'scan.fallback-delta-branch' = 'delta' ); ALTER TABLE spark_test.issue_c.`orders$branch_delta` SET TBLPROPERTIES ( 'scan.fallback-snapshot-branch' = 'snapshot', 'scan.fallback-delta-branch' = 'delta' ); INSERT INTO spark_test.issue_c.`orders$branch_snapshot` VALUES (1, 'old', 1, '20250810'), (2, 'keep', 1, '20250810'); INSERT INTO spark_test.issue_c.`orders$branch_delta` VALUES (1, 'new', 2, '20250811'); -- Ordinary columns: read the merged logical partition. SELECT id, payload, dt FROM spark_test.issue_c.orders WHERE dt = '20250811' ORDER BY id; -- Partition metadata on the snapshot-only chain read fails. SELECT id, __paimon_partition FROM spark_test.issue_c.orders WHERE dt = '20250810'; -- Partition metadata on the snapshot + delta chain read also fails. SELECT id, dt, __paimon_partition.dt AS metadata_dt FROM spark_test.issue_c.orders WHERE dt = '20250811'; -- A metadata-only projection and a metadata predicate encounter the same restriction. SELECT __paimon_partition.dt FROM spark_test.issue_c.orders WHERE dt = '20250811'; SELECT id FROM spark_test.issue_c.orders WHERE dt = '20250811' AND __paimon_partition.dt = '20250811'; ``` The failure is: ```text java.lang.UnsupportedOperationException: Partition metadata is only supported for DataSplit without query authorization. ``` There is a second affected path: an ordinary table read with authorization rules. Given a REST catalog that implements query authorization, enable `query-auth.enabled` on an ordinary partitioned table and grant the querying user access to `id` and `dt`. Return a row-filter policy such as `id > 0`, without column masks, and ensure the table contains a matching row. Reading `id, dt` is permitted, but adding `__paimon_partition.dt` encounters the same restriction because the scan produces `QueryAuthSplit(DataSplit)`. This case does not require a chain table or a masking policy. For ordinary tables, merely enabling authorization without any row-filter or masking rules does not trigger this split-type failure: the scan retains its original `DataSplit`. A valid authorization service and a policy that produces the wrapper are required for this second case. ### What doesn't meet your expectations? Queries should be able to project and filter `__paimon_partition` on chain tables and query-authorized reads, with the following behavior: - For a chain read, metadata should describe the logical partition of the returned row. In the merged `20250811` partition above, both the updated row and the row retained from the older snapshot should have `dt = metadata_dt = '20250811'`. - With query authorization, metadata must expose the same visible partition values as the corresponding ordinary columns. For example, if `dt` is masked to `'***'`, both `dt` and `__paimon_partition.dt` should return `'***'`. This also applies to masks that depend on another column or return `NULL`. - Accessing `__paimon_partition.dt`, including only in a predicate, must require permission on `dt` and must not bypass a denied column through metadata. With Spark's default `spark.sql.optimizer.nestedSchemaPruning.enabled=true`, accessing a single metadata subfield must not require permission on unrelated partition fields. Reading the complete partition struct must authorize all its exposed fields. - Row filters must still apply. Predicates on partition metadata, and queries combining them with `COUNT(*)` or `LIMIT`, should operate on the visible values without exposing or filtering against raw masked partition values. Scope when nested schema pruning is disabled: with `spark.sql.optimizer.nestedSchemaPruning.enabled=false`, Spark requests the complete partition struct even when the query only accesses `.dt`. Authorization therefore requires all partition fields, including fields such as `region` that the query does not explicitly access. Reads, row filters and column masks remain supported when those permissions are available. Precise subfield authorization with this setting disabled is a separate follow-up and is outside the scope of this issue's proposed fix. For example, after configuring a mask of `dt -> '***'`, this query should compare the metadata with the masked value: ```sql SELECT id, dt, __paimon_partition.dt AS metadata_dt FROM spark_test.issue_c.orders WHERE __paimon_partition.dt = '***'; ``` This does not require changing the existing restriction on explicit physical partition filters over masked columns. ### Anything else? The [reader](https://github.com/apache/paimon/blob/f35f2d84671a8c1429182d2c0f17ad8fd4f3efc2/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonRecordReaderIterator.scala#L40-L49) currently rejects partition metadata for chain and query-authorized splits. Metadata must come from visible rows to preserve logical partition values and column masks. Local validation of the proposed fix: 22 cases passed on each of Spark 3.2.4 and 3.3.4 (JDK 11), and Spark 4.0.3 and 4.1.2 (JDK 17), using real Spark reads with test-catalog authorization policies. Related PRs, which do not cover this partition metadata support: - [#10481](https://github.com/apache/paimon/pull/10481): partition and bucket metadata for ordinary merge reads. - [#10652](https://github.com/apache/paimon/pull/10652): chain bucket metadata; retains the partition metadata restriction. - [#10725](https://github.com/apache/paimon/pull/10725): core authorization and internal partition-selection prerequisites. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
