taoran92 opened a new pull request, #10838: URL: https://github.com/apache/paimon/pull/10838
### Purpose Fixes #10837. Spark queries that project or filter `__paimon_partition` currently fail on chain tables and query-authorized reads. Build partition metadata from visible reader rows so retained snapshot rows report the logical chain partition, while authorized reads preserve row filters, column permissions and column masks, including masks that depend on another field or return `NULL`. Read and authorize the requested metadata subfields, then remove any added ordinary fields from the Spark output. Preserve nested pruning and nullable masked partition fields, and keep fields added only for metadata out of ordinary runtime-filter attributes. Document the authorization scope when nested schema pruning is disabled. ### Tests Passed locally with Spark 3.5.8 and JDK 11: - `SparkPartitionMetadataAuthTest` and `SparkChainTableMetadataTest`: 22 cases covering ordinary, chain and fallback reads, row filters, denied columns, constant/field-dependent/NULL masks, both nested-pruning settings, metadata predicates, aggregates, limits, joins, typed partitions and blob descriptors. - `SplitUtilsQueryAuthTest`: 4 cases covering visible partition values, split wrappers and reader cleanup. - 16 additional SQL assertions covering Parquet, ORC and Avro, typed and NULL partitions, empty partition structs, mixed metadata projections, and NULL-mask counts and JSON output. Checkstyle, Spotless and Enforcer passed for the affected Spark modules and their dependencies without `fast-build`. Apache RAT and `git diff --check` also passed. Cross-implementation scope: the Rust/DataFusion schema and Python's own and Native reader paths were inspected separately. They do not expose the same Spark hidden partition struct, so no corresponding bug patch is needed for this change; adding that interface would be separate feature work. Rust/Python runtime tests were not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
