taoran92 commented on code in PR #10838:
URL: https://github.com/apache/paimon/pull/10838#discussion_r4409245773
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonSparkTableBase.scala:
##########
@@ -116,8 +117,28 @@ abstract class PaimonSparkTableBase(val table: Table)
capabilities
}
+ // Masks can return NULL even for partition keys declared NOT NULL. Expose
this on the
+ // relation as well as the metadata struct so Spark retains null checks and
IS NULL filters.
+ override lazy val schema: StructType = {
+ val tableType = SparkTypeUtils.fromPaimonRowType(table.rowType())
+ if (coreOptions.queryAuthEnabled()) {
+ StructType(tableType.fields.map {
+ field =>
+ if (table.partitionKeys().contains(field.name)) field.copy(nullable
= true) else field
Review Comment:
Hi @JingsongLi
Thanks for catching this. The second commit keeps partition columns nullable
when query authorization is enabled, but leaves partition-column `IS NOT NULL`
predicates as Spark residual filters. Spark evaluates them on the visible,
post-mask values, preventing inferred JOIN null checks from becoming unsafe
physical partition filters.
I reproduced your JOIN case and added regression coverage for constant,
cross-column, and NULL masks, including explicit IS NOT NULL queries. These
tests pass on Spark 3.2.4 and 3.5.8.
I also confirmed a separate DPP limitation: runtime filters on masked
partition columns can still reach physical partition pruning and trigger the
same protection. This also reproduces before this PR. I’ll address safe DPP
handling for query-authorized scans in a follow-up PR.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
