JingsongLi commented on code in PR #10838:
URL: https://github.com/apache/paimon/pull/10838#discussion_r4400490917
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonSparkTableBase.scala:
##########
@@ -116,8 +117,28 @@ abstract class PaimonSparkTableBase(val table: Table)
capabilities
}
+ // Masks can return NULL even for partition keys declared NOT NULL. Expose
this on the
+ // relation as well as the metadata struct so Spark retains null checks and
IS NULL filters.
+ override lazy val schema: StructType = {
+ val tableType = SparkTypeUtils.fromPaimonRowType(table.rowType())
+ if (coreOptions.queryAuthEnabled()) {
+ StructType(tableType.fields.map {
+ field =>
+ if (table.partitionKeys().contains(field.name)) field.copy(nullable
= true) else field
Review Comment:
[P2] Preserve ordinary joins on masked partition keys
Making ordinary partition fields nullable causes Spark to infer `IS NOT
NULL(dt)` for an equality join on `dt`. `PaimonBaseScanBuilder` pushes that
predicate as a physical partition filter, and
`AbstractDataTableScan.rejectMaskedPartitionFilter()` then rejects it when `dt`
is masked. This breaks an existing ordinary-column query even though it has no
explicit partition filter and does not access metadata.
Reproduction using the new test's `PolicySparkCatalog` and the ordinary
table initialized by `createTable(spark, false)`:
```sql
-- Disable DPP/AQE to isolate the newly inferred static predicate.
SET spark.sql.optimizer.dynamicPartitionPruning.enabled=false;
SET spark.sql.adaptive.enabled=false;
-- These test properties configure the PolicySparkCatalog fixture.
ALTER TABLE T SET TBLPROPERTIES (
'query-auth.enabled'='true',
'test.auth-mode'='constant',
'test.allowed'='id,dt,region'
);
SELECT id, dt FROM T; -- succeeds: two visible rows, both with dt = '***'
SELECT l.id, r.id
FROM T l JOIN T r ON l.dt = r.dt;
-- Expected: four rows, since both visible dt values are '***'.
-- Actual: UnsupportedOperationException:
-- A partition filter cannot be enforced on masked partition key 'dt'
```
I reproduced this on Spark 3.5.8/JDK 11 at `8116009d`. Restoring the
baseline schema declaration in an otherwise identical run returns the expected
four rows; excluding `InferFiltersFromConstraints` also makes the current
implementation succeed.
Please retain the nullable schema needed for NULL masks, but keep these
inferred null checks as residual filters evaluated after masking, or route them
through a safe post-mask data-filter path, instead of physical partition
filtering. A regression test joining on the masked partition column would cover
this; the current join tests use `id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
