kerwin-zk opened a new pull request, #9155:
URL: https://github.com/apache/paimon/pull/9155
### Purpose
`MergeIntoPaimonTable` extracts the target-only conjuncts of the merge
condition and uses them to prune the target table before the full outer join
(`targetOnlyCondition` / `filteredTargetPlan`,
`MergeIntoPaimonTable.scala:63-69`).
That pruning is sound for `WHEN MATCHED` and `WHEN NOT MATCHED`: a target
row that fails a target-only conjunct can never satisfy the whole merge
condition, so it can never be matched, and dropping it cannot change the
outcome of those actions.
It is **not** sound for `WHEN NOT MATCHED BY SOURCE`. The pruned-away rows
are exactly the population that clause is defined over, so their actions are
silently skipped — no error, no warning, just fewer rows changed.
Repro on a partitioned table holding `pt = 'p1'` and `pt = 'p2'` rows, with
a source that only contains `a = 1`:
```sql
MERGE INTO target t
USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'
```
Every `pt = 'p2'` row should be updated to `stale` (no source row can ever
match it), but none of them is:
```
Expected Actual
[1,100,c1,p1] [1,100,c1,p1]
[2,20,stale,p1] [2,20,stale,p1]
![3,30,stale,p2] [3,30,c3,p2]
![4,40,stale,p2] [4,40,c4,p2]
```
The pruning was introduced together with MERGE INTO itself in #2331
(2023-11-17), one month before `WHEN NOT MATCHED BY SOURCE` was added in #2517
(2023-12-22), and its safety argument was never revisited.
Note the V2 row-level paths (`ReplaceData` / `WriteDelta`) are rewritten by
Spark and are **not** affected, so today the same statement produces different
results depending on whether the table qualifies for
`SparkTable.supportsV2RowLevelOps`. Primary key tables never qualify, so they
always take the affected V1 path.
**Fix:** disable the pruning when the merge has any `WHEN NOT MATCHED BY
SOURCE` action. Setting `targetOnlyCondition` to `None` covers all three places
it feeds — `filteredTargetPlan`, `findCandidateDataSplits` and
`targetDSWithFilePathCol` — so there is no path left that can drop those rows.
A follow-up can restore part of the pruning by handling the excluded rows as
a separate not-matched-by-source-only stream: they are all
not-matched-by-source by definition, so they never need to be joined against
the source, only read and rewritten.
### Tests
Added `Paimon MergeInto: not matched by source is not narrowed by
target-only condition` to the `MergeIntoNotMatchedBySourceTest` trait, which is
mixed into all eight MERGE INTO test classes (V1/V2 x primary key/append x
bucketed/non-bucketed).
Verified both directions on `paimon-spark-3.5`:
| | with fix | without fix |
| --- | --- | --- |
| total | `succeeded 476, failed 0` | `succeeded 470, failed 6` |
| failures | none | the 6 V1 classes, all on the new test |
Without the fix, the split is exactly along the V1/V2 line, which also pins
down that primary key tables always take the V1 path:
```
FAILED MergeIntoPrimaryKeyBucketedTableTest
FAILED MergeIntoPrimaryKeyNonBucketTableTest
FAILED MergeIntoAppendBucketedTableTest
FAILED MergeIntoAppendNonBucketedTableTest
FAILED V2MergeIntoPrimaryKeyBucketedTableTest
FAILED V2MergeIntoPrimaryKeyNonBucketTableTest
passed V2MergeIntoAppendBucketedTableTest
passed V2MergeIntoAppendNonBucketedTableTest
```
### API and Format
No API or format change.
### Documentation
No documentation change; this restores the documented SQL semantics of `WHEN
NOT MATCHED BY SOURCE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
