JingsongLi commented on code in PR #10924:
URL: https://github.com/apache/paimon/pull/10924#discussion_r4438994089
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/lookup/LookupCompactDiffRead.java:
##########
@@ -45,6 +45,9 @@ public class LookupCompactDiffRead extends
AbstractDataTableRead {
public LookupCompactDiffRead(MergeFileSplitRead mergeRead, TableSchema
schema) {
super(schema);
this.incrementalDiffRead = new
IncrementalCompactDiffSplitRead(mergeRead);
+ // deleted rows must reach the lookup cache as DELETE records, both
when compaction keeps
+ // the delete record and when it drops the row and the row is only in
the files before
+ this.incrementalDiffRead.forceKeepDelete();
Review Comment:
[P2] Retract the cached secondary key before applying key-only deletes
This now sends sequence-group tombstones to `SecondaryIndexLookupTable`,
whose DELETE branch retracts the secondary key from the delete payload instead
of the cached row. With `merge-engine=partial-update`,
`fields.f2.sequence-group=f1`,
`partial-update.remove-record-on-sequence-group=f2`, `force-lookup=true`,
`changelog-producer=none`, and a lookup join on `f1`, write `(1,11,1)`,
compact/open the cache, then delete `(1,NULL,2)` and compact to an intermediate
level. The tombstone removes primary state but leaves the old `11 -> 1` index
entry. Reinserting `(1,22,3)` and compacting then makes lookup key `11` return
`(1,22,3)`, although that row only matches key `22`.
I reproduced the complete write/compaction/cache-refresh sequence in both
MEMORY and FULL caches, with an assertion that the actual mode is
`COMPACT_DELTA_MONITOR`. The final old-key assertion fails at this head and
passes with only this new `forceKeepDelete()` call removed. Please retract the
secondary key from the previously cached row before deleting primary state, and
cover delete/reinsert with a changed secondary key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
