LuciferYang opened a new pull request, #10924: URL: https://github.com/apache/paimon/pull/10924
### Purpose close #10923 With `lookup.cache = full` (or join keys that are not the primary keys), a primary key table with the aggregation or partial-update merge engine, `changelog-producer = none` and a lookup strategy refreshes the lookup cache from the difference between the files before and after each compaction. `LookupCompactDiffRead` read that difference without keeping delete records, so a deleted row produced no record and the cache kept returning it. The difference is now read with `forceKeepDelete()`, so a deleted row reaches the cache as a delete record, both when compaction keeps the delete record and when it drops the row. The initial full load is a separate read and does not change. Tables with deletion vectors are not covered: streaming plans carry no deletion files, so a delete that compaction only writes into a deletion vector is not visible in this difference. #10813 handles deletion vectors for the partial lookup, which is a different path. ### Tests `LookupJoinITCase#testLookupJoinDeletionPropagationAfterCompaction`, for both ways a compaction handles a delete on an aggregation table with `aggregation.remove-record-on-delete`: - `false`: the delete job compacts into an intermediate level and keeps the delete record. - `true`: the delete stays in level 0 (`write-only`), then a full compaction drops the row; `$files` shows that only one level-5 file is left. In both cases the lookup join must return `(1, null)` after the delete. On master it still returns `(1, 10)`. ### API and Format no ### Documentation no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
