LuciferYang opened a new issue, #10923: URL: https://github.com/apache/paimon/issues/10923
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (30d91278af) ### Compute Engine Flink ### Minimal reproduce step ```sql CREATE TABLE dim (i INT PRIMARY KEY NOT ENFORCED, cnt BIGINT) WITH ( 'merge-engine' = 'aggregation', 'fields.cnt.aggregate-function' = 'sum', 'aggregation.remove-record-on-delete' = 'true', 'force-lookup' = 'true', 'lookup.cache' = 'full' ); INSERT INTO dim VALUES (1, 10), (2, 20); -- a streaming lookup join against dim is running SELECT T.i, D.cnt FROM T LEFT JOIN dim FOR SYSTEM_TIME AS OF T.proctime AS D ON T.i = D.i; DELETE FROM dim WHERE i = 1; ``` ### What doesn't meet your expectations? After the delete, the lookup join keeps returning `(1, 10)` for key 1, while `SELECT * FROM dim` no longer returns it. Partial-update tables that remove rows show the same. A primary key table with the aggregation or partial-update merge engine, `changelog-producer = none` and a lookup strategy is refreshed in the `COMPACT_DELTA_MONITOR` mode when `lookup.cache = full` or the join keys are not the primary keys. The refresh reads the difference between the files before and after each compaction through `LookupCompactDiffRead`, which does not keep delete records. A deleted row then produces no record at all, whether compaction keeps its delete record in an intermediate level or drops the row so that it only exists in the files before the compaction, and the full cache never removes the key. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
