LuciferYang opened a new issue, #10831: URL: https://github.com/apache/paimon/issues/10831
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (ed8ecdc231) ### Compute Engine Flink ### Minimal reproduce step Run a streaming job that writes to a primary key table with `local-merge-buffer-size` set, through a filter (for example a `WHERE` clause) that drops the records while the source watermark keeps advancing. For example, a source that emits one record and a larger watermark every 100 ms, followed by a filter that only lets the first record through, writing to: ```sql CREATE TABLE T (k INT, v INT, PRIMARY KEY (k) NOT ENFORCED) WITH ( 'bucket' = '1', 'local-merge-buffer-size' = '5m', 'commit.force-create-snapshot' = 'true' ); ``` `commit.force-create-snapshot` only makes every checkpoint create a snapshot, so that the committed watermark can be seen while no data is written. ### What doesn't meet your expectations? The watermark of the snapshots stays at the value it had when the last record passed the filter, and only moves again when another record reaches the sink. Without `local-merge-buffer-size`, the snapshot watermark of the same job keeps advancing. Features driven by the watermark, such as `tag.automatic-creation = 'watermark'` and `partition.mark-done-action.mode = 'watermark'`, wait on it. `LocalMergeOperator.processWatermark` stores the watermark, and `flushBuffer()` emits it after the buffered rows. `flushBuffer()` returns early when the buffer is empty, so at a checkpoint with no buffered rows the watermark is not emitted. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
