Arvid Heise created FLINK-18405:
-----------------------------------
Summary: Add watermark support for unaligned checkpoints
Key: FLINK-18405
URL: https://issues.apache.org/jira/browse/FLINK-18405
Project: Flink
Issue Type: Improvement
Affects Versions: 1.12.0
Reporter: Arvid Heise
Currently, Flink generates the watermark as a first step of recovery instead of
storing the latest watermark in the operators to ease rescaling. In unaligned
checkpoints, that means on recovery, Flink generates watermarks after it
restores in-flight data. If your pipeline uses an operator that applies the
latest watermark on each record, it will produce incorrect results during
recovery if the watermark is not directly or indirectly part of the operator
state. Thus, SQL OVER operator should not be used with unaligned
checkpoints, while window operators are safe to use.
A possible solution is to store the watermark in the operator state. If
rescaling may occur, watermarks should be stored per key-group in a
union-state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
