[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401627#comment-16401627
]
Kostas Kloudas edited comment on FLINK-5601 at 3/16/18 9:13 AM:
----------------------------------------------------------------
This is an important and known issue.
It was also encountered during the implementation of the triggerDSL (which was
not finally merged).
Back then we did not have union state. I think that now, the
{{TimestampsAndPeriodicWatermarksOperator}} (and not the {{windowOperator}} as
users may play with the watermark in many places in their pipeline) or the
source (whoever has the watermark emitter) should checkpoint its watermark in a
union state, and upon restoring/rescaling, its task should take its previously
checkpointed watermark, and in case of scaling up, the new tasks should take
the minimum of the previously checkpointed watermarks.
What do you think?
was (Author: kkl0u):
This is an important and known issue.
It was also encountered during the implementation of the triggerDSL (which was
not finally merged).
Back then we did not have union state. I think that now, the
TimestampsAndPeriodicWatermarksOperator or the source (whoever has the
watermark emitter) should checkpoint its watermark in a union state, and upon
restoring/rescaling, its task should take its previously checkpointed
watermark, and in case of scaling up, the new tasks should take the minimum of
the previously checkpointed watermarks.
What do you think?
> Window operator does not checkpoint watermarks
> ----------------------------------------------
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Reporter: Ufuk Celebi
> Priority: Major
>
> During release testing [[email protected]] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
