[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324800#comment-17324800
]
Jiayi Liao commented on FLINK-5601:
-----------------------------------
[~liyu] Yes. My idea is to add watermark as state into {{WatermarksOperator}}
like what I did in the https://github.com/apache/flink/pull/7013. This will
change {{WatermarksOperator}} from stateless to stateful. It looks tricky but
it seems to be the best way to solve it now. I'll ping everyone involved in the
discussion before. Do you think we need to start a discussion on dev mail list
?
cc [~kkl0u] [~aljoscha][~klion26][~sewen] Do you have any new insights on this
issue ?
> Window operator does not checkpoint watermarks
> ----------------------------------------------
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.11.0
> Reporter: Ufuk Celebi
> Assignee: Jiayi Liao
> Priority: Critical
> Labels: pull-request-available, stale-assigned
>
> During release testing [[email protected]] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
