[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674048#comment-16674048
]
ASF GitHub Bot commented on FLINK-5601:
---------------------------------------
buptljy opened a new pull request #7013: [FLINK-5601][Checkpointing] Watermark
checkpointing
URL: https://github.com/apache/flink/pull/7013
## What is the purpose of the change
This pull request uses ListState in
TimestampsAndPeriodicWatermarksOperator.java and
TimestampsAndPunctuatedWatermarksOperator.java to checkpoint watermark. During
recovering process, it uses union state to find the lowest watermark and emit
it immediately.
## Brief change log
* Add ListState to store watermark in
TimestampsAndPeriodicWatermarksOperator.java and
TimestampsAndPunctuatedWatermarksOperator.java.
## Verifying this change
* Unit testing in WatermarkCheckpointingITCase.java.
* TimestampsAndPeriodicWatermarksOperator testing: Send five "Five",
then checkpoint, then make the job fail, then recover the job, and send five
"Three", the sum should be 25.
* TimestampsAndPunctuatedWatermarksOperator testing: Similar testing
logic machenism as above.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Window operator does not checkpoint watermarks
> ----------------------------------------------
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.5.0, 1.6.0, 1.7.0
> Reporter: Ufuk Celebi
> Assignee: Jiayi Liao
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> During release testing [[email protected]] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
