[
https://issues.apache.org/jira/browse/BEAM-6678?focusedWorklogId=199203&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199203
]
ASF GitHub Bot logged work on BEAM-6678:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/19 12:44
Start Date: 15/Feb/19 12:44
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #7849: [BEAM-6678]
Persist watermark holds view in Flink checkpoints
URL: https://github.com/apache/beam/pull/7849
The FlinkRunner maintains a partition view of the pending watermark holds.
This
view is used to hold back the watermark at the operator. It must be
persisted,
otherwise elements may be late after recovering a Beam job.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | ---
See [.test-infra/jenkins/README](../.test-infra/jenkins/README.md) for
trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 199203)
Time Spent: 10m
Remaining Estimate: 0h
> FlinkRunner does not checkpoint partition view of watermark holds
> -----------------------------------------------------------------
>
> Key: BEAM-6678
> URL: https://issues.apache.org/jira/browse/BEAM-6678
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The FlinkRunner does not persist its view of the per-partition Watermark
> holds. This can lead to elements to be considered late after restoring from a
> savepoint or resuming a failed pipeline.
> Similar to the approach in BEAM-6650, we can recover the Watermarks by
> iterating through the keys of the state backend during recovery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
