GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2707
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that the subtask that wrote a
temporary buffer to the
state backend upon checkpointing, will be also the one to commit it to the
third-party storage system.
This commit removes this assumption. To do this it changes the
CheckpointCommitter to dynamically take the subtaskIdx as a parameter when
committing something to the third-party storage system ( [void
commitCheckpoint(int subtaskIdx, long checkpointID)] ) and when asking
if a checkpoint was committed ( [boolean isCheckpointCommitted(int
subtaskIdx, long checkpointID)] ) and also changes the state kept by the
[GenericWriteAheadSink] to also
include that subtask index of the subtask that wrote the pending buffer.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink write_ahead_sink
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2707.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2707
----
commit 207c5239c9fab6ef09b7bdd410ee83d3d8ca105f
Author: kl0u <[email protected]>
Date: 2016-10-26T15:19:12Z
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that
the subtask that wrote a temporary buffer to the
state backend, will be also the one to commit it to
the third-party storage system.
This commit removes this assumption. To do this
it changes the CheckpointCommitter to dynamically
take the subtaskIdx as a parameter when asking
if a checkpoint was committed and also changes the
state kept by the GenericWriteAheadSink to also
include that subtask index of the subtask that wrote
the pending buffer.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
