[
https://issues.apache.org/jira/browse/FLINK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612488#comment-15612488
]
ASF GitHub Bot commented on FLINK-4939:
---------------------------------------
GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2707
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that the subtask that wrote a
temporary buffer to the
state backend upon checkpointing, will be also the one to commit it to the
third-party storage system.
This commit removes this assumption. To do this it changes the
CheckpointCommitter to dynamically take the subtaskIdx as a parameter when
committing something to the third-party storage system ( [void
commitCheckpoint(int subtaskIdx, long checkpointID)] ) and when asking
if a checkpoint was committed ( [boolean isCheckpointCommitted(int
subtaskIdx, long checkpointID)] ) and also changes the state kept by the
[GenericWriteAheadSink] to also
include that subtask index of the subtask that wrote the pending buffer.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink write_ahead_sink
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2707.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2707
----
commit 207c5239c9fab6ef09b7bdd410ee83d3d8ca105f
Author: kl0u <[email protected]>
Date: 2016-10-26T15:19:12Z
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that
the subtask that wrote a temporary buffer to the
state backend, will be also the one to commit it to
the third-party storage system.
This commit removes this assumption. To do this
it changes the CheckpointCommitter to dynamically
take the subtaskIdx as a parameter when asking
if a checkpoint was committed and also changes the
state kept by the GenericWriteAheadSink to also
include that subtask index of the subtask that wrote
the pending buffer.
----
> GenericWriteAheadSink: Decouple the creating from the committing subtask for
> a pending checkpoint
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-4939
> URL: https://issues.apache.org/jira/browse/FLINK-4939
> Project: Flink
> Issue Type: Improvement
> Components: Cassandra Connector
> Reporter: Kostas Kloudas
> Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> So far the GenericWriteAheadSink expected that
> the subtask that wrote a pending checkpoint to the
> state backend, will be also the one to commit it to
> the third-party storage system.
> This issue targets at removing this assumption. To do this
> the CheckpointCommitter has to be able to dynamically
> take the subtaskIdx as a parameter when asking
> if a checkpoint was committed and also change the
> state kept by the GenericWriteAheadSink to also
> include that subtask index of the subtask that wrote
> the pending checkpoint.
> This change is also necessary for making the operator rescalable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
