[
https://issues.apache.org/jira/browse/FLINK-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637222#comment-15637222
]
ASF GitHub Bot commented on FLINK-5020:
---------------------------------------
GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2759
[FLINK-5020] Make the GenericWriteAheadSink rescalable.
As the issue suggests, this is the final commit to make the
GenericWriteAheadSink re-scalable.
In essence, what it does is that it replaces the old snapshot()/restore()
cycle with the new snapshotState()/initializeState() and adds tests to show
that it works as expected.
R: @zentol
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink rescaling_wr_ahead
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2759.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2759
----
commit 1cdfff0f56a77616f36addfa1cb48b2efe2c9626
Author: kl0u <[email protected]>
Date: 2016-10-26T15:19:12Z
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that
the subtask that wrote a temporary buffer to the
state backend, will be also the one to commit it to
the third-party storage system.
This commit removes this assumption. To do this
it changes the CheckpointCommitter to dynamically
take the subtaskIdx as a parameter when asking
if a checkpoint was committed and also changes the
state kept by the GenericWriteAheadSink to also
include that subtask index of the subtask that wrote
the pending buffer.
commit 77c3892687f78780c19710a95f8830907fe67c86
Author: kl0u <[email protected]>
Date: 2016-11-03T11:28:37Z
Integrated PR comments
commit c8627568c375cdd37ac0a8314fc5bf56077a578e
Author: kl0u <[email protected]>
Date: 2016-11-04T15:05:19Z
Removes redundant safety check.
commit 2b9c28059e84ba09068ed0350680730f77119253
Author: kl0u <[email protected]>
Date: 2016-11-03T20:46:58Z
[FLINK-5020] Make the GenericWriteAheadSink rescalable.
----
> Make the GenericWriteAheadSink rescalable.
> ------------------------------------------
>
> Key: FLINK-5020
> URL: https://issues.apache.org/jira/browse/FLINK-5020
> Project: Flink
> Issue Type: Improvement
> Components: Cassandra Connector
> Reporter: Kostas Kloudas
> Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> This targets integrating the GenericWriteAheadSink with the new rescalable
> state abstractions so that the parallelism of the operator can change
> arbitrarily without jeopardizing the guarantees offered by it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
