GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2759
[FLINK-5020] Make the GenericWriteAheadSink rescalable.
As the issue suggests, this is the final commit to make the
GenericWriteAheadSink re-scalable.
In essence, what it does is that it replaces the old snapshot()/restore()
cycle with the new snapshotState()/initializeState() and adds tests to show
that it works as expected.
R: @zentol
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink rescaling_wr_ahead
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2759.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2759
----
commit 1cdfff0f56a77616f36addfa1cb48b2efe2c9626
Author: kl0u <[email protected]>
Date: 2016-10-26T15:19:12Z
[FLINK-4939] GenericWriteAheadSink: Decouple the creating from the
committing subtask for a pending checkpoint
So far the GenericWriteAheadSink expected that
the subtask that wrote a temporary buffer to the
state backend, will be also the one to commit it to
the third-party storage system.
This commit removes this assumption. To do this
it changes the CheckpointCommitter to dynamically
take the subtaskIdx as a parameter when asking
if a checkpoint was committed and also changes the
state kept by the GenericWriteAheadSink to also
include that subtask index of the subtask that wrote
the pending buffer.
commit 77c3892687f78780c19710a95f8830907fe67c86
Author: kl0u <[email protected]>
Date: 2016-11-03T11:28:37Z
Integrated PR comments
commit c8627568c375cdd37ac0a8314fc5bf56077a578e
Author: kl0u <[email protected]>
Date: 2016-11-04T15:05:19Z
Removes redundant safety check.
commit 2b9c28059e84ba09068ed0350680730f77119253
Author: kl0u <[email protected]>
Date: 2016-11-03T20:46:58Z
[FLINK-5020] Make the GenericWriteAheadSink rescalable.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
