GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1408
[FLINK-3051] Control the maximum number of concurrent checkpoints
This change introduces a parameter that lets users control at most how many
checkpoints
should be in progress at any given point in time. This is very useful for
cases where the best checkpoint interval is not easy to determine, or where
once in
a while checkpoints may take long. Previously, such situations lead to
checkpoint
queue-up, where the system ended up being more busy with checkpoints than
with processing.
After this change, the checkpoint rate will by default sort of
self-regulate: The checkpoint interval is the most frequent time at which
checkpoints will occur, but they will occur slower if they take longer than
that interval.
### Checkpoint Config
This also introduces the CheckpointConfig class that holds all
checkpointing related parameters to more simply pass them between environment,
streamgraph, etc...
### Default number of concurrent checkpoints
Previously, the maximum number of concurrent checkpoints was implicitly
infinite.
I would suspect that most people will want to run the system such that only
one checkpoint is ever concurrently active, so I set the default value for this
to 1.
In some corner cases, it may be interesting to set that value higher.
### WIP for delay between checkpoints
This also contains some WIP to define a minimum time between checkpoint
attempts. That flag would tell the system not only to not do more than one
checkpoint concurrently, but to leave at least a certain time between
completion of one checkpoint, and the triggering of the next. It basically
defines a guaranteed time that is "work only" between checkpoints.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink
checkpoints_configure
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1408.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1408
----
commit e4d063867ac09cbec6666ecbca92816cfb6faf6c
Author: Stephan Ewen <[email protected]>
Date: 2015-11-19T17:03:55Z
[hotfix] Java-7-ify the ExecutionConfig class
commit 79def44a2e973c8b6817821d5d7342731f7e7aa2
Author: Stephan Ewen <[email protected]>
Date: 2015-11-19T18:05:47Z
[FLINK-3051] [streaming] Add mechanisms to control the maximum number of
concurrent checkpoints
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
