GitHub user gyfora opened a pull request:
https://github.com/apache/flink/pull/676
Pluggable state backend added
This PR introduces the pluggable state backends using StateHandleProviders
and extends the StateHandle interface with a discard method for cleaning up the
unnecessary checkpoints.
It also adds a statehandle/provider implementation for storing checkpoints
in any flink supported file system such as HDFS or Tachyon.
The checkpoint coordinator has been modified to properly discard user state
handles using the following logic:
- If a pending checkpoint expires (by the timer thread) it discards the
user state
- When a successful checkpoint expires (by acquiring following successful
ones) it discards user states
- When the checkpoint coordinator is shut down it discards all pending and
successful states
Travis error:
I modified the recovery IT case to use the local file system for storing
the checkpoints. Afterwards it checks whether the directory is empty. The test
passes all the time ran locally, but it seems to fail on travis for no apparent
reason. Usually a couple of files (2-5) remain in the checkpoint directory,
meaning that almost all of them had been deleted but those. Also the
checkpointing and recovery logic runs fine without test failure.
I would appreciate some help figuring this out somehow, or trying to
reproduce it locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gyfora/flink statehandle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/676.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #676
----
commit 22b5996e046cf83612fac2cb5aa02f2fd76a7e7b
Author: Gyula Fora <gyf...@apache.org>
Date: 2015-05-07T12:29:23Z
[streaming] ByteStream and File state handle added
commit fed66675e2e824eee00b88197c5a73882415c919
Author: Gyula Fora <gyf...@apache.org>
Date: 2015-05-14T09:49:30Z
[streaming] Discard method added to state handle
commit 36474aafe74be9b61a89b5240bbc39f47226da77
Author: Gyula Fora <gyf...@apache.org>
Date: 2015-05-14T19:23:42Z
[streaming] StateHandleProvider added for configurable state backend
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
