GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/7279
[WIP] [SPARK-7292] Cheap checkpointing
Certain use cases of Spark involve RDDs with long lineages that must be
truncated periodically (e.g. GraphX). The existing way of doing it is through
`rdd.checkpoint()`, which is expensive because it writes to HDFS. This patch
provides an alternative to truncate lineages cheaply *without providing the
same level of fault tolerance*.
**Local checkpointing** writes checkpointed data to the local file system
through the block manager. It is much faster than replicating to a reliable
storage and provides the same semantics as long as executors do not terminate.
It is accessible through a new operator `rdd.localCheckpoint()` and leaves the
old one unchanged. Users may even decide to combine the two and call the
reliable one less frequently.
The bulk of this patch involves refactoring the checkpointing interface to
accept custom implementations of checkpointing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark local-checkpoint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7279.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7279
----
commit a8a106d318a97ed9e93611f3e52318bd74d4fad9
Author: Andrew Or <[email protected]>
Date: 2015-07-07T00:44:16Z
Refactor checkpoint interface for modularity
This commit makes two classes abstract: `RDDCheckpointData` and
`CheckpointRDD`. It implements the existing fault-tolerant
checkpointing by subclassing these abstract classes. The goal
of this commit is to retain as much functionality as possible.
Much of the code is just moved from one file to another.
The following commits will add an implementation for unreliable
checkpointing.
commit 8b291287774a62ef12030e53c481d105cae47061
Author: Andrew Or <[email protected]>
Date: 2015-07-07T01:07:16Z
Fix tests
The parent base class was not serializable while the child is,
causing some java invocation exception. Also, a prior code clean
up caused an array out of bounds exception, which is now fixed
in this commit.
commit c386e1ee739788cc1529815573f9e7a62a79c43a
Author: Andrew Or <[email protected]>
Date: 2015-07-07T18:02:47Z
First implementation of local checkpointing
The write path runs a job to put each partition into disk store,
while the read path simply reads these blocks back from the disk
store.
commit 192d5254838cee73e5c5cfb47bab63c62dc3f75b
Author: Andrew Or <[email protected]>
Date: 2015-07-07T22:01:29Z
Rename a few methods with awkward names (minor)
commit c741f13bfdd8aabcb726687fefaa32ee114f5278
Author: Andrew Or <[email protected]>
Date: 2015-07-07T22:30:55Z
Clean local checkpoint files through RDD cleanups
This commit simplifies the previous one in removing the special
`LocalCheckpointBlockId`, which is not needed if we use the
checkpoint RDD's ID instead of the parent RDD's. This allows us
to simply reuse the RDD cleanup code path, which is nice.
commit 9c43f99e4d4489250b2c8849c9c06a0e8a43010e
Author: Andrew Or <[email protected]>
Date: 2015-07-07T23:51:13Z
Refactor CheckpointSuite to test local checkpointing
This commit makes each test in CheckpointSuite run twice, once
for normal checkpointing and another time for local checkpointing.
This commit also fixes legitimate test failures after the
refactoring.
commit f67d3dafb6edaf5a63b37e251093d330bedd4e7f
Author: Andrew Or <[email protected]>
Date: 2015-07-08T01:39:38Z
Add local checkpoint clean up tests
commit 1324d25f1b0e6279ac5c317e0ecb69d26d76ed3c
Author: Andrew Or <[email protected]>
Date: 2015-07-08T01:48:22Z
Add a few warnings and clear exception messages
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
