GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/9258
[SPARK-8582][Core]Optimize checkpointing to avoid computing an RDD twice
Unlike #7021, this PR uses an approach similar to `persist` to compute and
checkpoint an RDD at the same job. When computing an RDD at the first time,
each partition will be checkpointed and read back as an iterator (maybe
compute it again, not sure which one is better). After finishing the job, we
also check if all partitions are checkpointed, if not, we still need to launch
a job to checkpoint the missing partitions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark optimize-checkpoint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9258.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9258
----
commit c909ef002bd38f716a7a3286a70497e22c445e84
Author: zsxwing <[email protected]>
Date: 2015-10-24T00:56:28Z
Optimize checkpointing to avoid computing an RDD twice
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
