GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/9428
Pr7021
Took over #7021 and fixed the following potential issues in the previous PR:
1. Make sure checkpointing all data even if the Iterator is not drained.
2. Make sure checkpointing all partitions even if some partitions are not
touched.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark pr7021
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9428.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9428
----
commit d8635168a9f01e3be2b53a27cc5918a1a0ed1612
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-06-25T16:50:45Z
Add CheckpointingIterator to optimize checkpointing.
commit 1a3055ea6ca67fcb23c1188b7c3344c726b054b3
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-06-25T17:10:52Z
Fix scala style.
commit 3c5b203fd2b85f4110795a1fc6ca3e289ca0d837
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-06-26T09:49:01Z
Write checkpoint data to disk if it is at the end of iterator.
commit a829a7d563d1a12e7d99cc1aecf87e264a016f7b
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-07-03T16:15:19Z
Merge remote-tracking branch 'upstream/master' into optimize_checkpoint
Conflicts:
core/src/main/scala/org/apache/spark/rdd/RDDCheckpointData.scala
commit 2f43ff3c6d1a4a428e5cbe8f4a4e4347274fc95c
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-07-03T16:23:16Z
Fix scala style.
commit 5c42503f84d496c4db8694f3bc78b9886b597f78
Author: zsxwing <[email protected]>
Date: 2015-11-02T22:37:28Z
Merge remote-tracking branch 'origin/master' into pr7021
commit 647162fd67d745108ee4816c13d38b71bc71cd59
Author: zsxwing <[email protected]>
Date: 2015-11-03T01:33:22Z
Fix the corner cases in CheckpointingIterator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
