GitHub user JerryLead opened a pull request:
https://github.com/apache/spark/pull/3545
[SPARK-4672][Core]Checkpoint() should clear f to shorten the serialization
chain
The related JIRA is https://issues.apache.org/jira/browse/SPARK-4672
The f closure of `PartitionsRDD(ZippedPartitionsRDD2)` contains a `$outer`
that references EdgeRDD/VertexRDD, which causes task's serialization chain
become very long in iterative GraphX applications. As a result, StackOverflow
error will occur. If we set "f = null" in `clearDependencies()`, checkpoint()
can cut off the long serialization chain. More details and explanation can be
found in the JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JerryLead/spark my_core
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3545.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3545
----
commit 52799e3ea2b22f4bcaec3d9cd4c8891e212be09e
Author: Lijie Xu <[email protected]>
Date: 2014-12-01T08:54:37Z
Merge pull request #1 from apache/master
update
commit c0169da181660281b3bd82678ae89a73f5926370
Author: JerryLead <[email protected]>
Date: 2014-12-02T03:19:31Z
Merge branch 'master' of https://github.com/apache/spark
update to the latest version
commit f7faea514f5e0749b2f11d27abd481dbd9879ce6
Author: JerryLead <[email protected]>
Date: 2014-12-02T03:41:00Z
checkpoint() should clear the f to avoid StackOverflow error
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
