Zheng Tan created SPARK-15060:
---------------------------------
Summary: Fix stack overflow when executing long linage transform
without checkpoint
Key: SPARK-15060
URL: https://issues.apache.org/jira/browse/SPARK-15060
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.6.1, 1.5.2, 2.0.0
Reporter: Zheng Tan
When executing long linage rdd transform, it is easy to get stack overflow
exception in driver end. This can be reproduced by the following example:
var rdd = sc.makeRDD(1 to 10, 10)
for (_ <- 1 to 1000) {
rdd = rdd.map(x => x)
}
rdd.reduce(_ + _)
SPARK-5955 solve this problem by checkpointing rdd for every 10~20 rounds. It
is not so convenient since it required checkpointing data to HDFS.
Another solution is cutting off the recursive rdd dependencies in driver end
and re-assembly them in executor end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
