Liang-Chi Hsieh created SPARK-2490:
--------------------------------------
Summary: StackOverflowError when RDD dependencies are too long
Key: SPARK-2490
URL: https://issues.apache.org/jira/browse/SPARK-2490
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Liang-Chi Hsieh
When performing some transformations on RDDs after many iterations, the
dependencies of RDDs could be very long. It can easily cause StackOverflowError
when recursively visiting these dependencies in Spark core. For example:
var rdd = sc.makeRDD(Array(1))
for (i <- 1 to 1000) {
rdd = rdd.coalesce(1).cache()
rdd.collect()
}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
