Liang-Chi Hsieh created SPARK-2490: -------------------------------------- Summary: StackOverflowError when RDD dependencies are too long Key: SPARK-2490 URL: https://issues.apache.org/jira/browse/SPARK-2490 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.0.0 Reporter: Liang-Chi Hsieh
When performing some transformations on RDDs after many iterations, the dependencies of RDDs could be very long. It can easily cause StackOverflowError when recursively visiting these dependencies in Spark core. For example: var rdd = sc.makeRDD(Array(1)) for (i <- 1 to 1000) { rdd = rdd.coalesce(1).cache() rdd.collect() } -- This message was sent by Atlassian JIRA (v6.2#6252)