Re: Any suggestion about JIRA 1006 "MLlib ALS gets stack overflow with too many iterations"?

[Ewen Cheslack-Postava]

The three obvious ones in DAGScheduler.scala are in: getParentStages getMissingParentStages stageDependsOn They all follow the same pattern though (def visit(), followed by visit(root)), so they should be easy to replace with a Scala stack in place of the call stack. Shao, Saisai January 25, 2014 at 10:52 PM In my test I found this phenomenon might be caused by RDD's long dependency chain, this dependency chain is serialized into task and sent to each executor, while deserializing this task will cause stack overflow. Especially in iterative job, like: var rdd = .. for (i <- 0 to 100) rdd = rdd.map(x=>x) rdd = rdd.cache Here rdd's dependency will be chained, at some point stack overflow will occur. You can check (https://groups.google.com/forum/?fromgroups#!searchin/spark-users/dependency/spark-users/-Cyfe3G6VwY/PFFnslzWn6AJ) and (https://groups.google.com/forum/?fromgroups#!searchin/spark-users/dependency/spark-users/NkxcmmS-DbM/c9qvuShbHEUJ) for details. Current workaround method is to cut the dependency chain by checkpointing RDD, maybe a better way is to clean the dependency chain after materialize stage is executed. Thanks Jerry -----Original Message----- From: Reynold Xin [mailto:r...@databricks.com] Sent: Sunday, January 26, 2014 2:04 PM To: dev@spark.incubator.apache.org Subject: Re: Any suggestion about JIRA 1006 "MLlib ALS gets stack overflow with too many iterations"? I'm not entirely sure, but two candidates are the visit function in stageDependsOn submitStage Reynold Xin January 25, 2014 at 10:03 PM I'm not entirely sure, but two candidates are the visit function in stageDependsOn submitStage Aaron Davidson January 25, 2014 at 10:01 PM I'm an idiot, but which part of the DAGScheduler is recursive here? Seems like processEvent shouldn't have inherently recursive properties. Reynold Xin January 25, 2014 at 9:57 PM It seems to me fixing DAGScheduler to make it not recursive is the better solution here, given the cost of checkpointing. Xia, Junluan January 25, 2014 at 9:49 PM Hi all The description about this Bug submitted by Matei is as following The tipping point seems to be around 50. We should fix this by checkpointing the RDDs every 10-20 iterations to break the lineage chain, but checkpointing currently requires HDFS installed, which not all users will have. We might also be able to fix DAGScheduler to not be recursive. regards, Andrew