That's what I thought, too. So is it not "hanging", just recalculating for a very long time? The log stops updating and it just gives the output I posted. If there are any suggestions as to parameters to change, or any other data, it would be appreciated.
Thank you, Shangyu. On Fri, Nov 1, 2013 at 11:31 AM, Shangyu Luo <[email protected]> wrote: > I think the missing parent may be not abnormal. From my understanding, > when a Spark task cannot find its parent, it can use some meta data to find > the result of its parent or recalculate its parent's value. Imaging in a > loop, a Spark task tries to find some value from the last iteration's > result. > > > 2013/11/1 Walrus theCat <[email protected]> > >> Are there heuristics to check when the scheduler says it is "missing >> parents" and just hangs? >> >> >> >> On Thu, Oct 31, 2013 at 4:56 PM, Walrus theCat <[email protected]>wrote: >> >>> Hi, >>> >>> I'm not sure what's going on here. My code seems to be working thus far >>> (map at SparkLR:90 completed.) What can I do to help the scheduler out >>> here? >>> >>> Thanks >>> >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: Completed >>> ShuffleMapTask(10, 211) >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: Stage 10 (map at >>> SparkLR.scala:90) finished in 0.923 s >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: looking for newly >>> runnable stages >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: running: Set(Stage 11) >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: waiting: Set(Stage 9, >>> Stage 8) >>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: failed: Set() >>> 13/10/31 02:10:16 INFO scheduler.DAGScheduler: Missing parents for Stage >>> 9: List(Stage 11) >>> 13/10/31 02:10:16 INFO scheduler.DAGScheduler: Missing parents for Stage >>> 8: List(Stage 9) >>> >>> >>> >>> >> > > > -- > -- > > Shangyu, Luo > Department of Computer Science > Rice University > > -- > Not Just Think About It, But Do It! > -- > Success is never final. > -- > Losers always whine about their best >
