Hi Nicolas, I've had suspicions about speculation causing problems on my cluster but don't have any hard evidence of it yet.
I'm also interested in why it's turned off by default. On Tue, Sep 16, 2014 at 3:01 PM, Nicolas Mai <nicolas....@gmail.com> wrote: > Hi, guys > > My current project is using Spark 0.9.1, and after increasing the level of > parallelism and partitions in our RDDs, stages and tasks seem to complete > much faster. However it also seems that our cluster becomes more "unstable" > after some time: > - stalled stages still showing under "active stages" in the Spark app web > dashboard > - incomplete stages showing under "completed stages" > - stages with failures > > I was thinking about reducing/tuning the number of parallelism, but I was > also considering using "spark.speculation" which is currently turned off > but > seems promising. > > Questions about speculation: > - Just wondering why it is turned off by default? > - Are there any risks using speculation? > - Is it possible that a speculative task straggles, and would trigger > another new speculative task to finish the job... and so on... (some kind > of > loop until there's no more executors available). > - What configuration do you guys usually use for spark.speculation? > (interval, quantile, multiplier) I guess it depends on the project, it may > give some ideas about how to use it properly. > > Thank you! :) > Nicolas > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark-speculation-tp14398.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
