tgravescs commented on issue #26614: [SPARK-29976][CORE] New conf for single task stage speculation URL: https://github.com/apache/spark/pull/26614#issuecomment-558773705 Yes you still need the config for the timeout, you just don't need a second one when to apply that config. ie when you have <= second config task number then use the config, otherwise use the normal speculation logic. I was originally thinking the total number of tasks <= number of slots on 1 executor, then apply the timeout config, seemed the most straightforward and obvious to the user. I'm fine with either way though as long as it can be explained to user. I think using delta does complicate things again as it uses the new algorithm sometimes and then the original algorithm at other times. My initial thought is to keep it simple in initial implementation, they can always turn the spark.speculation.quantile down when you have a larger number of tasks, but lots of corner cases again. The thing with 1 task is that the current settings will never work for it because you need at least 1 to compare against. Note you will ask for a new executor if you speculate and the executors are all used. It might not be on a different node though.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
