jiangxb1987 commented on issue #26614: [SPARK-29976][CORE] New conf for single task stage speculation URL: https://github.com/apache/spark/pull/26614#issuecomment-558843301 Ah good point, considering both the running and pending tasks would make the speculation strategy more stable (in the means once you entered the speculation task duration threshold branch, unless more tasks finished successfully, you would always choose the same speculation strategy in the next few scheduling iterations). Also, if runningTasks + pendingTasks > SlotsPerExecutor, that means either too many tasks have been running, or we are waiting for extra executors to be launched, in both cases we'd better not hurry into adding more speculative tasks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
