jiangxb1987 commented on issue #26614: [SPARK-29976][CORE] New conf for single task stage speculation URL: https://github.com/apache/spark/pull/26614#issuecomment-558783837 I was proposing something like this: ``` if (tasksSuccessful >= minFinishedForSpeculation && tasksSuccessful > 0) { // Try to add speculative tasks that has been running more than SPECULATION_MULTIPLIER * medianDuration. } else if (speculationTaskDurationThresOpt.isDefined && runningTasks <= conf.getInt("spark.executor.cores", 1) / conf.getInt("spark.task.cpus", 1)) { // Try to add speculative tasks that has been running more than a specified duration. } else { // Do not add speculative tasks. } ``` So this just introduces a new way to add speculative tasks that have been running more than a specified duration, which should be easy to reason about. Please note I only consider the number of running tasks in the TaskSet, because the original speculation logic didn't include pending tasks either. On the other hand, if we keep get those long running tasks, at the end more executors would be required to run speculative tasks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
