squito commented on a change in pull request #23677: [SPARK-26755][SCHEDULER] :
Optimize Spark Scheduler to dequeue speculative tasks…
URL: https://github.com/apache/spark/pull/23677#discussion_r303617604
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##########
@@ -1064,7 +979,8 @@ private[spark] class TaskSetManager(
val info = taskInfos(tid)
val index = info.index
if (!successful(index) && copiesRunning(index) == 1 &&
info.timeRunning(time) > threshold &&
- !speculatableTasks.contains(index)) {
+ !speculatableTasks.contains(index)) {
+ addPendingTask(index, speculative = true)
logInfo(
"Marking task %d in stage %s (on %s) as speculatable because it
ran more than %.0f ms"
.format(index, taskSet.id, info.host, threshold))
Review comment:
thinking a bit more about my comment that you should change this logline ...
if you have a ton of speculative tasks, isn't this horribly verbose? I am
wondering if it should be dropped to logDebug. But then I guess we lose all
info about speculatable assignments, which would be a bummer. But at the very
least, since this comment is here, its probably worth also throwing in the
total number of speculatable tasks? "... ran more than 100 ms (145
speculatable tasks in this taskset now)"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
