LuciferYang commented on code in PR #43890:
URL: https://github.com/apache/spark/pull/43890#discussion_r1403905311
##########
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##########
@@ -1196,15 +1196,17 @@ private[spark] class TaskSetManager(
val timeMs = clock.getTimeMillis()
if (numSuccessfulTasks >= minFinishedForSpeculation) {
val medianDuration = successfulTaskDurations.percentile()
- val threshold = max(speculationMultiplier * medianDuration,
minTimeToSpeculation)
+ val threshold = max(speculationMultiplier * medianDuration,
minTimeToSpeculation.toDouble)
// TODO: Threshold should also look at standard deviation of task
durations and have a lower
// bound based on that.
logDebug("Task length threshold for speculation: " + threshold)
foundTasks = checkAndSubmitSpeculatableTasks(timeMs, threshold)
} else if (isSpeculationThresholdSpecified &&
speculationTasksLessEqToSlots) {
val threshold = speculationTaskDurationThresOpt.get
logDebug(s"Tasks taking longer time than provided speculation threshold:
$threshold")
- foundTasks = checkAndSubmitSpeculatableTasks(timeMs, threshold,
customizedThreshold = true)
+ foundTasks = checkAndSubmitSpeculatableTasks(timeMs,
+ threshold.toDouble,
+ customizedThreshold = true)
Review Comment:
```scala
foundTasks = checkAndSubmitSpeculatableTasks(
timeMs, threshold.toDouble, customizedThreshold = true)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
