Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/21656#discussion_r200759539
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -772,6 +772,12 @@ private[spark] class TaskSetManager(
private[scheduler] def markPartitionCompleted(partitionId: Int): Unit = {
partitionToIndex.get(partitionId).foreach { index =>
if (!successful(index)) {
+ if (speculationEnabled) {
--- End diff --
yeah that is sort of what I was suggesting -- but I was thinking rather
than just a flag, maybe we separate out `tasksSuccessful` into
`tasksCompletedSuccessfully` (from this taskset) and `tasksNoLongerNecessary`
(from any taskset), perhaps with better names. If you just had a flag, you
would avoid the exception from the empty heap, but you still might decide to
enable speculation prematurely as you really haven't finished enough for
`SPECULATION_QUANTILE`:
https://github.com/apache/spark/blob/a381bce7285ec30f58f28f523dfcfe0c13221bbf/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
