Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21656#discussion_r200366359
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -772,6 +772,12 @@ private[spark] class TaskSetManager(
private[scheduler] def markPartitionCompleted(partitionId: Int): Unit = {
partitionToIndex.get(partitionId).foreach { index =>
if (!successful(index)) {
+ if (speculationEnabled) {
--- End diff --
IIUC in this case no task in this taskSet actually successfully finishes,
it's another task attempt from another taskSet for the same stage that
succeeded. In stead of changing this code path, I'd suggest we have another
flag to show whether any task succeeded in current taskSet, and if no task have
succeeded, skip L987.
WDYT @squito ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
