Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21698#discussion_r200220663
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1386,8 +1386,12 @@ class DAGScheduler(
)
}
}
- // Mark the map whose fetch failed as broken in the map stage
- if (mapId != -1) {
+
+ if (mapStage.rdd.recomputeAllPartitionsOnFailure()) {
--- End diff --
hmmm, what if we have a `map` after `repartition`? then the root RDD will
return false on `recomputeAllPartitionsOnFailure`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
