Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102996593
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with Timeou
* In this test, we run a map stage where one of the executors fails but
we still receive a
* "zombie" complete message from that executor. We want to make sure
the stage is not reported
* as done until all tasks have completed.
+ *
+ * Most of the functionality in this test is tested in "run trivial
shuffle with out-of-band
+ * executor failure and retry". However, that test uses
ShuffleMapStages that are followed by
+ * a ResultStage, whereas in this test, the ShuffleMapStage is tested in
isolation, without a
+ * ResultStage after it.
--- End diff --
the *final* map-stage can't have anything that follows it, but the job
overall can still have multiple stages, and the failure can occur during the
processing of those earlier map-stages, or the final one.
In any case, I agree you don't need to expand that test in this PR. and
even though this test doesn't do as much as I was hoping, I do still think it
adds value, and is worth leaving in, even though its very similar to the other
test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
