Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102282650
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with Timeou
* In this test, we run a map stage where one of the executors fails but
we still receive a
* "zombie" complete message from that executor. We want to make sure
the stage is not reported
* as done until all tasks have completed.
+ *
+ * Most of the functionality in this test is tested in "run trivial
shuffle with out-of-band
+ * executor failure and retry". However, that test uses
ShuffleMapStages that are followed by
+ * a ResultStage, whereas in this test, the ShuffleMapStage is tested in
isolation, without a
+ * ResultStage after it.
--- End diff --
I should have looked closer at this test earlier ... I was hoping this was
testing a *multi*-stage mapjob. That would really be a better test. ideally
we'd even have three stages, with a failure happening in the second stage, and
the last stage.
In any case, your changes still look good, no need to have to do those
other things now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
