Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102821513
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with Timeou
* In this test, we run a map stage where one of the executors fails but
we still receive a
* "zombie" complete message from that executor. We want to make sure
the stage is not reported
* as done until all tasks have completed.
+ *
+ * Most of the functionality in this test is tested in "run trivial
shuffle with out-of-band
+ * executor failure and retry". However, that test uses
ShuffleMapStages that are followed by
+ * a ResultStage, whereas in this test, the ShuffleMapStage is tested in
isolation, without a
+ * ResultStage after it.
--- End diff --
The original point of this test was to test a Map-only stage, so then it
can't have a stage that follows it, right? I thought your earlier comment
("That is testing directly submitting mapstage jobs. While a lot of the logic
is the same, I think its worth its own test (even earlier versions of #16620
had the behavior wrong for these jobs)." was saying that it was important /
useful to have this map-only test. Let me know if that comment was based on
the understanding that this tested multi-stage jobs and you think I should just
remove this map-only test.
I do agree that it would be useful to add another test that tests a job
with more stages, which seems like it could reveal more bugs, but I'll hold off
on doing that in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
