Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8466#issuecomment-146914758
Hi @mateiz sorry I totally missed your comment a while back. so I think
poorly explained the issue in the first place. Exceptions are already making
it to the joblisteners to the best of my knowledge (as part of
[`doCancelAllJobs`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1648)).
The issue that I'm trying to address is just confusing behavior in the
tests. Before this change, if there is an exception inside the event
processing loop, then all that happens is [some internal state is
set](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala#L172).
Furthermore, the event process loop [stops the
`SparkContext`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1652),
but it doesn't actually stop the `DAGScheduler` in use b/c in these tests the
[`DAGScheduler` is custom, not the one created by the
`SparkContext`](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala#L195).
The net effect is that when a test doesn't behave the way you expect it to,
its hard to unravel why. Unless tests include `assert(failure == null)` on
every other line, if you do hit an exception, the test will keep running quite
a bit further but with strange behavior which can be hard to make sense of.
The only goal here is just to make those issues much more immediately
obvious. For these tests, as soon as there is an exception in the event
process loop, throw the exception and fail the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
