Github user concretevitamin commented on the pull request:
https://github.com/apache/spark/pull/251#issuecomment-39004007
Hey @andrewor14 @pwendell -- so what I have been thinking and what Andrew
pointed out correctly is, the event **"the listener bus event queue is
currently empty"** and the event **"the shutdown message has been received by
the listener bus thread"** *are two different events*. We want `stop()` to do
the right thing which is to wait for the second event to happen. However, as
far as I can tell `waitUntilEmpty()` *only waits for the first event*. In the
current uses of `waitUntilEmpty()` in the unit tests, due to manual control of
the events posted to the listener bus the two events happen to be the same
thing.
In the general case, I think it's crucial that we distinguish these two.
For instance it's probably possible that during the execution of a job the
event queue becomes empty (for instance, the consumption of the events is
faster than the posting of the events) -- but that doesn't mean a call to
`stop()` at such a moment should succeed; rather we want the call to succeed
whenever the end-of-work `SparkListenerShutdown` message is processed. This is
where the condition variable comes in.
Let me know what you guys think ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
