GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/21356
[SPARK-24309][CORE] AsyncEventQueue should stop on interrupt.
EventListeners can interrupt the event queue thread. In particular,
when the EventLoggingListener writes to hdfs, hdfs can interrupt the
thread. When there is an interrupt, the queue should be removed and stop
accepting any more events. Before this change, the queue would continue
to take more events (till it was full), and then would not stop when the
application was complete because the PoisonPill couldn't be added.
Added a unit test which failed before this change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-24309
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21356.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21356
----
commit a689f527e6b9bf626cef0e318a37e7aa3c008bc3
Author: Imran Rashid <irashid@...>
Date: 2018-05-17T20:45:52Z
[SPARK-24309][CORE] AsyncEventQueue should stop on interrupt.
EventListeners can interrupt the event queue thread. In particular,
when the EventLoggingListener writes to hdfs, hdfs can interrupt the
thread. When there is an interrupt, the queue should be removed and stop
accepting any more events. Before this change, the queue would continue
to take more events (till it was full), and then would not stop when the
application was complete because the PoisonPill couldn't be added.
Added a unit test which failed before this change.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
