Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r215750952
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -218,6 +244,8 @@ private[spark] class Executor(
env.metricsSystem.report()
heartbeater.shutdown()
heartbeater.awaitTermination(10, TimeUnit.SECONDS)
+ executorPluginThread.interrupt()
+ executorPluginThread.join()
--- End diff --
We should be careful about semantics of `interrupt()` when overriding it -
here we are using it to trigger something in addition to thread interruption.
For example, the actual shutdown of plugin's is getting triggered in this
caller thread - and not in plugin init thread : which can cause issues since
plugin's are using a a different thread ctx classloader.
This usage is actually a bit confusing.
A better option, if we want to follow current design, would be:
* Subclass Thread - expose methods to trigger state changes.
* Once init completes, wait+notify/condition wait until plugin shutdown is
required.
* On stop(), invoke plugin shutdown in plugin thread (so that right
classloader is in use).
IIRC VM shutdown results in `executor.stop` - and so plugin shutdown should
get invoked even in that case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
