Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21923#discussion_r207323114
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -130,6 +130,12 @@ private[spark] class Executor(
private val urlClassLoader = createClassLoader()
private val replClassLoader = addReplClassLoaderIfNeeded(urlClassLoader)
+ Thread.currentThread().setContextClassLoader(replClassLoader)
--- End diff --
I was thinking another thread would at least prevent them from not allowing
the executor to run/start. If someone added a plugin that just blocked or did
something that took time and then you started to see timeouts during start,
those might not be as obvious what is going on. If we start it in a separate
thread, yes it still uses resources but it doesn't completely block the
executor from starting and trying to take tasks. It also just seems safer to
me as you could try to catch exceptions from there and possibly ignore them so
it doesn't affect the main running.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
