Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r213045752
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -130,6 +130,16 @@ private[spark] class Executor(
private val urlClassLoader = createClassLoader()
private val replClassLoader = addReplClassLoaderIfNeeded(urlClassLoader)
+ // One thread will handle loading all of the plugins on this executor
--- End diff --
I guess it does depend on what the intended use is here. If we have it in
the same thread it has the issue that it could block the executor or take to
long and things start timing out. It can have more direct impact on the
executor code itself, where as a separate thread isolates it more. But like
you say if its not here and we don't wait for it then we could have order issue
if certain plugins have to be initialized before other things happen. I can
see both arguments as well. So perhaps the api needs an init type function
that can be called more inline with a timeout to prevent from taking to long
and the main part of the plugin called in a separate thread?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
