Github user NiharS commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r213831980
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -130,6 +130,14 @@ private[spark] class Executor(
private val urlClassLoader = createClassLoader()
private val replClassLoader = addReplClassLoaderIfNeeded(urlClassLoader)
+ // Load plugins in the current thread, they are expected to not block.
+ // Heavy computation in plugin initialization should be done async.
+ Thread.currentThread().setContextClassLoader(replClassLoader)
+ conf.get(EXECUTOR_PLUGINS).foreach { classes =>
--- End diff --
@mridulm We've had some back and forth on that, open to changing it if
people agree it should be one way over another. Just checking, have you looked
at the earlier conversation at
https://github.com/apache/spark/pull/22192#discussion_r212746543? There's also
one on the old PR for the same topic.
As for moving plugins for after executors have initialized, I see no
problem with that. I don't think it should make a huge difference provided
either we do change to separate thread, or we keep it in the same thread but
writers respect not making the initialization blocking and computation heavy.
I'll see if there's a more natural place to move it
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
