Github user NiharS commented on a diff in the pull request: https://github.com/apache/spark/pull/22192#discussion_r213831980 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -130,6 +130,14 @@ private[spark] class Executor( private val urlClassLoader = createClassLoader() private val replClassLoader = addReplClassLoaderIfNeeded(urlClassLoader) + // Load plugins in the current thread, they are expected to not block. + // Heavy computation in plugin initialization should be done async. + Thread.currentThread().setContextClassLoader(replClassLoader) + conf.get(EXECUTOR_PLUGINS).foreach { classes => --- End diff -- @mridulm We've had some back and forth on that, open to changing it if people agree it should be one way over another. Just checking, have you looked at the earlier conversation at https://github.com/apache/spark/pull/22192#discussion_r212746543? There's also one on the old PR for the same topic. As for moving plugins for after executors have initialized, I see no problem with that. I don't think it should make a huge difference provided either we do change to separate thread, or we keep it in the same thread but writers respect not making the initialization blocking and computation heavy. I'll see if there's a more natural place to move it
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org