Github user NiharS commented on the issue:
https://github.com/apache/spark/pull/22192
I tested @vanzin's suggestion of storing the classloader, changing it to
replClassLoader, and changing back after init (and then changing again for
shutdown). It's working well (and this time I made sure it doesn't just shut
down immediately; I ran on a cluster and confirmed it runs the init code and
creates a thread, the thread stays alive for the duration of the executor, and
the shutdown code runs fully before exiting). While it looks a little weird,
the code is synchronous and shouldn't run into an issue with the classloader
being set to something it shouldn't be set to while executing the existing
Spark code, and ensures the plugins initialize fully during executor
initialization.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
