tgravescs commented on issue #26670: [SPARK-30033][core] Manage shuffle IO plugins using Spark's plugin system. URL: https://github.com/apache/spark/pull/26670#issuecomment-578253611 > The only downside is that initialization of the SortShuffleManager in executors > is a bit weird, because of the order in which things are initialized: the > shuffle manager is initialized by SparkEnv, and plugin initialization happens > after that. In any case, all initialization is done before any tasks are > allowed to run.. I haven't looked at the code yet, what exactly do you mean initialized by SparkEnv? Do you just mean that sparkEnv instantiates the shuffle manager class so its created before you possibly would want to initialize something in the plugin? within the executor plugin it seems like having other plugins initialize first would make sense in case shuffle relied on another plugin initialization, but I supposed the reverse could be true as well.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
