vanzin commented on a change in pull request #26170: [SPARK-29397][core] Extend
plugin interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#discussion_r338794778
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -539,6 +541,9 @@ class SparkContext(config: SparkConf) extends Logging {
_heartbeatReceiver = env.rpcEnv.setupEndpoint(
HeartbeatReceiver.ENDPOINT_NAME, new HeartbeatReceiver(this))
+ // Initialize any plugins before the task scheduler is initialized.
+ _plugins = PluginContainer(this)
Review comment:
I'll just move the conversation back here because github doesn't have
top-level threads and that's annoying.
> Just to clarify, I was thinking about registering the plugin source in the
driver somewhere "near" to what was done in #23838
That has the problem I ran into - that's done after `metricsSystem.start()`
is called, but in my case metrics added after that were not showing up in
`ConsoleSink`.
And the thing is, I can't just move the metrics registration to that spot.
I'd have to move all plugin initialization to that spot, otherwise there's no
place where the plugin can initialize the metrics before the registration. And
I can't move the plugin initialization to that spot, because then I can't send
the plugin data to executors via config. I could use RPCs for that but it would
slow down executor startup unnecessarily.
So, ignoring the `ConsoleSink` issue, I still think my suggestion is better:
if you need to initialize your metrics later, then use a listener that is
installed from your plugin's init method.
I can change the code to allow for this late initialization, so you don't
have to add a dummy metric in the plugin's init.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
