[
https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609832#comment-16609832
]
Jason Lowe commented on YARN-7018:
----------------------------------
Originally I was thinking this could be outside of the scheduler, examining
scheduler-agnostic settings like SchedulerNode, etc., then it could send
NODE_RESOURCE_UPDATE to adjust node capabilities which is also
scheduler-agnostic. However it would be lower overhead to have the scheduler
call the plugin directly to avoid the messaging overhead, but it does increase
coupling between the plugin and the scheduler a little. I'm fine if we want to
move the plugin interactions into each of the schedulers.
Back to the prototype patch, I assume NodeHeartBeatPluginImpl is just an
example and would not be part of the final commit?
There needs to be some lifecycle support around the plugin, i.e.: a way for the
plugin to know it is being initialized, shutdown, etc. Having a callback when
nodes are added and removed would also be helpful for some plugin
implementations, otherwise the plugin will have to track nodes redundantly to
know when it sees a new one and some other type of hack like timeouts to know
when one is no longer being tracked.
Similarly I think it would be nice to have explicit config refresh support in
the plugin like there is for the schedulers. One idea: if the plugin class we
load after refreshing is the same as the old one, do _not_ replace the plugin
object but rather invoke a refreshConfigs method or something similar that lets
the existing plugin refresh rather than forcing a load-from-scratch approach on
each refresh.
> Interface for adding extra behavior to node heartbeats
> ------------------------------------------------------
>
> Key: YARN-7018
> URL: https://issues.apache.org/jira/browse/YARN-7018
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Major
> Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch
>
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat
> processing. Adding a formal interface for additional node heartbeat
> processing would allow admins to configure new functionality that is
> scheduler-independent without needing to replace the entire scheduler. For
> example, both YARN-5202 and YARN-5215 had approaches where node heartbeat
> processing was extended to implement new functionality that was essentially
> scheduler-independent and could be implemented as a plugin with this
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
