[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754054#comment-16754054 ] Weiwei Yang commented on YARN-7018: --- Hi [~maniraj...@gmail.com], could you add some more background or use case for this? YARN-5202 and YARN-5215 both went stale for a while, I am not sure if they are still valid or not. Particularly for over-commitment, YARN-2877 and a large part of YARN-1011 are already committed. I am trying to understand if there is any work depending on such plugin. > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch, > YARN-7018.POC.003.patch, YARN-7018.POC.004.patch, YARN-7018.POC.005.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739033#comment-16739033 ] Manikandan R commented on YARN-7018: [~leftnoteasy] Can you please take a look? Thanks > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch, > YARN-7018.POC.003.patch, YARN-7018.POC.004.patch, YARN-7018.POC.005.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[
https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707561#comment-16707561
]
Manikandan R commented on YARN-7018:
{quote}Since the plugin has init and stop, wondering if we should just have the
plugin extend Service so we have the full init/start/stop lifecycle.\{quote}
Makes sense. Incorporated the same in attached patch.
Also fixed typo issue.
[~leftnoteasy] Can you please take a look?
> Interface for adding extra behavior to node heartbeats
> --
>
> Key: YARN-7018
> URL: https://issues.apache.org/jira/browse/YARN-7018
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch,
> YARN-7018.POC.003.patch, YARN-7018.POC.004.patch
>
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat
> processing. Adding a formal interface for additional node heartbeat
> processing would allow admins to configure new functionality that is
> scheduler-independent without needing to replace the entire scheduler. For
> example, both YARN-5202 and YARN-5215 had approaches where node heartbeat
> processing was extended to implement new functionality that was essentially
> scheduler-independent and could be implemented as a plugin with this
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702036#comment-16702036 ] Jason Lowe commented on YARN-7018: -- Thanks for updating the patch! Overall looks OK for a POC. Would be good to hear [~leftnoteasy]'s thoughts on the latest revision. Some minor points: Since the plugin has init and stop, wondering if we should just have the plugin extend Service so we have the full init/start/stop lifecycle. "PLIUGIN" s/b "PLUGIN" > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch, > YARN-7018.POC.003.patch, YARN-7018.POC.004.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647710#comment-16647710 ] Manikandan R commented on YARN-7018: Thanks for your comments. Incorporated all suggestions and kept the plugin also in Scheduler itself to make it simple. Please review. If required, we can open new jira for implementations! > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch, > YARN-7018.POC.003.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644967#comment-16644967 ] Jason Lowe commented on YARN-7018: -- Thanks for updating the patch! Interface looks good for now. This JIRA should define the initial interface to the plugin and provide a default implementation of the plugin that does nothing (i.e.: the null plugin). NodeHeartbeatPluginImpl and BaseNodeHeartBeatPluginImpl should not be part of this, IMHO. Loading the plugin should not be in a base class that implements the plugin -- that belongs in he AbstractYarnScheduler code that is responsible for loading and initializing the plugin. And once that's removed from BaseNodeHeartBeatPluginImpl I don't see a need for that class. bq. Where shall we keep the plugin? RMActiveService? or Scheduler like Wangda Tan suggested earlier? We can put it in the AbstractYarnScheduler which the latest patch essentially is doing. The reinitialize method is loading the plugin and setting it up in the RMContext. > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch, > YARN-7018.POC.003.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[
https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616398#comment-16616398
]
Manikandan R commented on YARN-7018:
[~jlowe] [~leftnoteasy] Thanks for your comments.
Addressed the following:
1. Introduced init, stop, addNode, removeNode methods in interface to handle
the cases described in earlier comment.
2. Introduced refreshConfigs as described in earlier comment.
3. Kept the plugin interactions into each of the schedulers.
{quote}Back to the prototype patch, I assume NodeHeartBeatPluginImpl is just an
example and would not be part of the final commit?\{quote}
Yes, thats the initial thought. But then, was thinking to mould it as concrete
implementation at least for one scheduler (for ex, say CS) so that it can act
as well defined example for future plugin development. Hence asked you to
suggest potential movable code parts from CS scheduler. Thoughts?
Also, Where shall we keep the plugin? RMActiveService? or Scheduler like
[~leftnoteasy] suggested earlier?
> Interface for adding extra behavior to node heartbeats
> --
>
> Key: YARN-7018
> URL: https://issues.apache.org/jira/browse/YARN-7018
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch
>
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat
> processing. Adding a formal interface for additional node heartbeat
> processing would allow admins to configure new functionality that is
> scheduler-independent without needing to replace the entire scheduler. For
> example, both YARN-5202 and YARN-5215 had approaches where node heartbeat
> processing was extended to implement new functionality that was essentially
> scheduler-independent and could be implemented as a plugin with this
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609832#comment-16609832 ] Jason Lowe commented on YARN-7018: -- Originally I was thinking this could be outside of the scheduler, examining scheduler-agnostic settings like SchedulerNode, etc., then it could send NODE_RESOURCE_UPDATE to adjust node capabilities which is also scheduler-agnostic. However it would be lower overhead to have the scheduler call the plugin directly to avoid the messaging overhead, but it does increase coupling between the plugin and the scheduler a little. I'm fine if we want to move the plugin interactions into each of the schedulers. Back to the prototype patch, I assume NodeHeartBeatPluginImpl is just an example and would not be part of the final commit? There needs to be some lifecycle support around the plugin, i.e.: a way for the plugin to know it is being initialized, shutdown, etc. Having a callback when nodes are added and removed would also be helpful for some plugin implementations, otherwise the plugin will have to track nodes redundantly to know when it sees a new one and some other type of hack like timeouts to know when one is no longer being tracked. Similarly I think it would be nice to have explicit config refresh support in the plugin like there is for the schedulers. One idea: if the plugin class we load after refreshing is the same as the old one, do _not_ replace the plugin object but rather invoke a refreshConfigs method or something similar that lets the existing plugin refresh rather than forcing a load-from-scratch approach on each refresh. > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595575#comment-16595575 ] Wangda Tan commented on YARN-7018: -- [~jlowe], given the fields need to be updated should all inside scheduler (Such as SchedulerNode#unallocatedResource). For this purpose, is it better to just add a plugin to scheduler instead of to RMActiveService? The plugin itself could be scheduler agnostic (just like AppPlacementAllocator), and to use these plugins, changing a little bit scheduler code seems fine. Adding to RMActiveService is additional cost to maintenance, we have to consider scheduler reinitialize (if it is stateful) and other scheduler related interactions. Just my $.02 > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch, YARN-7018.POC.002.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[
https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595356#comment-16595356
]
Manikandan R commented on YARN-7018:
[~jlowe] Thanks for detailed comments.
{quote}That's why it may be important to either pass the RMContext to the
plugin when it is called or at least pass the context when the plugin is
instantiated.{quote}
Taken care. I think we may require RMNode argument as well as it is the only
arg being passed to nodeUpdate() method. Can we start moving possible methods
from scheduler class to this plugin impl? Can you suggest few pieces to start
with?
{quote}The plugin should not be loaded on every heartbeat. The plugin needs to
be loaded when the RM initializes. Bonus points if we support re-initializing
it when a scheduler config refresh occurs (e.g.: refreshQueues command from
admin CLI). {quote}
Taken care. Plugin object would be loaded only once during RM initialization
and has been kept inside RMContext for future references. Since RMContext has
plugin object, when a scheduler config refresh occurs, loading the plugin
object again and store the same in RMContext.
{quote}I'm not sure what is meant by the isSchedulerExternal() comment in the
patch.{quote}
Example use case (YARN-5215) depends on this property. Hence used the same
example for this POC, but it can be ignored.
> Interface for adding extra behavior to node heartbeats
> --
>
> Key: YARN-7018
> URL: https://issues.apache.org/jira/browse/YARN-7018
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7018.POC.001.patch
>
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat
> processing. Adding a formal interface for additional node heartbeat
> processing would allow admins to configure new functionality that is
> scheduler-independent without needing to replace the entire scheduler. For
> example, both YARN-5202 and YARN-5215 had approaches where node heartbeat
> processing was extended to implement new functionality that was essentially
> scheduler-independent and could be implemented as a plugin with this
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[
https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595347#comment-16595347
]
Manikandan R commented on YARN-7018:
[~jlowe] Thanks for detailed comments.
{quote}That's why it may be important to either pass the RMContext to the
plugin when it is called or at least pass the context when the plugin is
instantiated.{quote}
Taken care. I think we may require RMNode argument as well as it is the only
arg being passed to nodeUpdate() method. Can we start moving possible methods
from scheduler class to this plugin impl?
{quote}The plugin should not be loaded on every heartbeat. The plugin needs to
be loaded when the RM initializes. Bonus points if we support re-initializing
it when a scheduler config refresh occurs (e.g.: refreshQueues command from
admin CLI). {quote}
Taken care. Plugin object would be loaded only once during RM initialization
and has been kept inside RMContext for future references. Since RMContext has
plugin object, when a scheduler config refresh occurs, loading the plugin
object again and storing it in
> Interface for adding extra behavior to node heartbeats
> --
>
> Key: YARN-7018
> URL: https://issues.apache.org/jira/browse/YARN-7018
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7018.POC.001.patch
>
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat
> processing. Adding a formal interface for additional node heartbeat
> processing would allow admins to configure new functionality that is
> scheduler-independent without needing to replace the entire scheduler. For
> example, both YARN-5202 and YARN-5215 had approaches where node heartbeat
> processing was extended to implement new functionality that was essentially
> scheduler-independent and could be implemented as a plugin with this
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584453#comment-16584453 ] Jason Lowe commented on YARN-7018: -- Thanks for the POC patch! At a very high level it's along the lines of what I was proposing. Specific comments on what's mentioned in the patch: If the intent of the plugin is to adjust node resources then I don't think we'll need setter methods beyond what is already there. The RM already supports dynamic changes to the node via the updateNodeResource YARN admin CLI, so this plugin can worst-case use the same methods to adjust the node. Note that it's usually not sufficient to just adjust the node, one also has to adjust the scheduler's bookkeeping (e.g.: potentially queue capacities, etc.). That's why it may be important to either pass the RMContext to the plugin when it is called or at least pass the context when the plugin is instantiated. The plugin should not be loaded on every heartbeat. The plugin needs to be loaded when the RM initializes. Bonus points if we support re-initializing it when a scheduler config refresh occurs (e.g.: refreshQueues command from admin CLI). That will require some APIs to the plugin so it can be called when a scheduler config refresh occurs. We can also avoid the null check if we fallback to a null plugin instance that simply does nothing for the onNodeHeartbeat call. I'm not sure what is meant by the isSchedulerExternal() comment in the patch. > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Attachments: YARN-7018.POC.001.patch > > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571985#comment-16571985 ] Manikandan R commented on YARN-7018: [~jlowe] Attaching POC patch based on earlier discussion (and YARN-5215 as an use case). Please review the structure and share your comments. Based on your feedback, we can iterate further to make it as concrete patch. For example, Probably, we can move methods like AbstractYarnScheduler#updateNodeResourceUtilization as well? > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats
[ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571962#comment-16571962 ] Manikandan R commented on YARN-7018: [~jlowe] and I had a very preliminary offline discussion on approaches. Thank you [~jlowe] for explaining the requirements in detail. Summary of the discussion: (Copying [~jlowe]'s suggestion as is) The point of YARN-7018 is to create a hook that runs before the scheduler and, as much as possible, is *not* scheduler-dependent. If we have to do something scheduler-dependent then we might as well just modify the schedulers directly. I haven't had much time to think about it, but one potential way to implement this is a very simple plugin API. Something like this: - Admin configures the plugin via a conf key - Plugin only has one API call, onNodeHeartbeat or something like that. It's called when a NODE_UPDATE event is processed and before the main scheduler loop is called - Call arguments would include the SchedulerNode and maybe the RMContext. >From the RMContext the plugin can get the scheduler and do scheduler-specific >things if it wants to, or can be scheduler-specific if it wants to. I'd rather not force plugin writers to implement every scheduler supported. It's a burden to writing a plugin and an obstacle to providing new schedulers in YARN. > Interface for adding extra behavior to node heartbeats > -- > > Key: YARN-7018 > URL: https://issues.apache.org/jira/browse/YARN-7018 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > > This JIRA tracks an interface for plugging in new behavior to node heartbeat > processing. Adding a formal interface for additional node heartbeat > processing would allow admins to configure new functionality that is > scheduler-independent without needing to replace the entire scheduler. For > example, both YARN-5202 and YARN-5215 had approaches where node heartbeat > processing was extended to implement new functionality that was essentially > scheduler-independent and could be implemented as a plugin with this > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
