[
https://issues.apache.org/jira/browse/YARN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176644#comment-15176644
]
Karthik Kambatla commented on YARN-4719:
----------------------------------------
[~leftnoteasy] - thanks for chiming in, you make some valid points.
Since we are building a library for node tracking, I would like for us to
restrict access to the map/set of nodes tracked only through addNode and
removeNode so total_cluster_resources, total_inflated_cluster_resources (for
YARN-1011), max_cluster_resources are not affected by other scheduler code. Do
you think this is a reasonable goal? At least, as long as it doesn't hurt
performance?
If yes, we should decide on how to handle cases where the scheduler code needs
to iterate through the nodes: (1) we could provide a snapshot copy of the
map/set of nodes/nodeIds, or (2) provide a way to do the same with the right
locks by adding additional methods or an abstraction (similar to lambdas) that
applies to multiple methods.
Thoughts?
PS: By the way, thanks for pointing out the javadoc for values(). I will clean
that up based on the discussion output here.
> Add a helper library to maintain node state and allows common queries
> ---------------------------------------------------------------------
>
> Key: YARN-4719
> URL: https://issues.apache.org/jira/browse/YARN-4719
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 2.8.0
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: yarn-4719-1.patch, yarn-4719-2.patch, yarn-4719-3.patch
>
>
> The scheduler could use a helper library to maintain node state and allowing
> matching/sorting queries. Several reasons for this:
> # Today, a lot of the node state management is done separately in each
> scheduler. Having a single library will take us that much closer to reducing
> duplication among schedulers.
> # Adding a filtering/matching API would simplify node labels and locality
> significantly.
> # An API that returns a sorted list for a custom comparator would help
> YARN-1011 where we want to sort by allocation and utilization for
> continuous/asynchronous and opportunistic scheduling respectively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
