[
https://issues.apache.org/jira/browse/YARN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174594#comment-15174594
]
Karthik Kambatla commented on YARN-4719:
----------------------------------------
bq. For ClusterNodeTracker#nodes, can we use lock-free data structure to avoid
copying the whole set?
Not sure I understand the suggestion. Elaborate?
bq. We'd better not add addBlacklistedNodeIdsToList to ClusterNodeTracker since
it calls application's logic, we should only include node related stuffs to
ClusterNodeTracker.
I feel any logic that has to iterate through all nodes should go through
ClusterNodeTracker - that way, we don't run into cases where we access the list
of nodes without a lock. Alternatively, we could get a list of nodeIDs from
ClusterNodeTracker and then look up individual nodes. I am not particular about
which approach, but also I don't quite see an issue with it being a part of
ClusterNodeTracker. Any particular reason you think this doesn't belong here?
> Add a helper library to maintain node state and allows common queries
> ---------------------------------------------------------------------
>
> Key: YARN-4719
> URL: https://issues.apache.org/jira/browse/YARN-4719
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 2.8.0
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: yarn-4719-1.patch, yarn-4719-2.patch
>
>
> The scheduler could use a helper library to maintain node state and allowing
> matching/sorting queries. Several reasons for this:
> # Today, a lot of the node state management is done separately in each
> scheduler. Having a single library will take us that much closer to reducing
> duplication among schedulers.
> # Adding a filtering/matching API would simplify node labels and locality
> significantly.
> # An API that returns a sorted list for a custom comparator would help
> YARN-1011 where we want to sort by allocation and utilization for
> continuous/asynchronous and opportunistic scheduling respectively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
