[
https://issues.apache.org/jira/browse/HADOOP-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007915#comment-16007915
]
Rohith Sharma K S commented on HADOOP-14412:
--------------------------------------------
Thanks Jason for finding this issue. I am +1 for using AtomicReference and for
the patch. It looks cleaner and better solution now.
> HostsFileReader#getHostDetails is very expensive on large clusters
> ------------------------------------------------------------------
>
> Key: HADOOP-14412
> URL: https://issues.apache.org/jira/browse/HADOOP-14412
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 2.8.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: HADOOP-14412.001.patch
>
>
> After upgrading one of our large clusters to 2.8 we noticed many IPC server
> threads of the resourcemanager spending time in NodesListManager#isValidNode
> which in turn was calling HostsFileReader#getHostDetails. The latter is
> creating complete copies of the include and exclude sets for every node
> heartbeat, and these sets are not small due to the size of the cluster.
> These copies are causing multiple resizes of the underlying HashSets being
> filled and creating lots of garbage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
