[
https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13767431#comment-13767431
]
Junping Du commented on YARN-1200:
----------------------------------
Thanks Harsh for nice catch on this. We do need a central topology view for
consistence and easy maintenances.
I like Steve's comments, but a few more to add:
bq. +1 to this idea -without it AMs can't track topologies that change over
time.
That's right. The worse case is YARN (and HDFS) doesn't provide a mechanism to
track topology changes (remove the network location that previous cached). The
only use case now to trigger cache clear is fault topology in Datanode
registering (it also should be optimised later to clear only one item instead
of clear all items now). I will file a JIRA and work on it if no one file it
before.
bq. Actually, having a notification that the topology has changed could be even
more useful for some applications, but that is very much feature creep.
That's a great idea. We do need a broadcasting mechanism for YARN rather than
individual heartbeat between AMs/RM. Here is a typical case. I guess we should
file some JIRA against this. Thoughts?
> Provide a central view for rack topologies
> ------------------------------------------
>
> Key: YARN-1200
> URL: https://issues.apache.org/jira/browse/YARN-1200
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 2.1.0-beta
> Reporter: Harsh J
>
> It appears that with YARN, any AM (such as the MRv2 AM) that tries to do
> rack-info-based work, will need to resolve racks locally rather than get rack
> info from YARN directly:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054
> and its use of a simple implementation of
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
> This is a regression, as we've traditionally only had users maintain rack
> mappings and its associated script on a single master role node (JobTracker),
> not at every compute node. Task spawning hosts have never done/needed rack
> resolution of their own.
> It is silly to have to maintain rack configs and their changes on all nodes.
> We should have the RM host a stable interface service so that there's only a
> single view of the topology across the cluster, and document for AMs to use
> that instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
