Dr. Martin Menzel commented on YARN-1226:

I extended the Test class given on the bug page of oracle to

import java.net.InetAddress;
import java.net.UnknownHostException;

public class HostNameTest {

        public static void main(String[] args) throws UnknownHostException {
                String version = System.getProperty("java.version");
                System.out.println(version + " getHostName(): " + 
                System.out.println(version + " getCanonicalHostName() : 
"+InetAddress.getLocalHost().getCanonicalHostName() );
                System.out.println(version + " www.google.com.getHostName(): " 
+ InetAddress.getByName("www.google.com").getHostName());

If I use getCanonicalHostName() instead of getHostName() I get on ipv6 and ipv4 
based hosts the FQDN.


May I ask the community why in such cases the hostname is used instead of the 
always unique IP address directly? I think round robin DNS IP assignments to 
the same name will not be relevant for hadoop nodes (or am I wrong here?).


Most of the hadoop programs running on datanodes are started by script using 
the entries in 


Using those entries as a common basis we could just ask the (forward) DNS which 
slave entry matches the local IP address. This mapping could be cached for 
further use.

> Inconsistent hostname leads to low data locality on IPv6 hosts
> --------------------------------------------------------------
>                 Key: YARN-1226
>                 URL: https://issues.apache.org/jira/browse/YARN-1226
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 0.23.3, 2.0.0-alpha, 2.1.0-beta
>         Environment: Linux, IPv6
>            Reporter: Kaibo Zhou
> When I run a mapreduce job which use TableInputFormat to scan a hbase table 
> on yarn cluser with 140+ nodes, I consistently get very low data locality 
> around 0~10%. 
> The scheduler is Capacity Scheduler. Hbase and hadoop are integrated in the 
> cluster with NodeManager, DataNode and HRegionServer run on the same node.
> The reason of low data locality is: most machines in the cluster uses IPV6, 
> few machines use IPV4. NodeManager use 
> "InetAddress.getLocalHost().getHostName()" to get the host name, but the 
> return result of this function depends on IPV4 or IPV6, see 
> ["InetAddress.getLocalHost().getHostName() returns 
> FQDN"|http://bugs.sun.com/view_bug.do?bug_id=7166687]. 
> On machines with ipv4, NodeManager get hostName as: 
> search042097.sqa.cm4.site.net
> But on machines with ipv6, NodeManager get hostName as: search042097.sqa.cm4
> if run with IPv6 disabled, -Djava.net.preferIPv4Stack=true, then returns 
> search042097.sqa.cm4.site.net.
> ----
> For the mapred job which scan hbase table, the InputSplit contains node 
> locations of [FQDN|http://en.wikipedia.org/wiki/FQDN], e.g. 
> search042097.sqa.cm4.site.net. Because in hbase, the RegionServers' hostnames 
> are allocated by HMaster. HMaster communicate with RegionServers and get the 
> region server's host name use java NIO: 
> clientChannel.socket().getInetAddress().getHostName().
> Also see the startup log of region server:
> 13:06:21,200 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master 
> passed us hostname to use. Was=search042024.sqa.cm4, 
> Now=search042024.sqa.cm4.site.net
> ----
> As you can see, most machines in the Yarn cluster with IPV6 get the short 
> hostname, but hbase always get the full hostname, so the Host cannot matched 
> (see RMContainerAllocator::assignToMap).This can lead to poor locality.
> After I use java.net.preferIPv4Stack to force IPv4 in yarn, I get 70+% data 
> locality in the cluster.
> Thanks,
> Kaibo

This message was sent by Atlassian JIRA

Reply via email to