[
https://issues.apache.org/jira/browse/ACCUMULO-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith Turner updated ACCUMULO-2036:
-----------------------------------
Description:
I ran listscans in the Accumulo shell while running continuous verify on small
cluster and almost not map task were running locally.
I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT. Before
that change Accumulo would always store IP addrs. Code like the following in
o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits() would translate IPs to
hostnames.
{code:java}
if (location == null) {
InetAddress inetAddress = InetAddress.getByName(ip);
location = inetAddress.getHostName();
hostNameCache.put(ip, location);
}
{code}
In my case I configured Accumulo to use hostnames, but not fully qualified
ones. So I think the above code was just passing the non-quallified hostname
through. I suspected hadoop wanted FQDN and changed the code to the following
and mappers ran locally. I need to confirm what hadoop is expecting. I think
the above code will result in a FQDN if given an IP, so this is not an issue
for 1.4 or 1.5.
{code:java}
if (location == null) {
InetAddress inetAddress = InetAddress.getByName(ip);
location = inetAddress.getCanonicalHostName();
hostNameCache.put(ip, location);
}
{code}
was:
I ran listscans in the Accumulo shell while running continuous verify on small
cluster and almost not map task were running locally.
I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT. Before
that change Accumulo would always store IP addrs. Code like the following in
o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits() would translate IPs to
hostnames.
{code:java}
if (location == null) {
InetAddress inetAddress = InetAddress.getByName(ip);
location = inetAddress.getHostName();
hostNameCache.put(ip, location);
}
{code}
In my case I configured Accumulo to use hostnames, but not fully qualified
ones. So I think the above code was just passing the non-quallified hostname
through. I suspected that maybe hadoop wanted FQDN and changed the code the
following and mappers ran locally. I need to confirm what hadoop is expecting.
I think the above code will result in a FQDN if given an IP, but not a
non-quallified hostname. So this is not an issue for 1.4 or 1.5.
{code:java}
if (location == null) {
InetAddress inetAddress = InetAddress.getByName(ip);
location = inetAddress.getCanonicalHostName();
hostNameCache.put(ip, location);
}
{code}
> Mappers are not running locally
> -------------------------------
>
> Key: ACCUMULO-2036
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2036
> Project: Accumulo
> Issue Type: Bug
> Reporter: Keith Turner
> Fix For: 1.6.0
>
>
> I ran listscans in the Accumulo shell while running continuous verify on
> small cluster and almost not map task were running locally.
> I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT. Before
> that change Accumulo would always store IP addrs. Code like the following in
> o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits() would translate IPs
> to hostnames.
> {code:java}
> if (location == null) {
> InetAddress inetAddress = InetAddress.getByName(ip);
> location = inetAddress.getHostName();
> hostNameCache.put(ip, location);
> }
> {code}
> In my case I configured Accumulo to use hostnames, but not fully qualified
> ones. So I think the above code was just passing the non-quallified hostname
> through. I suspected hadoop wanted FQDN and changed the code to the
> following and mappers ran locally. I need to confirm what hadoop is
> expecting. I think the above code will result in a FQDN if given an IP, so
> this is not an issue for 1.4 or 1.5.
> {code:java}
> if (location == null) {
> InetAddress inetAddress = InetAddress.getByName(ip);
> location = inetAddress.getCanonicalHostName();
> hostNameCache.put(ip, location);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
