Ming Ma created HDFS-10203:
------------------------------
Summary: Excessive topology lookup for large number of client
machines in namenode
Key: HDFS-10203
URL: https://issues.apache.org/jira/browse/HDFS-10203
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ming Ma
In the {{ClientProtocol#getBlockLocations}} call, DatanodeManager computes the
network distance between the client machine and the datanodes. As part of that,
it needs to resolve the network location of the client machine. If the client
machine isn't a datanode, it needs to ask {{DNSToSwitchMapping}} to resolve it.
{noformat}
public void sortLocatedBlocks(final String targethost,
final List<LocatedBlock> locatedblocks) {
//sort the blocks
// As it is possible for the separation of node manager and datanode,
// here we should get node but not datanode only .
Node client = getDatanodeByHost(targethost);
if (client == null) {
List<String> hosts = new ArrayList<> (1);
hosts.add(targethost);
List<String> resolvedHosts = dnsToSwitchMapping.resolve(hosts);
if (resolvedHosts != null && !resolvedHosts.isEmpty()) {
....
}
}
}
{noformat}
When there are ten of thousands of non-datanode client machines hitting the
namenode which uses {{ScriptBasedMapping}}, it causes the following issues:
* After namenode startup, {{CachedDNSToSwitchMapping}} only has datanodes in
the cache. Calls from many different client machines means lots of process fork
to run the topology script and can cause spike in namenode load.
* Cache size of {{CachedDNSToSwitchMapping}} can grow large. Under normal
workload say < 100k client machines and each entry in the cache uses < 200
bytes, it will take up to 20MB, not much compared to NN's heap size. But in
theory it can still blow up NN if there is misconfiguration or malicious attack
with millions of IP addresses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
