[
https://issues.apache.org/jira/browse/HDFS-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867011#comment-16867011
]
Íñigo Goiri edited comment on HDFS-14579 at 6/18/19 8:22 PM:
-------------------------------------------------------------
So here you can see this stuck at getting the address:
{code}
"main" #1 prio=5 os_prio=0 tid=0x0000000001b38800 nid=0x6a0c runnable
[0x0000000001b1e000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.parseEntry(HostFileManager.java:94)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.readFile(HostFileManager.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:157)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:70)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.<init>(DatanodeManager.java:274)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:416)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:408)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:792)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:673)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:750)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1000)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:979)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1726)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1794)
{code}
The code path is not exactly the same but it's the same DNS resolution issue.
This is Hadoop 2.9.1 BTW.
was (Author: elgoiri):
So here you can see this stuck at getting the address:
{code}
"main" #1 prio=5 os_prio=0 tid=0x0000000001b38800 nid=0x6a0c runnable
[0x0000000001b1e000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.parseEntry(HostFileManager.java:94)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.readFile(HostFileManager.java:80)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:157)
at
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:70)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.<init>(DatanodeManager.java:274)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:416)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:408)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:792)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:673)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:750)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1000)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:979)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1726)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1794)
{code}
The code path is not exactly the same but it's the same DNS resolution issue.
> In refreshNodes, avoid performing a DNS lookup while holding the write lock
> ---------------------------------------------------------------------------
>
> Key: HDFS-14579
> URL: https://issues.apache.org/jira/browse/HDFS-14579
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.3.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Attachments: HDFS-14579.001.patch
>
>
> When refreshNodes is called on a large cluster, or a cluster where DNS is not
> performing well, it can cause the namenode to hang for a long time. This is
> because the refreshNodes operation holds the global write lock while it is
> running. Most of refreshNodes code is simple and hence fast, but
> unfortunately it performs a DNS lookup for each host in the cluster while the
> lock is held.
> Right now, it calls:
> {code}
> public void refreshNodes(final Configuration conf) throws IOException {
> refreshHostsReader(conf);
> namesystem.writeLock();
> try {
> refreshDatanodes();
> countSoftwareVersions();
> } finally {
> namesystem.writeUnlock();
> }
> }
> {code}
> The line refreshHostsReader(conf); reads the new config file and does a DNS
> lookup on each entry - the write lock is not held here. Then the main work is
> done here:
> {code}
> private void refreshDatanodes() {
> final Map<String, DatanodeDescriptor> copy;
> synchronized (this) {
> copy = new HashMap<>(datanodeMap);
> }
> for (DatanodeDescriptor node : copy.values()) {
> // Check if not include.
> if (!hostConfigManager.isIncluded(node)) {
> node.setDisallowed(true);
> } else {
> long maintenanceExpireTimeInMS =
> hostConfigManager.getMaintenanceExpirationTimeInMS(node);
> if (node.maintenanceNotExpired(maintenanceExpireTimeInMS)) {
> datanodeAdminManager.startMaintenance(
> node, maintenanceExpireTimeInMS);
> } else if (hostConfigManager.isExcluded(node)) {
> datanodeAdminManager.startDecommission(node);
> } else {
> datanodeAdminManager.stopMaintenance(node);
> datanodeAdminManager.stopDecommission(node);
> }
> }
> node.setUpgradeDomain(hostConfigManager.getUpgradeDomain(node));
> }
> }
> {code}
> All the isIncluded(), isExcluded() methods call node.getResolvedAddress()
> which does the DNS lookup. We could probably change things to perform all the
> DNS lookups outside of the write lock, and then take the lock and process the
> nodes. Also change or overload isIncluded() etc to take the inetAddress
> rather than the datanode descriptor.
> It would not shorten the time the operation takes to run overall, but it
> would move the long duration out of the write lock and avoid blocking the
> namenode for the entire time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
