[
https://issues.apache.org/jira/browse/HADOOP-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228080#comment-15228080
]
Steve Loughran commented on HADOOP-13005:
-----------------------------------------
Stacks of NN and DN
{code}
2016-04-06 11:00:35,796 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception in secureMain java.io.IOException: java.lang.RuntimeException: Could
not resolve Kerberos principal name: java.net.UnknownHostException: xubunty:
xubunty: unknown error
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:347)
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:114)
at
org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.<init>(DatanodeHttpServer.java:108)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:781)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1138)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:432)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2423)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2310)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2357)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2538)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2562)
Caused by: java.lang.RuntimeException: Could not resolve Kerberos principal
name: java.net.UnknownHostException: xubunty: xubunty: unknown error
at
org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:90)
at
org.apache.hadoop.http.HttpServer2.getFilterProperties(HttpServer2.java:455)
at
org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:445)
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:340)
... 11 more
Caused by: java.net.UnknownHostException: xubunty: xubunty: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at
org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:224)
at
org.apache.hadoop.security.SecurityUtil.replacePattern(SecurityUtil.java:192)
at
org.apache.hadoop.security.SecurityUtil.getServerPrincipal(SecurityUtil.java:147)
at
org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:87)
... 14 more
Caused by: java.net.UnknownHostException: xubunty: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 18 more2016-04-06 11:00:35,799 INFO
org.apache.hadoop.util.ExitUtil: Exiting with status 12016-04-06 11:00:35,806
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
{code}
{code}
2016-04-06 10:59:31,714 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
Web-server for hdfs at: http://xubunty.cotham.uk:50070
2016-04-06 10:59:31,790 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-04-06 10:59:51,817 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping NameNode metrics system...
2016-04-06 10:59:51,818 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system stopped.
2016-04-06 10:59:51,819 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system shutdown complete.
2016-04-06 10:59:51,819 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Failed to start namenode.
java.io.IOException: java.lang.RuntimeException: Could not resolve Kerberos
principal name: java.net.UnknownHostException: xubunty: xubunty: unknown error
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:347)
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:114)
at
org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:892)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:720)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
Caused by: java.lang.RuntimeException: Could not resolve Kerberos principal
name: java.net.UnknownHostException: xubunty: xubunty: unknown error
at
org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:90)
at
org.apache.hadoop.http.HttpServer2.getFilterProperties(HttpServer2.java:455)
at
org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:445)
at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:340)
... 9 more
Caused by: java.net.UnknownHostException: xubunty: xubunty: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at
org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:224)
at
org.apache.hadoop.security.SecurityUtil.replacePattern(SecurityUtil.java:192)
at
org.apache.hadoop.security.SecurityUtil.getServerPrincipal(SecurityUtil.java:147)
at
org.apache.hadoop.security.AuthenticationFilterInitializer.getFilterConfigMap(AuthenticationFilterInitializer.java:87)
... 12 more
Caused by: java.net.UnknownHostException: xubunty: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 16 more
2016-04-06 10:59:51,822 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2016-04-06 10:59:51,825 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/************************************************************
{code}
> if local hostname doesn't resolve, HttpServer2 fails with a kerberos exception
> ------------------------------------------------------------------------------
>
> Key: HADOOP-13005
> URL: https://issues.apache.org/jira/browse/HADOOP-13005
> Project: Hadoop Common
> Issue Type: Bug
> Components: net
> Affects Versions: 2.7.2
> Environment: a VM whose hostname was no longer in /etc/hosts
> Reporter: Steve Loughran
> Priority: Minor
>
> When a service (NN, DN) is started and the local hostname isn't in
> /etc/hosts, things will, obviously, fail.
> However, the failure is reported as a wrapped Kerberos Exception, rather than
> going via the NetUtils.wrapException handling code
> {{java.lang.RuntimeException: Could not resolve Kerberos principal name:
> java.net.UnknownHostException: xubunty: xubunty: unknown error}}
> The text is a dangerous distraction from the real cause. Especially as it
> appears even on systems which aren't running kerberos.
> A fail-fast check could go into the http server startup process ahead of the
> security setup; NetUtils could add special reporting of the
> unknown-local-host problem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
