[
https://issues.apache.org/jira/browse/HDFS-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amithsha updated HDFS-10728:
----------------------------
Comment: was deleted
(was: This is because of java network ip cache,
*networkaddress.cache.ttl = -1 (property under java.security )*
you can find the above config in jdk-<Version>/jre/lib/security/java.secEven
though
Eventhough if it is commented on that particular file in hadoop code its
default value is *-1 (FOREVER)*
So remove the comment or add the config with value *networkaddress.cache.ttl =
0*)
> DN can't communicate with NN when IP address of NN changed since Auth failed
> ----------------------------------------------------------------------------
>
> Key: HDFS-10728
> URL: https://issues.apache.org/jira/browse/HDFS-10728
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ipc
> Affects Versions: 2.7.1
> Reporter: He Xiaoqiao
> Priority: Major
>
> In security authentication hadoop cluster with Kerberos, when I changed SBN's
> ip address, hostname not changed and the routing tables are also refresh, DN
> can't communicate with NN since auth failed.
> Trace info of DN as following:
> {code:xml}
> 2016-08-04 22:45:25,719 WARN org.apache.hadoop.ipc.Client: Address change
> detected. Old: {hostname}/{ip1}:8020 New: {hostname}/{ip2}:8020
> 2016-08-04 22:45:25,719 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: {hostname}/{ip2}:8020. Already tried 0 time(s); maxRetries=45
> ......
> 2016-08-04 22:45:38,010 WARN org.apache.hadoop.ipc.Client: Couldn't setup
> connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
> at java.lang.Thread.run(Thread.java:745)
> 2016-08-04 22:45:38,014 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in offerService
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020;
> Host Details : local host is: "{dn hostname}/{dn ip}"; destination host is:
> "{hostname}":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
> at org.apache.hadoop.ipc.Client.call(Client.java:1474)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Couldn't setup connection for hdfs/{dn
> hostname}@REALMS.COM to {hostname}/{ip1}:8020
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> ... 8 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
> ... 11 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
