yunjiong zhao created HDFS-13441:
------------------------------------
Summary: DataNode missed BlockKey update from NameNode due to
HeartbeatResponse was dropped
Key: HDFS-13441
URL: https://issues.apache.org/jira/browse/HDFS-13441
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, namenode
Affects Versions: 2.7.1
Reporter: yunjiong zhao
Assignee: yunjiong zhao
After NameNode failover, lots of application failed due to some DataNodes can't
re-compute password from block token.2018-04-11 20:10:52,448 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error processing
unknown operation src: /10.142.74.116:57404 dst: /10.142.77.45:50010
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
[Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
re-compute password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist.] at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at
com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't re-compute
password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist. at
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
at
org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 7 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
