yunjiong zhao created HDFS-13441: ------------------------------------ Summary: DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped Key: HDFS-13441 URL: https://issues.apache.org/jira/browse/HDFS-13441 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.7.1 Reporter: yunjiong zhao Assignee: yunjiong zhao
After NameNode failover, lots of application failed due to some DataNodes can't re-compute password from block token.2018-04-11 20:10:52,448 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error processing unknown operation src: /10.142.74.116:57404 dst: /10.142.77.45:50010 javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't re-compute password for block_token_identifier (expiryDate=1523538652448, keyId=1762737944, userId=hadoop, blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700, access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't exist.] at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598) at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't re-compute password for block_token_identifier (expiryDate=1523538652448, keyId=1762737944, userId=hadoop, blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700, access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't exist. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241) at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) ... 7 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org