yunjiong zhao created HDFS-13441:
------------------------------------

             Summary: DataNode missed BlockKey update from NameNode due to 
HeartbeatResponse was dropped
                 Key: HDFS-13441
                 URL: https://issues.apache.org/jira/browse/HDFS-13441
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, namenode
    Affects Versions: 2.7.1
            Reporter: yunjiong zhao
            Assignee: yunjiong zhao


After NameNode failover, lots of application failed due to some DataNodes can't 
re-compute password from block token.2018-04-11 20:10:52,448 ERROR 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error processing 
unknown operation  src: /10.142.74.116:57404 dst: /10.142.77.45:50010 
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password 
[Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't 
re-compute password for block_token_identifier (expiryDate=1523538652448, 
keyId=1762737944, userId=hadoop, 
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700, 
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't 
exist.]         at 
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
         at 
com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
         at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)    
     at java.lang.Thread.run(Thread.java:745) Caused by: 
org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't re-compute 
password for block_token_identifier (expiryDate=1523538652448, 
keyId=1762737944, userId=hadoop, 
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700, 
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't 
exist.         at 
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
         at 
org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
         at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
         at 
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
         ... 7 more

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Reply via email to