hfutatzhanghb opened a new pull request, #6530:
URL: https://github.com/apache/hadoop/pull/6530
### Description of PR
Refer to HDFS-17372.
Recently, we met a critical problem in our production cluster which have
lots of small files. In that cluster, per datanode has almost six million
blocks.
After deleting large dir or recommision, some datanodes's
SumOfActorCommandQueueLength metrics became very high as below picture shows.
After a while, we found some datanodes's write block ops became zero. That
means client can not write to those datanodes. We found some logs on those
datanodes:
```
2024-02-05 17:30:11,657 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification
failed: op=WRITE_BLOCK, remoteAddress=/x.x.x.x:63990, message=Can't re-compute
password for block_token_identifier (expiryDate=1707161411033, keyId=596193458,
userId=hdfs, blockPoolId=BP-1169917699-x.x.x.x-1678688680604,
blockId=11354988786, access modes=[WRITE], storageTypes= [DISK, DISK, DISK,
DISK, DISK, DISK, DISK], storageIds= [DS-2664b73d-1cc9-4613-b6e3-cd58ec5ae8d5,
DS-8f18621e-46c1-49b3-8bf5-7f5b8e47fe80,
DS-527817e1-c5d0-4085-929f-9100a423223f,
DS-d3f850c9-18ab-43cc-86d9-779b9d71a79f,
DS-e4aa15ce-0c5f-4645-a582-e187e46e7183,
DS-3fbbd401-58d2-49ac-a258-da7769417593,
DS-d76a51e4-73db-4db6-a4e0-f82a38c1ad82]), since the required block key
(keyID=596193458) doesn't exist.
```
That is to say, DNA_ACCESSKEYUPDATE command was blocked in
CommandProcessingThread#queue. This can be deadly.
So, we should guarantee that command with high-priority should be processed
in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
