[
https://issues.apache.org/jira/browse/HDFS-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897137#comment-13897137
]
Aaron T. Myers commented on HDFS-5922:
--------------------------------------
In the heartbeat thread in BPServiceActor, we have the following:
{code}
if (waitTime > 0 && pendingReceivedRequests == 0) {
try {
pendingIncrementalBRperStorage.wait(waitTime);
{code}
This means that if for some reason the value of {{pendingReceivedRequests}}
permanently stays positive then we will never sleep in the heartbeat thread.
The question, then, is what can cause this value to stay positive.
I believe the issue is that in
{{BPServiceActor#addPendingReplicationBlockInfo}} we might not increase the
size of the {{PerStoragePendingIncrementalBR}} if there is already an entry for
a given block in there:
{code}
// Make sure another entry for the same block is first removed.
// There may only be one such entry.
for (Map.Entry<String, PerStoragePendingIncrementalBR> entry :
pendingIncrementalBRperStorage.entrySet()) {
if (entry.getValue().removeBlockInfo(bInfo)) {
break;
}
}
getIncrementalBRMapForStorage(storageUuid).putBlockInfo(bInfo);
{code}
But in {{BPServiceActor#notifyNamenodeBlockImmediately}} we will always
increment {{pendingReceivedRequests}} regardless of whether or not there was
already an entry for the block:
{code}
void notifyNamenodeBlockImmediately(
ReceivedDeletedBlockInfo bInfo, String storageUuid) {
synchronized (pendingIncrementalBRperStorage) {
addPendingReplicationBlockInfo(bInfo, storageUuid);
pendingReceivedRequests++;
pendingIncrementalBRperStorage.notifyAll();
}
}
{code}
Then, in {{BPServiceActor#reportReceivedDeletedBlocks}}, we will only subtract
the number of blocks that are actually in the
{{PerStoragePendingIncrementalBR}} from {{pendingReceivedRequests}}:
{code}
ReceivedDeletedBlockInfo[] rdbi = perStorageMap.dequeueBlockInfos();
pendingReceivedRequests =
(pendingReceivedRequests > rdbi.length ?
(pendingReceivedRequests - rdbi.length) : 0);
{code}
This means that if we ever call
{{BPServiceActor#notifyNamenodeBlockImmediately}} twice without calling
{{BPServiceActor#reportReceivedDeletedBlocks}} in between, we will have
{{pendingReceivedRequests}} at 2, but then only subtract 1 from it.
[~andrew.wang] also pointed out offline that it is perhaps incorrect to be
subtracting the number of _deleted_ blocks from {{pendingReceivedRequests}} in
{{BPServiceActor#reportReceivedDeletedBlocks}}, but the result of that is
somewhat less serious, since in that case the worst case is just that we send a
somewhat delayed IBR.
> DN heartbeat thread can get stuck in tight loop
> -----------------------------------------------
>
> Key: HDFS-5922
> URL: https://issues.apache.org/jira/browse/HDFS-5922
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.3.0
> Reporter: Aaron T. Myers
>
> We saw an issue recently on a test cluster where one of the DN threads was
> consuming 100% of a single CPU. Running jstack indicated that it was the DN
> heartbeat thread. I believe I've tracked down the cause to a bug in the
> accounting around the value of {{pendingReceivedRequests}}.
> More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
