[
https://issues.apache.org/jira/browse/HDFS-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190108#comment-17190108
]
huhaiyang edited comment on HDFS-15556 at 9/3/20, 12:42 PM:
------------------------------------------------------------
3.the cause of occurred the problem is:
{quote}
1.One DataNode reports heartbeat to NN timed out, The DNA_REGISTER will be
occurred when the service is restored：
BPServiceActor#run->offerService->processCommand->reRegister->sendHeartBeat
2.NN run registerDatanode will DatanodeDescriptor#pruneStorageMap (remove
storageMap) for the registered DN
3.DN reRegister it took about a minute, after the heartbeat exceeds 9 seconds,
the Lifeline reports to NN,
But at this point, the storageMap is null of the DN is recorded at the NN
occurred NPE
{quote}
4. detailed execution log
[^NN_DN.LOG]
5.Source code is：
{code:java}
HeartbeatManager#updateLifeline
synchronized void updateLifeline(final DatanodeDescriptor node,StorageReport[]
reports, long cacheCapacity, long cacheUsed,int xceiverCount, int failedVolumes,
VolumeFailureSummary volumeFailureSummary) {
stats.subtract(node);
//Every time DN heartbeat report，nodesInServiceXceiverCount will be minus
the XceiverCount of the DN of the current
...
node.updateHeartbeatState(reports, cacheCapacity, cacheUsed, xceiverCount,
failedVolumes, volumeFailureSummary);
//NPE exception occurred here throws
stats.add(node); //Here logic is never executed
}
{code}
{code:java}
BlockPlacementPolicyDefault#excludeNodeByLoad
boolean excludeNodeByLoad(DatanodeDescriptor node){
final double maxLoad = considerLoadFactor *
stats.getInServiceXceiverAverage();
//stats.getInServiceXceiverAverage()=
heartbeatManager.getInServiceXceiverCount()/getNumDatanodesInService()
//the final maxLoad value will be affected
final int nodeLoad = node.getXceiverCount();
if ((nodeLoad > maxLoad) && (maxLoad > 0)) {
logNodeIsNotChosen(node, NodeNotChosenReason.NODE_TOO_BUSY,
"(load: " + nodeLoad + " > " + maxLoad + ")");
return true;
}
return false;
}
{code}
was (Author: haiyang hu):
3.the cause of occurred the problem is:
{quote}
1.One DataNode reports heartbeat to NN timed out, The DNA_REGISTER will be
occurred when the service is restored：
BPServiceActor#run->offerService->processCommand->reRegister->sendHeartBeat
2.NN run registerDatanode will DatanodeDescriptor#pruneStorageMap (remove
storageMap) for the registered DN
3.DN reRegister it took about a minute, after the heartbeat exceeds 9 seconds,
the Lifeline reports to NN,
But at this point, the storageMap is null of the DN is recorded at the NN
occurred NPE
{quote}
4. detailed execution log
[^NN_DN.LOG]
5.Source code is：
{code:java}
HeartbeatManager#updateLifeline
synchronized void updateLifeline(final DatanodeDescriptor
node,StorageReport[] reports, long cacheCapacity, long cacheUsed,
int xceiverCount, int failedVolumes,
VolumeFailureSummary volumeFailureSummary) {
stats.subtract(node); //Every time DN heartbeat
report，nodesInServiceXceiverCount will be minus the XceiverCount of the DN of
the current
...
node.updateHeartbeatState(reports, cacheCapacity, cacheUsed,
xceiverCount, failedVolumes, volumeFailureSummary); //NPE exception
occurred here throws
stats.add(node); //Here logic is never executed
}
BlockPlacementPolicyDefault#excludeNodeByLoad
boolean excludeNodeByLoad(DatanodeDescriptor node){
final double maxLoad = considerLoadFactor *
stats.getInServiceXceiverAverage();
//stats.getInServiceXceiverAverage()=
heartbeatManager.getInServiceXceiverCount()/getNumDatanodesInService()
//the final maxLoad value will be affected
final int nodeLoad = node.getXceiverCount();
if ((nodeLoad > maxLoad) && (maxLoad > 0)) {
logNodeIsNotChosen(node, NodeNotChosenReason.NODE_TOO_BUSY,
"(load: " + nodeLoad + " > " + maxLoad + ")");
return true;
}
return false;
}
{code}
> Fix NPE in DatanodeDescriptor#updateStorageStats when handle DN Lifeline
> ------------------------------------------------------------------------
>
> Key: HDFS-15556
> URL: https://issues.apache.org/jira/browse/HDFS-15556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.2.0
> Reporter: huhaiyang
> Priority: Critical
> Attachments: HDFS-15556.001.patch, NN-CPU.png, NN_DN.LOG
>
>
> In our cluster, the NameNode appears NPE when processing lifeline messages
> sent by the DataNode, which will cause an maxLoad exception calculated by NN.
> because DataNode is identified as busy and unable to allocate available nodes
> in choose DataNode, program loop execution results in high CPU and reduces
> the processing performance of the cluster.
> *NameNode the exception stack*:
> {code:java}
> 2020-08-25 00:59:02,977 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 8022, call Call#20535 Retry#0
> org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol.sendLifeline
> from xxxxx:34766
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateStorageStats(DatanodeDescriptor.java:460)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateHeartbeatState(DatanodeDescriptor.java:390)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.updateLifeline(HeartbeatManager.java:254)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleLifeline(DatanodeManager.java:1805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleLifeline(FSNamesystem.java:4039)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendLifeline(NameNodeRpcServer.java:1761)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeLifelineProtocolServerSideTranslatorPB.sendLifeline(DatanodeLifelineProtocolServerSideTranslatorPB.java:62)
> at
> org.apache.hadoop.hdfs.protocol.proto.DatanodeLifelineProtocolProtos$DatanodeLifelineProtocolService$2.callBlockingMethod(DatanodeLifelineProtocolProtos.java:409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:886)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:828)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1903)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2717)
> {code}
> {code:java}
> // DatanodeDescriptor#updateStorageStats
> ...
> for (StorageReport report : reports) {
> DatanodeStorageInfo storage = null;
> synchronized (storageMap) {
> storage =
> storageMap.get(report.getStorage().getStorageID());
> }
> if (checkFailedStorages) {
> failedStorageInfos.remove(storage);
> }
> storage.receivedHeartbeat(report); // NPE exception occurred here
> // skip accounting for capacity of PROVIDED storages!
> if (StorageType.PROVIDED.equals(storage.getStorageType())) {
> continue;
> }
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
