JiangHua Zhu created HDFS-15794:
-----------------------------------
Summary: IBR and FBR use different queues to load data.
Key: HDFS-15794
URL: https://issues.apache.org/jira/browse/HDFS-15794
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: JiangHua Zhu
When DataNode reports data to NameNode, IBR and FBR are included here.
After the NameNode receives the DataNode request, it temporarily stores the
data in a queue, here it refers to
BlockManager#BlockReportProcessingThread#queue.
NameNodeRpcServer#blockReport()
for (int r = 0; r <reports.length; r++) {
final BlockListAsLongs blocks = reports[r].getBlocks();
final int index = r;
noStaleStorages = bm.runBlockOp(() ->
bm.processReport(nodeReg, reports[index].getStorage(),
blocks, context));
}
NameNodeRpcServer#blockReport()
for (final StorageReceivedDeletedBlocks r: receivedAndDeletedBlocks) {
bm.enqueueBlockOp(new Runnable() {
@Override
public void run() {
try {
namesystem.processIncrementalBlockReport(nodeReg, r);
} catch (Exception ex) {
// usually because the node is unregistered/dead. next heartbeat
// will correct the problem
blockStateChangeLog.error(
"*BLOCK* NameNode.blockReceivedAndDeleted: "
+ "failed from "+ nodeReg + ":" + ex.getMessage());
}
}
});
}
The problem here is that when the NameNode is blocked in processing the IBR,
the FBR requested by the DN from the NameNode will be affected. Similarly, when
the NameNode processing FBR is blocked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
