[
https://issues.apache.org/jira/browse/HDFS-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272716#comment-17272716
]
JiangHua Zhu edited comment on HDFS-15794 at 1/27/21, 9:45 AM:
---------------------------------------------------------------
When DataNode requests IBR and FBR from NameNode, NameNode uses different queue
processing (BlockManager). This can improve the performance of NameNode
processing these two types of requests.
[~weichiu] [~elgoiri] Do you have different ideas?
was (Author: jianghuazhu):
When DataNode requests IBR and FBR from NameNode, NameNode uses different queue
processing (BlockManager). This can improve the performance of NameNode
processing these two types of requests.
[~weichiu] [~elgoiri] Do you have different opinions?
> IBR and FBR use different queues to load data.
> ----------------------------------------------
>
> Key: HDFS-15794
> URL: https://issues.apache.org/jira/browse/HDFS-15794
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
>
> When DataNode reports data to NameNode, IBR and FBR are included here.
> After the NameNode receives the DataNode request, it temporarily stores the
> data in a queue, here it refers to
> BlockManager#BlockReportProcessingThread#queue.
> NameNodeRpcServer#blockReport()
> for (int r = 0; r <reports.length; r++) {
> final BlockListAsLongs blocks = reports[r].getBlocks();
> final int index = r;
> noStaleStorages = bm.runBlockOp(() ->
> bm.processReport(nodeReg, reports[index].getStorage(),
> blocks, context));
> }
> NameNodeRpcServer#blockReport()
> for (final StorageReceivedDeletedBlocks r: receivedAndDeletedBlocks) {
> bm.enqueueBlockOp(new Runnable() {
> @Override
> public void run() {
> try {
> namesystem.processIncrementalBlockReport(nodeReg, r);
> } catch (Exception ex) {
> // usually because the node is unregistered/dead. next heartbeat
> // will correct the problem
> blockStateChangeLog.error(
> "*BLOCK* NameNode.blockReceivedAndDeleted: "
> + "failed from "+ nodeReg + ":" + ex.getMessage());
> }
> }
> });
> }
> The problem here is that when the NameNode is blocked in processing the IBR,
> the FBR requested by the DN from the NameNode will be affected. Similarly,
> when the NameNode processing FBR is blocked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
