[
https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241420#comment-17241420
]
GeoffreyStark edited comment on HDFS-13831 at 12/1/20, 12:43 PM:
-----------------------------------------------------------------
[~linyiqun], [~jianliang.wu] [~weichiu] Hello, recently, in A production
environment with more than 1000 nodes(HDP3.1.0;HADOOP-3.1.0), when A large
number of files were deleted, RPC response was particularly high. I detected
the phenomenon in the following figure with tool Arthas(Alibaba)
[Arthas screenShot|https://gaofeng.blog.csdn.net/article/details/110442949]
The HDP code is as follows
{code:java}
static int BLOCK_DELETION_INCREMENT = 1000;
....
.....
......
boolean delete(String src, boolean recursive, boolean logRetryCache)
throws IOException {
final String operationName = "delete";
BlocksMapUpdateInfo toRemovedBlocks = null;
checkOperation(OperationCategory.WRITE);
final FSPermissionChecker pc = getPermissionChecker();
writeLock();//————high latency time
boolean ret = false;
try {
checkOperation(OperationCategory.WRITE);
checkNameNodeSafeMode("Cannot delete " + src);
toRemovedBlocks = FSDirDeleteOp.delete(
this, pc, src, recursive, logRetryCache);
ret = toRemovedBlocks != null;
} catch (AccessControlException e) {
logAuditEvent(false, operationName, src);
throw e;
} finally {
writeUnlock(operationName);
}
getEditLog().logSync();
if (toRemovedBlocks != null) {
//删除数据块操作,循环刚才收集到的toRemovedBlocks,然后调用blockManager的removeBlock来处理要删除的数据块.
removeBlocks(toRemovedBlocks); // Incremental deletion of blocks
}
logAuditEvent(true, operationName, src);
return ret;
}
....
.....
......
void removeBlocks(BlocksMapUpdateInfo blocks) {
List<BlockInfo> toDeleteList = blocks.getToDeleteList();
Iterator<BlockInfo> iter = toDeleteList.iterator();
while (iter.hasNext()) {
writeLock();//————> high latency time
try {
for (int i = 0; i < BLOCK_DELETION_INCREMENT && iter.hasNext(); i++) {
blockManager.removeBlock(iter.next());
}
} finally {
writeUnlock("removeBlocks");
}
}
}{code}
Should I inject your patch and make the
parameters("dfs.namenode.block.deletion.increment") bigger?
was (Author: gaofeng6):
[~linyiqun], [~jianliang.wu] [~weichiu] Hello, recently, in A production
environment with more than 1000 nodes(HDP3.1.0;HADOOP-3.1.0), when A large
number of files were deleted, RPC response was particularly high. I detected
the phenomenon in the following figure with tool Arthas(Alibaba)
[Arthas screenShot|https://gaofeng.blog.csdn.net/article/details/110442949]
The HDP code is as follows
{code:java}
static int BLOCK_DELETION_INCREMENT = 1000;
....
.."...
......
void removeBlocks(BlocksMapUpdateInfo blocks)
{//写入editlog后，循环刚才收集到的blocks,然后调用blockManager的removeBlock来处理要删除的数据块.
List<BlockInfo> toDeleteList = blocks.getToDeleteList();
Iterator<BlockInfo> iter = toDeleteList.iterator();
while (iter.hasNext()) {
writeLock();
try {
//循环收集到的块，这里双重限制：常量限制(BLOCK_DELETION_INCREMENT)和块数量限制(iter.hasNext())
//每次默认限制删除块的增量是BLOCK_DELETION_INCREMENT(1000).
for (int i = 0; i < BLOCK_DELETION_INCREMENT && iter.hasNext(); i++) {
blockManager.removeBlock(iter.next());//用blockManager的removeBlock来处理要删除的数据块.
}
} finally {
writeUnlock("removeBlocks");
}
}
}{code}
Should I inject your patch and make the
parameters("dfs.namenode.block.deletion.increment") bigger?
> Make block increment deletion number configurable
> -------------------------------------------------
>
> Key: HDFS-13831
> URL: https://issues.apache.org/jira/browse/HDFS-13831
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: Yiqun Lin
> Assignee: Ryan Wu
> Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch,
> HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch
>
>
> When NN deletes a large directory, it will hold the write lock long time. For
> improving this, we remove the blocks in a batch way. So that other waiters
> have a chance to get the lock. But right now, the batch number is a
> hard-coded value.
> {code}
> static int BLOCK_DELETION_INCREMENT = 1000;
> {code}
> We can make this value configurable, so that we can control the frequency of
> other waiters to get the lock chance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
