[
https://issues.apache.org/jira/browse/HDFS-16596?focusedWorklogId=775303&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775303
]
ASF GitHub Bot logged work on HDFS-16596:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/22 00:59
Start Date: 27/May/22 00:59
Worklog Time Spent: 10m
Work Description: ZanderXu commented on code in PR #4360:
URL: https://github.com/apache/hadoop/pull/4360#discussion_r883193453
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -3003,10 +3003,10 @@
</property>
<property>
- <name>dfs.datanode.fsdatasetasyncdisk.max.threads.per.volume</name>
- <value>4</value>
+ <name>dfs.datanode.fsdatasetasyncdisk.core.threads.per.volume</name>
+ <value>1</value>
<description>
- The maximum number of threads per volume used to process async disk
+ The number of core threads per volume used to process async disk
Review Comment:
@saintstack Thanks for your comment.
> During bulk delete, the executor did not blossom up to the max number of
threads per volume? Upping the max wouldn't help w/ your case?
The threadPoolExecutor used one unbounded LinkedBlockingQueue, so the
executor will not blossom up to the max number of threads per volume, and
upping the max wouldn't help this case.
Because the threadPoolExecutor use one unbounded LinkedBlockingQueue, so I
want to set the maximum number of threads same with the number of core threads.
Do you have any good suggestions?
Issue Time Tracking
-------------------
Worklog Id: (was: 775303)
Time Spent: 40m (was: 0.5h)
> Improve the processing capability of FsDatasetAsyncDiskService
> --------------------------------------------------------------
>
> Key: HDFS-16596
> URL: https://issues.apache.org/jira/browse/HDFS-16596
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In our production environment, when DN needs to delete a large number blocks,
> we find that many deletion tasks are backlogged in the queue of
> threadPoolExecutor in FsDatasetAsyncDiskService. We can't improve its
> throughput because the number of core threads is hard coded.
> So DN needs to support the number of core threads of
> FsDatasetAsyncDiskService can be configured.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
