[ https://issues.apache.org/jira/browse/HDFS-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17055739#comment-17055739 ]
zhuqi edited comment on HDFS-15180 at 3/10/20, 9:44 AM: -------------------------------------------------------- !image-2020-03-10-17-22-57-391.png|width=604,height=137! Hi [~sodonnell] It's the thread blocked avg per day monitor. I just gray 3 nodes in busy cluster in different time , the green one is the longest, and there are almost no blocked thread now. The two one has a good improvement from it has beed changed to RW Lock. And i analyze the log to find the long lock time happen when : !image-2020-03-10-17-31-58-830.png|width=611,height=148! DirectoryScanner scan operation. And other does not cause too much time when very busy: !image-2020-03-10-17-34-26-368.png|width=812,height=136! Such as the deep copy for the caculation of dfsUsed. was (Author: zhuqi): !image-2020-03-10-17-22-57-391.png|width=604,height=137! Hi [~sodonnell] I just gray 3 nodes in busy cluster in different time , the green one is the longest, and there are almost no blocked thread now. The two one has a good improvement from it has beed changed to RW Lock. And i analyze the log to find the long lock time happen when : !image-2020-03-10-17-31-58-830.png|width=611,height=148! DirectoryScanner scan operation. And other does not cause too much time when very busy: !image-2020-03-10-17-34-26-368.png|width=812,height=136! Such as the deep copy for the caculation of dfsUsed. > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool. > ----------------------------------------------------------- > > Key: HDFS-15180 > URL: https://issues.apache.org/jira/browse/HDFS-15180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 3.2.0 > Reporter: zhuqi > Assignee: zhuqi > Priority: Major > Attachments: image-2020-03-10-17-22-57-391.png, > image-2020-03-10-17-31-58-830.png, image-2020-03-10-17-34-26-368.png > > > Now the FsDatasetImpl datasetLock is heavy, when their are many namespaces in > big cluster. If we can split the FsDatasetImpl datasetLock via blockpool. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org