[ https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865274#comment-16865274 ]
Lisheng Sun commented on HDFS-14313: ------------------------------------ [~jojochuang] Thanks for your review. {quote}FsDatasetImpl#deepCopyReplica() needs to be synchronized on datasetLock. Additionally I am not sure the impact of creating creating millions of ReplicaInfo for dense DataNodes. That sounds quite heavy. {quote} I consider this point when developing the feature. And do test. ||Count(ReplicaInfo) of DataNode ||Time of synchronized on datasetLock|| |1039864|314ms| |343866|89ms| We get same result hundred milliseconds of millions ReplicaInfos in production environment. The feature has been running for a long time for XiaoMi HBase service and does not affect their performance. Thanks. > Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory > instead of df/du > ---------------------------------------------------------------------------------------- > > Key: HDFS-14313 > URL: https://issues.apache.org/jira/browse/HDFS-14313 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, performance > Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0 > Reporter: Lisheng Sun > Assignee: Lisheng Sun > Priority: Major > Attachments: HDFS-14313.000.patch, HDFS-14313.001.patch > > > There are two ways of DU/DF getting used space that are insufficient. > # Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. > # Running DF is inaccurate when the disk sharing by multiple datanode or > other servers. > Getting hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfos in memory > is very small and accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org