[ https://issues.apache.org/jira/browse/HDFS-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gang Xie updated HDFS-13378: ---------------------------- Attachment: HDFS-13378_v1.diff > Add the option to set the du reserve by percent on datanode > ----------------------------------------------------------- > > Key: HDFS-13378 > URL: https://issues.apache.org/jira/browse/HDFS-13378 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 3.0.1 > Reporter: Gang Xie > Priority: Major > Attachments: HDFS-13378_v1.diff > > > Recently, we could event we reserved some space(dfs.datanode.du.reserved), > this disk still gets full, in turn, the performance is downgraded. After some > test, we found that we need consider the space used by ext4 (we use ext4 as > local fs). I.E, if the datanode reports there is 8T space available, you > could not write 8T data actually. About %4-%5 space is used for the ext4 > metadata. The following is the metrics of ext4 meta usage in disks with > different capacity (GB, reserved block for root is set to 1%, 120MB blocks): > |*FileSystem Capacity*|*Ext4 Data*|*Ex4 metadata*|*Metadata Ratio*| > |3666|3629|37|0.0101| > |7392|7318|74|0.0100| > |733|725|8|0.0109| > > So, it seems that the ext4 most likely has the same meta data ratio with the > same about data and dir tree structure. > > On the other hand, in our data center, there are several disk types with > different capacity. It's inefficient to set a absolute value for each > datanode according to the disk capacity. To us, it makes sense to set the > reservation with a ratio since we always use ext4 as the local fs. I think > it's the same case for most of other data center. > > Our idea is to leave the default behavior unchanged. User could still use > dfs.datanode.du.reserved to set the reservation, but switch to use a ratio if > some other specified param is set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org