zhihaixu2012 opened a new pull request #3175: URL: https://github.com/apache/hadoop/pull/3175
…avoid failed volumes at datanodes. Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a JIRA: https://issues.apache.org/jira/browse/HDFS-16111 there is a potential bug when picking a disk volume to write a new block file(replica). By default, Hadoop uses RoundRobinVolumeChoosingPolicy, The code to select a disk will check whether the available space on the selected disk is more than the size bytes of block file to store (https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java#L86) But when creating a new block, there will be two files created: one is the block file blk_XXXX, the other is block metadata file blk_XXXX_XXXX.meta, this is the code when finalizing a block, both block file size and meta data file size will be updated: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java#L391 the current code only considers the size of block file and doesn't consider the size of block metadata file, wh en choosing a disk in RoundRobinVolumeChoosingPolicy. There can be a lot of on-going blocks received at the same time, the default maximum number of DataXceiver threads is 4096. This will underestimate the total size needed to write a block, which will potentially cause the disk full error(No space left on device) when writing a replica. Since the size of the block metadata file is not fixed, I suggest to add a configuration(dfs.datanode.round-robin-volume-choosing-policy.additional-available-space) to safeguard the disk space when choosing a volume to write a new block data in RoundRobinVolumeChoosingPolicy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
