Eli Collins created HDFS-4054:
---------------------------------
Summary: Increase the invalidation rate based on the pending
deletion count
Key: HDFS-4054
URL: https://issues.apache.org/jira/browse/HDFS-4054
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Eli Collins
Priority: Minor
It is currently possible to run out of free space on a cluster before HDFS
fills up because when deleting a file we adjust the HDFS quota immediately but
don't actually delete the block until the NN issues an invalidation to the DN
(this is what pending deletions tracks). Therefore it's possible for a cluster
with a high ingest and deletion rate to fill up its local disks even though
there should be sufficient space in HDFS. One way to address this would be to
simply wait to adjust the quota until the DNs have confirmed the blocks have
been deleted. However that makes the quota tracking more complicated and
doesn't help this use case. Increasing the invalidation rate to keep up with
the pending deletion count would address this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
