HDFS now notifies the client when the block file has been deleted. See HDFS-6750 for details.
Best, Colin On Jun 26, 2014 11:39 AM, "Giuseppe Reina" <[email protected]> wrote: > On Thu, Jun 26, 2014 at 1:02 AM, Enis Söztutar <[email protected]> wrote: > > > Agreed, this seems like an hdfs issue unless hbase itself does not close > > the hfiles properly. But judging from the fact that you were able to > > circumvent the problem by getting reducing the cache size, it does seem > > unlikely. > > > > > Well, we had the same issue with two different HDP 2.1 installations (with > default parameters) and two different client applications on top of HBase. > I agree this is an HDFS issue, I posted on the HBase mailing list because I > was wondering if anyone else had the same problem. Thanks for forwarding > it. > > In both applications we are injesting lots of data on one HBase table with > one column family and about 10 regions for each RS. > So the RegionServer does a lot of flushes, merges and split which involves > lots of HDFS block creations and deletions. > > What was your dfs.client.read.shortcircuit.streams.cache.size ? > > > The default value is 4096 on HDP 2.1 (not 256 as in the vanilla > distribution). > When the shortcircuit cache is too big (like in this case), the deleted > files remain open until the cache is full. This is bad when big WAL/HFiles > are merged or split as the space on disk is still occupied and the file is > never accessed (so it's a leak). Of course any application that perform > file creation, read, and deletion of big hdfs files will be affected by the > same problem. > > > > > > I don't think the local block reader will be notified when a file/block > is > > deleted. > > > > That's what I thought too, but I would be easy for the HDFS client library > to periodically scan through all the open file descriptors verifying that > the file has not been deleted (by "stat"-ing the filename for example). It > is useless to keep it in memory as the file will never be accessed, and if > it will this could potentially lead to inconsistent states. > > > > Giuseppe > > > > > > > > Enis > > > > > > On Wed, Jun 25, 2014 at 3:29 PM, Andrew Purtell <[email protected]> > > wrote: > > > > > Forwarded > > > > > > ---------- Forwarded message ---------- > > > From: Vladimir Rodionov <[email protected]> > > > Date: Wed, Jun 25, 2014 at 12:03 PM > > > Subject: RE: Disk space leak when using HBase and HDFS ShortCircuit > > > To: "[email protected]" <[email protected]> > > > > > > > > > >> Apparently those file descriptors were stored by the HDFS > > > >> ShortCircuit cache. > > > > > > As far as I understand this is issue of HDFS shorty-circuit-reads > > > implementation not HBase. HBase uses HDFS API to access > > > files. Did you ask this question on hdfs dev list? This looks like a > very > > > serious bug. > > > > > > Best regards, > > > Vladimir Rodionov > > > Principal Platform Engineer > > > Carrier IQ, www.carrieriq.com > > > e-mail: [email protected] > > > > > > ________________________________________ > > > From: Giuseppe Reina [[email protected]] > > > Sent: Wednesday, June 25, 2014 2:54 AM > > > To: [email protected] > > > Subject: Disk space leak when using HBase and HDFS ShortCircuit > > > > > > Hi all, > > > we have been experiencing the same problem with 2 of our clusters. > We > > > are currently using HDP 2.1 that comes with HBase 0.98. > > > > > > The problem manifested by showing a huge differences (hundreds of GB) > > > between the output of "df" and "du" of the hdfs data directories. > > > Eventually, other systems complained for the lack of space before > > shutting > > > down. We identified the problem and discovered that all the > RegionServers > > > were holding lots of open file descriptors to deleted files, which > > > prevented the OS to free the disk space occupied (hence the difference > > > between "df" and "du"). The deleted files were pointing to the local > HDFS > > > blocks of old HFiles deleted from HDFS during the compaction and/or > split > > > operations. Apparently those file descriptors were stored by the HDFS > > > ShortCircuit cache. > > > > > > My question is, isn't the shortcircuit feautre supposed to get > "notified" > > > somehow of file deletion on a file on HDFS so it can remove the open > fds > > > from the cache? This creates huge leaks whenever HBase is heavily > loaded > > > and we had to restart the RegionServer periodically until before > > > identifying the problem. We solved the problem first by disabling > > > shortcircuit from HDFS and then enabling it and reducing the cache size > > so > > > to trigger often the caching policies (this leads to some performance > > > loss). > > > > > > > > > p.s. I am aware of the " > > > dfs.client.read.shortcircuit.streams.cache.expiry.ms > > > " directoparameter, but for some reason the default value (5 mins) does > > not > > > work out-of-the-box on HDP 2.1, moreover the problem persists for high > > > timeouts and big cache sizes. > > > > > > Kind Regards > > > > > > Confidentiality Notice: The information contained in this message, > > > including any attachments hereto, may be confidential and is intended > to > > be > > > read only by the individual or entity to whom this message is > addressed. > > If > > > the reader of this message is not the intended recipient or an agent or > > > designee of the intended recipient, please note that any review, use, > > > disclosure or distribution of this message or its attachments, in any > > form, > > > is strictly prohibited. If you have received this message in error, > > please > > > immediately notify the sender and/or [email protected] and > > > delete > > > or destroy any copy of this message and its attachments. > > > > > > > > > > > > -- > > > Best regards, > > > > > > - Andy > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > (via Tom White) > > > > > >
