I see exactly the same thing on one of our clusters, also running HBase
0.98 (not sure of the rest of the version number since I'm not in the
office right now). The non-hdfs disk space slowly fills up and I failed
to locate the actual files using 'du'. I did a lot of googling but
couldn't find any other mentions of the problem at the time.
Mike.
On 24/10/2015 04:08, Otis Gospodnetić wrote:
Hi Ted,
0.98.6-cdh5.3.0
I did actually try to use lsof, but I didn't see anything unusual there.
Is there something specific I should look for? Things owned by hbase user
or hdfs or yarn? Hm, here, I don't really see anything interesting
<snip>
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <[email protected]> wrote:
Which specific release of 0.98 are you using ?
Have you used lsof to see which files were being held onto ?
Thanks
On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić <
[email protected]> wrote:
Hello,
Is/was there a known issue with HBase 0.98 "holding onto" files?
We noticed the used disk space metric going up, up and up and we could
not
stop it with major compaction.
But we noticed that if we restart a RegionServer 2 things happen:
1) its disk usage immediately drops a lot
2) the disk usage of other RegionServers drops some as well
Have a look at this chart:
https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq
At 1:54 we restarted the first RS (blue line)
At 2:03 we restarted the second RS (dark green line)
Is/was this a known HBase 0.98 issue?
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
