Thanks for writing this note. I'd seen the IOE and presumed it a known
issue in hadoop 0.17. I then went looking for it but couldn't find
one. Turns out its a silly issue in hbase (HBASE-761).
How many regions you have loaded up in your regionserver?
You are using default heap size? You might try upping it a little.
Also make sure you have upped open file descriptors limit too (See #4
and #6 in the FAQ, http://wiki.apache.org/hadoop/Hbase/FAQ).
St.Ack
leith wrote:
looking at our region logs, we've noticed that the compaction thread
constantly runs into exceptions. the entire log is filled with
something like this:
----------------------------------
2008-07-22 12:29:52,759 WARN
org.apache.hadoop.hbase.regionserver.HStore: Exception closing reader
for 242866774/new
java.io.IOException: Stream closed
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.close(DFSClient.java:1319)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at
org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1581)
at org.apache.hadoop.io.MapFile$Reader.close(MapFile.java:577)
at
org.apache.hadoop.hbase.regionserver.HStore.closeCompactionReaders(HStore.java:917)
at
org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:910)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:787)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:887)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:847)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84)
-------------------------------------
the regionserver is taking about up a good amount of memory on our
system, and nothing is happening except for i assume compaction/split
processes.
these only seem to be warnings, but there is so many of them, it would
be nice to get a second opinion on this.
we've also gotten an 'out of memory' exception a few times from the
compaction thread, and those actually ended up killing the thread,
resulting in the region server shutting itself down.
thanks,
/leith
