On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze <marnan...@gmail.com> wrote: > Hi, > I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM > allocated to hbase region server. Basically, doing a bulk import processing > large files,
How large? Unless very large, it should not be OOMEing. but some imports require to do gets and scans as well. In the master UI I see that the heap used gets very close to the 6GB limit, but I know hbase is eager for memory and will use the heap as much as possible.I use block caching. Looking at similar posts I see that modifying the handler count and memory store upper/ower limits may be key to solving this issue. Nevertheless I wanted to ask if there is a way to estimate the extra memory used by hbase that makes it crash and if there are other configuration settings I should be looking into to prevent OOME. The job runs correctly for some time but region servers eventually crash. > > More information about the cluster: > > - All nodes have 16GM total memory. > - 7 nodes running region server (6GB) + datanodes (1GB) + task trackers (1GB > Heap). Map reduce jobs running w/ 756MB tops each. Good. How many MR child tasks can run on each node concurrently? > - 1 node running hbase master (2GB Heap allocated), namenode (4GB), Secondary > Namenode (4GB), JobTracker (4GB) and Master (2GB). > - 3 of the nodes have zookeeper running with 512MB Heap > > Many thanks, > Martin > Can we see the lines before the below is thrown? Also, do a listing (ls -r) on this region in hdfs and lets see if anything pops out about files sizes, etc. You'll need to manually map the below region name to its encoded name to figure the region but the encoded name should be earlier in the log. You'll do something like: bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME Thanks, St.Ack > > > 2010-08-26 07:19:14,859 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening > table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.readIndex(HFile.java:1538) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:806) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273) > at > org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129) > at > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410) > at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1571) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1538) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1458) > at java.lang.Thread.run(Thread.java:619) >
