Does the RegionServer OOME opening same file each time it happens? If so, something is up with the file. Move it aside to get your cluster online and then lets try and figure whats in this file that brings on OOME -- a very large record or perhaps a corruption?
St.Ack On Mon, Apr 11, 2011 at 9:45 PM, 陈加俊 <[email protected]> wrote: > I wan to know why openRegion can case the heap OME ,how to calculate the > size of heap space . > > On Tue, Apr 12, 2011 at 10:49 AM, Jean-Daniel Cryans > <[email protected]>wrote: > >> Were they opening the same region by any chance? >> >> On Mon, Apr 11, 2011 at 6:13 PM, 陈加俊 <[email protected]> wrote: >> > There is no big scan,and just norma load. Also strange is when one RS >> exited >> > then another RS exited and others RS like that. >> > >> > On Tue, Apr 12, 2011 at 8:55 AM, Jean-Daniel Cryans <[email protected] >> >wrote: >> > >> >> Ok that looks "fine", did the region server die under heavy load by >> >> any chance? Or was it big scans? Or just normal load? >> >> >> >> J-D >> >> >> >> On Mon, Apr 11, 2011 at 5:50 PM, 陈加俊 <[email protected]> wrote: >> >> > my configuration is follows: >> >> > <property> >> >> > <name>hbase.client.write.buffer</name> >> >> > <value>2097152</value> >> >> > <description> >> >> > 1024*1024*2=2097152 >> >> > </description> >> >> > </property> >> >> > <property> >> >> > <name>hbase.hstore.blockingStoreFiles</name> >> >> > <value>14</value> >> >> > </property> >> >> > <property> >> >> > <name>hbase.hregion.memstore.block.multiplier</name> >> >> > <value>2</value> >> >> > </property> >> >> > <property> >> >> > <name>hbase.regionserver.handler.count</name> >> >> > <value>30</value> >> >> > </property> >> >> > <property> >> >> > <name>zookeeper.session.timeout</name> >> >> > <value>90000</value> >> >> > </property> >> >> > <property> >> >> > <name>hbase.zookeeper.property.tickTime</name> >> >> > <value>9000</value> >> >> > </property> >> >> > <property> >> >> > <name>hbase.hregion.max.filesize</name> >> >> > <value>1073741824</value> >> >> > <description> >> >> > Maximum HStoreFile size. If any one of a column >> families' >> >> > HStoreFiles has grown to exceed this value, >> >> > the hosting HRegion is split in two. Default: 256M. >> >> > 1024*1024*1024=1073741824 (1G) >> >> > </description> >> >> > </property> >> >> > >> >> > On Tue, Apr 12, 2011 at 8:43 AM, Jean-Daniel Cryans < >> [email protected] >> >> > >> >> > wrote: >> >> >> >> >> >> And where will they go? The issue isn't the number of regions per >> say, >> >> >> it's the amount of data being served by that region server. Also I >> >> >> still don't know if that's really your issue or it's a configuration >> >> >> issue (which I have yet to see). >> >> >> >> >> >> J-D >> >> >> >> >> >> On Mon, Apr 11, 2011 at 5:41 PM, 陈加俊 <[email protected]> wrote: >> >> >> > Can I limit the numbers of regions on one RegionServer ? >> >> >> > >> >> >> > On Tue, Apr 12, 2011 at 8:37 AM, Jean-Daniel Cryans >> >> >> > <[email protected]>wrote: >> >> >> > >> >> >> >> It's really a lot yes, but it could also be weird configurations >> or >> >> >> >> too big values. >> >> >> >> >> >> >> >> J-D >> >> >> >> >> >> >> >> On Mon, Apr 11, 2011 at 5:35 PM, 陈加俊 <[email protected]> >> wrote: >> >> >> >> > Is it too many regions ? Is the memory enough ? >> >> >> >> > HBase-0.20.6 >> >> >> >> > >> >> >> >> > 2011-04-12 00:16:31,844 FATAL >> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: >> >> OutOfMemoryError, >> >> >> >> > aborting. >> >> >> >> > java.lang.OutOfMemoryError: Java heap space >> >> >> >> > at >> >> >> >> java.io.BufferedInputStream.<init>(BufferedInputStream.java:178) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1380) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1648) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767) >> >> >> >> > at >> >> java.io.DataInputStream.readFully(DataInputStream.java:178) >> >> >> >> > at >> >> java.io.DataInputStream.readFully(DataInputStream.java:152) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1368) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:848) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:793) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410) >> >> >> >> > at >> >> >> >> org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1566) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1533) >> >> >> >> > at >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1453) >> >> >> >> > at java.lang.Thread.run(Thread.java:619) >> >> >> >> > 2011-04-12 00:16:31,847 INFO >> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of >> >> metrics: >> >> >> >> > request=0.0, regions=1624, stores=3075, storefiles=2526, >> >> >> >> > storefileIndexSize=374, memstoreSize=339, compactionQueueSize=0, >> >> >> >> > usedHeap=3990, maxHeap=3991, blockCacheSize=706107016, >> >> >> >> > blockCacheFree=131010552, blockCacheCount=10631, >> >> >> >> > blockCacheHitRatio=58, >> >> >> >> > fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0 >> >> >> >> > 2011-04-12 00:16:31,848 INFO >> >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: worker >> thread >> >> >> >> > exiting >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Thanks & Best regards >> >> >> >> > jiajun >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Thanks & Best regards >> >> >> > jiajun >> >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > Thanks & Best regards >> >> > jiajun >> >> > >> >> > >> >> >> > >> > >> > >> > -- >> > Thanks & Best regards >> > jiajun >> > >> > > > > -- > Thanks & Best regards > jiajun >
