FYI, a detailed report from a Trend RD team as email thread below.
Also, a weird NPE at HRegionServer.getClosestRowBefore(HRegionServer.java:1758)
during compaction of META.
2009-10-28 09:59:10,653 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region .META.,,1
2009-10-28 09:59:10,687 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
2009-10-28 09:59:10,691 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 0 on 60020, call getClosestRowBefore([...@431d7fb1,
[...@45aa5fe3, [...@347d53d0) from 172.21.12.41:44655: error:
java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1758)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
2009-10-28 09:59:10,697 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region .META.,,1 in 0sec
---- Forwarded Message ----
From: Andrew Purtell (RD-US)
Sent: Thu 10/29/2009 4:44 PM
Subject: RE: jhat (was: RE: HBase 0.20.0 region server crash due to out
of memory error)
Top item on the regionserver which went down due to OOME is byte[] -- class [B
2879025 3938815412 (3.6 GB) -- with top referencer
org.apache.hadoop.hbase.KeyValue 1539539 (about 53% of all references). That is
value data.
There are a lot of ConcurrentSkipListMap$Node objects also. The major user of this
data type if you trace it through in jhat most likely will be KeyValueSkipListSet
-> MemStore.
According to the metrics dumped at OOME time:
> regions=342, stores=1506, storefiles=892, storefileIndexSize=196,
memstoreSize=606, usedHeap=3983, maxHeap=3983,
> blockCacheSize=6853968, blockCacheFree=828520304, blockCacheCount=0,
blockCacheHitRatio=0
606 MB memstore plus 7 MB block cache plus 196 MB storeFileIndex != 3.6 GB of
byte[]. Something does not add up.
(Do disable the block cache as I described in the other email. It is not
helping at all with a hit ratio of 0%.)
The MemStoreScanner line reminds me of a recently filed issue:
> class org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner 77723
2798028
I heard about this issue while traveling but forgot about it. See
https://issues.apache.org/jira/browse/HBASE-1927 . Fixed already on the 0.20
branch and scheduled for 0.20.2 release. Current branch head is available to
test now if this has any impact on the OOME issue. There could be some impact
because MemStoreScanners cache one row of values, which will hold references to
ConcurrentSkipListMap$Node. This is not accounted for in the metrics.
That the problem manifests during compaction with associated messages such as:
> 2009-10-28 09:54:18,365 WARN org.apache.hadoop.hbase.regionserver.HLog:
> IPC Server handler 3 on 60020 took 4786ms appending an edit to hlog;
editcount=6175
> 2009-10-28 09:55:29,502 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
24610ms,
> ten times longer than scheduled: 1000
indicating very slow HDFS (~5 seconds per write) and thread starvation also
suggests MemStore flushing may have fallen behind. Combine long delayed
flushing with KeyValue heap load on the read side (scanners), and this may
explain all of the heap usage, simply insufficient RAM/heap to contain all of
the workload. Scanner leak may be a contributing factor. Snapshots for flushing
are not accounted for in the metrics either.
Let's try to sort out what may be memory leak related and what may be just
overloading. Will you consider testing the latest 0.20 branch revision in HBase
project SVN?
I will forward this to hbase-dev@ with your email addresses removed.
Best regards,
- Andy
-----Original Message-----
Sent: Thu 10/29/2009 12:51 PM
To: Andrew Purtell (RD-US)
Subject: RE: jhat (was: RE: HBase 0.20.0 region server crash due to out of
memory error)
Dear Andy,
Here is the top ten classes in heap histogram got from jhat. Please kindly tell
me if you need more information.
Class Instance Count Total Size
class [B 2879025 3938815412
class java.util.concurrent.ConcurrentSkipListMap$Node 1472184 35332416
class org.apache.hadoop.hbase.KeyValue 1539539 24632624
class java.util.concurrent.ConcurrentSkipListMap$Index 734903 17637672
class [J 895 10660592
class [[B 892 10660120
class [Ljava.lang.Object; 82608 8947128
class [I 2992 5467740
class [C 37504 3125808
class org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner 77723
2798028
References by Type of class[B:
class [B [0x2aaba8220738]
Referrers by Type
Class Count
org.apache.hadoop.hbase.KeyValue 1539539
[[B 1330731
org.apache.hadoop.hbase.io.ImmutableBytesWritable 22504
org.apache.hadoop.hdfs.DFSClient$BlockReader 2605
java.util.TreeMap$Entry 1938
java.io.DataInputStream 1790
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream 1784
java.util.concurrent.ConcurrentSkipListMap$Node 1559
org.apache.hadoop.hbase.HColumnDescriptor 1509
org.apache.hadoop.hbase.regionserver.Store 1506
org.apache.hadoop.hbase.HRegionInfo 1032
Best Regards,
Robin Tang
-----Original Message-----
From: Andrew Purtell (RD-US)
Sent: 2009?10?29? 11:56
Subject: jhat (was: RE: HBase 0.20.0 region server crash due to out of memory
error)
Hi Robin,
Do you know how to use jhat to read a heap dump from a regionserver (written
automatically when it OOMEs) and get a histogram of per object heap usage?
Best regards,
- Andy
-----Original Message-----
Sent: 2009?10?28? 19:17
To: Andrew Purtell (RD-US)
Subject: HBase 0.20.0 region server crash due to out of memory error
Importance: High
Dear Andy,
In TSS testing, after processing about 1.5 million urls, two of four region
servers crashed.
69.174.253.200:
2009-10-28 09:52:06,082 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
urls,002007360827102009-0000000000000000037-172.21.12.42-c74ec67ed7f2c42b948ae96a33fd59af,1256636471071
in 15sec
2009-10-28 09:52:06,082 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region result_hash,,1256524756227
2009-10-28 09:52:09,814 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256723488679,
entries=27549, calcsize=63900120, filesize=60961579. New hlog
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256723529786
2009-10-28 09:52:09,814 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256722212390 whose
highest sequence/edit id is 322578992
2009-10-28 09:52:09,820 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256722224590 whose
highest sequence/edit id is 322702111
2009-10-28 09:52:09,826 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256722235891 whose
highest sequence/edit id is 322817821
2009-10-28 09:52:09,833 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256722254773 whose
highest sequence/edit id is 322908772
2009-10-28 09:52:09,842 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/data/hbase/.logs/tss-poc2-3,60020,1256627509381/hlog.dat.1256722320154 whose
highest sequence/edit id is 323030355
2009-10-28 09:54:18,365 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 4786ms appending an edit to hlog; editcount=6175
2009-10-28 09:55:29,502 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
24610ms, ten times longer than scheduled: 1000
2009-10-28 09:55:29,503 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=342, stores=1506, storefiles=892, storefileIndexSize=196,
memstoreSize=606, usedHeap=3983, maxHeap=3983, blockCacheSize=6853968,
blockCacheFree=828520304, blockCacheCount=0, blockCacheHitRatio=0
2009-10-28 09:55:29,503 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop flag in
regionserver/172.21.12.39:60020.compactor
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:76)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:185)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:875)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:738)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:779)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:732)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:104)
2009-10-28 09:55:33,171 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-10-28 09:55:33,172 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020, call get([...@12249bf9,
row=024003550728102009-00000000000000004
74-172.21.12.42-edf985c42366636e09f74f03d945dbbe, maxVersions=1,
timeRange=[0,9223372036854775807), families=ALL) from 172.21.12.42:58762:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1764)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 09:55:33,171 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/172.21.12.39:60020.cacheFlusher exiting
2009-10-28 09:55:33,171 INFO org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/172.21.12.39:60020.logFlusher exiting
69.174.253.201:
2009-10-28 09:59:10,653 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region .META.,,1
2009-10-28 09:59:10,687 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
2009-10-28 09:59:10,691 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call getClosestRowBefore([...@431d7fb1, [...@45aa5fe3,
[...@347d53d0) from 172.21.12.41:44655: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1758)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
2009-10-28 09:59:10,697 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region .META.,,1 in 0sec
2009-10-28 10:04:54,207 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1066294637410862793
lease expired
2009-10-28 10:04:54,209 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 6680747753024842647
lease expired
2009-10-28 10:04:54,209 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-6935590141578792585 lease expired
2009-10-28 10:05:52,261 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -547447698255259149
lease expired
2009-10-28 10:06:47,273 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 2776617092001158844
lease expired
2009-10-28 10:06:51,596 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -547447698255259149
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:06:51,598 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call next(-547447698255259149, 100) from
172.21.12.38:54959: error: org.apache.hadoop.hbase.UnknownScannerException:
Name: -547447698255259149
org.apache.hadoop.hbase.UnknownScannerException: Name: -547447698255259149
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:07:03,997 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -6935590141578792585
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:07:03,998 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call next(-6935590141578792585, 100) from
172.21.12.39:59669:error: org.apache.hadoop.hbase.UnknownScannerException:
Name: -6935590141578792585
org.apache.hadoop.hbase.UnknownScannerException: Name: -6935590141578792585
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:08:31,829 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
result_url,016912590527102009-0000000000000000004-172.21.12.42c9828c1ba938237e64402dcb08e38ae3,1256637802087
2009-10-28 10:08:39,130 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
result_url,016912590527102009-0000000000000000004-172.21.12.42c9828c1ba938237e64402dcb08e38ae3,1256637802087
in 7sec
2009-10-28 10:08:39,130 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
urls,048403360827102009-0000000000000000034-172.21.12.42-24ca13c9888cfa83b50bdbccdbf41d20,1256637340764
2009-10-28 10:08:48,056 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-6964642614101173789 lease expired
2009-10-28 10:08:51,372 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
urls,048403360827102009-0000000000000000034-172.21.12.42-24ca13c9888cfa83b50bdbccdbf41d20,1256637340764
in 12sec
2009-10-28 10:08:51,373 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region result_hash,,1256524756227
2009-10-28 10:09:13,732 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: 6680747753024842647
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:13,736 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020, call next(6680747753024842647, 100) from
172.21.12.37:47309: error: org.apache.hadoop.hbase.UnknownScannerException:
Name: 6680747753024842647
org.apache.hadoop.hbase.UnknownScannerException: Name: 6680747753024842647
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:17,595 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: 1066294637410862793
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:17,597 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call next(1066294637410862793, 100) from
172.21.12.37:47311: error: org.apache.hadoop.hbase.UnknownScannerException:
Name: 1066294637410862793
org.apache.hadoop.hbase.UnknownScannerException: Name: 1066294637410862793
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:43,343 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -6964642614101173789
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:43,344 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call next(-6964642614101173789, 100) from
172.21.12.41:42371:error: org.apache.hadoop.hbase.UnknownScannerException:
Name: -6964642614101173789
org.apache.hadoop.hbase.UnknownScannerException: Name: -6964642614101173789
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:43,448 WARN org.apache.hadoop.hbase.regionserver.Store: Not in
setorg.apache.hadoop.hbase.regionserver.storescan...@50498912
2009-10-28 10:09:43,887 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: 2776617092001158844
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:09:43,888 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call next(2776617092001158844, 100) from
172.21.12.39:59763: error: org.apache.hadoop.hbase.UnknownScannerException:
Name: 2776617092001158844
org.apache.hadoop.hbase.UnknownScannerException: Name: 2776617092001158844
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1911)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:10:01,444 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 1 on 60020 took 3706ms appending an edit to hlog; editcount=68777
2009-10-28 10:11:01,838 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 0 on 60020 took 4219ms appending an edit to hlog; editcount=75191
2009-10-28 10:11:34,202 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 9 on 60020 took 4287ms appending an edit to hlog; editcount=77350
2009-10-28 10:11:43,711 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 4 on 60020 took 2553ms appending an edit to hlog; editcount=77532
2009-10-28 10:11:59,666 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 5 on 60020 took 3756ms appending an edit to hlog; editcount=77740
2009-10-28 10:12:43,878 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
21384ms, ten times longer than scheduled: 1000
2009-10-28 10:12:43,896 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:76)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:103)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1794)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1733)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1927)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:12:43,912 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError, aborting.
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:76)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:103)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1794)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1733)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1927)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-28 10:12:49,264 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=454, stores=1841, storefiles=1075, storefileIndexSize=242,
memstoreSize=497, usedHeap=3961, maxHeap=3983, blockCacheSize=6853968,
blockCacheFree=828520304, blockCacheCount=0, blockCacheHitRatio=0
2009-10-28 10:12:49,267 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020, call next(1793460901757128664, 100) from
172.21.12.40:50800: error: java.io.IOException: java.lang.OutOfMemoryError:
Java heap space
java.io.IOException: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1940)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:76)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:103)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1794
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1733)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1927)
... 5 more
2009-10-28 10:12:49,268 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call put([...@3e2a9413,
[Lorg.apache.hadoop.hbase.client.Put;@5d5f8b47) from 172.21.12.38:57580: error:
java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
