Hi all,

We've been having issues for a few days with HBase region servers crashing
when under load from mapreduce jobs.

There are a few different errors in the region server logs - I've attached a
sample log of 4 different region servers crashing within an hour of each
other.

Some details:
- This happens when a full table scan from a mapreduce is in progress.
- We are running HBase 0.20.3, with a 16-slave cluster, on EC2.
- Some of the region server errors are NPEs which look a lot like
https://issues.apache.org/jira/browse/HBASE-2077. I'm not sure if that is
the exact problem or if this issue is fixed in 0.20.5. Is it worth upgrading
to 0.20.5 to fix this?
- Some of the region server errors are scanner lease expired errors:
2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
86246ms, ten times longer than scheduled: 1000
2010-07-12 15:10:03,299 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x229c72b89360001 to sun.nio.ch.selectionkeyi...@7f712b3a
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2010-07-12 15:10:03,299 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
1779060682963568676 lease expired
2010-07-12 15:10:03,406 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: 1779060682963568676
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)

We tried increasing hbase.regionserver.lease.period to 2 minutes but that
didn't seem to make a difference here.

- Our configuration and table size haven't changed significantly in those
days.
- We're running a 3-node Zookeeper cluster collocated on the same machines
as the HBase/Hadoop cluster.
- Based on Ganglia output, it doesn't look like the regionservers (or any of
the machines) are swapping.
- At the time of the crash, it doesn't appear that the network was
overloaded (i.e. we've seen higher network traffic without crashes). So it
doesn't seem that this is a problem communicating with Zookeeper.
- We have "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" enabled, so it
doesn't seem like we should be pausing due to GC too much.

Any thoughts?

Thanks,

- Dmitry
================================================================================
Region server 1
================================================================================
================================================================================
2010-07-12 15:10:03,299 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=326.5849MB (342449072), Free=72.14012MB (75644400), 
Max=398.725MB (418093472), Counts: Blocks=5144, Access=227050, Hit=0, 
Miss=227050, Evictions=341, Evicted=221904, Ratios: Hit Ratio=0.0%, Miss 
Ratio=100.0%, Evicted/Run=650.744873046875
2010-07-12 15:10:03,299 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 8265998820716754316 
lease expired
2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
86246ms, ten times longer than scheduled: 1000
2010-07-12 15:10:03,299 WARN org.apache.zookeeper.ClientCnxn: Exception closing 
session 0x229c72b89360001 to sun.nio.ch.selectionkeyi...@7f712b3a
java.io.IOException: TIMED OUT
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2010-07-12 15:10:03,299 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1779060682963568676 
lease expired
2010-07-12 15:10:03,406 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: 1779060682963568676
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:10:03,406 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 83 on 60020, call next(1779060682963568676, 1000) from 
10.215.186.63:46854: error: org.apache.hadoop.hbase.UnknownScannerException: 
Name: 1779060682963568676
org.apache.hadoop.hbase.UnknownScannerException: Name: 1779060682963568676
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:10:03,406 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: 
Disconnected, type: None, path: null
2010-07-12 15:10:03,406 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP
2010-07-12 15:10:03,406 DEBUG 
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region 
tid_sle,/qL2w3VUXm32CICo+O51Xg==_20091206114011_G6N5nvSf6+WH3cPkZuTo9A==,1275828722336
2010-07-12 15:10:03,466 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Closing 
tid_sle,/qL2w3VUXm32CICo\x2BO51Xg==_20091206114011_G6N5nvSf6\x2BWH3cPkZuTo9A==,1275828722336:
 compactions & flushes dis


================================================================================
Region Server 2
================================================================================
================================================================================

2010-07-12 15:50:15,598 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction completed. Freed 41826272 bytes.  Priority Sizes: 
Single=186.45094MB (195507984), Multi=149.51476MB (156777584),Memory=0.0MB (0)
2010-07-12 15:50:15,820 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
        at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:594)
        at java.util.PriorityQueue.siftUp(PriorityQueue.java:572)
        at java.util.PriorityQueue.offer(PriorityQueue.java:274)
        at java.util.PriorityQueue.add(PriorityQueue.java:251)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:111)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1807)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1771)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:50:15,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 11 on 60020, call next(7992420212601154337, 1000) from 
10.214.183.143:49532: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:872)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:862)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1918)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
        at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:594)
        at java.util.PriorityQueue.siftUp(PriorityQueue.java:572)
        at java.util.PriorityQueue.offer(PriorityQueue.java:274)
        at java.util.PriorityQueue.add(PriorityQueue.java:251)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:111)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1807)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1771)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
        ... 5 more
2010-07-12 15:50:25,129 INFO 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: 
regionserver/10.192.81.155:60020.cacheFlusher exiting
2010-07-12 15:50:25,129 INFO org.apache.hadoop.hbase.regionserver.LogFlusher: 
regionserver/10.192.81.155:60020.logFlusher exiting
2010-07-12 15:50:25,129 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2010-07-12 15:50:25,129 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction started.  Attempting to free 41821848 bytes
2010-07-12 15:50:25,129 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1924)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:50:25,129 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1880)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)  
      at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:50:25,181 INFO org.apache.hadoop.hbase.regionserver.LogRoller: 
LogRoller exiting.
2010-07-12 15:50:25,181 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction completed. Freed 41832000 bytes.  Priority Sizes: 
Single=186.3927MB (195446912), Multi=149.51476MB (1567775
84),Memory=0.0MB (0)
2010-07-12 15:50:25,181 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server 
on 60020
2010-07-12 15:50:25,181 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 15 on 60020, call close(7992420212601154337) from 10.214.183.143:49532: 
error: java.io.IOException: Server not running
java.io.IOException: Server not running
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1924)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:50:25,181 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC 
Server Responder
2010-07-12 15:50:25,181 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer


================================================================================
Region Server 3
================================================================================
================================================================================

2010-07-12 15:50:29,942 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
Responder, call getRegionInfo([...@2b3f6979) from 10.192.81.155:44186: output 
error
2010-07-12 15:50:29,995 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 17 on 60020 caught: java.nio.channels.ClosedChannelException
        at 
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)

2010-07-12 15:50:30,046 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction started.  Attempting to free 42322520 bytes
2010-07-12 15:50:30,046 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction completed. Freed 42335552 bytes.  Priority Sizes: 
Single=184.72813MB (193701488), Multi=151.59412MB (158957960),Memory=0.0MB (0)
2010-07-12 15:50:30,098 INFO org.apache.hadoop.io.compress.CodecPool: Got 
brand-new decompressor
2010-07-12 15:50:30,098 INFO org.apache.hadoop.io.compress.CodecPool: Got 
brand-new decompressor
2010-07-12 15:50:40,397 INFO org.apache.hadoop.io.compress.CodecPool: Got 
brand-new decompressor
2010-07-12 15:50:40,397 INFO org.apache.hadoop.io.compress.CodecPool: Got 
brand-new decompressor
2010-07-12 15:50:40,617 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction started.  Attempting to free 41834608 bytes
2010-07-12 15:50:40,617 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction completed. Freed 41843904 bytes.  Priority Sizes: 
Single=181.24695MB (190051192), Multi=154.42139MB (161922560),Memory=0.0MB (0)
2010-07-12 15:51:01,682 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
10548ms, ten times longer than scheduled: 1000
2010-07-12 15:51:11,683 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=327.32544MB (343225592), Free=71.399574MB (74867880), 
Max=398.725MB (418093472), Counts: Blocks=5164, Access=1844993, Hit=177832, 
Miss=1667161, Evictions=2347, Evicted=1519107, Ratios: Hit 
Ratio=9.638627618551254%, Miss Ratio=90.36137461662292%, 
Evicted/Run=647.2548217773438
2010-07-12 15:51:22,196 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 7221146436600755895 
lease expired
2010-07-12 15:51:22,196 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
10460ms, ten times longer than scheduled: 1000
2010-07-12 15:51:22,196 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction started.  Attempting to free 42713240 bytes
2010-07-12 15:51:22,248 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Block cache LRU eviction completed. Freed 42771000 bytes.  Priority Sizes: 
Single=179.195MB (187899584), Multi=158.44287MB (166139400),Memory=0.0MB (0)
2010-07-12 15:51:22,300 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 5334034567156682857 
lease expired
2010-07-12 15:51:22,300 WARN org.apache.hadoop.hbase.regionserver.Store: Not in 
setorg.apache.hadoop.hbase.regionserver.storescan...@9631824
2010-07-12 15:51:32,355 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 
-8096524385547771427 lease expired
2010-07-12 15:51:42,973 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 
-4577540683658373681 lease expired
2010-07-12 15:51:43,025 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
Responder, call next(-4819670754969059742, 1000) from 10.198.78.63:43367: 
output error
2010-07-12 15:51:43,025 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 43 on 60020 caught: java.nio.channels.ClosedChannelException
        at 
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1164)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)

2010-07-12 15:51:43,078 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
        at 
java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:644)
        at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
        at java.util.PriorityQueue.poll(PriorityQueue.java:523)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1807)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1771)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:51:43,078 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
        at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:594)
        at java.util.PriorityQueue.siftUp(PriorityQueue.java:572)
        at java.util.PriorityQueue.offer(PriorityQueue.java:274)
        at java.util.PriorityQueue.add(PriorityQueue.java:251)
        at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:111)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1807)
        at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1771)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 15:51:53,486 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
10408ms, ten times longer than scheduled: 1000
2010-07-12 15:51:53,486 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server 
on 60020
2010-07-12 15:51:53,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 8 on 60020: exiting
2010-07-12 15:51:53,486 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC 
Server Responder
2010-07-12 15:51:53,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 17 on 60020: exiting


================================================================================
Region Server 4
================================================================================
================================================================================

2010-07-12 16:48:48,058 DEBUG org.apache.hadoop.hbase.regionserver.Store: Major 
compaction triggered on store ad; time since last major compaction 103167101ms
2010-07-12 16:48:48,058 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
Started compaction of 3 file(s)  into /hbase/tid_sle/compaction.dir/724953388, 
seqid=1070810368
2010-07-12 16:48:51,040 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain 
block blk_4179738052807336295_836974 from any node:  java.io.IOException: No 
live nodes contain current block
2010-07-12 16:48:53,364 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain 
block blk_4179738052807336295_836974 from any node:  java.io.IOException: No 
live nodes contain current block
2010-07-12 16:48:55,146 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain 
block blk_4179738052807336295_836974 from any node:  java.io.IOException: No 
live nodes contain current block
2010-07-12 16:48:55,146 WARN org.apache.zookeeper.ClientCnxn: Exception closing 
session 0x129c72b89e10000 to sun.nio.ch.selectionkeyi...@3b1d15b4
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 
cap=4]
        at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-07-12 16:52:26,127 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
213231ms, ten times longer than scheduled: 10000
2010-07-12 16:52:26,127 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=320.59167MB (336164752), Free=78.133316MB (81928720), 
Max=398.725MB (418093472), Counts: Blocks=5017, Access=1554779, Hit=69267, 
Miss=1485512, Evictions=1745, Evicted=1125439, Ratios: Hit 
Ratio=4.4551026076078415%, Miss Ratio=95.54489850997925%, 
Evicted/Run=644.9507446289062
2010-07-12 16:52:26,127 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=320.59167MB (336164752), Free=78.133316MB (81928720), 
Max=398.725MB (418093472), Counts: Blocks=5017, Access=1554779, Hit=69267, 
Miss=1485512, Evictions=1745, Evicted=1125439, Ratios: Hit 
Ratio=4.4551026076078415%, Miss Ratio=95.54489850997925%, 
Evicted/Run=644.9507446289062
2010-07-12 16:52:26,127 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=320.59167MB (336164752), Free=78.133316MB (81928720), 
Max=398.725MB (418093472), Counts: Blocks=5017, Access=1554779, Hit=69267, 
Miss=1485512, Evictions=1745, Evicted=1125439, Ratios: Hit 
Ratio=4.4551026076078415%, Miss Ratio=95.54489850997925%, 
Evicted/Run=644.9507446289062
2010-07-12 16:52:26,127 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=320.59167MB (336164752), Free=78.133316MB (81928720), 
Max=398.725MB (418093472), Counts: Blocks=5017, Access=1554779, Hit=69267, 
Miss=1485512, Evictions=1745, Evicted=1125439, Ratios: Hit 
Ratio=4.4551026076078415%, Miss Ratio=95.54489850997925%, 
Evicted/Run=644.9507446289062
2010-07-12 16:52:26,235 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: 
Disconnected, type: None, path: null
2010-07-12 16:52:26,366 INFO org.apache.zookeeper.ClientCnxn: Attempting 
connection to server 
domU-12-31-39-0B-8C-51.compute-1.internal/10.214.147.159:2181
2010-07-12 16:52:26,418 INFO org.apache.zookeeper.ClientCnxn: Priming 
connection to java.nio.channels.SocketChannel[connected 
local=/10.214.183.143:36645 
remote=domU-12-31-39-0B-8C-51.compute-1.internal/10.214.147.159:2181]
2010-07-12 16:52:26,418 INFO org.apache.zookeeper.ClientCnxn: Server connection 
successful
2010-07-12 16:52:26,418 WARN org.apache.zookeeper.ClientCnxn: Exception closing 
session 0x129c72b89e10000 to sun.nio.ch.selectionkeyi...@54736ad
java.io.IOException: Session Expired
        at 
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
        at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-07-12 16:52:26,418 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: 
Expired, type: None, path: null
2010-07-12 16:52:26,418 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired
2010-07-12 16:52:26,418 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: 
request=0.0, regions=119, stores=769, storefiles=1384, storefileIndexSize=232, 
memstoreSize=681, compactionQueue
Size=59, usedHeap=1751, maxHeap=1993, blockCacheSize=335901360, 
blockCacheFree=82192112, blockCacheCount=5013, blockCacheHitRatio=4, 
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
2010-07-12 16:52:26,549 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1880)
        at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)        
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 16:52:26,549 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 56 on 60020, call next(-8627048760844633473, 1000) from 
10.214.183.143:46074: error: java.io.IOException: Server not running, a
borting
java.io.IOException: Server not running, aborting
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1880)
        at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 16:52:26,549 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1924)
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)        
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-12 16:52:26,605 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server 
handler 57 on 60020, call close(-8627048760844633473) from 
10.214.183.143:46074: error: java.io.IOException: Server not running, aborti
ng
java.io.IOException: Server not running, aborting
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2304)
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1924)
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)


Reply via email to