Region Server not Exiting after failing
---------------------------------------
Key: HBASE-617
URL: https://issues.apache.org/jira/browse/HBASE-617
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.1.2
Reporter: Daniel Leffel
Priority: Minor
If a region server was overloaded, the process never exists. Also, this causes
an hbase shutdown to wait indefinitely (as called from the shell script).
This is the thread dump from the regionserver:
--------------------------------------------
Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_14-b03 mixed mode):
"IPC Server handler 9 on 60020" daemon prio=1 tid=0x00002aab1c29eb40 nid=0x45db
waiting on condition [0x0000000042835000..0x0000000042835c80]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:219)
at org.apache.hadoop.hbase.HLog.append(HLog.java:390)
- locked <0x00002aaab70634d0> (a java.lang.Integer)
at org.apache.hadoop.hbase.HRegion.update(HRegion.java:1602)
at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1405)
at
org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1544)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
"regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher" daemon prio=1
tid=0x00002aab1c1cb800 nid=0x45c8 waiting for monitor entry
[0x0000000041522000..0x0000000041522b00]
at org.apache.hadoop.hbase.HLog.completeCacheFlush(HLog.java:481)
- waiting to lock <0x00002aaab70634d0> (a java.lang.Integer)
at org.apache.hadoop.hbase.HRegion.internalFlushcache(HRegion.java:1103)
at org.apache.hadoop.hbase.HRegion.flushcache(HRegion.java:998)
at
org.apache.hadoop.hbase.HRegionServer$Flusher.flushRegion(HRegionServer.java:447)
- locked <0x00002aaab70681f8> (a java.util.HashSet)
at
org.apache.hadoop.hbase.HRegionServer$Flusher.run(HRegionServer.java:390)
"regionserver/0:0:0:0:0:0:0:0:60020.logRoller" daemon prio=1
tid=0x00002aab1c1cc2b0 nid=0x45c7 waiting on condition
[0x0000000041421000..0x0000000041421a80]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:219)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionServer.java:618)
- locked <0x00002aaab70632b8> (a java.lang.Integer)
"[EMAIL PROTECTED]" daemon prio=1 tid=0x00002aab1c0893c0 nid=0x45c5 waiting on
condition [0x000000004121f000..0x000000004121fd80]
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:605)
at java.lang.Thread.run(Thread.java:595)
"org.apache.hadoop.io.ObjectWritable Connection Culler" daemon prio=1
tid=0x00002aab1c10ad00 nid=0x45c3 waiting on condition
[0x000000004101d000..0x000000004101dc80]
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:423)
"org.apache.hadoop.hbase.io.HbaseObjectWritable Connection Culler" daemon
prio=1 tid=0x000000005093f200 nid=0x45b5 waiting on condition
[0x0000000040e1b000..0x0000000040e1bc80]
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:423)
"DestroyJavaVM" prio=1 tid=0x00002aab1c00d930 nid=0x458f waiting on condition
[0x0000000000000000..0x00007fff1680e7f0]
"regionserver/0:0:0:0:0:0:0:0:60020" prio=1 tid=0x00002aab1c00dfe0 nid=0x45b4
waiting for monitor entry [0x0000000040d1a000..0x0000000040d1ac00]
at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:867)
- waiting to lock <0x00002aaab70632b8> (a java.lang.Integer)
at java.lang.Thread.run(Thread.java:595)
"Low Memory Detector" daemon prio=1 tid=0x00000000505e9c90 nid=0x45b2 runnable
[0x0000000000000000..0x0000000000000000]
"CompilerThread1" daemon prio=1 tid=0x00000000505e8060 nid=0x45b1 waiting on
condition [0x0000000000000000..0x0000000040a16360]
"CompilerThread0" daemon prio=1 tid=0x00000000505e6ab0 nid=0x45b0 waiting on
condition [0x0000000000000000..0x00000000409156e0]
"AdapterThread" daemon prio=1 tid=0x00000000505e5350 nid=0x45af waiting on
condition [0x0000000000000000..0x0000000000000000]
"Signal Dispatcher" daemon prio=1 tid=0x00000000505e4070 nid=0x45ae runnable
[0x0000000000000000..0x0000000000000000]
"Finalizer" daemon prio=1 tid=0x00000000505cfef0 nid=0x45ad in Object.wait()
[0x0000000040613000..0x0000000040613c80]
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab70bc3d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked <0x00002aaab70bc3d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x00000000505cf6c0 nid=0x45ac in
Object.wait() [0x0000000040512000..0x0000000040512c00]
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab70b3ac8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab70b3ac8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=1 tid=0x00000000505cb1d0 nid=0x45ab runnable
"GC task thread#0 (ParallelGC)" prio=1 tid=0x000000005051e4d0 nid=0x45a9
runnable
"GC task thread#1 (ParallelGC)" prio=1 tid=0x000000005051f350 nid=0x45aa
runnable
"VM Periodic Task Thread" prio=1 tid=0x0000000050515430 nid=0x45b3 waiting on
condition
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
