Attaching the dumps from the HBase UI :
Debug dump :
55 active threads
Thread 10536 (IPC Client (47) connection to cldx-1139-1033/172.25.6.69:9000
from hduser):
State: TIMED_WAITING
Blocked count: 1
Waited count: 2
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
Thread 10535 (IPC Client (47) connection to cldx-1139-1033/172.25.6.69:60000
from hduser):
State: TIMED_WAITING
Blocked count: 3
Waited count: 4
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
Thread 10121 (1840929932@qtp-1863048490-7):
State: RUNNABLE
Blocked count: 77
Waited count: 415
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
org.mortbay.jetty.Server.handle(Server.java:326)
Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
State: WAITING
Blocked count: 2
Waited count: 4
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
State: WAITING
Blocked count: 2
Waited count: 4
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 251 (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
State: WAITING
Blocked count: 25
Waited count: 31
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
State: TIMED_WAITING
Blocked count: 5247
Waited count: 6944
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
State: TIMED_WAITING
Blocked count: 39
Waited count: 1736
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
State: WAITING
Blocked count: 2
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
State: WAITING
Blocked count: 2
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
State: WAITING
Blocked count: 1
Waited count: 5
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 76
(master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
State: RUNNABLE
Blocked count: 40
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 52060
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 75 (Timer-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 17354
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
[email protected]:60010):
State: RUNNABLE
Blocked count: 92
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 71 (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
State: TIMED_WAITING
Blocked count: 8718
Waited count: 17353
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
State: TIMED_WAITING
Blocked count: 8934
Waited count: 17353
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
State: TIMED_WAITING
Blocked count: 0
Waited count: 52061
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 45 (LeaseChecker):
State: TIMED_WAITING
Blocked count: 1
Waited count: 520519
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
java.lang.Thread.run(Thread.java:722)
Thread 43 (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 520520
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.Chore.run(Chore.java:75)
java.lang.Thread.run(Thread.java:722)
Thread 9 (master-cldx-1139-1033,60000,1365517046931):
State: TIMED_WAITING
Blocked count: 322
Waited count: 520958
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
java.lang.Thread.run(Thread.java:722)
Thread 40 (Timer thread for monitoring jvm):
State: TIMED_WAITING
Blocked count: 0
Waited count: 52062
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 39 (Timer thread for monitoring hbase):
State: TIMED_WAITING
Blocked count: 0
Waited count: 52062
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 38 (REPL IPC Server handler 2 on 60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 37 (REPL IPC Server handler 1 on 60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 36 (REPL IPC Server handler 0 on 60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 35 (IPC Server handler 9 on 60000):
State: WAITING
Blocked count: 33
Waited count: 17376
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 34 (IPC Server handler 8 on 60000):
State: WAITING
Blocked count: 0
Waited count: 17343
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 33 (IPC Server handler 7 on 60000):
State: WAITING
Blocked count: 50
Waited count: 17393
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 32 (IPC Server handler 6 on 60000):
State: WAITING
Blocked count: 13
Waited count: 17355
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 31 (IPC Server handler 5 on 60000):
State: WAITING
Blocked count: 27
Waited count: 17368
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 30 (IPC Server handler 4 on 60000):
State: WAITING
Blocked count: 26
Waited count: 17370
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 29 (IPC Server handler 3 on 60000):
State: WAITING
Blocked count: 13
Waited count: 17357
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 28 (IPC Server handler 2 on 60000):
State: WAITING
Blocked count: 57
Waited count: 17401
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 27 (IPC Server handler 1 on 60000):
State: WAITING
Blocked count: 12
Waited count: 17356
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 26 (IPC Server handler 0 on 60000):
State: WAITING
Blocked count: 13
Waited count: 17357
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 10 (IPC Server listener on 60000):
State: RUNNABLE
Blocked count: 17
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
Thread 23 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
Thread 25 (main-EventThread):
State: WAITING
Blocked count: 30
Waited count: 55
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 24 (main-SendThread(cldx-1140-1034:2181)):
State: RUNNABLE
Blocked count: 50
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Thread 21 (Timer thread for monitoring rpc):
State: TIMED_WAITING
Blocked count: 0
Waited count: 52062
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 20 (IPC Reader 9 on port 60000):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 19 (IPC Reader 8 on port 60000):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 18 (IPC Reader 7 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 17 (IPC Reader 6 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 16 (IPC Reader 5 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 15 (IPC Reader 4 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 14 (IPC Reader 3 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 13 (IPC Reader 2 on port 60000):
State: RUNNABLE
Blocked count: 2
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 12 (IPC Reader 1 on port 60000):
State: RUNNABLE
Blocked count: 3
Waited count: 2
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 11 (IPC Reader 0 on port 60000):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Thread 5 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 539
Waited count: 540
Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 538
Waited count: 539
Waiting on java.lang.ref.Reference$Lock@6f8a34e6
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread 1 (main):
State: WAITING
Blocked count: 16
Waited count: 17
Waiting on java.lang.Thread@3a246847
Stack:
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1258)
java.lang.Thread.join(Thread.java:1332)
org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
Regards,
Omkar Joshi
Technology Office
Phone : 022-67954364
-----Original Message-----
From: Omkar Joshi [mailto:[email protected]]
Sent: Monday, April 15, 2013 9:12 AM
To: [email protected]
Subject: Unable to connect from windows desktop to HBase
Hi,
I'm trying to connect from a Java client running on a Windows desktop machine
to a remote HBase cluster running in the distributed mode(atop 2-node Hadoop
cluster).
1. On the master(namenode)node :
hduser@cldx-1139-1033:~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
jps
9161 HMaster
4536 SecondaryNameNode
4368 DataNode
4645 JobTracker
8395 Jps
4813 TaskTracker
4179 NameNode
7717 Main
2. On the slave(datanode)node :
hduser@cldx-1140-1034:~$ jps
5677 HRegionServer
5559 HQuorumPeer
2634 TaskTracker
3260 Jps
2496 DataNode
3. I also connected to the shell and created a hbase table and also able
to scan it :
hbase(main):004:0> scan 'CUSTOMERS'
ROW COLUMN+CELL
CUSTID12345 column=CUSTOMER_INFO:EMAIL,
timestamp=1365600369284, [email protected]
CUSTID12345 column=CUSTOMER_INFO:NAME,
timestamp=1365600052104, value=Omkar Joshi
CUSTID614 column=CUSTOMER_INFO:NAME,
timestamp=1365601350972, value=Prachi Shah
2 row(s) in 0.8760 seconds
4. The hbase-site.xml has the following configurations:
<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>cldx-1140-1034</value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
<description>Property from ZooKeeper's config zoo.cfg.
The directory where the snapshot is stored.
</description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://cldx-1139-1033:9000/hbase</value>
<description>The directory shared by RegionServers.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are false:
standalone and pseudo-distributed setups with managed Zookeeper true:
fully-distributed with unmanaged Zookeeper Quorum (see
hbase-env.sh)</description>
</property>
</configuration>
5. The Hadoop's core-site.xml has the following settings :
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://cldx-1139-1033:9000</value>
</property>
</configuration>
6. My java client class is :
package client.hbase;
import java.io.IOException;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.RegexStringComparator;
import org.apache.hadoop.hbase.filter.ValueFilter;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseCRUD {
private static Configuration config;
private static HBaseAdmin hbaseAdmin;
private static HTablePool hTablePool;
static {
config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "172.25.6.69");
hTablePool = new HTablePool(config, 2);
}
/**
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
HBaseCRUD hbaseCRUD = new HBaseCRUD();
/*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
hbaseCRUD.populateTableData("CUSTOMERS");*/
hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
}
private void createTables(String tableName, String... columnFamilyNames)
throws IOException {
// TODO Auto-generated method stub
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
if (!(columnFamilyNames == null || columnFamilyNames.length == 0)) {
for (String columnFamilyName : columnFamilyNames) {
HColumnDescriptor columnFamily = new HColumnDescriptor(
columnFamilyName);
tableDesc.addFamily(columnFamily);
}
}
hbaseAdmin.createTable(tableDesc);
}
private void populateTableData(String tableName) throws IOException {
HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));
List<Put> tblRows = getTableData(tableName);
tbl.close();
}
private List<Put> getTableData(String tableName) {
// TODO Auto-generated method stub
if (tableName == null || tableName.isEmpty())
return null;
/* Pull data from wherever located */
if (tableName.equalsIgnoreCase("CUSTOMERS")) {
/*
* Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),
* Bytes.toBytes("NAME"), value)
*/
}
return null;
}
private void scanTable(String tableName, String columnFamilyName,
String... columnNames) throws IOException {
System.out.println("In HBaseCRUD.scanTable(...)");
Scan scan = new Scan();
if (!(columnNames == null || columnNames.length <= 0)) {
for (String columnName : columnNames) {
scan.addColumn(Bytes.toBytes(columnFamilyName),
Bytes.toBytes(columnName));
}
Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL,
new RegexStringComparator("lntinfotech"));
scan.setFilter(filter);
}
HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));
ResultScanner scanResults = tbl.getScanner(scan);
for (Result result : scanResults) {
System.out.println("The result is " + result);
}
tbl.close();
}
}
7. The exception I get is :
In HBaseCRUD.scanTable(...)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Apr 10, 2013 4:24:54 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
<init>
INFO: The identifier of this process is 3648@INFVA03351
Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
retryOrThrow
WARNING: Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase/hbaseid
Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
sleepUntilNextRetry
INFO: Sleeping 2000ms before retry #1...
Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
retryOrThrow
WARNING: Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase/hbaseid
Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
sleepUntilNextRetry
INFO: Sleeping 4000ms before retry #2...
I have also made the master's entry in my local machine's host file. What can
be the error?
Thanks and regards !
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or
privileged information for the intended recipient(s). Unintended recipients are
prohibited from taking action on the basis of information in this e-mail and
using or disseminating the information, and must notify the sender and delete
it from their system. L&T Infotech will not accept responsibility or liability
for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"
