solrry... http://pastebin.com/m5aa78a09
Ananth T Sarathy On Wed, Oct 21, 2009 at 2:41 PM, Ananth T. Sarathy < [email protected]> wrote: > here's what I get > > Heap > par new generation total 19136K, used 13714K [0x00002aaaafc40000, > 0x00002aaab1100000, 0x00002aaab4f70000) > eden space 17024K, 79% used [0x00002aaaafc40000, 0x00002aaab09816d8, > 0x00002aaab0ce0000) > from space 2112K, 6% used [0x00002aaab0ef0000, 0x00002aaab0f13470, > 0x00002aaab1100000) > to space 2112K, 0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000, > 0x00002aaab0ef0000) > concurrent mark-sweep generation total 63872K, used 25097K > [0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000) > concurrent-mark-sweep perm gen total 30220K, used 18263K > [0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000) > > 2009-10-21 14:40:58 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode): > > "pool-1-thread-1" prio=10 tid=0x00002aaafc01a000 nid=0x49f waiting on > condition [0x0000000042e56000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6559490> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > > "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaafc01f800 > nid=0x49e in Object.wait() [0x0000000042d55000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab6519890> (a > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread) > at java.lang.Object.wait(Object.java:485) > at > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512) > - locked <0x00002aaab6519890> (a > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread) > > "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00002aaaf7c12000 > nid=0x49d waiting on condition [0x0000000042c54000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00002aaaf7c0f800 > nid=0x49c waiting on condition [0x0000000042b53000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00002aaaf7c0d800 > nid=0x49b waiting on condition [0x0000000042a52000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00002aaafd08b800 > nid=0x49a waiting on condition [0x0000000042951000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00002aaafd089800 > nid=0x499 waiting on condition [0x0000000042850000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00002aaafd087000 > nid=0x498 waiting on condition [0x000000004274f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00002aaafd085000 > nid=0x497 waiting on condition [0x000000004264e000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00002aaaf7adf800 > nid=0x496 waiting on condition [0x000000004254d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00002aaaf7adb800 > nid=0x495 waiting on condition [0x000000004244c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00002aaaf7adb000 > nid=0x494 waiting on condition [0x000000004234b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434798> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) > > "IPC Server listener on 60020" daemon prio=10 tid=0x00002aaaf7de9800 > nid=0x493 runnable [0x000000004224a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x00002aaab643b548> (a sun.nio.ch.Util$1) > - locked <0x00002aaab643b530> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00002aaab63ea7e0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) > > "IPC Server Responder" daemon prio=10 tid=0x00002aaaf7de9000 nid=0x492 > runnable [0x0000000042149000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x00002aaab643b458> (a sun.nio.ch.Util$1) > - locked <0x00002aaab643b440> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00002aaab63ea768> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) > > "regionserver/10.244.9.171:60020.safeMode" daemon prio=10 > tid=0x00002aaaf7c84000 nid=0x491 in Object.wait() [0x0000000042048000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab63911f8> (a > java.util.concurrent.atomic.AtomicBoolean) > at java.lang.Object.wait(Object.java:485) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionLimitThread.run(HRegionServer.java:902) > - locked <0x00002aaab63911f8> (a > java.util.concurrent.atomic.AtomicBoolean) > > "Timer-0" daemon prio=10 tid=0x00002aaaf7de4000 nid=0x490 in Object.wait() > [0x0000000041f47000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab6558e60> (a java.util.TaskQueue) > at java.util.TimerThread.mainLoop(Timer.java:509) > - locked <0x00002aaab6558e60> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:462) > > "853527...@qtp0-1 - Acceptor0 [email protected]:60030" > prio=10 tid=0x00002aaafcd3a800 nid=0x48f runnable [0x0000000041e46000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x00002aaab651bf38> (a sun.nio.ch.Util$1) > - locked <0x00002aaab651bf50> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00002aaab6620c60> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) > at > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) > at > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > at > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) > > "1096304...@qtp0-0" prio=10 tid=0x00002aaaf7d5b800 nid=0x48e in > Object.wait() [0x0000000041d45000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab653acd0> (a > org.mortbay.thread.QueuedThreadPool$PoolThread) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) > - locked <0x00002aaab653acd0> (a > org.mortbay.thread.QueuedThreadPool$PoolThread) > > "regionserver/10.244.9.171:60020.leaseChecker" prio=10 > tid=0x00002aaafc16a000 nid=0x48d waiting on condition [0x0000000041c44000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6434518> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) > at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209) > at org.apache.hadoop.hbase.Leases.run(Leases.java:81) > > "regionserver/10.244.9.171:60020.majorCompactionChecker" daemon prio=10 > tid=0x00002aaafc168000 nid=0x48c waiting on condition [0x0000000041b43000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) > at org.apache.hadoop.hbase.Chore.run(Chore.java:76) > > "regionserver/10.244.9.171:60020.worker" daemon prio=10 > tid=0x00002aaafc228000 nid=0x48b waiting on condition [0x0000000041a42000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab63a3b68> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1459) > at java.lang.Thread.run(Thread.java:619) > > "regionserver/10.244.9.171:60020.compactor" daemon prio=10 > tid=0x00002aaafc226000 nid=0x48a waiting on condition [0x0000000041941000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:79) > > "regionserver/10.244.9.171:60020.cacheFlusher" daemon prio=10 > tid=0x00002aaafc225000 nid=0x489 waiting on condition [0x0000000041840000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:137) > > "regionserver/10.244.9.171:60020.logFlusher" daemon prio=10 > tid=0x00002aaafc212000 nid=0x488 waiting on condition [0x000000004173f000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) > at org.apache.hadoop.hbase.Chore.run(Chore.java:76) > > "regionserver/10.244.9.171:60020.logRoller" daemon prio=10 > tid=0x00002aaafc211800 nid=0x487 in Object.wait() [0x000000004153d000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab63913f0> (a > java.util.concurrent.atomic.AtomicBoolean) > at > org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65) > - locked <0x00002aaab63913f0> (a > java.util.concurrent.atomic.AtomicBoolean) > > "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 > tid=0x00002aaaf7e14000 nid=0x486 in Object.wait() [0x000000004163e000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab65446e0> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > - locked <0x00002aaab65446e0> (a java.lang.ref.ReferenceQueue$Lock) > at > org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082) > > "IPC Client (47) connection to /10.244.9.171:60000 from an unknown user" > daemon prio=10 tid=0x00002aaaf80b6800 nid=0x46d in Object.wait() > [0x000000004143c000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab642ba08> (a > org.apache.hadoop.hbase.ipc.HBaseClient$Connection) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:405) > - locked <0x00002aaab642ba08> (a > org.apache.hadoop.hbase.ipc.HBaseClient$Connection) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448) > > "DestroyJavaVM" prio=10 tid=0x00002aaaf80b4800 nid=0x455 waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "regionserver/10.244.9.171:60020" prio=10 tid=0x00002aaaf80b3800 nid=0x467 > waiting on condition [0x000000004133b000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:596) > at java.lang.Thread.run(Thread.java:619) > > "main-EventThread" daemon prio=10 tid=0x00002aaaf80ab800 nid=0x466 waiting > on condition [0x000000004123a000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab6391598> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:376) > > "main-SendThread" daemon prio=10 tid=0x00002aaaf80ab000 nid=0x465 runnable > [0x0000000041139000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x00002aaab6418908> (a sun.nio.ch.Util$1) > - locked <0x00002aaab64188f0> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00002aaab63fb0a0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:873) > > "Low Memory Detector" daemon prio=10 tid=0x00002aaaf7858800 nid=0x463 > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread1" daemon prio=10 tid=0x00002aaaf7856000 nid=0x462 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread0" daemon prio=10 tid=0x00002aaaf7852800 nid=0x461 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Signal Dispatcher" daemon prio=10 tid=0x00002aaaf76b9800 nid=0x460 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaaf76b7800 > nid=0x45f waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Finalizer" daemon prio=10 tid=0x00002aaaf7695800 nid=0x45e in > Object.wait() [0x0000000040a32000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab6396438> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > - locked <0x00002aaab6396438> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > "Reference Handler" daemon prio=10 tid=0x00002aaaf7693800 nid=0x45c in > Object.wait() [0x0000000040931000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:485) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock) > > "VM Thread" prio=10 tid=0x00002aaaf768d000 nid=0x45b runnable > > "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040119000 > nid=0x456 runnable > > "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x000000004011b000 > nid=0x457 runnable > > "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x000000004011c800 > nid=0x458 runnable > > "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x000000004011e800 > nid=0x459 runnable > > "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaf3d9d800 nid=0x45a > runnable > "VM Periodic Task Thread" prio=10 tid=0x00002aaaf785b000 nid=0x464 waiting > on condition > > JNI global references: 806 > > Heap > par new generation total 19136K, used 14022K [0x00002aaaafc40000, > 0x00002aaab1100000, 0x00002aaab4f70000) > eden space 17024K, 81% used [0x00002aaaafc40000, 0x00002aaab09ce548, > 0x00002aaab0ce0000) > from space 2112K, 6% used [0x00002aaab0ef0000, 0x00002aaab0f13470, > 0x00002aaab1100000) > to space 2112K, 0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000, > 0x00002aaab0ef0000) > concurrent mark-sweep generation total 63872K, used 25097K > [0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000) > concurrent-mark-sweep perm gen total 30220K, used 18263K > [0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000) > > > > Ananth T Sarathy > > > > On Wed, Oct 21, 2009 at 1:24 PM, stack <[email protected]> wrote: > >> If thats all you get, a thread dump on 10.242.71.191:60020 would help ($ >> kill -QUIT RS_PID). Thread dump will be in the RS .out file. Do a few. >> Paste to pastebin. >> Thanks Ananth, >> St.Ack >> >> On Wed, Oct 21, 2009 at 8:05 AM, Ananth T. Sarathy < >> [email protected]> wrote: >> >> > yeah, >> > I just don't understand why getScanner("Column:X") returns the iterator >> > and >> > process them yet getScanner("Column:Y") just spins and spins, yet >> Column:Y >> > is a much denser result. >> > >> > When I load from shell >> > >> > *Version: 0.20.0, r810752, Thu Sep 3 00:06:18 PDT 2009 >> > hbase(main):001:0> count 'GS_Applications' >> > 09/10/21 11:04:48 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode >> > /hbase/root-region-server got 10.244.9.171:60020 >> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$ClientZKWatcher: Got >> > ZooKeeper event, state: SyncConnected, type: None, path: null >> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Found >> ROOT >> > at 10.244.9.171:60020 >> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Cached >> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME => >> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE >> => >> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', >> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION >> => >> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', >> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION >> => >> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', >> > BLOCKCACHE => 'false'}]}} >> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached >> > location address: 10.242.71.191:60020, regioninfo: REGION => {NAME => >> > 'GS_Applications,,1255020109210', STARTKEY => '', ENDKEY => '', ENCODED >> => >> > 1732076772, TABLE => {{NAME => 'GS_Applications', FAMILIES => [{NAME => >> > 'Application', COMPRESSION => 'NONE', VERSIONS => '1', TTL => >> '2147483647', >> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME >> => >> > 'BinaryData', COMPRESSION => 'NONE', VERSIONS => '1', TTL => >> '2147483647', >> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME >> => >> > 'BinaryRetrieval', COMPRESSION => 'NONE', VERSIONS => '1', TTL => >> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >> > 'true'}, {NAME => 'Files', COMPRESSION => 'NONE', VERSIONS => '1', TTL >> => >> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >> > 'true'}, {NAME => 'Info', COMPRESSION => 'NONE', VERSIONS => '1', TTL => >> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >> > 'true'}, {NAME => 'Network', COMPRESSION => 'NONE', VERSIONS => '1', TTL >> => >> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >> > 'true'}, {NAME => 'Registry', VERSIONS => '1', COMPRESSION => 'NONE', >> TTL >> > => >> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >> > 'true'}]}} >> > 09/10/21 11:04:49 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode >> > /hbase/root-region-server got 10.244.9.171:60020 >> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Found >> ROOT >> > at 10.244.9.171:60020 >> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached >> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME => >> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE >> => >> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', >> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION >> => >> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', >> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION >> => >> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', >> > BLOCKCACHE => 'false'}]}} >> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Creating scanner >> over >> > GS_Applications starting at key '' >> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Advancing internal >> > scanner to startKey at '' >> > 09/10/21 11:04:50 DEBUG client.HConnectionManager$TableServers: Cache >> hit >> > for row <> in tableName GS_Applications: location server >> > 10.242.71.191:60020, >> > location region name GS_Applications,,1255020109210 >> > >> > >> > *Ananth T Sarathy >> > >> > >> > On Wed, Oct 21, 2009 at 11:00 AM, stack <[email protected]> wrote: >> > >> > > In both cases you are doing a full table scan? >> > > >> > > Try from shell with DEBUG enable. You'll see the regions being >> loaded. >> > > May >> > > help you narrow in on problem region or at least on problem >> regionserver. >> > > >> > > St.Ack >> > > >> > > On Wed, Oct 21, 2009 at 7:19 AM, Ananth T. Sarathy < >> > > [email protected]> wrote: >> > > >> > > > Anyone have any further thoughts on this? >> > > > Ananth T Sarathy >> > > > >> > > > >> > > > On Tue, Oct 20, 2009 at 6:37 PM, Ananth T. Sarathy < >> > > > [email protected]> wrote: >> > > > >> > > > > Well that's not the case. Every Row has that column. In fact the >> > > second >> > > > > snippet i sent is with a column with many less rows. (1k vs 25k) >> but >> > > > comes >> > > > > back pretty quickly. >> > > > > >> > > > > By forever, I mean i have watched my logs do nothing for a half >> hour >> > > > before >> > > > > giving up. >> > > > > >> > > > > >> > > > > Ananth T Sarathy >> > > > > >> > > > > >> > > > > >> > > > > On Tue, Oct 20, 2009 at 5:03 PM, Ryan Rawson <[email protected]> >> > > wrote: >> > > > > >> > > > >> If you are asking for a column that is very sparse and doesnt >> exist, >> > > > >> it will cause HBase to read through the entire region to find 100 >> > > > >> matching rows. This could take a while, you said 'forever', but >> > could >> > > > >> you quantify that? >> > > > >> >> > > > >> On Tue, Oct 20, 2009 at 1:58 PM, Jean-Daniel Cryans < >> > > > [email protected]> >> > > > >> wrote: >> > > > >> > Scanner pre-fetching is always faster, so something must be >> wrong >> > > with >> > > > >> > your region server. Check the logs, top, etc >> > > > >> > >> > > > >> > WRT to row size, it's pretty much a matter of how many bytes >> you >> > > have >> > > > >> > in each column and sum them up (plus some overhead with the >> keys). >> > > > >> > >> > > > >> > You want filters, check the filter package in the javadoc. >> > > > >> > >> > > > >> > J-D >> > > > >> > >> > > > >> > On Tue, Oct 20, 2009 at 1:52 PM, Ananth T. Sarathy >> > > > >> > <[email protected]> wrote: >> > > > >> >> Ok, but how come >> > > > >> >> when I run a similiar call (with less returned rows 1000 vs >> 25k >> > in >> > > > the >> > > > >> >> previous one) it runs through the iterator very quickly? (See >> > > Below) >> > > > >> >> >> > > > >> >> Also, how do I determine the row size? It's just text data, >> and >> > > > really >> > > > >> not >> > > > >> >> much. >> > > > >> >> >> > > > >> >> Finally, is there a way to query for rows that do not have a >> > > column? >> > > > >> (Ie all >> > > > >> >> rows without Files:path1) >> > > > >> >> >> > > > >> >> HBaseTableDataManagerImpl htdmni = new >> > > > >> HBaseTableDataManagerImpl( >> > > > >> >> "GS_Applications"); >> > > > >> >> >> > > > >> >> String[] columns = { "Files:path1" }; >> > > > >> >> log.info("Getting all Rows with Files"); >> > > > >> >> Scanner s = htdmni.getScannerForAllRows(columns); >> > > > >> >> log.info("Got all Rows with Files"); >> > > > >> >> >> > > > >> >> Iterator<RowResult> iter = s.iterator(); >> > > > >> >> out >> > > > >> >> >> > > > >> >> >> > > > >> >> > > > >> > > >> > >> .write("Application_Full_Name,Version,Application_installer_name,Operating >> > > > >> >> System, Application_platform >> > > > >> >> >> > > > >> >> > > > >> > > >> > >> ,Application_sub_category,md5Hash,Sha1Hash,Sha256Hash,filepath,fileName,modified,size,operation\n"); >> > > > >> >> out.write("<BR>"); >> > > > >> >> while (iter.hasNext()) >> > > > >> >> { >> > > > >> >> >> > > > >> >> Ananth T Sarathy >> > > > >> >> >> > > > >> >> >> > > > >> >> On Tue, Oct 20, 2009 at 4:44 PM, Jean-Daniel Cryans < >> > > > >> [email protected]>wrote: >> > > > >> >> >> > > > >> >>> If you have a very slow data source (S3), then it fetches 100 >> > row >> > > > >> >>> before coming back to your client with all of them and that >> can >> > > take >> > > > a >> > > > >> >>> lot of time. Also make sure that 100 of your rows can fit in >> a >> > > > region >> > > > >> >>> server's memory. How big is each row? >> > > > >> >>> >> > > > >> >>> J-D >> > > > >> >>> >> > > > >> >>> On Tue, Oct 20, 2009 at 1:32 PM, Ananth T. Sarathy >> > > > >> >>> <[email protected]> wrote: >> > > > >> >>> > I am running this code where >> > > > >> >>> > >> > > > >> >>> > getScannerForAllRows(columns) just does return >> > > > >> table.getScanner(columns); >> > > > >> >>> > >> > > > >> >>> > and the table has setScannerCaching(100); >> > > > >> >>> > >> > > > >> >>> > But it spins forever after getting the iterator. Why would >> > that >> > > > be? >> > > > >> How >> > > > >> >>> can >> > > > >> >>> > I speed it up? >> > > > >> >>> > >> > > > >> >>> > HBaseTableDataManagerImpl htdmni = new >> > > > >> HBaseTableDataManagerImpl( >> > > > >> >>> > "GS_Applications"); >> > > > >> >>> > >> > > > >> >>> > String[] columns = { "Files:Name" }; >> > > > >> >>> > log.info("Getting all Rows with Files"); >> > > > >> >>> > Scanner s = htdmni.getScannerForAllRows(columns); >> > > > >> >>> > log.info("Got all Rows with Files"); >> > > > >> >>> > log.info("Getting Iterator"); >> > > > >> >>> > >> > > > >> >>> > Iterator<RowResult> iter = s.iterator(); >> > > > >> >>> > log.info("Got Iterator"); >> > > > >> >>> > >> > > > >> >>> > while (iter.hasNext()) >> > > > >> >>> > { >> > > > >> >>> > log.info("Getting next Row"); >> > > > >> >>> > RowResult rr = iter.next(); >> > > > >> >>> > >> > > > >> >>> > >> > > > >> >>> > Ananth T Sarathy >> > > > >> >>> > >> > > > >> >>> >> > > > >> >> >> > > > >> > >> > > > >> >> > > > > >> > > > > >> > > > >> > > >> > >> > >
