why do they hang up?
On Thu, Jan 27, 2011 at 12:25 AM, Schubert Zhang <[email protected]> wrote: > The update: > > If I start 1 or 2 or 4 client threads (each have a HTable instance), normal. > > If I start 8 or more client threads (each have a HTable instance), the put > operations hang-up. > > On Thu, Jan 27, 2011 at 1:19 AM, Schubert Zhang <[email protected]> wrote: > >> The "Thread-Opr0" the client thread to put data into hbase, it is waiting. >> >> "Thread-Opr0-EventThread" daemon prio=10 tid=0x00002aaafc7a8000 nid=0xe08 >> waiting on condition [0x000000004383f000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00002aaab632ae50> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) >> "Thread-Opr0-SendThread(nd1-rack2-cloud:2181)" daemon prio=10 >> tid=0x00002aaafc7a6800 nid=0xe07 runnable [0x000000004373e000] >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked <0x00002aaab6304410> (a sun.nio.ch.Util$1) >> - locked <0x00002aaab6304428> (a >> java.util.Collections$UnmodifiableSet) >> - locked <0x00002aaab632abd0> (a sun.nio.ch.EPollSelectorImpl) >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107) >> >> "Thread-Opr0" prio=10 tid=0x00002aab0402a000 nid=0xdf2 in Object.wait() >> [0x000000004262d000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> - waiting on <0x00002aaab04302d0> (a >> org.apache.hadoop.hbase.ipc.HBaseClient$Call) >> at java.lang.Object.wait(Object.java:485) >> at >> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739) >> - locked <0x00002aaab04302d0> (a >> org.apache.hadoop.hbase.ipc.HBaseClient$Call) >> at >> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >> at $Proxy0.getClosestRowBefore(Unknown Source) >> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:517) >> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:515) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1000) >> at >> org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:514) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:645) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:699) >> - locked <0x00002aaab6294660> (a java.lang.Object) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819) >> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675) >> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660) >> at com.bigdata.bench.hbase.HBaseWriter$Operator.operateTo(Unknown >> Source) >> at com.bigdata.bench.hbase.HBaseWriter$Operator.run(Unknown Source) >> >> On Thu, Jan 27, 2011 at 12:06 AM, Schubert Zhang <[email protected]>wrote: >> >>> Even though cannot put more data into table, I can read the existing data. >>> >>> And I stop and re-start the HBase, still cannot put more data. >>> >>> hbase(main):031:0> status 'simple' >>> 8 live servers >>> nd5-rack2-cloud:60020 1296057544120 >>> requests=0, regions=32, usedHeap=130, maxHeap=8973 >>> nd8-rack2-cloud:60020 1296057544350 >>> requests=0, regions=31, usedHeap=128, maxHeap=8983 >>> nd2-rack2-cloud:60020 1296057543346 >>> requests=0, regions=32, usedHeap=130, maxHeap=8973 >>> nd3-rack2-cloud:60020 1296057544224 >>> requests=0, regions=32, usedHeap=133, maxHeap=8973 >>> nd6-rack2-cloud:60020 1296057544482 >>> requests=0, regions=32, usedHeap=130, maxHeap=8983 >>> nd9-rack2-cloud:60020 1296057544565 >>> requests=174, regions=32, usedHeap=180, maxHeap=8983 >>> nd7-rack2-cloud:60020 1296057544617 >>> requests=0, regions=32, usedHeap=126, maxHeap=8983 >>> nd4-rack2-cloud:60020 1296057544138 >>> requests=0, regions=32, usedHeap=126, maxHeap=8973 >>> 0 dead servers >>> Aggregate load: 174, regions: 255 >>> >>> >>> On Wed, Jan 26, 2011 at 11:58 PM, Schubert Zhang <[email protected]>wrote: >>> >>>> I am using 0.90.0 (8 RS + 1Master) >>>> and the HDFS is CDH3b3 >>>> >>>> During the first hours of running, I puts many (tens of millions entites, >>>> each about 200 bytes), it worked well. >>>> >>>> But then, the client cannot put more data. >>>> >>>> I checked all log files of hbase, no abnormal is found, I will continue >>>> to check this issue. >>>> >>>> It seems related to ZooKeeper...... >>>> >>> >>> >> >
