The "Thread-Opr0" the client thread to put data into hbase, it is waiting.
"Thread-Opr0-EventThread" daemon prio=10 tid=0x00002aaafc7a8000 nid=0xe08
waiting on condition [0x000000004383f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab632ae50> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
"Thread-Opr0-SendThread(nd1-rack2-cloud:2181)" daemon prio=10
tid=0x00002aaafc7a6800 nid=0xe07 runnable [0x000000004373e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab6304410> (a sun.nio.ch.Util$1)
- locked <0x00002aaab6304428> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab632abd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
"Thread-Opr0" prio=10 tid=0x00002aab0402a000 nid=0xdf2 in Object.wait()
[0x000000004262d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab04302d0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739)
- locked <0x00002aaab04302d0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy0.getClosestRowBefore(Unknown Source)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:517)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:515)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1000)
at
org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:514)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:645)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:699)
- locked <0x00002aaab6294660> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
at com.bigdata.bench.hbase.HBaseWriter$Operator.operateTo(Unknown
Source)
at com.bigdata.bench.hbase.HBaseWriter$Operator.run(Unknown Source)
On Thu, Jan 27, 2011 at 12:06 AM, Schubert Zhang <[email protected]> wrote:
> Even though cannot put more data into table, I can read the existing data.
>
> And I stop and re-start the HBase, still cannot put more data.
>
> hbase(main):031:0> status 'simple'
> 8 live servers
> nd5-rack2-cloud:60020 1296057544120
> requests=0, regions=32, usedHeap=130, maxHeap=8973
> nd8-rack2-cloud:60020 1296057544350
> requests=0, regions=31, usedHeap=128, maxHeap=8983
> nd2-rack2-cloud:60020 1296057543346
> requests=0, regions=32, usedHeap=130, maxHeap=8973
> nd3-rack2-cloud:60020 1296057544224
> requests=0, regions=32, usedHeap=133, maxHeap=8973
> nd6-rack2-cloud:60020 1296057544482
> requests=0, regions=32, usedHeap=130, maxHeap=8983
> nd9-rack2-cloud:60020 1296057544565
> requests=174, regions=32, usedHeap=180, maxHeap=8983
> nd7-rack2-cloud:60020 1296057544617
> requests=0, regions=32, usedHeap=126, maxHeap=8983
> nd4-rack2-cloud:60020 1296057544138
> requests=0, regions=32, usedHeap=126, maxHeap=8973
> 0 dead servers
> Aggregate load: 174, regions: 255
>
>
> On Wed, Jan 26, 2011 at 11:58 PM, Schubert Zhang <[email protected]>wrote:
>
>> I am using 0.90.0 (8 RS + 1Master)
>> and the HDFS is CDH3b3
>>
>> During the first hours of running, I puts many (tens of millions entites,
>> each about 200 bytes), it worked well.
>>
>> But then, the client cannot put more data.
>>
>> I checked all log files of hbase, no abnormal is found, I will continue to
>> check this issue.
>>
>> It seems related to ZooKeeper......
>>
>
>
