My hbase version is 0.98.6, i will try update client to 0.98.14 and keep server at 0.98.6, Thanks very much!
2015-07-23 15:00 GMT+08:00 Victor Xu <[email protected]>: > Client-side dead loop can cause sending wrong read/write requests to the > region servers, and you've got exactly the same log output as myself when > the the bug happens. However, this only occurs when you are using 0.98.X > version. 1.0 and above do not have this problem. > > On Thu, Jul 23, 2015 at 2:55 PM Louis Hust <[email protected]> wrote: > >> It seems that the HBASE-13896 >> <https://issues.apache.org/jira/browse/HBASE-13896> is client-side dead >> loop, >> but my problem is the regionserver-side dead lock for get row lock, >> >> 2015-07-23 11:23 GMT+08:00 Victor Xu <[email protected]>: >> >>> FYI >>> >>> ---------- Forwarded message --------- >>> From: Victor Xu <[email protected]> >>> Date: Thu, Jul 23, 2015 at 11:22 AM >>> Subject: Re: Dead loop for batch put when get WrongRegionException >>> To: [email protected] <[email protected]> >>> >>> >>> Any chance that this would be your problem? >>> https://issues.apache.org/jira/browse/HBASE-13896 >>> >>> On Thu, Jul 23, 2015 at 11:17 AM Louis Hust <[email protected]> >>> wrote: >>> >>>> Hi ,all >>>> >>>> We are using batch put to insert rows, and sometimes get the following >>>> WARN >>>> in the region server log: >>>> >>>> <log> >>>> 2015-07-23 10:08:49,684 WARN >>>> [B.defaultRpcServer.handler=5,queue=5,port=60020] regionserver.HRegion: >>>> Failed getting lock in batch put, row=BHXYHZFIHHR3ECON101002150723999999 >>>> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row >>>> out of range for row lock on HRegion >>>> atpco:ttf_fare,C,1437145538123.9c2b8cb846b318045f2ad6b5c87fef21., >>>> startKey='C', getEndKey()='D', row='BHXYHZFIHHR3ECON101002150723999999' >>>> at >>>> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588) >>>> at >>>> >>>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477) >>>> at >>>> >>>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) >>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) >>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) >>>> at >>>> >>>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) >>>> </log> >>>> >>>> And the WARN message is logged no-stop. I think the batch put dived into >>>> the dead loop. >>>> >>>> And i look up into the source code, and find the batch put will never >>>> stop >>>> if got WrongRegionException for some row. >>>> >>>> Any body know how to avoid this situation? >>>> >>>> Any idea will be appreciated! >>>> >>> >>
