A regionserver today again this error?? any idea??I hope to help.
------------------ ???????? ------------------ ??????: "????-????";<[email protected]>; ????????: 2015??7??12??(??????) ????7:15 ??????: "user"<[email protected]>; ????: ?????? regionserver.HRegion: Failed getting lock in batch put hi,stack: Quite simply,Can you explain why all regionserver has been reported to this waring.Until I restarted the regionserver on the l-hbase1.dba.cn1 to resume. Regionserver on the log is as follows?? 2015-07-10 16:34:35,672 WARN [B.defaultRpcServer.handler=18,queue=0,port=60020] regionserver.HRegion: Failed getting lock in batch put, row=LEDHKTABPNCB4RU004001150711999999 org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for row lock on HRegion atpco:atp_fare,M,1435898852855.e2f9f303a434f3f27dfc839c01c3d9eb., startKey='M', getEndKey()='N', row='LEDHKTABPNCB4RU004001150711999999' at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456) at org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) ------------------ ???????? ------------------ ??????: "Stack";<[email protected]>; ????????: 2015??7??11??(??????) ????0:55 ??????: "Hbase-User"<[email protected]>; ????: Re: regionserver.HRegion: Failed getting lock in batch put On Fri, Jul 10, 2015 at 4:35 AM, ???? <[email protected]> wrote: > hi??all??I use the HBase version is hbase-0.98.6-cdh5.2.0. > > This afternoon,HBase access failure??use command 'hbase hbck' Check data > inconsistencies?? > Hmaster on the log is as follows?? > 2015-07-10 16:24:41,285 INFO > [FifoRpcScheduler.handler1-thread-43]master.HMaster: Client=hadoop// > 192.168.28.12 assign > dama:lowprice_intl_roundtrip_20150311,002687951001426005490000,1426090250918.01b4e069174f6bb786b4e0632f4e4740. > 2015-07-10 16:24:41,285 DEBUG > [FifoRpcScheduler.handler1-thread-43]master.AssignmentManager: Force region > state offline > {01b4e069174f6bb786b4e0632f4e4740 state=OPEN, ts=1436433547853, server= > l-hbase1.dba.cn1.qunar.com,60020,1436433434134} > 2015-07-10 16:24:56,285 INFO > [FifoRpcScheduler.handler1-thread-43]master.AssignmentManager: Server > l-hbase1.dba.cn1.qunar.com,60020,1436433434134 returned > java.net.ConnectException: Connection timed out for > dama:lowprice_intl_roundtrip_20150311,002687951001426005490000,1426090250918.01b4e069174f6bb786b4e0632f4e4740., > try=1 of 10 > java.net.ConnectException: Connection timed out > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) > at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1717) > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:730) > at > org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1705) > at > org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1822) > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1453) > at org.apache.hadoop.hbase.master.HMaster.assignRegion(HMaster.java:2500) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42259) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > at > org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > Regionserver on the log is as follows?? > 2015-07-10 16:34:35,672 > WARN[B.defaultRpcServer.handler=18,queue=0,port=60020]regionserver.HRegion: > Failed getting lock in batch put, row=LEDHKTABPNCB4RU004001150711999999 > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row > out of range for row lock on HRegion > atpco:atp_fare,M,1435898852855.e2f9f303a434f3f27dfc839c01c3d9eb., > startKey='M', getEndKey()='N', row='LEDHKTABPNCB4RU004001150711999999' > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456) > at > org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474) > at > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) > at java.lang.Thread.run(Thread.java:744) > > What's the problem?? You are pasting two seemingly unrelated problems -- master unable to connect to a server (is that server up? did it come back? -- and the client having trouble finding the right region to insert into: did hbck show inconsistencies? St.Ack
