Hi Biju
Yes local index
. It all started when 1 box crashed.
When I brought up a new one the error was localized to the new box.
After cluster restart - it's spread to all servers.
I shall attempt to clear system stats and increase meta cache size
Vishnu,
Are you using "local index" on any of the tables? We have seen similar
issues while using "local index".
On Wed, Jun 22, 2016 at 12:25 PM, vishnu rao <[email protected]> wrote:
> the server dies when trying to take the thread dump.
>
> i believe i am experiencing this bug
>
> https://issues.apache.org/jira/browse/PHOENIX-2508
>
> On Wed, Jun 22, 2016 at 5:03 PM, Heng Chen <[email protected]>
> wrote:
>
> > which thread hold the row lock? could you dump the jstack with 'jstack
-l
> > pid' ?
> >
> > 2016-06-22 16:14 GMT+08:00 vishnu rao <[email protected]>:
> >
> > > hi Heng.
> > >
> > > 2016-06-22 08:13:42,256 WARN
> > > [B.defaultRpcServer.handler=32,queue=2,port=16020]
> regionserver.HRegion:
> > > Failed getting lock in batch put,
> > > row=\x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z
> > >
> > > java.io.IOException: Timed out waiting for lock for row:
> > > \x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051)
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944)
> > >
> > > at
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> > >
> > > at
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> > >
> > > at
> > >
> > >
> >
>
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > On Wed, Jun 22, 2016 at 3:50 PM, Heng Chen <[email protected]>
> > > wrote:
> > >
> > > > Could you paste the whole jstack and relates rs log? It seems row
> > write
> > > > lock was occupied by some thread. Need more information to find it.
> > > >
> > > > 2016-06-22 13:48 GMT+08:00 vishnu rao <[email protected]>:
> > > >
> > > > > need some help. this has happened for 2 of my servers
> > > > > -------------
> > > > >
> > > > > *[B.defaultRpcServer.handler=2,queue=2,port=16020]
> > > regionserver.HRegion:
> > > > > Failed getting lock in batch put,
> > > > > row=a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF*
> > > > >
> > > > > *java.io.IOException: Timed out waiting for lock for row:
> > > > > a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF*
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> > > > >
> > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > > > >
> > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> > > > >
> > > > > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > > --
> > > > > with regards,
> > > > > ch Vishnu
> > > > > mash213.wordpress.com
> > > > > doodle-vishnu.blogspot.in
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > with regards,
> > > ch Vishnu
> > > mash213.wordpress.com
> > > doodle-vishnu.blogspot.in
> > >
> >
>
>
>
> --
> with regards,
> ch Vishnu
> mash213.wordpress.com
> doodle-vishnu.blogspot.in
>
