[ 
https://issues.apache.org/jira/browse/PHOENIX-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaanai updated PHOENIX-4915:
----------------------------
    Description: 
The client has got stuck when using the multi-thread writes the same rows data 
into a data table which has a global index.

I find that rows lock of the data table will not be released under highly 
writing load and throwing " ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to 
find cached index metadata." exception information. Most of the threads will be 
waiting for getting the row lock in Jstack information.

The following are exceptions on the server side:
{code:java}
[B.defaultRpcServer.handler=37,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3455 service: ClientService meth
odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
 Index update failed
        at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
        at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
        at 
org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
        at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
        at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
        at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)

2018-09-20 17:35:39,254 INFO  
[B.defaultRpcServer.handler=13,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3848 service: ClientService meth
odName: Multi size: 27.2 K connection: 192.168.199.7:52042 param: 
actionCount=52#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out waiting for 
lock for row: 0e 30 64 32 65 34 35 63 37 2d 63 63 33 64 2d 34 36 61 35 2d 61 34 
38 64 2d 31 38 61 62 36 31 61 31 30 63 30 39
        at 
org.apache.phoenix.hbase.index.LockManager.lockRow(LockManager.java:96)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:425)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
{code}
{code:java}
[B.defaultRpcServer.handler=37,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3455 service: ClientService meth
odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
 Index update failed
        at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
        at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
        at 
org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
        at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
        at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
        at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)
{code}
 

 

The left shows the RPC handler queue was filled with requests which have total 
120 handler size on all RS processes.  The middle and right are normal after 
fixed this bug.

!image-2018-09-21-19-30-12-989.png|width=434,height=225!

  was:
The client has got stuck when using the multi-thread writes the same rows data 
into a data table which has a global index.

I find that rows lock of the data table will not be released under highly 
writing load and throwing " ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to 
find cached index metadata." exception information.  Most of the threads will 
be waiting for getting the row lock in Jstack information. 

The following are exceptions on the server side:

{code:java}
[B.defaultRpcServer.handler=37,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3455 service: ClientService meth
odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
 Index update failed
        at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
        at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
        at 
org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
        at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
        at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
        at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)

2018-09-20 17:35:39,254 INFO  
[B.defaultRpcServer.handler=13,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3848 service: ClientService meth
odName: Multi size: 27.2 K connection: 192.168.199.7:52042 param: 
actionCount=52#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out waiting for 
lock for row: 0e 30 64 32 65 34 35 63 37 2d 63 63 33 64 2d 34 36 61 35 2d 61 34 
38 64 2d 31 38 61 62 36 31 61 31 30 63 30 39
        at 
org.apache.phoenix.hbase.index.LockManager.lockRow(LockManager.java:96)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:425)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
{code}



{code:java}
[B.defaultRpcServer.handler=37,queue=1,port=16020] 
regionserver.RSRpcServices(103): Failed doing multi operation, current call is 
: callId: 3455 service: ClientService meth
odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
 Index update failed
        at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
        at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
        at 
org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
        at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
index metadata.  key=-727998515684050837 
region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
        at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
        at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
        at 
org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)
{code}





 


> The client gets stuck when using same rows concurrently writing data table
> --------------------------------------------------------------------------
>
>                 Key: PHOENIX-4915
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4915
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.12.0, 4.14.0
>            Reporter: Jaanai
>            Assignee: Jaanai
>            Priority: Blocker
>         Attachments: Screen Shot 2018-09-21 at 19.19.28.png, 
> image-2018-09-21-19-30-12-989.png, test.java, test.sql
>
>
> The client has got stuck when using the multi-thread writes the same rows 
> data into a data table which has a global index.
> I find that rows lock of the data table will not be released under highly 
> writing load and throwing " ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to 
> find cached index metadata." exception information. Most of the threads will 
> be waiting for getting the row lock in Jstack information.
> The following are exceptions on the server side:
> {code:java}
> [B.defaultRpcServer.handler=37,queue=1,port=16020] 
> regionserver.RSRpcServices(103): Failed doing multi operation, current call 
> is : callId: 3455 service: ClientService meth
> odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
> actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-727998515684050837 
> region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
> 1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
>  Index update failed
>         at 
> org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
>         at 
> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
>         at 
> org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
>         at 
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
>         at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
>         at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
> Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
> index metadata.  key=-727998515684050837 
> region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
> 6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
>         at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
>         at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)
> 2018-09-20 17:35:39,254 INFO  
> [B.defaultRpcServer.handler=13,queue=1,port=16020] 
> regionserver.RSRpcServices(103): Failed doing multi operation, current call 
> is : callId: 3848 service: ClientService meth
> odName: Multi size: 27.2 K connection: 192.168.199.7:52042 param: 
> actionCount=52#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out waiting for 
> lock for row: 0e 30 64 32 65 34 35 63 37 2d 63 63 33 64 2d 34 36 61 35 2d 61 
> 34 38 64 2d 31 38 61 62 36 31 61 31 30 63 30 39
>         at 
> org.apache.phoenix.hbase.index.LockManager.lockRow(LockManager.java:96)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:425)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
>         at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
>         at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
> {code}
> {code:java}
> [B.defaultRpcServer.handler=37,queue=1,port=16020] 
> regionserver.RSRpcServices(103): Failed doing multi operation, current call 
> is : callId: 3455 service: ClientService meth
> odName: Multi size: 23.1 K connection: 192.168.199.7:52050 param: 
> actionCount=44#regionCount=8#LOCK,\x02,1537434393195.ee6d441a04ee6a59b24262f22f618d88.#####
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-727998515684050837 
> region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f
> 1c7a20c73a2.host=hb-bp1v2q830426r6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
>  Index update failed
>         at 
> org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:88)
>         at 
> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:62)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:87)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:103)
>         at 
> org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:95)
>         at 
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:80)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:528)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:374)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1032)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1746)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1028)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.asyncBatchMutate(HRegion.java:3236)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doAsyncBatchOp(RSRpcServices.java:2147)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchMutationCrossRegions(RSRpcServices.java:2308)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2578)
>         at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
>         at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178)
> Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached 
> index metadata.  key=-727998515684050837 
> region=LOCK,\x0E,1537434393195.f4de29d4b36775589a49f1c7a20c73a2.host=hb-bp1v2q830426r
> 6763-004.hbase.rds.aliyuncs.com,16020,1537434304031
>         at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:493)
>         at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
>         at 
> org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:85)
> {code}
>  
>  
> The left shows the RPC handler queue was filled with requests which have 
> total 120 handler size on all RS processes.  The middle and right are normal 
> after fixed this bug.
> !image-2018-09-21-19-30-12-989.png|width=434,height=225!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to