Ankit Singhal created PHOENIX-6331:
--------------------------------------
Summary: Increase index retry from 1 to 2 incase of
NotServingRegionException
Key: PHOENIX-6331
URL: https://issues.apache.org/jira/browse/PHOENIX-6331
Project: Phoenix
Issue Type: Bug
Reporter: Ankit Singhal
Currently, we move the index to PENDING_DISABLE whenever the single write to
index failed and carry out a retry at the client but this can be optimized for
NotServingRegionException as Index regions can move very frequently depending
on the balancer and one more retry at the server could avoid unnecessary
handling of index states and retries at the client.
{code:java}
2021-01-20 06:54:58,682 WARN org.apache.hadoop.hbase.client.AsyncProcess: #277,
table=<index_table>, attempt=1/1 failed=1ops, last exception:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region <region> is not
online on <region_server>
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2997)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1069)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2100)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
on <region_server>, tracking started Wed Jan 20 06:54:58 CET 2021; not
retrying 1 - final failure
2021-01-20 06:54:58,690 INFO
org.apache.phoenix.index.PhoenixIndexFailurePolicy: Successfully update
INDEX_DISABLE_TIMESTAMP for <index_table> due to an exception while writing
updates. indexState=PENDING_DISABLE
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
disableIndexOnFailure=true, Failed to write to multiple index tables:
[<index_table>]
at
org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
at
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
at
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
at
org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:617)
at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:577)
at
org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:560)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3421)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2944)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2886)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:765)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:716)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2146)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
2021-01-20 06:54:58,691 INFO
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 1121 (XCL21): Write to the
index failed. disableIndexOnFailure=true, Failed to write to multiple index
tables: [<index_table>] ,serverTimestamp=1611122098649,
2021-01-20 06:55:01,296 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Auth successful for hbase (auth:SIMPLE) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
