[
https://issues.apache.org/jira/browse/PHOENIX-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268793#comment-17268793
]
Kadir OZDEMIR commented on PHOENIX-6331:
----------------------------------------
[[email protected]], FYI ... This behavior is specific to the old index design.
Currently, both the old and new index implementation coexists, however, by
default, indexes created on the new data tables use the new design. In the new
index design, indexes are not disabled due to IO errors. You can upgrade the
existing tables to the new design using IndexUpgradeTool.
> Increase index retry from 1 to 2 incase of NotServingRegionException
> --------------------------------------------------------------------
>
> Key: PHOENIX-6331
> URL: https://issues.apache.org/jira/browse/PHOENIX-6331
> Project: Phoenix
> Issue Type: Bug
> Reporter: Ankit Singhal
> Priority: Major
>
> Currently, we move the index to PENDING_DISABLE whenever the single write to
> index failed and carry out a retry at the client but this can be optimized
> for NotServingRegionException as Index regions can move very frequently
> depending on the balancer and one more retry at the server could avoid
> unnecessary handling of index states and retries at the client.
>
> {code:java}
> 2021-01-20 06:54:58,682 WARN org.apache.hadoop.hbase.client.AsyncProcess:
> #277, table=<index_table>, attempt=1/1 failed=1ops, last exception:
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region <region> is not
> online on <region_server>
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2997)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1069)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2100)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
> on <region_server>, tracking started Wed Jan 20 06:54:58 CET 2021; not
> retrying 1 - final failure
> 2021-01-20 06:54:58,690 INFO
> org.apache.phoenix.index.PhoenixIndexFailurePolicy: Successfully update
> INDEX_DISABLE_TIMESTAMP for <index_table> due to an exception while writing
> updates. indexState=PENDING_DISABLE
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
> disableIndexOnFailure=true, Failed to write to multiple index tables:
> [<index_table>]
> at
> org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:617)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:577)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:560)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3421)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2944)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2886)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:765)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:716)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2146)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
> 2021-01-20 06:54:58,691 INFO
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 1121 (XCL21): Write to
> the index failed. disableIndexOnFailure=true, Failed to write to multiple
> index tables: [<index_table>] ,serverTimestamp=1611122098649,
> 2021-01-20 06:55:01,296 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
> Auth successful for hbase (auth:SIMPLE) {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
