Brian Thomas created PHOENIX-4075:
-------------------------------------

             Summary: Executor already shutdown building secondary index
                 Key: PHOENIX-4075
                 URL: https://issues.apache.org/jira/browse/PHOENIX-4075
             Project: Phoenix
          Issue Type: Bug
         Environment: 3 Node HBase cluster running version 0.98
Phoenix version 4.7.0 using thin client + query server
            Reporter: Brian Thomas


I started to experience an issue trying to insert some data into our HBase 
cluster using phoenix. This seems to have start occurring after I added a 
secondary index on one of my columns. 

The issue seems to resolve temporarily with an HBase restart. However, it 
eventually falls back into this state and once it gets into this state, it's 
not possible to insert data anymore. When my application is running, it tries 
to insert 80k rows and then issues a single commit. However, the batch size 
doesn't seem to affect it as I cannot even insert a single row when it gets 
into this state. The full stack trace I see is

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 131563 actions: 
org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to 
build index for unexp
ected reason!
        at 
org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:180)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:205)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
        at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3088)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2817)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:751)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:713)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2142)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.RejectedExecutionException: Executor already 
shutdown
        at 
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.startTask(MoreExecutors.java:327)
        at 
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:251)
        at 
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
        at 
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:58)
        at 
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
        at 
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:143)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:273)
        at 
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:202)
        ... 17 more
: 131563 times, 
        at 
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
        at 
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
        at 
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1663)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:982)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:996)
        at org.apache.phoenix.execute.MutationState.send(MutationState.java:951)
        ... 22 more


        at 
org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2315)
        at 
org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:46)
        at 
org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:107)
        at 
org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:377)
        at 
org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:375)
        at 
org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:666)
        at 
org.apache.calcite.avatica.remote.RemoteMeta.commit(RemoteMeta.java:375)
        at 
org.apache.calcite.avatica.AvaticaConnection.commit(AvaticaConnection.java:175)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to