[ 
https://issues.apache.org/jira/browse/PHOENIX-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119077#comment-16119077
 ] 

Sergey Soldatov commented on PHOENIX-4075:
------------------------------------------

That's may be a side effect of PHOENIX-3360 and follows jiras. To properly set 
the configuration you need to have a separate copy of hbase-site.xml for PQS 
with removed controllerfactory property. 

> Executor already shutdown building secondary index
> --------------------------------------------------
>
>                 Key: PHOENIX-4075
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4075
>             Project: Phoenix
>          Issue Type: Bug
>         Environment: 3 Node HBase cluster running version 1.2.0-cdh5.7.2
> Phoenix version 4.7.0 using thin client + query server
>            Reporter: Brian Thomas
>
> I started to experience an issue trying to insert some data into our HBase 
> cluster using phoenix. This seems to have start occurring after I added a 
> secondary index on one of my columns. 
> The issue seems to resolve temporarily with an HBase restart. However, it 
> eventually falls back into this state and once it gets into this state, it's 
> not possible to insert data anymore. When my application is running, it tries 
> to insert 80k rows and then issues a single commit. However, the batch size 
> doesn't seem to affect it as I cannot even insert a single row when it gets 
> into this state. The full stack trace I see is
> Caused by: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 
> 131563 actions: 
> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed 
> to build index for unexp
> ected reason!
>         at 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:180)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:205)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
>         at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3088)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2817)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:751)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:713)
>         at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2142)
>         at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
>         at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
>         at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.RejectedExecutionException: Executor already 
> shutdown
>         at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.startTask(MoreExecutors.java:327)
>         at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:251)
>         at 
> com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
>         at 
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:58)
>         at 
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
>         at 
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:143)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:273)
>         at 
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:202)
>         ... 17 more
> : 131563 times, 
>         at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
>         at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
>         at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1663)
>         at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:982)
>         at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:996)
>         at 
> org.apache.phoenix.execute.MutationState.send(MutationState.java:951)
>         ... 22 more
>         at 
> org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2315)
>         at 
> org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:46)
>         at 
> org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:107)
>         at 
> org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:377)
>         at 
> org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:375)
>         at 
> org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:666)
>         at 
> org.apache.calcite.avatica.remote.RemoteMeta.commit(RemoteMeta.java:375)
>         at 
> org.apache.calcite.avatica.AvaticaConnection.commit(AvaticaConnection.java:175)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to