Your stack trace indicates that your search pool was exhausted, not your admin action thread pool. I don't think that restarting servers will cause searches to execute, but hopefully someone else would chime in and correct me. Do you have that many searches going on?
-- Ivan On Fri, Aug 1, 2014 at 5:45 AM, Anand kumar <[email protected]> wrote: > > Thank you Ivan. > > If so i'm having 500 indices, each splitted into 5 shards. And while > starting the instance, it'll update each shards. In that case, will it > create 500*5 requests? > > Also i don't have any warmer or percolator. As percolator are fixed on > processor size, is there any chance of automatic updation on these? > > Thanks again for reporting this. > > On Thursday, 31 July 2014 22:41:19 UTC+5:30, Ivan Brusic wrote: > >> The thread pool will reject any search requests when there are 1000 >> actions already queued. >> >> http://www.elasticsearch.org/guide/en/elasticsearch/ >> reference/current/modules-threadpool.html >> >> Do you have this many search requests at one time? Do you have warmers >> and/or percolators running since you mentioned it occurred at startup? >> >> -- >> Ivan >> >> >> On Thu, Jul 31, 2014 at 12:36 AM, Anand kumar <[email protected]> wrote: >> >>> Hi All, >>> >>> In my cluster, I'm having around 500 indices. When i'm trying to start >>> the elasticsearch instance, its showing the following exception. >>> Why its happening, what should be done to resolve it? >>> >>> Thanks, >>> Anand >>> >>> >>> [2014-07-31 11:50:01,551][DEBUG][action.search.type ] [ESCS_NODE] >>> [components][3], node[VQMkI5CBRB2hDEnZ5yVLcw], [P], s[STARTED]: Failed to >>> execute [org.elasticsearch.action.search.SearchRequest@c34bbb8] >>> lastShard [true] >>> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: >>> rejected execution (queue capacity 1000) on org.elasticsearch.search. >>> action.SearchServiceTransportAction$23@60cfc409 >>> at org.elasticsearch.common.util.concurrent.EsAbortPolicy. >>> rejectedExecution(EsAbortPolicy.java:62) >>> at java.util.concurrent.ThreadPoolExecutor.reject( >>> ThreadPoolExecutor.java:821) >>> at java.util.concurrent.ThreadPoolExecutor.execute( >>> ThreadPoolExecutor.java:1372) >>> at org.elasticsearch.search.action.SearchServiceTransportAction. >>> execute(SearchServiceTransportAction.java:509) >>> at org.elasticsearch.search.action.SearchServiceTransportAction. >>> sendExecuteQuery(SearchServiceTransportAction.java:203) >>> at org.elasticsearch.action.search.type. >>> TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase( >>> TransportSearchQueryThenFetchAction.java:80) >>> at org.elasticsearch.action.search.type.TransportSearchTypeAction$ >>> BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171) >>> at org.elasticsearch.action.search.type.TransportSearchTypeAction$ >>> BaseAsyncAction.start(TransportSearchTypeAction.java:153) >>> at org.elasticsearch.action.search.type. >>> TransportSearchQueryThenFetchAction.doExecute( >>> TransportSearchQueryThenFetchAction.java:59) >>> at org.elasticsearch.action.search.type. >>> TransportSearchQueryThenFetchAction.doExecute( >>> TransportSearchQueryThenFetchAction.java:49) >>> at org.elasticsearch.action.support.TransportAction. >>> execute(TransportAction.java:63) >>> at org.elasticsearch.action.search.TransportSearchAction.doExecute( >>> TransportSearchAction.java:101) >>> at org.elasticsearch.action.search.TransportSearchAction.doExecute( >>> TransportSearchAction.java:43) >>> at org.elasticsearch.action.support.TransportAction. >>> execute(TransportAction.java:63) >>> at org.elasticsearch.client.node.NodeClient.execute(NodeClient. >>> java:92) >>> at org.elasticsearch.client.support.AbstractClient.search( >>> AbstractClient.java:212) >>> at org.elasticsearch.rest.action.search.RestSearchAction. >>> handleRequest(RestSearchAction.java:75) >>> at org.elasticsearch.rest.RestController.executeHandler( >>> RestController.java:159) >>> at org.elasticsearch.rest.RestController.dispatchRequest( >>> RestController.java:142) >>> at org.elasticsearch.http.HttpServer.internalDispatchRequest( >>> HttpServer.java:121) >>> at org.elasticsearch.http.HttpServer$Dispatcher. >>> dispatchRequest(HttpServer.java:83) >>> at org.elasticsearch.http.netty.NettyHttpServerTransport. >>> dispatchRequest(NettyHttpServerTransport.java:294) >>> at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived( >>> HttpRequestHandler.java:44) >>> at org.elasticsearch.common.netty.channel. >>> SimpleChannelUpstreamHandler.handleUpstream( >>> SimpleChannelUpstreamHandler.java:70) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. >>> sendUpstream(DefaultChannelPipeline.java:564) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ >>> DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: >>> 791) >>> at org.elasticsearch.common.netty.handler.codec.http. >>> HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) >>> at org.elasticsearch.common.netty.channel. >>> SimpleChannelUpstreamHandler.handleUpstream( >>> SimpleChannelUpstreamHandler.java:70) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. >>> sendUpstream(DefaultChannelPipeline.java:564) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ >>> DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: >>> 791) >>> at org.elasticsearch.common.netty.channel.Channels. >>> fireMessageReceived(Channels.java:296) >>> at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. >>> unfoldAndFireMessageReceived(FrameDecoder.java:459) >>> at org.elasticsearch.common.netty.handler.codec.replay. >>> ReplayingDecoder.callDecode(ReplayingDecoder.java:536) >>> at org.elasticsearch.common.netty.handler.codec.replay. >>> ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) >>> at org.elasticsearch.common.netty.channel. >>> SimpleChannelUpstreamHandler.handleUpstream( >>> SimpleChannelUpstreamHandler.java:70) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. >>> sendUpstream(DefaultChannelPipeline.java:564) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ >>> DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: >>> 791) >>> at org.elasticsearch.common.netty.OpenChannelsHandler. >>> handleUpstream(OpenChannelsHandler.java:74) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. >>> sendUpstream(DefaultChannelPipeline.java:564) >>> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. >>> sendUpstream(DefaultChannelPipeline.java:559) >>> at org.elasticsearch.common.netty.channel.Channels. >>> fireMessageReceived(Channels.java:268) >>> at org.elasticsearch.common.netty.channel.Channels. >>> fireMessageReceived(Channels.java:255) >>> at org.elasticsearch.common.netty.channel.socket.nio. >>> NioWorker.read(NioWorker.java:88) >>> at org.elasticsearch.common.netty.channel.socket.nio. >>> AbstractNioWorker.process(AbstractNioWorker.java:108) >>> at org.elasticsearch.common.netty.channel.socket.nio. >>> AbstractNioSelector.run(AbstractNioSelector.java:318) >>> at org.elasticsearch.common.netty.channel.socket.nio. >>> AbstractNioWorker.run(AbstractNioWorker.java:89) >>> at org.elasticsearch.common.netty.channel.socket.nio. >>> NioWorker.run(NioWorker.java:178) >>> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( >>> ThreadRenamingRunnable.java:108) >>> at org.elasticsearch.common.netty.util.internal. >>> DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>> ThreadPoolExecutor.java:1145) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>> ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/075e105b-9140-4e6a-8518-7e8be2481f13%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/075e105b-9140-4e6a-8518-7e8be2481f13%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA5rUhB7vG%2BT6uOzAHdTodsPK4%3DcYMPg7aTkvTsiWPb6g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
