That's a massive amount of indexes to create at the one time, and to have on your cluster, so it's no surprise it's timing out.
How big are your nodes? Can you add a few more or collapse your index count? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 6 May 2014 19:48, nicktgr15 <[email protected]> wrote: > Hello, > > We are having an issue with elasticsearch index creation. In our cluster > we create new indices everyday at midnight and at the moment we create > about 150 new indices every time. > Lately we have started getting log lines like the following during index > creation: > > [2014-05-05 00:00:35,596][DEBUG][action.admin.indices.create] [Amber Hunt] > [indexname-2014-05-05] failed to create > org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:failed > to process cluster > event (create-index [indexname-2014-05-05], cause [auto(bulk api)])within > 30s > at org.elasticsearch.cluster.service.InternalClusterService$2$1. > run(InternalClusterService.java:248) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > It's not very clear what this error means and how we could stop seeing > this in our logs. Is there a config parameter for the timeout value of bulk > requests? (if that's the problem) > > Our cluster at the moment has the following stats and we are using > elasticsearch 1.1.1: > > 12 Nodes (3 master, 6 data, 3 search) > 13,140 Total Shards > 13,140 Successful Shards > 2,196 Indices > 434,248,844 Documents > 194.8GB Size > > We have noticed that around the same time we see the above "failed to > create" message, flume elasticsearh-sink (used on the client side) stops > working, so we are trying to understand if there is any correlation between > these two events (index creation failure, flume elasticsearch-sink failure). > > Any help/suggestions would be appreciated! > > Regards, > Nick > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/79eaa859-5583-40ea-8af7-9f5e5878d298%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/79eaa859-5583-40ea-8af7-9f5e5878d298%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YY3OcGuWQFw38FXd%2BkuXyPU6ecPAmnew8z1h8NagQ-Jw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
