How many nodes do you have in the cluster? Jörg
On Thu, Jan 8, 2015 at 6:57 PM, Tom <[email protected]> wrote: > Hi, we'd been using ES for a while now. Specifically version 0.90.3. A > couple of months ago we decided to migrate to the latest version which was > finally frozen to be 1.4.1. No data migration was necessary because we have > a redundant MongoDB, but yesterday we enabled data writing to the new ES > cluster. All was running smoothly when we noticed that at o'clock times > there were bursts of four or five log messages of the following kinds: > > Error indexing None into index ind-analytics-2015.01.08. Total elapsed > time: 1065 ms. > org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: > failed to process cluster event (acquire index lock) within 1s > at > org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_17] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_17] > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17] > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - Scattering > to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > {[ind-2015.01.08.00][search][...]} > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_17] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_17] > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17] > > This occurs at o'clock times because we write over hour-based indices. For > example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > so on. > > With 0.90.3 version of ES, automatic index creation was working flawlessly > (with no complaints) but the new version doesn't seem to handle that > feature very well. It looks like, when all those concurrent writes competes > to be the first to create the index, all but one fails. Of course we could > just create such indices manually to avoid this situation altogether, but > this would only be a workaround for a feature that previously worked. > > Also, we use ES through the native Java client and the configuration for > all our indices is > > settings = { > number_of_shards = 5, > number_of_replicas = 2 > } > > Any ideas? > > Thanks in advance, > Tom; > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEL7i_6Nugjw4XDMAuAK9o6a14%2BDiah9wA37gCtpmf%3DwQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
