Sorry, didn't mean to say "the same indexing operation" but multiple indexing operations (distinct data) on the same non-existent index.
El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió: > > Well yes. We also have a cluster for the app where each node talks to the > elastic cluster independently. > > Remember that we are not creating the index manually. Each app node issues > an index operation on an index that may yet not exist and we expect ES to > take care of the index creation on demand. Many processes may issue the > same indexing operation on the ES cluster "simultaneously" and only one of > them must succeed in triggering the index creation. > > Tom; > > El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió: >> >> It seems there are more than one process trying to create the index, it >> that possible? >> >> Jörg >> >> On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi <[email protected]> >> wrote: >> >>> We enlarged our cluster to 5 nodes and now the QUORUM error message >>> seems to have disappeared. >>> "failed to process cluster event (acquire index lock) within 1s" kind >>> of messages are still happening though. >>> >>> :( >>> >>> Tom; >>> >>> On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi <[email protected]> >>> wrote: >>> >>>> We enlarged our cluster to 5 nodes and now the QUORUM error message >>>> seems to have disappeared. >>>> "failed to process cluster event (acquire index lock) within 1s" kind >>>> of messages are still happening though. >>>> >>>> :( >>>> >>>> Tom; >>>> >>>> On Fri, Jan 9, 2015 at 1:25 PM, [email protected] <[email protected]> >>>> wrote: >>>> >>>>> Exactly, with 3 nodes, the error will be gone. >>>>> >>>>> Please, always use an odd number of data nodes, in particular with >>>>> replica > 0, in order not to confuse ES quorum formula, and also to avoid >>>>> split brains with minimun_master_nodes >>>>> >>>>> Jörg >>>>> >>>>> On Fri, Jan 9, 2015 at 3:17 PM, Tom <[email protected]> wrote: >>>>> >>>>>> Also, we have another cluster (for different purposes) that has 3 >>>>>> nodes but we didn't experience such errors with it (for this ES we >>>>>> create >>>>>> indices on a daily basis). >>>>>> >>>>>> El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: >>>>>> >>>>>>> 4 >>>>>>> >>>>>>> El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: >>>>>>>> >>>>>>>> How many nodes do you have in the cluster? >>>>>>>> >>>>>>>> Jörg >>>>>>>> >>>>>>>> On Thu, Jan 8, 2015 at 6:57 PM, Tom <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, we'd been using ES for a while now. Specifically version >>>>>>>>> 0.90.3. A couple of months ago we decided to migrate to the latest >>>>>>>>> version >>>>>>>>> which was finally frozen to be 1.4.1. No data migration was necessary >>>>>>>>> because we have a redundant MongoDB, but yesterday we enabled data >>>>>>>>> writing >>>>>>>>> to the new ES cluster. All was running smoothly when we noticed that >>>>>>>>> at >>>>>>>>> o'clock times there were bursts of four or five log messages of the >>>>>>>>> following kinds: >>>>>>>>> >>>>>>>>> Error indexing None into index ind-analytics-2015.01.08. Total >>>>>>>>> elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. >>>>>>>>> ProcessClusterEventTimeoutException: failed to process cluster >>>>>>>>> event (acquire index lock) within 1s >>>>>>>>> at org.elasticsearch.cluster.metadata. >>>>>>>>> MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) >>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> >>>>>>>>> ~[na:1.7.0_17] >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> >>>>>>>>> ~[na:1.7.0_17] >>>>>>>>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17] >>>>>>>>> >>>>>>>>> [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - >>>>>>>>> Scattering to failed in 1043ms >>>>>>>>> org.elasticsearch.action.UnavailableShardsException: >>>>>>>>> [ind-2015.01.08.00][0] Not enough active copies to meet write >>>>>>>>> consistency >>>>>>>>> of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index >>>>>>>>> {[ind-2015.01.08.00][search][...]} >>>>>>>>> at org.elasticsearch.action.support.replication. >>>>>>>>> TransportShardReplicationOperationAction$ >>>>>>>>> AsyncShardOperationAction.retryBecauseUnavailable( >>>>>>>>> TransportShardReplicationOperationAction.java:784) >>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] >>>>>>>>> at org.elasticsearch.action.support.replication. >>>>>>>>> TransportShardReplicationOperationAction$ >>>>>>>>> AsyncShardOperationAction.raiseFailureIfHaveNotEnoughAct >>>>>>>>> iveShardCopies(TransportShardReplicationOperationAction.java:776) >>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] >>>>>>>>> at org.elasticsearch.action.support.replication. >>>>>>>>> TransportShardReplicationOperationAction$ >>>>>>>>> AsyncShardOperationAction.performOnPrimary( >>>>>>>>> TransportShardReplicationOperationAction.java:507) >>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] >>>>>>>>> at org.elasticsearch.action.support.replication. >>>>>>>>> TransportShardReplicationOperationAction$ >>>>>>>>> AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) >>>>>>>>> >>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> >>>>>>>>> ~[na:1.7.0_17] >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> >>>>>>>>> ~[na:1.7.0_17] >>>>>>>>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17] >>>>>>>>> >>>>>>>>> This occurs at o'clock times because we write over hour-based >>>>>>>>> indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 >>>>>>>>> goes to >>>>>>>>> ind-2015.01.08.18. At 19:00:00 all writes will go to >>>>>>>>> ind-2015.01.08.19, and >>>>>>>>> so on. >>>>>>>>> >>>>>>>>> With 0.90.3 version of ES, automatic index creation was working >>>>>>>>> flawlessly (with no complaints) but the new version doesn't seem to >>>>>>>>> handle >>>>>>>>> that feature very well. It looks like, when all those concurrent >>>>>>>>> writes >>>>>>>>> competes to be the first to create the index, all but one fails. Of >>>>>>>>> course >>>>>>>>> we could just create such indices manually to avoid this situation >>>>>>>>> altogether, but this would only be a workaround for a feature that >>>>>>>>> previously worked. >>>>>>>>> >>>>>>>>> Also, we use ES through the native Java client and the >>>>>>>>> configuration for all our indices is >>>>>>>>> >>>>>>>>> settings = { >>>>>>>>> number_of_shards = 5, >>>>>>>>> number_of_replicas = 2 >>>>>>>>> } >>>>>>>>> >>>>>>>>> Any ideas? >>>>>>>>> >>>>>>>>> Thanks in advance, >>>>>>>>> Tom; >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/4deefb09- >>>>>>>>> bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Tom; >>>> >>> >>> >>> >>> -- >>> Tom; >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
