Ok, do you know if there are clear indicators when limits are being reached?
We don't see errors in the logs (apart from the 30s timeout) but if there are system or ES provided metrics that we can track to know when we need to scale it would be really useful. Thanks, Paul. On Tuesday, May 13, 2014 11:24:06 AM UTC+1, Mark Walkom wrote: > > Empty or not, there is still metadata that ES needs to maintain in the > cluster state. So the more indexes you have open the bigger that is and the > more resources required to track it. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] <javascript:> > web: www.campaignmonitor.com > > > On 13 May 2014 20:16, Paul <[email protected] <javascript:>> wrote: > >> In testing and replicating the issue, this slow down has been seen >> occurring with empty indices. >> >> The running cluster is at present ~100 GB across 2,200 Indices with a >> total of 13,500 shards and ~430,000,000 documents. >> >> We have 7GB RAM and 5GB heap on the data nodes - haven't looked overly >> carefully but don't think the heap is maxing out on any of the nodes when >> this occurs. >> >> >> Thanks, >> >> Paul. >> >> >> On Tuesday, May 13, 2014 11:02:32 AM UTC+1, Mark Walkom wrote: >>> >>> Sounds like the inevitable "add more nodes" situation. >>> >>> How much RAM on each node, how big is your data set? >>> >>> Regards, >>> Mark Walkom >>> >>> Infrastructure Engineer >>> Campaign Monitor >>> email: [email protected] >>> web: www.campaignmonitor.com >>> >>> >>> On 13 May 2014 19:59, Paul <[email protected]> wrote: >>> >>>> We are seeing a slow down in shard initialization speed as the number >>>> of shards/indices grows in our cluster. >>>> >>>> With 0-100's of indices/shards existing in the cluster a new bulk >>>> creation of indices up the 100's at a time is fine, we see them pass >>>> through the states and get a green cluster in a reasonable amount of time. >>>> >>>> As the total cluster size grows to 1000+ indices (3000+ shards) we >>>> begin to notice that the first rounds of initialization take longer to >>>> process, it seems to speed up after the first few batches, but this slow >>>> down leads to "failed to process cluster event (create-index >>>> [index_1112], cause [auto(bulk api)]) within 30s" type messages in the >>>> Master logs - the indices are eventually created. >>>> >>>> >>>> Has anyone else experienced this? (did you find the cause / way to fix?) >>>> >>>> Is this somewhat expected behaviour? - are we approaching something >>>> incorrectly? (there are 3 data nodes involved, with 3 shards per index) >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667% >>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9e52d337-7b5d-411b-904d-477c0806f99d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
