You will want to obtain Marvel ( http://www.elasticsearch.org/guide/en/marvel/current/) and then wait till you have a history and start digging.
Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 13 May 2014 20:29, Paul <[email protected]> wrote: > Ok, do you know if there are clear indicators when limits are being > reached? > > We don't see errors in the logs (apart from the 30s timeout) but if there > are system or ES provided metrics that we can track to know when we need to > scale it would be really useful. > > > Thanks, > > Paul. > > > > On Tuesday, May 13, 2014 11:24:06 AM UTC+1, Mark Walkom wrote: >> >> Empty or not, there is still metadata that ES needs to maintain in the >> cluster state. So the more indexes you have open the bigger that is and the >> more resources required to track it. >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 13 May 2014 20:16, Paul <[email protected]> wrote: >> >>> In testing and replicating the issue, this slow down has been seen >>> occurring with empty indices. >>> >>> The running cluster is at present ~100 GB across 2,200 Indices with a >>> total of 13,500 shards and ~430,000,000 documents. >>> >>> We have 7GB RAM and 5GB heap on the data nodes - haven't looked overly >>> carefully but don't think the heap is maxing out on any of the nodes when >>> this occurs. >>> >>> >>> Thanks, >>> >>> Paul. >>> >>> >>> On Tuesday, May 13, 2014 11:02:32 AM UTC+1, Mark Walkom wrote: >>>> >>>> Sounds like the inevitable "add more nodes" situation. >>>> >>>> How much RAM on each node, how big is your data set? >>>> >>>> Regards, >>>> Mark Walkom >>>> >>>> Infrastructure Engineer >>>> Campaign Monitor >>>> email: [email protected] >>>> web: www.campaignmonitor.com >>>> >>>> >>>> On 13 May 2014 19:59, Paul <[email protected]> wrote: >>>> >>>>> We are seeing a slow down in shard initialization speed as the number >>>>> of shards/indices grows in our cluster. >>>>> >>>>> With 0-100's of indices/shards existing in the cluster a new bulk >>>>> creation of indices up the 100's at a time is fine, we see them pass >>>>> through the states and get a green cluster in a reasonable amount of time. >>>>> >>>>> As the total cluster size grows to 1000+ indices (3000+ shards) we >>>>> begin to notice that the first rounds of initialization take longer to >>>>> process, it seems to speed up after the first few batches, but this slow >>>>> down leads to "failed to process cluster event (create-index >>>>> [index_1112], cause [auto(bulk api)]) within 30s" type messages in the >>>>> Master logs - the indices are eventually created. >>>>> >>>>> >>>>> Has anyone else experienced this? (did you find the cause / way to >>>>> fix?) >>>>> >>>>> Is this somewhat expected behaviour? - are we approaching something >>>>> incorrectly? (there are 3 data nodes involved, with 3 shards per index) >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40goo >>>>> glegroups.com<https://groups.google.com/d/msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2% >>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/9e52d337-7b5d-411b-904d-477c0806f99d%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/9e52d337-7b5d-411b-904d-477c0806f99d%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zs2t_o-UH1%3DU0GEsD%3DbbihGGNaFpeC_RVztNC8sTK1Cg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
