I'm on 1.4.1 and still seeing the same behavior. There should be a better practice than remove all shards at the same time and try to move a few. We are going to apply the same solution you mentioned, add more disk. Thank's for your help.
2015-01-15 16:09 GMT-03:00 Kimbro Staken <[email protected]>: > So is this still happening with 1.4.2? > > Here's the ticket. Looks like the fix was supposed to be in 1.4.1 > > https://github.com/elasticsearch/elasticsearch/issues/8538 > > On Thu, Jan 15, 2015 at 10:55 AM, Matías Waisgold <[email protected]> > wrote: > >> Great, thank you. We are creating another cluster with more disk space to >> avoid this situations. >> By any chance do you have the link to the issue? >> >> 2015-01-15 13:26 GMT-03:00 Kimbro Staken <[email protected]>: >> >>> I've experienced what you're describing. I called it a "shard relocation >>> storm" and it's really tough to get under control. I opened a ticket on the >>> issue and a fix was supposedly included in 1.4.2. What version are you >>> running? >>> >>> If you want to truly manually manage this situation you could set >>> cluster.routing.allocation.disk.threshold_enabled to false but that will >>> likely cause other issues. I ended up just setting >>> cluster.routing.allocation.disk.watermark.high to a really low value and >>> actively managed shard allocations to prevent nodes from getting anywhere >>> near that value. This is tricky as the way ES allocates shards it can >>> easily run nodes out of disk if you're regularly creating new indices and >>> those grow rapidly. >>> >>> Kimbro >>> >>> On Thu, Jan 15, 2015 at 6:14 AM, Matías Waisgold <[email protected]> >>> wrote: >>> >>>> Yes, I've seen that but the problem is that when the threshold is >>>> reached it removes all shards from the server instead of just removing 1 >>>> and balance. And when that happens the cluster starts to move shards over >>>> everywhere and it never stops. >>>> >>>> Another problem we are having is that in the file storage we see data >>>> from shards that are not assigned to itself so it can´t allocate anything >>>> in this dirty state. >>>> >>>> 2015-01-15 0:09 GMT-03:00 Mark Walkom <[email protected]>: >>>> >>>>> You could do this, but it's a lot of manual overhead to have to deal >>>>> with. >>>>> However ES does have some disk space awareness during allocation, take >>>>> a look at >>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/index-modules-allocation.html#disk >>>>> >>>>> On 15 January 2015 at 10:57, Matías Waisgold <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi is there any setting that I can put to ES that it automatically >>>>>> assigns shards that are unassigned but never ever rebalance the cluster? >>>>>> I´ve found several issues when rebalancing and prefer to do it >>>>>> manually. >>>>>> If I set cluster.routing.allocation.enable to "none" nothing happens. >>>>>> If I set it to "all" then it starts rebalancing. >>>>>> >>>>>> Is it ok to combine cluster.routing.allocation.allow_rebalance to >>>>>> "none" and cluster.routing.allocation.enable to "all". >>>>>> >>>>>> The issue is mainly because we are running low on disk and when that >>>>>> happens elasticsearch removes all shards from an instance, that doesn´t >>>>>> care about cluster.routing.allocation.cluster_concurrent_rebalance and >>>>>> starts moving shards like crazy around the entire cluster, filling the >>>>>> storage on other instances in the way that it will never stop balancing. >>>>>> >>>>>> Kind regards >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAMaTqYq%2B_S1_CaPJ-_HN8F%3DizX9VjKmF_W5dhDFpAmS6kxw0WA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYq%2B_S1_CaPJ-_HN8F%3DizX9VjKmF_W5dhDFpAmS6kxw0WA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAA0DmXar33evahvzhzbQ2zgO_kv%3Du4YtMaUMfZcj9MNJWC%3D_MA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAA0DmXar33evahvzhzbQ2zgO_kv%3Du4YtMaUMfZcj9MNJWC%3D_MA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqTcALa3zKGVLXmobjbFkEMMR8L03xVR9r-1pBWsn451A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
