I've experienced what you're describing. I called it a "shard relocation storm" and it's really tough to get under control. I opened a ticket on the issue and a fix was supposedly included in 1.4.2. What version are you running?
If you want to truly manually manage this situation you could set cluster.routing.allocation.disk.threshold_enabled to false but that will likely cause other issues. I ended up just setting cluster.routing.allocation.disk.watermark.high to a really low value and actively managed shard allocations to prevent nodes from getting anywhere near that value. This is tricky as the way ES allocates shards it can easily run nodes out of disk if you're regularly creating new indices and those grow rapidly. Kimbro On Thu, Jan 15, 2015 at 6:14 AM, Matías Waisgold <[email protected]> wrote: > Yes, I've seen that but the problem is that when the threshold is reached > it removes all shards from the server instead of just removing 1 and > balance. And when that happens the cluster starts to move shards over > everywhere and it never stops. > > Another problem we are having is that in the file storage we see data from > shards that are not assigned to itself so it can´t allocate anything in > this dirty state. > > 2015-01-15 0:09 GMT-03:00 Mark Walkom <[email protected]>: > >> You could do this, but it's a lot of manual overhead to have to deal with. >> However ES does have some disk space awareness during allocation, take a >> look at >> http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/index-modules-allocation.html#disk >> >> On 15 January 2015 at 10:57, Matías Waisgold <[email protected]> wrote: >> >>> Hi is there any setting that I can put to ES that it automatically >>> assigns shards that are unassigned but never ever rebalance the cluster? >>> I´ve found several issues when rebalancing and prefer to do it manually. >>> If I set cluster.routing.allocation.enable to "none" nothing happens. >>> If I set it to "all" then it starts rebalancing. >>> >>> Is it ok to combine cluster.routing.allocation.allow_rebalance to "none" >>> and cluster.routing.allocation.enable to "all". >>> >>> The issue is mainly because we are running low on disk and when that >>> happens elasticsearch removes all shards from an instance, that doesn´t >>> care about cluster.routing.allocation.cluster_concurrent_rebalance and >>> starts moving shards like crazy around the entire cluster, filling the >>> storage on other instances in the way that it will never stop balancing. >>> >>> Kind regards >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
