Document TTL can be a resource hog, you might want to look at logstash's curator instead to manage expiration. It might also be beneficial to increase your shard count so you have one shard per node, alternatively look at routing.allocation.awareness.attributes.
ES will distribute things as evenly as possible based on the shard count, however sometimes that doesn't work; eg we have one node now with >20GB of 500GB disk free, despite other nodes having more free space. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 17 April 2014 00:13, Dipesh Patel <[email protected]>wrote: > Hi all > > We've noticed that a couple of our data nodes have been having memory > issues eventually leading to them dropping out of the cluster. Other nodes > in the cluster are nowhere near their limits in terms of memory usage. > Looking at the gc logs of those problem boxes it seems like it reaches a > point after some time ( about 3 days ) where the nodes start garbage > collecting every 30 seconds. We also see some out of memory errors in the > main log file and we think this may be due to when new logs are created at > midnight for the start of the new day - however we are still in the process > of confirming this. > > I'm sure it's some configuration issues but not too sure where to even > look to figure out what needs tweaking. I'm going to describe our set up > below in the hopes that you may know how to help. > > * Deploying to AWS > * Using Elasticsearch 1.1.0 and the equivalent aws-elasticsearch-plugin. > * Using Java 7 on centos6 > * 6 Data nodes are m1.large > * 3 Master nodes m1.medium > * All nodes have ES_HEAP_SIZE = 5g > * All nodes have MAX_OPEN_FILES=65535 > * 3 shards per index > * 1 replica > * ttl is set globally to 30 days. > * Flume is used to push log data into the cluster. > > The big one is the number of indices. We are using it for our application > logging data. We have multiple applications and we write each application > log for each day into a separate index. So there will be new applications > that will fire up and send its logs to our cluster over time. It's similar > to the logstash set up, the difference being each application writes to > it's own index on a daily basis not one big global one on a daily basis if > that makes sense. Currently we have about 250 indices. > > One of the things we have been considering is scaling up to deal with the > issue as we are on AWS. However we would like to understand how > elasticsearch distributes the load amongst the data nodes as during scaling > up we would like to distribute amongst the data nodes based on load, else > the scaling up may not have a significant effect. > > Any help would be appreciated. > > Dip > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/580e6a37-aa00-41c1-88cd-6f54865eccdb%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/580e6a37-aa00-41c1-88cd-6f54865eccdb%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Yco9G8E%3DNXuVorL8NL0bkJ3WfPL2a6DeZbD_yDSQzQCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
