I forgot to mention that we're also using the S3 snapshot plugin to store the snapshots in an S3 bucket. Perhaps this might be part of the performance problems?
On Monday, March 9, 2015 at 1:58:33 PM UTC-5, Mark Walkom wrote: > > How many indices are there, are you using the default shard count (5)? Are > you optimising older indices? > > The snapshot takes segments, so it may be that there is a lot of them to > copy. You could try optimising your old indices, eg older than 7 days, down > to a single segment and then see if that helps. > > Be aware though, the optimise is a resource heavy operation, so unless you > have a lot of resources you should only run one at a time. > > On 10 March 2015 at 05:18, Andy Nemzek <[email protected] <javascript:>> > wrote: > >> Hello, >> >> My company is using the ELK stack. Right now we have a very small amount >> of data actually being sent to elastic search (probably a couple hundred >> logstash entries a day if that), however, the data that is getting logged >> is very important. I recently set up snapshots to help protect this data. >> >> I take 1 snapshot a day, I delete snapshots that are older than 20 days, >> and each snapshot is comprised of all the logstash indexes in >> elasticsearch. It's also a business requirement that we are able to search >> at least a year's worth of data, so I can't close logstash indexes unless >> they're older than at least a year. >> >> Now, we've been using logstash for several months and each day it creates >> a new index. We've found that even though there is very little data in >> these indexes, it's taking upwards of 30 minutes to take a snapshot of all >> of them and each day it appears to take 20 - 100 seconds longer than the >> last. It is also taking about 30 minutes to delete a single snapshot, >> which is done each day as a part of cleaning up old snapshots. So, the >> whole process is is taking about an hour each day and appears to be growing >> longer very quickly. >> >> Am I doing something wrong here or is this kind of thing expected? It's >> seems pretty strange that it's taking so long with the little amount of >> data we have. I've looked through the snapshot docs several times and >> there doesn't appear to be much talk about how the process scales. >> >> Thanks! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e5844cef-e8e2-4822-b77b-b6606b409eb8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
