Hi! Sorry to keep you waiting, but I've been traveling. I completely misunderstood how snapshots worked when they first came out, so when I first wrote Curator's snapshot module it would only snap a complete index once, and never re-snapshot an index if it appeared in the repository. Then I learned more deeply about how snapshots worked and rewrote Curator's snapshot functionality to what it is now. You can refer to this issue on GitHub <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> if you like, where I explain how this all works with a thought experiment. A direct quote:
When I initially coded snapshot behavior, I thought that snapshots would > pile up additional copies of indices, somehow, even though I knew they > supported incremental backups. What I learned is that snapshots capture at > the *segment* level. As long as the segment is referenced by *any* other > snapshot in the system, deleting a snapshot will not delete the referenced > data. Subsequent snapshots of the same indices simply back up any new > segments since the last snapshot in that repository (incrementals). So to answer your question number 1, Elasticsearch *does* optimize for this. Though curator *references* indices in a snapshot, only new, not-yet-snapshotted segments are added to the new snapshot. It's completely incremental. The referenced GitHub issue <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> answers question number 2 by showing how keeping 60 days of snapshots would allow you to restore the most recent 60 days of indices from only the snapshot taken on day 60. The answer to question 3 may be addressed by my suggested use-case for time-series indices in that same GitHub <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> link. I also responded to the thread you linked <https://groups.google.com/forum/#!searchin/elasticsearch/snapshot$20duration/elasticsearch/bCKenCVFf2o/TFK-Es0wxSwJ> : Snapshots are at the segment level. The more segments stored in the > repository, the more segments will have to be compared to those in each > successive snapshot. With merges taking place continually in an active > index, you may end up with a considerable number of "orphaned" segments > stored in your repository, i.e. segments "backed up," but no longer > directly correlating to a segment in your index. Checking through these > may be contributing to the increased amount of time between snapshots. > > Consider pruning older snapshots. "Orphaned" segments will be deleted, > and any segments still referenced will be preserved. > This does not affect you in the same way if you have time-series indices (e.g. one per day). My suggested use-case, as mentioned previously, is to have 2 repositories: one for frequent snapshots (before daily index optimization [force merge to smaller number of segments per shard]), and one for optimized daily snapshots. In this way you could keep hourly snapshots of the last few daily indices in one repository, and daily snapshots of your optimized indices in another. This prevents the slow-down by reducing the number of segments the repositories must search through for both hourly *and* daily snapshots. --Aaron On Tuesday, December 2, 2014 1:20:25 PM UTC-5, Matt Hughes wrote: > > As noted here -- > https://groups.google.com/forum/#!searchin/elasticsearch/snapshot$20duration/elasticsearch/bCKenCVFf2o/TFK-Es0wxSwJ > > -- the time it takes to perform a snapshot increases the more snapshots you > take. This eventually can become untenable. So far, the only solution > seems to be either: trim snapshots or snapshot into a new repository, > resetting the performance. > > > 1) When I perform snapshots, I want to snapshot all indices. However, all > of my indices are timestamped logstash-style. The only index that receives > new documents is todays. I would think Elasticsearch could optimize for > this and not look through all the snapshots if the index is older than > today. If there was some mechanism to indicate an index was frozen > (read-only), then snapshotting could be very fast. Query the 'frozenTime' > for all indices and only try to update the unfrozen snapshots. > > > 2) I can sort of solve the above problem by just snapshotting today's > indices, but then restore is cumbersome. Say I retained 60 days worth of > data; that means I'd have to retain 60 days worth of snapshots. And to do > the restore, I'd have to restore all 60 snapshots. The situation gets > worse if I wanted to snapshot multiple times a day. > > > 3) While not really fixing the crux of the problem, the curator script > could help here. Right now, you can only trim snapshots --older-than some > date. But what if there was a --thin option. Say I take snapshots every > hour; I'm only really interested in that precision of snapshots for the > past 24 hours. A backup from a month ago at 12pm is not much different to > me than a month ago at 1pm. The proposed --thin option would look > something like this: > > curator snapshot --thin-older-than 1 --retain-copies 1. > > This would delete all but the last snapshot for each day. > > > > I'd love to hear thoughts on this and how people are currently solving > this problem in an automated way. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79a199d9-1ba7-47ae-a602-8e05e4a02472%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
