Thanks for the speedy reply. As for 1, I understand that ES optimizes for *storage* as snapshots of the same index accumulate; I just wish it could also optimize for performance. Right now, with a measly 4.5 gig cluster, the difference between the snapshot 1 and snapshot 24 is 8 minutes. If I kept 15 days worth of hourlies, I could see the snapshot time going past an hour.
Your suggestion of having two repositories is very helpful. Do you have any interest in a PR for the --thin-older-than feature I proposed above? I believe that would solve the same problem as the two repositories but only require one. So say you wanted 15 days worth of dailys and 2 days worth of hourlies, you could run something like this every hour: curator snapshot --all-indices --repository backup curator snapshot --delete-older-than 15 --repository backup curator snapshot --thin-older-than 2 --repository backup As described above, the --thin-older-than would remove all but N snapshots per time-unit where N is defaulted to 1. On Wednesday, December 3, 2014 7:35:49 AM UTC-5, Aaron Mildenstein wrote: > > Hi! Sorry to keep you waiting, but I've been traveling. I completely > misunderstood how snapshots worked when they first came out, so when I > first wrote Curator's snapshot module it would only snap a complete index > once, and never re-snapshot an index if it appeared in the repository. > Then I learned more deeply about how snapshots worked and rewrote > Curator's snapshot functionality to what it is now. You can refer to > this issue on GitHub > <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> > if you like, where I explain how this all works with a thought experiment. > A direct quote: > > When I initially coded snapshot behavior, I thought that snapshots would >> pile up additional copies of indices, somehow, even though I knew they >> supported incremental backups. What I learned is that snapshots capture at >> the *segment* level. As long as the segment is referenced by *any* other >> snapshot in the system, deleting a snapshot will not delete the referenced >> data. Subsequent snapshots of the same indices simply back up any new >> segments since the last snapshot in that repository (incrementals). > > > So to answer your question number 1, Elasticsearch *does* optimize for > this. Though curator *references* indices in a snapshot, only new, > not-yet-snapshotted segments are added to the new snapshot. It's > completely incremental. > > The referenced GitHub issue > <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> > answers question number 2 by showing how keeping 60 days of snapshots would > allow you to restore the most recent 60 days of indices from only the > snapshot taken on day 60. > > The answer to question 3 may be addressed by my suggested use-case for > time-series indices in that same GitHub > <https://github.com/elasticsearch/curator/issues/174#issuecomment-57056621> > link. > > I also responded to the thread you linked > <https://groups.google.com/forum/#!searchin/elasticsearch/snapshot$20duration/elasticsearch/bCKenCVFf2o/TFK-Es0wxSwJ> > : > > Snapshots are at the segment level. The more segments stored in the >> repository, the more segments will have to be compared to those in each >> successive snapshot. With merges taking place continually in an active >> index, you may end up with a considerable number of "orphaned" segments >> stored in your repository, i.e. segments "backed up," but no longer >> directly correlating to a segment in your index. Checking through these >> may be contributing to the increased amount of time between snapshots. >> >> Consider pruning older snapshots. "Orphaned" segments will be deleted, >> and any segments still referenced will be preserved. >> > > This does not affect you in the same way if you have time-series indices > (e.g. one per day). My suggested use-case, as mentioned previously, is to > have 2 repositories: one for frequent snapshots (before daily index > optimization [force merge to smaller number of segments per shard]), and > one for optimized daily snapshots. In this way you could keep hourly > snapshots of the last few daily indices in one repository, and daily > snapshots of your optimized indices in another. This prevents the > slow-down by reducing the number of segments the repositories must search > through for both hourly *and* daily snapshots. > > --Aaron > > On Tuesday, December 2, 2014 1:20:25 PM UTC-5, Matt Hughes wrote: >> >> As noted here -- >> https://groups.google.com/forum/#!searchin/elasticsearch/snapshot$20duration/elasticsearch/bCKenCVFf2o/TFK-Es0wxSwJ >> >> -- the time it takes to perform a snapshot increases the more snapshots you >> take. This eventually can become untenable. So far, the only solution >> seems to be either: trim snapshots or snapshot into a new repository, >> resetting the performance. >> >> >> 1) When I perform snapshots, I want to snapshot all indices. However, >> all of my indices are timestamped logstash-style. The only index that >> receives new documents is todays. I would think Elasticsearch could >> optimize for this and not look through all the snapshots if the index is >> older than today. If there was some mechanism to indicate an index was >> frozen (read-only), then snapshotting could be very fast. Query the >> 'frozenTime' for all indices and only try to update the unfrozen >> snapshots. >> >> >> 2) I can sort of solve the above problem by just snapshotting today's >> indices, but then restore is cumbersome. Say I retained 60 days worth of >> data; that means I'd have to retain 60 days worth of snapshots. And to do >> the restore, I'd have to restore all 60 snapshots. The situation gets >> worse if I wanted to snapshot multiple times a day. >> >> >> 3) While not really fixing the crux of the problem, the curator script >> could help here. Right now, you can only trim snapshots --older-than some >> date. But what if there was a --thin option. Say I take snapshots every >> hour; I'm only really interested in that precision of snapshots for the >> past 24 hours. A backup from a month ago at 12pm is not much different to >> me than a month ago at 1pm. The proposed --thin option would look >> something like this: >> >> curator snapshot --thin-older-than 1 --retain-copies 1. >> >> This would delete all but the last snapshot for each day. >> >> >> >> I'd love to hear thoughts on this and how people are currently solving >> this problem in an automated way. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8e76eaba-97d5-4447-89d9-46cfb65ba6d0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
