Worried about data loss makes sense. If I get the way solr behaves, the new directory should only have missing/changed segments. I guess since our application is extremely write heavy, with lot of inserts and deletes, almost every segment is touched even during a short window, so it appears like for our deployment every segment is copied over when replicas get out of sync.
Thanks for clarifying this behaviour from solr cloud so we can put in external steps to resolve when this situation arises. -----Original Message----- From: Ramkumar R. Aiyengar <andyetitmo...@gmail.com> To: solr-user <solr-user@lucene.apache.org> Sent: Tue, May 5, 2015 4:52 am Subject: Re: Multiple index.timestamp directories using up disk space Yes, data loss is the concern. If the recovering replica is not able to retrieve the files from the leader, it at least has an older copy. Also, the entire index is not fetched from the leader, only the segments which have changed. The replica initially gets the file list from the replica, checks against what it has, and then downloads the difference -- then moves it to the main index. Note that this process can fail sometimes (say due to I/O errors, or due to a problem with the leader itself), in which case the replica drops all accumulated files from the leader, and starts from scratch. If that happens, it needs to look back at its old index again to figure out what it needs to download on the next attempt. May be with a fair number of assumptions which should usually hold good, you can still come up with a mechanism to drop existing files, but those won't hold good in case of serious issues with the cloud, you could end up losing data. That's worse than using a bit more disk space! On 4 May 2015 11:56, "Rishi Easwaran" <rishi.easwa...@aol.com> wrote: Thanks for the responses Mark and Ramkumar. The question I had was, why does Solr need 2 copies at any given time, leading to 2x disk space usage. Not sure if this information is not published anywhere, and makes HW estimation almost impossible for large scale deployment. Even if the copies are temporary, this becomes really expensive, especially when using SSD in production, when the complex size is over 400TB indexes, running 1000's of solr cloud shards. If a solr follower has decided that it needs to do replication from leader and capture full copy snapshot. Why can't it delete the old information and replicate from scratch, not requiring more disk space. Is the concern data loss (a case when both leader and follower lose data)?. Thanks, Rishi. -----Original Message----- From: Mark Miller <markrmil...@gmail.com> To: solr-user <solr-user@lucene.apache.org> Sent: Tue, Apr 28, 2015 10:52 am Subject: Re: Multiple index.timestamp directories using up disk space If copies of the index are not eventually cleaned up, I'd fill a JIRA to address the issue. Those directories should be removed over time. At times there will have to be a couple around at the same time and others may take a while to clean up. - Mark On Tue, Apr 28, 2015 at 3:27 AM Ramkumar R. Aiyengar < andyetitmo...@gmail.com> wrote: > SolrCloud does need up to twice the amount of disk space as your usual > index size during replication. Amongst other things, this ensures you have > a full copy of the index at any point. There's no way around this, I would > suggest you provision the additional disk space needed. > On 20 Apr 2015 23:21, "Rishi Easwaran" <rishi.easwa...@aol.com> wrote: > > > Hi All, > > > > We are seeing this problem with solr 4.6 and solr 4.10.3. > > For some reason, solr cloud tries to recover and creates a new index > > directory - (ex:index.20150420181214550), while keeping the older index > as > > is. This creates an issues where the disk space fills up and the shard > > never ends up recovering. > > Usually this requires a manual intervention of bouncing the instance and > > wiping the disk clean to allow for a clean recovery. > > > > Any ideas on how to prevent solr from creating multiple copies of index > > directory. > > > > Thanks, > > Rishi. > > >
