Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
On Fri, Jun 11, 2010 at 11:48 AM, Paul Winkler wrote: > On Fri, Jun 11, 2010 at 10:34 AM, Hanno Schlichting wrote: > >> >> I tend to run rsync via "rsync -rP --rsh=ssh". The Data.fs is an >> append-only file, so rsync is very efficient at handling it. Only >> zeopack rewrites things all across the file and causes a subsequent >> rsync to be slow again. >> >> > Thanks. I'll do a trial run of this today. > It seems that a second rsync isn't exactly blazing fast with a few changes on the end of the 32G Data.fs. Near as I can tell, it spends a good 10 minutes or so just comparing the files to see if it has any work to do. Once that phase is done, it seems to spend a lot of its time in IO since by default it builds a new file and replaces the existing file when it's done. Total time ~ 25 minutes. The rsync man page paid off though: Using the --append option (or --append-verify on recent enough versions of rsync) seems to reduce the IO a lot, as it's tailor-made for this use case: updating in-place when the source file has only been appended to and potentially losing the target file on failure is OK. (We can manually make a pristine copy prior to starting our downtime, just in case we need to do it over for any reason). FWIW total time for the second `rsync -z --append Data.fs` was: real7m50.253s Last time I had to rebuild the index file it took ~ 30 minutes, so this looks like a win. We'll go with rsync. -- http://www.slinkp.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
On Fri, Jun 11, 2010 at 10:34 AM, Hanno Schlichting wrote: > On Fri, Jun 11, 2010 at 3:25 PM, Paul Winkler wrote: > > I'm preparing to move a zope site from one host to another. We've been > > planning to use repozo backups to copy the filestorage from the old > > host to the new one. I'd like to figure out how to minimize downtime > > while not losing any data. My plan was: > > For moving stuff to a new host where a short downtime is planned > anyways, I would just use rsync. > > 1. While the old site is still running, rsync both the Data.fs and > index file to the new host > 2. Make sure you don't run any zeopack operation between that point > and the actual switchover > 3. You can keep running rsync a couple times shortly before the > downtime to get the most recent state over > 4. Shut down the old site > 5. Do a final rsync of both the .fs and .fs.index (this should take a > couple seconds or minutes at most) > 6. Start the new site, since the index is up-to-date and matches the > Data.fs it should be seconds as well > > I tend to run rsync via "rsync -rP --rsh=ssh". The Data.fs is an > append-only file, so rsync is very efficient at handling it. Only > zeopack rewrites things all across the file and causes a subsequent > rsync to be slow again. > > Thanks. I'll do a trial run of this today. > > 1) would it be safe to copy the index file from the old host and just > > use that with the filestorage generated by repozo? > > It's save, but it will be ignored and a new index be created, so you > don't gain anything. > Hm, ok, so whether you're right or Chris is right, it's not going to work :) > But at some point you could upgrade to ZODB 3.10 (currently in beta), > where repozo does save the index file for each backup. It will still > be slower than rsync for this use-case, though. > Roger that. Thanks. -- http://www.slinkp.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
On Fri, Jun 11, 2010 at 10:18 AM, Chris Withers wrote: > Paul Winkler wrote: > >> 3 should be pretty fast as its an incremental backup. >> 4 may take a little while but iirc it's not too bad for this db - maybe >> 10 minutes. >> Most of the downtime is caused by rebuilding the index which takes way >> longer (30 min?) >> >> So I had two questions: >> >> 1) would it be safe to copy the index file from the old host and just use >> that with the filestorage generated by repozo? >> > > Almost certainly not. > OK, good to know, thanks. > > 2) should I maybe ditch repozo and just rsync everything? That'd replace >> step 3 with a second call to rsync, and eliminate steps 4 and 6. But I'm >> not sure how fast it is at updating a big binary file. I think I'll do an >> experiment today and report back on this option. >> > > I suppose you could look at zeoraid. > Hadn't thought of that, will look at it, thanks! > Have you tried just starting with an empty index file? > Yes - zeo takes ages to be usable while it builds the new index from scratch. That's the whole problem. -- http://www.slinkp.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
On Fri, Jun 11, 2010 at 3:25 PM, Paul Winkler wrote: > I'm preparing to move a zope site from one host to another. We've been > planning to use repozo backups to copy the filestorage from the old > host to the new one. I'd like to figure out how to minimize downtime > while not losing any data. My plan was: For moving stuff to a new host where a short downtime is planned anyways, I would just use rsync. 1. While the old site is still running, rsync both the Data.fs and index file to the new host 2. Make sure you don't run any zeopack operation between that point and the actual switchover 3. You can keep running rsync a couple times shortly before the downtime to get the most recent state over 4. Shut down the old site 5. Do a final rsync of both the .fs and .fs.index (this should take a couple seconds or minutes at most) 6. Start the new site, since the index is up-to-date and matches the Data.fs it should be seconds as well I tend to run rsync via "rsync -rP --rsh=ssh". The Data.fs is an append-only file, so rsync is very efficient at handling it. Only zeopack rewrites things all across the file and causes a subsequent rsync to be slow again. > 1) would it be safe to copy the index file from the old host and just > use that with the filestorage generated by repozo? It's save, but it will be ignored and a new index be created, so you don't gain anything. But at some point you could upgrade to ZODB 3.10 (currently in beta), where repozo does save the index file for each backup. It will still be slower than rsync for this use-case, though. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
Paul Winkler wrote: > 3 should be pretty fast as its an incremental backup. > 4 may take a little while but iirc it's not too bad for this db - > maybe 10 minutes. > Most of the downtime is caused by rebuilding the index which takes way > longer (30 min?) > > So I had two questions: > > 1) would it be safe to copy the index file from the old host and just > use that with the filestorage generated by repozo? Almost certainly not. > 2) should I maybe ditch repozo and just rsync everything? That'd > replace step 3 with a second call to rsync, and eliminate steps 4 and > 6. But I'm not sure how fast it is at updating a big binary file. I > think I'll do an experiment today and report back on this option. I suppose you could look at zeoraid. Have you tried just starting with an empty index file? Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Restoring from repozo and reusing an index file?
Hi folks, I'm preparing to move a zope site from one host to another. We've been planning to use repozo backups to copy the filestorage from the old host to the new one. I'd like to figure out how to minimize downtime while not losing any data. My plan was: 1. Make a backup, copy it to new host. 2. Shut down old host. 3. Make a final incremental backup, copy to new host. 4. Restore on new host. 5. Start new host and 6. wait for index file to get built. 3 should be pretty fast as its an incremental backup. 4 may take a little while but iirc it's not too bad for this db - maybe 10 minutes. Most of the downtime is caused by rebuilding the index which takes way longer (30 min?) So I had two questions: 1) would it be safe to copy the index file from the old host and just use that with the filestorage generated by repozo? 2) should I maybe ditch repozo and just rsync everything? That'd replace step 3 with a second call to rsync, and eliminate steps 4 and 6. But I'm not sure how fast it is at updating a big binary file. I think I'll do an experiment today and report back on this option. Paul Winkler -- Typed on my phone. Apologies for typos, brevity, and/or bizarre word substitutions. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
