Re: [ZODB-Dev] repozo full backup and index files
On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but also rather inefficient way. In my case it takes only about 15 minutes of rather heavy I/O to create the file, but for larger files this time usually goes up. From what I understand repozo does the following: 1. it opens the real file storage file as a read-only FileStorage 2. calculates the byte position of end of the last complete transaction 3. closes the file storage 4. opens the fs file as a normal binary file 5. copies over all bytes up to the calculated position into a temp file 6. closes the fs file and temp file and renames the temp file according to some timestamp What I'm wondering is, if you could copy or otherwise create the index as part of the second step. In step 5 we only deal with bytes and cannot parse those, but in step 2 we have a proper ZODB FileStorage object and can to some degree decide which transaction we want. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but also rather inefficient way. In my case it takes only about 15 minutes of rather heavy I/O to create the file, but for larger files this time usually goes up. From what I understand repozo does the following: 1. it opens the real file storage file as a read-only FileStorage 2. calculates the byte position of end of the last complete transaction 3. closes the file storage 4. opens the fs file as a normal binary file 5. copies over all bytes up to the calculated position into a temp file 6. closes the fs file and temp file and renames the temp file according to some timestamp What I'm wondering is, if you could copy or otherwise create the index as part of the second step. In step 5 we only deal with bytes and cannot parse those, but in step 2 we have a proper ZODB FileStorage object and can to some degree decide which transaction we want. This strategy should be workable for both full and incremental backups: the storage opened in step 1 has an attribute, '_index', which is an fsIndex instance. We should be able to save the index into the repository direcotry with a timestamped filename, something like:: index_file = os.path.splitext(dest)[0] + '.index' fs._index.save(fs._pas, index_file) At restore time, we would look for the index corresponding to the full or incremental being restored, and copy it into place followi[ng the successful copy of the data. As usual, the tedious part about implementing this feature will be adding / updating tests. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtV0EACgkQ+gerLs4ltQ4JFgCfYZ3PV4pZCP7NInG1lW36F0ke sRMAoL24vUmOIpb2W2XOps7I9TuxmGRk =aMi4 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: Hanno Schlichting wrote: On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but also rather inefficient way. In my case it takes only about 15 minutes of rather heavy I/O to create the file, but for larger files this time usually goes up. From what I understand repozo does the following: 1. it opens the real file storage file as a read-only FileStorage 2. calculates the byte position of end of the last complete transaction 3. closes the file storage 4. opens the fs file as a normal binary file 5. copies over all bytes up to the calculated position into a temp file 6. closes the fs file and temp file and renames the temp file according to some timestamp What I'm wondering is, if you could copy or otherwise create the index as part of the second step. In step 5 we only deal with bytes and cannot parse those, but in step 2 we have a proper ZODB FileStorage object and can to some degree decide which transaction we want. This strategy should be workable for both full and incremental backups: the storage opened in step 1 has an attribute, '_index', which is an fsIndex instance. We should be able to save the index into the repository direcotry with a timestamped filename, something like:: index_file = os.path.splitext(dest)[0] + '.index' fs._index.save(fs._pas, index_file) At restore time, we would look for the index corresponding to the full or incremental being restored, and copy it into place followi[ng the successful copy of the data. As usual, the tedious part about implementing this feature will be adding / updating tests. I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtX0gACgkQ+gerLs4ltQ4dWQCgs3Icg+omT7O3xfxCPoIiIyCm zuEAoKprV5ggwEx9xZHC7bWVX5Zfuogc =AKk0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
On Fri, May 14, 2010 at 4:33 PM, Tres Seaver tsea...@palladion.com wrote: I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Oh, awesome! Best possible outcome I could hope for - someone else wants to do the work for me ;-) Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] repozo full backup and index files
Hi. I was wondering if there's a specific reason why repozo does not backup or create index files, whenever it does a full backup. I understand that creating index files for incremental backups is probably hard, but for a full backup it should be possible. Recreating the index file after restoring a backup can take forever and increases the potential downtime quite a bit. So is this just something nobody ever had time to implement or are there technical problems with this? Thanks, Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
