Re: [ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
On Thu, Feb 4, 2010 at 7:22 AM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jim Fulton wrote: >> Another issue is that, as things are now, while an index is being saved, no transactions can be committed. This is less serious now that saving indexes is much faster, however, saving a large index may still take several seconds, and that might be a problem for some applications. I'd be happy to add an option to save indexes after every N records written, where N would be given by the option. >>> So what is the difference between such a new option and fixing the >>> "flawed" >>> algorithm as mentioned above? >> >> There would be explicit control. You's be able to say how often (in >> terms of records written) you'd save indexes. > > Assuming we could get this feature with ZODB 3.10 (perhaps with a > funding).. > could ZODB 3.8 clients (Zope 2.11) safely talk with a ZODB 3.10 ZEO > server? Yes. The only hitch is that 3.10 servers will not support undo for older clients. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > >>> Another issue is that, as things are now, while an index is being >>> saved, no transactions can be committed. This is less serious now >>> that saving indexes is much faster, however, saving a large index >>> may still take several seconds, and that might be a problem for >>> some applications. >>> >>> I'd be happy to add an option to save indexes after every N >>> records written, where N would be given by the option. >> So what is the difference between such a new option and fixing the >> "flawed" >> algorithm as mentioned above? > > There would be explicit control. You's be able to say how often (in > terms of records written) you'd save indexes. Assuming we could get this feature with ZODB 3.10 (perhaps with a funding).. could ZODB 3.8 clients (Zope 2.11) safely talk with a ZODB 3.10 ZEO server? Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktqu/sACgkQCJIWIbr9KYzQhwCdFvlwf60TovhujXkchwAFp/Up lfoAoLbHUp1BI/3l/GogvG7PH5FpJFPh =7XhT -END PGP SIGNATURE- <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
On Thu, Feb 4, 2010 at 1:12 AM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jim Fulton wrote: >> On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung >> wrote: ... >> Also, note that yesterday, I checked in a change to file-storage >> index saving logic and file format that makes index saving and >> loading *much* faster. >> > Would it be easy to backport this to ZODB (3.8) (we could eventually > handle this)? It would be easy to backport it as a local patch. I wouldn't want to release it in 3.8, as it changes the index format. I wouldn't want a situation where 3.9 releases couldn't read indexes created with 3.8. >>> So at startup time the complete index files had to be recreated >>> which takes pretty long for our storages. >>> >>> The question came up if it would be possible to write regular >>> snapshots of the in-memory index back to disk (e.g. every 15 >>> minutes) and then to modify the ZEO startup procedure to >>> optionally start a ZEO server with such a snapshoted index file? >> >> What is the difference between an index and a snapshot? >> >> > In our understanding the in-memory index is written back to disk only > at ZEO shutdown time (proper closing of the Filestorage). A snapshot > would be written to disk regularly while the Filestorage is open. The index is also written when the database is packed. The only limitation to writing the index while the file storage is open is that we can't commit transactions while writing the index. ... >> Yes, FileStorage actually has some logic now to save indexes >> periodically. Unfortunately, the algorithm is flawed and almost >> never fires. > Aha - this explains the experience discussed above. But this information > about peridically savings is actually new to me (and perhaps many > other people). The information is moot, since the algorithm never fires. :) >> Another issue is that, as things are now, while an index is being >> saved, no transactions can be committed. This is less serious now >> that saving indexes is much faster, however, saving a large index >> may still take several seconds, and that might be a problem for >> some applications. >> >> I'd be happy to add an option to save indexes after every N >> records written, where N would be given by the option. > So what is the difference between such a new option and fixing the > "flawed" > algorithm as mentioned above? There would be explicit control. You's be able to say how often (in terms of records written) you'd save indexes. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung > wrote: >> we had in the past several issues with hour-long outages of our >> ZEO servers. The reasons are pretty easy: with have several big >> storages (60-80GB) and in case of a cluster failures we were not >> able to shutdown the ZEO servers in a clean way. > > Note that we had a similar problem until we realized that zdaemon > gets impatient and sends a KILL signal to a process shutting down > after 10 seconds. We have since set the zdaemon backoff-limit to > 300 which gives our servers plenty of time to shut down cleanly. oh - that's interesting > > Also, note that yesterday, I checked in a change to file-storage > index saving logic and file format that makes index saving and > loading *much* faster. > Would it be easy to backport this to ZODB (3.8) (we could eventually handle this)? >> So at startup time the complete index files had to be recreated >> which takes pretty long for our storages. >> >> The question came up if it would be possible to write regular >> snapshots of the in-memory index back to disk (e.g. every 15 >> minutes) and then to modify the ZEO startup procedure to >> optionally start a ZEO server with such a snapshoted index file? > > What is the difference between an index and a snapshot? > > In our understanding the in-memory index is written back to disk only at ZEO shutdown time (proper closing of the Filestorage). A snapshot would be written to disk regularly while the Filestorage is open. >> This may cause some data loss (transactions between the last >> snapshot and the crash would be ignored - or perhaps the ZEO >> server could take the snapshot file and read all transactions >> happened after the snapshot until the crash in order to update >> the index with the completed transactions after the snapshot... > > That's what happens when reading indexes now. Of course, an index > *is* just a snapshot. Sure but ZEO/Filestorage does not written the in-memory index as snaphot to disk - except at shutdown - or? >> Just some ideas...would that be doable? > > Yes, FileStorage actually has some logic now to save indexes > periodically. Unfortunately, the algorithm is flawed and almost > never fires. Aha - this explains the experience discussed above. But this information about peridically savings is actually new to me (and perhaps many other people). > > Another issue is that, as things are now, while an index is being > saved, no transactions can be committed. This is less serious now > that saving indexes is much faster, however, saving a large index > may still take several seconds, and that might be a problem for > some applications. > > I'd be happy to add an option to save indexes after every N > records written, where N would be given by the option. So what is the difference between such a new option and fixing the "flawed" algorithm as mentioned above? Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktqZV8ACgkQCJIWIbr9KYxRzQCff5/ZTVOXHx/msZS5VB+JT66L 6S8AnjLAYOlju2bIHFDnST4rOtLqd4eK =ggTh -END PGP SIGNATURE- <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung wrote: > we had in the past several issues with hour-long outages of our > ZEO servers. The reasons are pretty easy: with have several big > storages (60-80GB) and in case of a cluster failures we were not able > to shutdown the ZEO servers in a clean way. Note that we had a similar problem until we realized that zdaemon gets impatient and sends a KILL signal to a process shutting down after 10 seconds. We have since set the zdaemon backoff-limit to 300 which gives our servers plenty of time to shut down cleanly. Also, note that yesterday, I checked in a change to file-storage index saving logic and file format that makes index saving and loading *much* faster. > So at startup time the > complete index files had to be recreated which takes pretty long > for our storages. > > The question came up if it would be possible to write regular snapshots > of the in-memory index back to disk (e.g. every 15 minutes) and then > to modify > the ZEO startup procedure to optionally start a ZEO server with such a > snapshoted > index file? What is the difference between an index and a snapshot? > This may cause some data loss (transactions between the last > snapshot and the crash would be ignored - or perhaps the ZEO server > could take > the snapshot file and read all transactions happened after the > snapshot until > the crash in order to update the index with the completed transactions > after > the snapshot... That's what happens when reading indexes now. Of course, an index *is* just a snapshot. > When I talk of ZEO here then this related almost everything to > FileStorage > in generell since the zodb index file is a FIleStorage feature. Yup. > Just some ideas...would that be doable? Yes, FileStorage actually has some logic now to save indexes periodically. Unfortunately, the algorithm is flawed and almost never fires. Another issue is that, as things are now, while an index is being saved, no transactions can be committed. This is less serious now that saving indexes is much faster, however, saving a large index may still take several seconds, and that might be a problem for some applications. I'd be happy to add an option to save indexes after every N records written, where N would be given by the option. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, we had in the past several issues with hour-long outages of our ZEO servers. The reasons are pretty easy: with have several big storages (60-80GB) and in case of a cluster failures we were not able to shutdown the ZEO servers in a clean way. So at startup time the complete index files had to be recreated which takes pretty long for our storages. The question came up if it would be possible to write regular snapshots of the in-memory index back to disk (e.g. every 15 minutes) and then to modify the ZEO startup procedure to optionally start a ZEO server with such a snapshoted index file? This may cause some data loss (transactions between the last snapshot and the crash would be ignored - or perhaps the ZEO server could take the snapshot file and read all transactions happened after the snapshot until the crash in order to update the index with the completed transactions after the snapshot... When I talk of ZEO here then this related almost everything to FileStorage in generell since the zodb index file is a FIleStorage feature. Just some ideas...would that be doable? Andreas - -- ZOPYX Limited \ zopyx group Charlottenstr. 37/1 \ The full-service network for your D-72070 Tübingen \ Python, Zope and Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyxgroup.com - E-Publishing, Python, Zope & Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktpNlcACgkQCJIWIbr9KYwRaACbBcyT/QTWgMhIHL6Viw56rL7e v+sAn3sCqUkBaVy6aT6AF03+VJ2rl6ZF =elE4 -END PGP SIGNATURE- <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
