-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote: > On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung <li...@zopyx.com> > wrote: >> we had in the past several issues with hour-long outages of our >> ZEO servers. The reasons are pretty easy: with have several big >> storages (60-80GB) and in case of a cluster failures we were not >> able to shutdown the ZEO servers in a clean way. > > Note that we had a similar problem until we realized that zdaemon > gets impatient and sends a KILL signal to a process shutting down > after 10 seconds. We have since set the zdaemon backoff-limit to > 300 which gives our servers plenty of time to shut down cleanly. oh - that's interesting > > Also, note that yesterday, I checked in a change to file-storage > index saving logic and file format that makes index saving and > loading *much* faster. > Would it be easy to backport this to ZODB (3.8) (we could eventually handle this)? >> So at startup time the complete index files had to be recreated >> which takes pretty long for our storages. >> >> The question came up if it would be possible to write regular >> snapshots of the in-memory index back to disk (e.g. every 15 >> minutes) and then to modify the ZEO startup procedure to >> optionally start a ZEO server with such a snapshoted index file? > > What is the difference between an index and a snapshot? > > In our understanding the in-memory index is written back to disk only at ZEO shutdown time (proper closing of the Filestorage). A snapshot would be written to disk regularly while the Filestorage is open. >> This may cause some data loss (transactions between the last >> snapshot and the crash would be ignored - or perhaps the ZEO >> server could take the snapshot file and read all transactions >> happened after the snapshot until the crash in order to update >> the index with the completed transactions after the snapshot... > > That's what happens when reading indexes now. Of course, an index > *is* just a snapshot.
Sure but ZEO/Filestorage does not written the in-memory index as snaphot to disk - except at shutdown - or? >> Just some ideas...would that be doable? > > Yes, FileStorage actually has some logic now to save indexes > periodically. Unfortunately, the algorithm is flawed and almost > never fires. Aha - this explains the experience discussed above. But this information about peridically savings is actually new to me (and perhaps many other people). > > Another issue is that, as things are now, while an index is being > saved, no transactions can be committed. This is less serious now > that saving indexes is much faster, however, saving a large index > may still take several seconds, and that might be a problem for > some applications. > > I'd be happy to add an option to save indexes after every N > records written, where N would be given by the option. So what is the difference between such a new option and fixing the "flawed" algorithm as mentioned above? Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktqZV8ACgkQCJIWIbr9KYxRzQCff5/ZTVOXHx/msZS5VB+JT66L 6S8AnjLAYOlju2bIHFDnST4rOtLqd4eK =ggTh -----END PGP SIGNATURE-----
<<attachment: lists.vcf>>
_______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
