Hash: SHA1

Jim Fulton wrote:
> On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung <li...@zopyx.com>
> wrote:
>> we had in the past several issues with hour-long outages of our
>> ZEO servers. The reasons are pretty easy: with have several big
>> storages (60-80GB) and in case of a cluster failures we were not
>> able to shutdown the ZEO servers in a clean way.
> Note that we had a similar problem until we realized that zdaemon
> gets impatient and sends a KILL signal to a process shutting down
> after 10 seconds.  We have since set the zdaemon backoff-limit to
> 300 which gives our servers plenty of time to shut down cleanly.
oh - that's interesting
> Also, note that yesterday, I checked in a change to file-storage
> index saving logic and file format that makes index saving and
> loading *much* faster.
Would it be easy to backport this to ZODB (3.8) (we could eventually
handle this)?
>> So at startup time the complete index files had to be recreated
>> which takes pretty long for our storages.
>> The question came up if it would be possible to write regular
>> snapshots of the in-memory index back to disk (e.g. every 15
>> minutes) and then to modify the ZEO startup procedure to
>> optionally start a ZEO server with such a snapshoted index file?
> What is the difference between an index and a snapshot?
In our understanding the in-memory index is written back to disk only
at ZEO shutdown time (proper closing of the Filestorage). A snapshot
would be written to disk regularly while the Filestorage is open.
>> This may cause some data loss (transactions between the last
>> snapshot and the crash would be ignored - or perhaps the ZEO
>> server could take the snapshot file and read all transactions
>> happened after the snapshot until the crash in order to update
>> the index with the completed transactions after the snapshot...
> That's what happens when reading indexes now. Of course, an index
> *is* just a snapshot.

Sure but ZEO/Filestorage does not written the in-memory index
as snaphot to disk - except at shutdown - or?
>> Just some ideas...would that be doable?
> Yes, FileStorage actually has some logic now to save indexes
> periodically. Unfortunately, the algorithm is flawed and almost
> never fires.
Aha - this explains the experience discussed above. But this information
about peridically savings is actually new to me (and perhaps many
other people).

> Another issue is that, as things are now, while an index is being
> saved, no transactions can be committed.  This is less serious now
> that saving indexes is much faster, however, saving a large index
> may still take several seconds, and that might be a problem for
> some applications.
> I'd be happy to add an option to save indexes after every N
> records written, where N would be given by the option.
So what is the difference between such a new option and fixing the
algorithm as mentioned above?


Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/


<<attachment: lists.vcf>>

For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to