We have an area of our product that can, depending on certain conditions,
produce lots of rapid transactions all at once. We’re getting many reports of
POSKeyErrors at sites where the transaction volume is higher than others. They
appear to coincide (in many cases, at least) with zodbpack runs.
After some investigation, it appears that it’s a timing issue. If an object is
unreferenced during the pre_pack() stage, it will be marked for GC. If it then
becomes referenced before the actual DELETE FROM is executed, it will be
deleted nonetheless and POSKeyErrors will result.
Now, I don’t know how the object is unreferenced and referenced in separate
transactions; as far as I can tell, there are no two-transaction moves or
anything like that happening. Nonetheless, it’s entirely possible we can solve
this by tracking down the code that sets up the race condition.
But is there any lower-level way around the race condition? A good amount of
our application is pluggable by the end user, and I’d like to avoid vague
technical warnings about transaction boundaries if possible :) If we could
verify that no POSKeyErrors will result from the DELETE before running it, that
would be much simpler.
I should also mention that running zodbpack with days=1 isn’t an
across-the-board solution for us. We have some customers for whom rapid growth
of the database is an issue, and they pack more frequently.
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org