Thanks for pursuing this.
I have lots of other ideas now, but I don't know which to pursue. I need a
> lot more information. It would be helpful if you sent me your database to
> analyze. Some possible causes:
> - Have you looked for filesystem-level corruption yet? I asked this before
> and I am waiting for an answer.
Yep, I've checked for file system consistency and Mysql consistency without
any error reported.
> - Although there is a pack lock, that lock unfortunately gets released
> automatically if MySQL disconnects prematurely. Therefore, it is possible
> to force RelStorage to run multiple pack operations in parallel, which would
> have unpredictable effects. Is there any possibility that you accidentally
> ran multiple pack operations in parallel? For example, maybe you have a
> cron job, or you were setting up a cron job at the time, and you started a
> pack while the cron job was running. (Normally, any attempt to start
> parallel pack operations will just generate an error, but if MySQL
> disconnects in just the right way, you'll get a mess.)
That's not unlikely! I've actually seen traces of packing invoked TTW,
however the cron job uses zodbpack. I will try to figure out if the PosKeys
starts to surface right after that.
> - Every SQL database has nasty surprises. Oracle, for example, has a nice
> "read only" mode, but it turns out that mode works differently in RAC
> environments, leading to silent corruption. As a result, we never use that
> feature of Oracle anymore. Maybe MySQL has some nasty surprises I haven't
> yet discovered; maybe the MySQL-specific "delete using" statement doesn't
> work as expected.
That could also be the case. In fact we have also seen Mysql locking up
longer than expected, but that's another story.
> - Applications can accidentally cause POSKeyErrors in a variety of ways.
> For example, persistent objects cached globally can cause POSKeyErrors.
> Maybe Plone 4 or some add-on uses ZODB incorrectly.
I was not aware of that.
Next step here would probably be to inspect log files further and grab a
copy of the dabase before PosKeys started to appear and see if it is
possible to recreate the incident.
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org