Hello ZODB dev, I was recently trying to GC a large multi-database setup for the first time using zc.zodbdgc. The process wouldn't complete (or really even get started) because of an IndexError being thrown from `zc.zodbdgc.getrefs` (__init__.py line 287). As I traced through it, it began to look like the combination of `cPickle.Unpickler.noload` and multi-database persistent ids (which in ZODB are list objects) fails, generating an empty list instead of the expected [ref type, args] list documented in `ZODB.serialize`. This makes it impossible to correctly GC a multi-database.
I was curious if anyone else had seen this, or maybe I'm just doing something wrong? We solved our problem by using `load` instead of `noload`, but I wondered if there might be a better way? Details: I'm working under Python 2.7.6 and 2.7.3 with ZODB 4.0.0, zc.zodbdgc 0.6.1 and eventually zodbpickle 0.5.2. Most of my results were repeated on both Mac OS X and Linux. After hitting the IndexError, I began debugging the problem. When it became clear that the persistent_load callback was simply getting the wrong persistent ids passed to it (empty lists instead of complete multi-db refs), I tried swapping in zodbpickle for the stock cPickle to the same effect. Here's some code demonstrating the problem: This pickle data came right out of ZODB, captured during a debug session of zc.zodbdgc. It has three persistent ids, two cross database and one in the same database: >>> p = 'cBTrees.OOBTree\nOOBTree\nq\x01.((((X\x0c\x00\x00\x00Users_1_Prodq\x02]q\x03(U\x01m(U\x0cUsers_1_Prodq\x04U\x08\x00\x00\x00\x00\x00\x00\x00\x01q\x05czope.site.folder\nFolder\nq\x06tq\x07eQX\x0c\x00\x00\x00Users_2_Prodq\x08]q\t(U\x01m(U\x0cUsers_2_Prodq\nU\x08\x00\x00\x00\x00\x00\x00\x00\x01q\x0bh\x06tq\x0ceQX\x0b\x00\x00\x00dataserver2q\r(U\x08\x00\x00\x00\x00\x00\x00\x00\x10q\x0eh\x06tQttttq\x0f.' This code is copy-and-pasted out of zc.zodbgc getrefs. It's supposed to find all the persistent refs and put them inside the `refs` list: >>> import cPickle >>> import cStringIO >>> refs =  >>> u = cPickle.Unpickler(cStringIO.StringIO(p)) >>> u.persistent_load = refs >>> u.noload() >>> u.noload() But if we look at `refs`, we see that the first two cross-database refs are returned as empty lists, not the correct value: >>> refs [, , ('\x00\x00\x00\x00\x00\x00\x00\x10', None)] If instead we use `load` to read the state, we get the correct references: >>> refs =  >>> u = cPickle.Unpickler(cStringIO.StringIO(p)) >>> u.persistent_load = refs >>> u.noload() >>> u.load() >>> refs [['m', ('Users_1_Prod', '\x00\x00\x00\x00\x00\x00\x00\x01', <class 'zope.site.folder.Folder'>)], ['m', ('Users_2_Prod', '\x00\x00\x00\x00\x00\x00\x00\x01', <class 'zope.site.folder.Folder'>)], ('\x00\x00\x00\x00\x00\x00\x00\x10', <class 'zope.site.folder.Folder'>)] The results are the same using zodbpickle or using an actual callback function instead of the append-directly-to-list shortcut. If we fix the IndexError by checking the size of the list first, we miss all the cross-db references, meaning that a GC is going to be too aggressive. But using `load` is slower and requires access to all of the classes referenced. If anyone has run into this before or has other suggestions, I'd appreciate hearing them. Thanks, Jason _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev