Re: [ZODB-Dev] zeopack error
On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote: > On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote: > > Recap: last week I examined problems I had packing our 4GB users > > storage. ... > >>> unp = pickle.Unpickler(f) > >>> unp.persistent_load = lambda oid: '' % oid > >>> pprint.pprint(unp.load()) > {'data': {" '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', )]>": 1, > " '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', '__main__.EnergyPack'>)]>": 1}} > > Those look like cross-database references to me. > > The original error (aaaugh Mutt makes it hard for me to look upthread > while I'm writing a response) was something about non-hashable lists? > Looks like a piece of code is trying to put persistent references into a > dict, which can't possibly work in all cases. ... > > During my checks I realized that running the pack in a Python 2.7 > > environment (using the same ZODB version - 3.10.3) works fine, the > > pack reduces our 4GB storage to 1GB. But our production server uses > > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test > > had been done on OS X 10.7.3 - 64bit, and the production server is > > Debian Squeeze 32bit). > > I've no idea why running the same ZODB version on Python 2.7 instead of > 2.6 would make this error go away. Duh! The code that fails is in the standard library -- in the cPickle module: > > Traceback (most recent call last): ... > > File > > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", > > line 328, in findrefs > > return self.referencesf(self._file.read(dh.plen)) > > File > > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py", > > line 630, in referencesf > > u.noload() > > TypeError: unhashable type: 'list' Since the bug is in the stdlib, it's not surprising that the newer stdlib cPickle from Python 2.7 fixes it. Looks like it was this one: http://bugs.python.org/issue1101399 (hg changeset 56532:d0f005e6fadd) When I diff Modules/cPickle.c between the 2.6 and 2.7 branches in the hg repository, I see that noload() handles the APPEND, APPENDS, SETITEM and SETITEMS differently in 2.7. hg blame attributes that change to 56532. The SETITEM/SETITEMS opcode in the pickle now no longer tries to actually stuff anything into any dicts, which is why the list never ends up being hashed, which explains the error and why it is gone. The fix is present in 2.7, but not in 2.6 That was fun. Marius Gedminas -- Critical sections are Windows's lightweight alternative to mutexes, thus cleverly getting more use out of a previously well-defined phrase by using it to mean something related to but slightly different from the earlier general understanding -- Verity Stob signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack error
On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote: > Recap: last week I examined problems I had packing our 4GB users > storage. With Martijn's help I was able to fix zeo's exception output > and write out the first broken pickle that throws an exception. ... > You can download the broken pickle from here: > http://www.reversepanda.com/download/brokenpickle > > If someone has more experience in parsing and understanding pickles in > regards to ZODB3, any help would be appreciated. I don't have much experience here, but I love a puzzle >>> import pickletools >>> f = open('brokenpickle', 'rb') A ZODB record consists of two pickles: the first stores the class of the object, the other stores the state of the object >>> pickletools.dis(f) 0: cGLOBAL 'rp.odb.containers EntityMapping' 33: qBINPUT 1 35: .STOP highest protocol among opcodes = 1 >>> pickletools.dis(f) 36: }EMPTY_DICT 37: qBINPUT 2 39: USHORT_BINSTRING 'data' 45: qBINPUT 3 47: }EMPTY_DICT 48: qBINPUT 4 50: (MARK 51: ]EMPTY_LIST 52: qBINPUT 5 54: (MARK 55: USHORT_BINSTRING 'm' 58: (MARK 59: USHORT_BINSTRING 'game' 65: qBINPUT 6 67: USHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\tT' 77: qBINPUT 7 79: cGLOBAL 'game.objects.item Tool' 103: qBINPUT 8 105: tTUPLE (MARK at 58) 106: qBINPUT 9 108: eAPPENDS(MARK at 54) 109: QBINPERSID 110: KBININT11 112: ]EMPTY_LIST 113: qBINPUT 10 115: (MARK 116: USHORT_BINSTRING 'm' 119: (MARK 120: hBINGET 6 122: USHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\x12\x03' 132: qBINPUT 11 134: cGLOBAL 'game.objects.item EnergyPack' 164: qBINPUT 12 166: tTUPLE (MARK at 119) 167: qBINPUT 13 169: eAPPENDS(MARK at 115) 170: QBINPERSID 171: KBININT11 173: uSETITEMS (MARK at 50) 174: sSETITEM 175: .STOP highest protocol among opcodes = 1 No secret calls to instantiate 'os.system' with 'rm -rf' as an argument, so I feel safe to try and unpickle it ;-) >>> import sys, pickle, pprint >>> sys.modules['rp.odb.containers'] = sys.modules['__main__'] # hack >>> sys.modules['rp.odb'] = sys.modules['__main__'] # hack >>> sys.modules['rp'] = sys.modules['__main__'] # hack >>> class EntityMapping(object): pass ... >>> f.seek(0) >>> pickle.load(f) (this is a good place to do a f.tell() and remember the position -- 36 in this case -- so you can f.seek(36) as you iterate trying to make the second pickle load) >>> sys.modules['game.objects.item'] = sys.modules['__main__'] # hack >>> sys.modules['game.objects'] = sys.modules['__main__'] # hack >>> sys.modules['game'] = sys.modules['__main__'] # hack >>> class Tool(object): pass ... >>> class EnergyPack(object): pass ... >>> unp = pickle.Unpickler(f) >>> unp.persistent_load = lambda oid: '' % oid >>> pprint.pprint(unp.load()) {'data': {")]>": 1, ")]>": 1}} Those look like cross-database references to me. The original error (aaaugh Mutt makes it hard for me to look upthread while I'm writing a response) was something about non-hashable lists? Looks like a piece of code is trying to put persistent references into a dict, which can't possibly work in all cases. See ZODB.serialize.ObjectReader._persistent_load for the canonical parser of the various possible formats. ZODB.ConflictResolution.PersistentReference.__init__ is much clearer, though perhaps a tiny bit less canonical. > During my checks I realized that running the pack in a Python 2.7 > environment (using the same ZODB version - 3.10.3) works fine, the > pack reduces our 4GB storage to 1GB. But our production server uses > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test > had been done on OS X 10.7.3 - 64bit, and the production server is > Debian Squeeze 32bit). I've no idea why running the same ZODB version on Python 2.7 instead of 2.6 would make this error go away. Incidentally, since you use cross-database references, please make sure they continue to work after you pack your storage. I've lost data that way (the ZODB garbage collector doesn't see references that exist in other storages, and can assume objects are garbage when it shouldn't). Packing with GC disabled ought to b
Re: [ZODB-Dev] zeopack error
Hi there, Recap: last week I examined problems I had packing our 4GB users storage. With Martijn's help I was able to fix zeo's exception output and write out the first broken pickle that throws an exception. During my checks I realized that running the pack in a Python 2.7 environment (using the same ZODB version - 3.10.3) works fine, the pack reduces our 4GB storage to 1GB. But our production server uses Python 2.6 (same ZODB3.10.3) which yields the problem (though the test had been done on OS X 10.7.3 - 64bit, and the production server is Debian Squeeze 32bit). Currently I see only three ways to resolve that problem: - Upgrade production server from Python 2.6 to Python 2.7 which would probably solve the current problem (if it's really a Python 2.6/2.7 problem). - Pack the storage with Python 2.7/ZODB3.10.3 on the working system and deploy it back to the Python 2.6 production environment - very time consuming (because I have to download/pack/upload a big storage and during that time the production server wouldn't be available to avoid changes during that time)/ potentially risky(?). - Find out what is broken regarding that pickle and possibly the Python 2.6/ZODB3.10.3 combination and deploy a fix. You can download the broken pickle from here: http://www.reversepanda.com/download/brokenpickle If someone has more experience in parsing and understanding pickles in regards to ZODB3, any help would be appreciated. Kind regards, Kaweh ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev