Re: [ZODB-Dev] zeopack error
Hi there, Recap: last week I examined problems I had packing our 4GB users storage. With Martijn's help I was able to fix zeo's exception output and write out the first broken pickle that throws an exception. During my checks I realized that running the pack in a Python 2.7 environment (using the same ZODB version - 3.10.3) works fine, the pack reduces our 4GB storage to 1GB. But our production server uses Python 2.6 (same ZODB3.10.3) which yields the problem (though the test had been done on OS X 10.7.3 - 64bit, and the production server is Debian Squeeze 32bit). Currently I see only three ways to resolve that problem: - Upgrade production server from Python 2.6 to Python 2.7 which would probably solve the current problem (if it's really a Python 2.6/2.7 problem). - Pack the storage with Python 2.7/ZODB3.10.3 on the working system and deploy it back to the Python 2.6 production environment - very time consuming (because I have to download/pack/upload a big storage and during that time the production server wouldn't be available to avoid changes during that time)/ potentially risky(?). - Find out what is broken regarding that pickle and possibly the Python 2.6/ZODB3.10.3 combination and deploy a fix. You can download the broken pickle from here: http://www.reversepanda.com/download/brokenpickle If someone has more experience in parsing and understanding pickles in regards to ZODB3, any help would be appreciated. Kind regards, Kaweh ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack error
On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
Recap: last week I examined problems I had packing our 4GB users
storage. With Martijn's help I was able to fix zeo's exception output
and write out the first broken pickle that throws an exception.
...
You can download the broken pickle from here:
http://www.reversepanda.com/download/brokenpickle
If someone has more experience in parsing and understanding pickles in
regards to ZODB3, any help would be appreciated.
I don't have much experience here, but I love a puzzle
import pickletools
f = open('brokenpickle', 'rb')
A ZODB record consists of two pickles: the first stores the class of
the object, the other stores the state of the object
pickletools.dis(f)
0: cGLOBAL 'rp.odb.containers EntityMapping'
33: qBINPUT 1
35: .STOP
highest protocol among opcodes = 1
pickletools.dis(f)
36: }EMPTY_DICT
37: qBINPUT 2
39: USHORT_BINSTRING 'data'
45: qBINPUT 3
47: }EMPTY_DICT
48: qBINPUT 4
50: (MARK
51: ]EMPTY_LIST
52: qBINPUT 5
54: (MARK
55: USHORT_BINSTRING 'm'
58: (MARK
59: USHORT_BINSTRING 'game'
65: qBINPUT 6
67: USHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\tT'
77: qBINPUT 7
79: cGLOBAL 'game.objects.item Tool'
103: qBINPUT 8
105: tTUPLE (MARK at 58)
106: qBINPUT 9
108: eAPPENDS(MARK at 54)
109: QBINPERSID
110: KBININT11
112: ]EMPTY_LIST
113: qBINPUT 10
115: (MARK
116: USHORT_BINSTRING 'm'
119: (MARK
120: hBINGET 6
122: USHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\x12\x03'
132: qBINPUT 11
134: cGLOBAL 'game.objects.item EnergyPack'
164: qBINPUT 12
166: tTUPLE (MARK at 119)
167: qBINPUT 13
169: eAPPENDS(MARK at 115)
170: QBINPERSID
171: KBININT11
173: uSETITEMS (MARK at 50)
174: sSETITEM
175: .STOP
highest protocol among opcodes = 1
No secret calls to instantiate 'os.system' with 'rm -rf' as an argument,
so I feel safe to try and unpickle it ;-)
import sys, pickle, pprint
sys.modules['rp.odb.containers'] = sys.modules['__main__'] # hack
sys.modules['rp.odb'] = sys.modules['__main__'] # hack
sys.modules['rp'] = sys.modules['__main__'] # hack
class EntityMapping(object): pass
...
f.seek(0)
pickle.load(f)
class '__main__.EntityMapping'
(this is a good place to do a f.tell() and remember the position -- 36
in this case -- so you can f.seek(36) as you iterate trying to make the
second pickle load)
sys.modules['game.objects.item'] = sys.modules['__main__'] # hack
sys.modules['game.objects'] = sys.modules['__main__'] # hack
sys.modules['game'] = sys.modules['__main__'] # hack
class Tool(object): pass
...
class EnergyPack(object): pass
...
unp = pickle.Unpickler(f)
unp.persistent_load = lambda oid: 'persistent reference %r' % oid
pprint.pprint(unp.load())
{'data': {persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\tT', class '__main__.Tool')]: 1,
persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', class '__main__.EnergyPack')]:
1}}
Those look like cross-database references to me.
The original error (aaaugh Mutt makes it hard for me to look upthread
while I'm writing a response) was something about non-hashable lists?
Looks like a piece of code is trying to put persistent references into a
dict, which can't possibly work in all cases.
See ZODB.serialize.ObjectReader._persistent_load for the canonical
parser of the various possible formats.
ZODB.ConflictResolution.PersistentReference.__init__ is much clearer,
though perhaps a tiny bit less canonical.
During my checks I realized that running the pack in a Python 2.7
environment (using the same ZODB version - 3.10.3) works fine, the
pack reduces our 4GB storage to 1GB. But our production server uses
Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
had been done on OS X 10.7.3 - 64bit, and the production server is
Debian Squeeze 32bit).
I've no idea why running the same ZODB version on Python 2.7 instead of
2.6 would make this error go away.
Incidentally, since you use cross-database references, please make sure
they continue to work after you pack your storage. I've lost data
Re: [ZODB-Dev] zeopack error
On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
Recap: last week I examined problems I had packing our 4GB users
storage.
...
unp = pickle.Unpickler(f)
unp.persistent_load = lambda oid: 'persistent reference %r' % oid
pprint.pprint(unp.load())
{'data': {persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\tT', class '__main__.Tool')]: 1,
persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', class
'__main__.EnergyPack')]: 1}}
Those look like cross-database references to me.
The original error (aaaugh Mutt makes it hard for me to look upthread
while I'm writing a response) was something about non-hashable lists?
Looks like a piece of code is trying to put persistent references into a
dict, which can't possibly work in all cases.
...
During my checks I realized that running the pack in a Python 2.7
environment (using the same ZODB version - 3.10.3) works fine, the
pack reduces our 4GB storage to 1GB. But our production server uses
Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
had been done on OS X 10.7.3 - 64bit, and the production server is
Debian Squeeze 32bit).
I've no idea why running the same ZODB version on Python 2.7 instead of
2.6 would make this error go away.
Duh! The code that fails is in the standard library -- in the cPickle
module:
Traceback (most recent call last):
...
File
/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py,
line 328, in findrefs
return self.referencesf(self._file.read(dh.plen))
File
/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py,
line 630, in referencesf
u.noload()
TypeError: unhashable type: 'list'
Since the bug is in the stdlib, it's not surprising that the newer
stdlib cPickle from Python 2.7 fixes it.
Looks like it was this one: http://bugs.python.org/issue1101399
(hg changeset 56532:d0f005e6fadd)
When I diff Modules/cPickle.c between the 2.6 and 2.7 branches in the hg
repository, I see that noload() handles the APPEND, APPENDS, SETITEM and
SETITEMS differently in 2.7. hg blame attributes that change to 56532.
The SETITEM/SETITEMS opcode in the pickle now no longer tries to
actually stuff anything into any dicts, which is why the list never ends
up being hashed, which explains the error and why it is gone.
The fix is present in 2.7, but not in 2.6
That was fun.
Marius Gedminas
--
Critical sections are Windows's lightweight alternative to mutexes, thus
cleverly getting more use out of a previously well-defined phrase by using it
to mean something related to but slightly different from the earlier general
understanding
-- Verity Stob
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
