On 9 February 2012 11:24, Jim Fulton <j...@zope.com> wrote:
> I'm sorry I haven't had time to look at this. Still don't really.
>
> Thanks Marius!!!
>
> On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas <mar...@gedmin.as> wrote:
>> On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
>>> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
>>> > Recap: last week I examined problems I had packing our 4GB users
>>> > storage.
>> ...
>>> >>> unp = pickle.Unpickler(f)
>>> >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>>> >>> pprint.pprint(unp.load())
>>> {'data': {"<persistent reference ['m', ('game',
>>> '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', <class '__main__.Tool'>)]>": 1,
>>> "<persistent reference ['m', ('game',
>>> '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class
>>> '__main__.EnergyPack'>)]>": 1}}
>
> Note the reference to __main__. This is almost certainly the root problem.
> Classes shouldn't be defined in __main__ (except when experimenting).
>
> At one time, I thought pickle disallowed pickling classes from __main__.
> ZODB probably should. It's a bug magnet.
>
>
>>>
>>> Those look like cross-database references to me.
>>>
>>> The original error (aaaugh Mutt makes it hard for me to look upthread
>>> while I'm writing a response) was something about non-hashable lists?
>>> Looks like a piece of code is trying to put persistent references into a
>>> dict, which can't possibly work in all cases.
>> ...
>>> > During my checks I realized that running the pack in a Python 2.7
>>> > environment (using the same ZODB version - 3.10.3) works fine, the
>>> > pack reduces our 4GB storage to 1GB. But our production server uses
>>> > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
>>> > had been done on OS X 10.7.3 - 64bit, and the production server is
>>> > Debian Squeeze 32bit).
>>>
>>> I've no idea why running the same ZODB version on Python 2.7 instead of
>>> 2.6 would make this error go away.
>>
>> Duh! The code that fails is in the standard library -- in the cPickle
>> module:
>>
>>> > Traceback (most recent call last):
>> ...
>>> > File
>>> > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py",
>>> > line 328, in findrefs
>>> > return self.referencesf(self._file.read(dh.plen))
>>> > File
>>> > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py",
>>> > line 630, in referencesf
>>> > u.noload()
>>> > TypeError: unhashable type: 'list'
>>
>> Since the bug is in the stdlib, it's not surprising that the newer
>> stdlib cPickle from Python 2.7 fixes it.
>
> I suspect a bug in the application (defining persistent classes in __main__)
> is the root problem that's aggravated by the cPickle problem.
The pickle's classes were defined in a normal module, I think Marius
just aliased those to modules to __main__ and defined the classes
there in order to load the pickle without the original code:
>>> sys.modules['game.objects.item'] = sys.modules['__main__'] # hack
>>> sys.modules['game.objects'] = sys.modules['__main__'] # hack
>>> sys.modules['game'] = sys.modules['__main__'] # hack
Laurence
_______________________________________________
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
