I'm sorry I haven't had time to look at this. Still don't really.
Thanks Marius!!!
On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas <mar...@gedmin.as> wrote:
> On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
>> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
>> > Recap: last week I examined problems I had packing our 4GB users
>> > storage.
> ...
>> >>> unp = pickle.Unpickler(f)
>> >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>> >>> pprint.pprint(unp.load())
>> {'data': {"<persistent reference ['m', ('game',
>> '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', <class '__main__.Tool'>)]>": 1,
>> "<persistent reference ['m', ('game',
>> '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class
>> '__main__.EnergyPack'>)]>": 1}}
Note the reference to __main__. This is almost certainly the root problem.
Classes shouldn't be defined in __main__ (except when experimenting).
At one time, I thought pickle disallowed pickling classes from __main__.
ZODB probably should. It's a bug magnet.
>>
>> Those look like cross-database references to me.
>>
>> The original error (aaaugh Mutt makes it hard for me to look upthread
>> while I'm writing a response) was something about non-hashable lists?
>> Looks like a piece of code is trying to put persistent references into a
>> dict, which can't possibly work in all cases.
> ...
>> > During my checks I realized that running the pack in a Python 2.7
>> > environment (using the same ZODB version - 3.10.3) works fine, the
>> > pack reduces our 4GB storage to 1GB. But our production server uses
>> > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
>> > had been done on OS X 10.7.3 - 64bit, and the production server is
>> > Debian Squeeze 32bit).
>>
>> I've no idea why running the same ZODB version on Python 2.7 instead of
>> 2.6 would make this error go away.
>
> Duh! The code that fails is in the standard library -- in the cPickle
> module:
>
>> > Traceback (most recent call last):
> ...
>> > File
>> > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py",
>> > line 328, in findrefs
>> > return self.referencesf(self._file.read(dh.plen))
>> > File
>> > "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py",
>> > line 630, in referencesf
>> > u.noload()
>> > TypeError: unhashable type: 'list'
>
> Since the bug is in the stdlib, it's not surprising that the newer
> stdlib cPickle from Python 2.7 fixes it.
I suspect a bug in the application (defining persistent classes in __main__)
is the root problem that's aggravated by the cPickle problem.
> That was fun.
I'm glad. :)
Thanks.
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
