[ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
I noticed my DB had swelled to 132 gigabytes (as of 3 days ago; it's 160 gigabytes today) and it seems to be because zeopack has started failing: tsa@sp2772c:~/db$ /home/tsa/env/bin/zeopack -u /home/tsa/db/zeo.sock Traceback (most recent call last): File

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
How do I go about fixing this? Let me know if I can provide any other information that would be helpful. I took the advice in this thread: https://mail.zope.org/pipermail/zodb-dev/2012-January/014526.html The exception that comes up, from the zeo server log, is: 2013-01-07T13:01:49 ERROR

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Jim Fulton
On Mon, Jan 7, 2013 at 1:04 PM, Claudiu Saftoiu csaft...@gmail.com wrote: How do I go about fixing this? Let me know if I can provide any other information that would be helpful. I took the advice in this thread: https://mail.zope.org/pipermail/zodb-dev/2012-January/014526.html The

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
I'm afraid this doesn't seem to help me figure out what's wrong... I suspect your database is corrupted. You'd probably want to look at the record in question to be sure. Sure, I've re-run the pack and will dump the pickled object to a file to inspect it - is that what you meant? (How

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Jim Fulton
I'm sorry I haven't had time to look at this. Still don't really. Thanks Marius!!! On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote: On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote: Recap:

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Laurence Rowe
On 9 February 2012 11:24, Jim Fulton j...@zope.com wrote: I'm sorry I haven't had time to look at this. Still don't really. Thanks Marius!!! On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote: On Wed, Feb

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Kaweh Kazemi
I suspect a bug in the application (defining persistent classes in __main__) is the root problem that's aggravated by the cPickle problem. The pickle's classes were defined in a normal module, I think Marius just aliased those to modules to __main__ and defined the classes there in order

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Marius Gedminas
On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: Thanks a lot for the investigation, Marius. That was awesome. Seems to me, I need to upgrade to Python 2.7 and be done with the problem then. :/ I wonder, though, if ZODB ought to use the 'noload' method of cPickle.Unpickler,

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Jim Fulton
On Thu, Feb 9, 2012 at 11:32 AM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: Thanks a lot for the investigation, Marius. That was awesome. Seems to me, I need to upgrade to Python 2.7 and be done with the problem then. :/ I wonder,

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread Darryl Dixon - Winterhouse Consulting
On Thu, Feb 9, 2012 at 11:32 AM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: Thanks a lot for the investigation, Marius. That was awesome. Seems to me, I need to upgrade to Python 2.7 and be done with the problem then. :/ I wonder,

Re: [ZODB-Dev] zeopack error

2012-02-09 Thread David Glick (GW)
On Feb 9, 2012, at 12:24 PM, Darryl Dixon - Winterhouse Consulting wrote: On Thu, Feb 9, 2012 at 11:32 AM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: Thanks a lot for the investigation, Marius. That was awesome. Seems to me, I need

Re: [ZODB-Dev] zeopack error

2012-02-08 Thread Kaweh Kazemi
Hi there, Recap: last week I examined problems I had packing our 4GB users storage. With Martijn's help I was able to fix zeo's exception output and write out the first broken pickle that throws an exception. During my checks I realized that running the pack in a Python 2.7 environment (using

Re: [ZODB-Dev] zeopack error

2012-02-08 Thread Marius Gedminas
On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote: Recap: last week I examined problems I had packing our 4GB users storage. With Martijn's help I was able to fix zeo's exception output and write out the first broken pickle that throws an exception. ... You can download the broken

Re: [ZODB-Dev] zeopack error

2012-02-08 Thread Marius Gedminas
On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote: On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote: Recap: last week I examined problems I had packing our 4GB users storage. ... unp = pickle.Unpickler(f) unp.persistent_load = lambda oid: 'persistent

Re: [ZODB-Dev] zeopack error

2012-02-01 Thread Kaweh Kazemi
To debug the pickle, I copied the 4.7GB users.fs to my local machine. I changed the code as proposed by Martijn, started zeo und zeopack - hours later - everything was packed without any troubles. I was puzzled. I then realized that I was running the whole thing with Python 2.7.1 on my local

Re: [ZODB-Dev] zeopack error

2012-01-31 Thread Kaweh Kazemi
Yes, I did that, unfortunately without avail. I don't get more than a None (see below). Our ZEO configuration inludes this: eventlog logfile path logs/zeo.log format [%(levelname)s] %(asctime)s: %(message)s /logfile /eventlog Do I miss something

Re: [ZODB-Dev] zeopack error

2012-01-31 Thread Martijn Pieters
On Tue, Jan 31, 2012 at 12:23, Kaweh Kazemi ka...@me.com wrote: [ERROR] 2012-01-31T09:20:25: (22455) Error raised in delayed method None Ugh, a quick look at the code reveals that this must've been raised by a SlowMethodThread handler, and it indeed doesn't provide the exception info in the log

Re: [ZODB-Dev] zeopack error

2012-01-31 Thread Martijn Pieters
On Tue, Jan 31, 2012 at 14:20, Kaweh Kazemi ka...@me.com wrote: I did the change and here we go: Right, that's a ZEO bug report right there then; my change should go into ZEO trunk. Jim, did you catch it? I have one assumption - seeing this traceback - but I can't prove if it's correct: one

[ZODB-Dev] zeopack error

2012-01-30 Thread Kaweh Kazemi
Hi zodb-dev, during our latest zeopack runs, zeopack for our users storage fails giving this error: Traceback (most recent call last): File /usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZEO/scripts/zeopack.py, line 159, in _main cs.pack(packt, wait=True)

Re: [ZODB-Dev] zeopack error

2012-01-30 Thread Kaweh Kazemi
Unfortunately I'm not seeing anything useful, which is my problem: [ERROR] 2012-01-30T13:36:18: (22455) Error raised in delayed method None Traceback (most recent call last): File /usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZEO/scripts/zeopack.py, line 159, in

Re: [ZODB-Dev] zeopack error

2012-01-30 Thread Martijn Pieters
On Mon, Jan 30, 2012 at 13:41, Kaweh Kazemi ka...@me.com wrote: Unfortunately I'm not seeing anything useful, which is my problem: That's because that's not the ZEO server log output, but the output from zeopack. Your ZEO server keeps logs too, Jim is asking for the information you'll find

Re: [ZODB-Dev] zeopack error

2012-01-30 Thread Kaweh Kazemi
On 30.01.2012, at 14:01, Martijn Pieters wrote: On Mon, Jan 30, 2012 at 13:41, Kaweh Kazemi ka...@me.com wrote: Unfortunately I'm not seeing anything useful, which is my problem: That's because that's not the ZEO server log output, but the output from zeopack. Your ZEO server keeps logs