Tested this patched stackless against my full server expressing this unpickle problem. The pickle/unpickle now works perfectly! Thankyou very much for this high speed fix!
All the best Crispin 2009/10/12 Kristján Valur Jónsson <[email protected]> > This has been fixed in the stackless trunk, and merged into 2.6, 3.0, 3.1 > and py3k in revisions 75377-75381 > > K > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Kristján Valur Jónsson > *Sent:* 12. október 2009 11:19 > *To:* Crispin Wellington; Richard Tew > > *Cc:* [email protected] > *Subject:* Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected > cell, found stackless._wrap.cell when unpickling stackless thread > > > > Ok, I understand the issue. > > Cell objects potentially can be part in a cycle. Therefore for pickling, > the three argument __reduce__ protocol is used, with an initial ‚new‘, > followed by a ‚setstate‘ later. > > Only in the ‚setstate‘ call will the type of the object be changed from the > special „_wrap“ type. But precisely because a reference cycle is being > constructed, a fresh cell is being inserted into a function, before having > had its state set, and the function constructor balks at this. > > I see no way around this using the current wrapped pickle technique except > to fix the test in the function object. Once I have created a simple > regression test, I will submit a fix. > > > > K > > > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Crispin Wellington > *Sent:* 12. október 2009 05:27 > *To:* Richard Tew > *Cc:* [email protected] > *Subject:* Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected > cell, found stackless._wrap.cell when unpickling stackless thread > > > > OK. I boiled down the code to a small snippet that triggers the bug. It is > a bug that is exposed through my use of Twisted. If i create a factory, and > then two callbacks that are closures, each callback setting thread level > variables. Then I add the callbacks to the factory's deferred, and then I > sleep the threadlet forever. > > Initiate the tasklet, pickle it, then unpickle it, and BAM, you have your > error: > > $ /usr/local/stackless/bin/python break-stackless.py > Traceback (most recent call last): > File "break-stackless.py", line 47, in <module> > test() > File "break-stackless.py", line 37, in test > obj = pickle.loads(data) > File "/usr/local/stackless/lib/python2.6/pickle.py", line 1409, in loads > return Unpickler(file).load() > File "/usr/local/stackless/lib/python2.6/pickle.py", line 893, in load > dispatch[key](self) > File "/usr/local/stackless/lib/python2.6/pickle.py", line 1252, in > load_build > setstate(state) > TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell > > The code is.... > > ----------------------------- > > import stackless > import pickle > > from twisted.web import client > > def tasklet(): > # create an example state > factory = client.HTTPClientFactory("http://www.google.com/";, agent = > "stackless/1.0" ) > > get_complete = [False] > get_failed = [False] > > def _doFailure(data): > get_failed[0] = factory.status > > def _doSuccess(data): > get_complete[0] = factory.status > > factory.deferred.addCallback(_doSuccess).addErrback(_doFailure) > > while True: > stackless.schedule() > > def test(): > # run the task > task = stackless.tasklet(tasklet) > task.setup() > task.run() > > # pump once > stackless.schedule() > > # now serialise > data = pickle.dumps(task) > > # now deserialise > obj = pickle.loads(data) > > # resume > task.kill() > obj.insert() > > # pump again > stackless.schedule() > > if __name__=="__main__": > test() > > ----------------------- > > If code formatting is an issue, I've attached the code to the email as a > file... > > Can anyone shed any light on this bug? > > Kind Regards > > Crispin > > On Sat, Oct 10, 2009 at 2:21 AM, Richard Tew <[email protected]> > wrote: > > On Fri, Oct 9, 2009 at 10:38 PM, Crispin Wellington > <[email protected]> wrote: > > I have written a server that contains stackless threads. When the server > > shuts down these threads are serialised to disk. This all happens without > > incident and I end up with a reasonably sized file per pickled object. > > > > When I go to deserialise them I hit a problem. I get this... > > ------------------- > > File > > > "/home/cwellington/Development/yabi/yabi-be-twisted/trunk/TaskManager/Tasklets.py", > > line 54, in load > > task = pickle.loads(data) > > File "/usr/local/stackless/lib/python2.6/pickle.py", line 1415, in > > loads > > return Unpickler(file).load() > > File "/usr/local/stackless/lib/python2.6/pickle.py", line 898, in > load > > dispatch[key](self) > > File "/usr/local/stackless/lib/python2.6/pickle.py", line 1258, in > > load_build > > setstate(state) > > exceptions.TypeError: arg 5 (closure) expected cell, found > > stackless._wrap.cell > > -------------------- > > > > I think this might be a bug. The arguments are being passed back into the > > constructor of the stackless._wrap.function and the constructor is > throwing > > that exception. So in essence, the __reduce__ or __reduce_ex__ functions > are > > creating instatiation values that do not work on reinstantiation. > > > > I changed pickle.py to print out some debug. setstate is... <built-in > method > > __setstate__ of stackless._wrap.function object at 0x21ca0c8> and > (state)[4] > > is... (<cell at 0x2605f30: empty>, <cell at 0x2940718: list object at > > 0x259f518>, <cell at 0x2940670: str object at 0x2940768>, <cell at > > 0x2940750: str object at 0x29407d8>, <cell at 0x29407c0: int object at > > 0x25cd338>) > > > > These all seem to be "cell" objects. Why aren't they > (stackless._wrap.cell) > > recognised as "cells"? > > > > I can provide a pickletools.dis() dump of the pickle stream if wanted. > > > > Using Stackless Python 2.6.2 on 64 bit linux (x86_64). > > > > Can anybody help me decode my objects? or even find a way to encode them > so > > the are decodable? Is this a bug, or am I doing it wrong? > > It definitely looks like a bug. If you can provide the smallest > possible reproduction case, it would help us look into it and we could > add it to the test suite as a regression test. > > If you want your objects back in the meantime, you might try compiling > 2.6 with the relevant code raising these errors commented out. > > Richard. > > >
_______________________________________________ Stackless mailing list [email protected] http://www.stackless.com/mailman/listinfo/stackless
