Thank you, I'll do nothing in my code for the moment (working on a prototype). By the time i get closer to production i'll do some calculation and set recursion limit accordingly.
Do you know by chance whether it is possible to change the python recursionlimit dynamically (mid-run)? I could then catch the exception, log the exception, double the limit, call pickle again ( for now). Cheers, lars ==================================== Lars van Gemerden [email protected] +31 6 26 88 55 39 ==================================== On 3 okt. 2012, at 15:25, Kristján Valur Jónsson <[email protected]> wrote: > Hello Lars. > First off, deep recursion isn’t really a problem if you are using a single > thread, since most operating systems will just allocate more stack as you > need it. > In the unlikely situation that you get into OS problems, though, this is when > Stackless will help more than regular python will. > > 1) for pickle, just make sure that your sys.setrecursionlimit() is high > enough. for cPickle you don’t need any such thing. > 2) Nothing mitigates pytyon’s recursionlimit. Stackless’ stack spilling > is a behind-the-scenes way of trimming the c stack, that is all. But > python’s recursionlimit only applies to regular python code. C code is > unarffercted by it and cPickle is c code. > 3) python 2 and 3 are the same, except that in 3, “pickle” is > implemented in C and “_pickle” is a pure python reference implementation. > 4) Nothing special, it is the same as using the regular (non-C) pickle > module. Make sure that sys.setrecursionlimit() is set high. > > So, to be clear: sys.*recursionlimit is really an artificial construct in > python, there to help avoid infinite recursion. Normally, you just increase > it to help with your deep recursion, the OS then allocates stack space. > Stackless is the same, but it will try to use less OS stack space. This will > help on “threads” (that have fixed stacks, normally) and on systems that run > out of c stack easily. > > K > > From: [email protected] > [mailto:[email protected]] On Behalf Of lars van Gemerden > Sent: 3. október 2012 11:44 > To: The Stackless Python Mailing List > Subject: Re: [Stackless] Pickling and recursion > > Hi Kristjan, > > Thanks for your response, it is somewhat clearer now. However I have some > more questions about the subject (because some of my data structures are very > recursive and redesigning them would be a lot of work): > > - under what conditions can stackless mitigate these recursion limits (for > pickle and cPickle, what do i need to avoid)? > - does this mitigation circumvent pythons recursion limit or does it only > kick in after the python recursion limit has been disabled? > - i am working on stackless python 2.7, what are the differences with > stackless python 3 in this respect? > - What happens if i write my own highly recursive persistence module (i wrote > one that converts most python objects to XML via the etree module)? > > This is far from the only reason I use stackless, but it would be a great > bonus if i could be reasonable sure that i won't run into recursion problems > in the near future (this has been one of those nagging uncertainties in the > back of my mind). > > Cheers, Lars > > > > On Tue, Oct 2, 2012 at 11:34 AM, Kristján Valur Jónsson > <[email protected]> wrote: > Not necessarily. > the term "stackless" is historic in this respect. Originally, the attempt > was made to have pure python code work independently of the C stack. > In practice, this change was simply too big to be maintainable and so, > nowadays, stackless does try to do this when possible, but otherwise relents. > Pickling in version 3 of python is by default done using C code, and C code > normally also uses the c stack for recursion. > > Two opposite caveats exists: > 1) Stackless does attempt to mitigate C recursion by using "stack spilling" > where very deep c invocation stacks are stored away on the heap. This > process is not foolproof and can become degenerated, but it is designed to > work with regular python function invocation and pickling. > 2) Python imposes its own recursion limit. This has to be disabled to go > very deep. But again, this does not apply to c pickling. > > So no, stackless python does not promise stackless python evaluation in the > literal sense, but it might help you for very deep pickling. Since it is not > the primary purpose of Stackless, your mileage may vary. > K > > > -----Original Message----- > > From: [email protected] [mailto:stackless- > > [email protected]] On Behalf Of Lars van Gemerden > > Sent: 1. október 2012 08:16 > > To: [email protected] > > Subject: [Stackless] Pickling and recursion > > > > Hello, > > > > I was wandering whether using stackless also means that there is > > (practically) > > no limit to the recursion depth of highly recursive objects when using > > pickle/cPickle? > > > > Cheers, Lars > > > > ==================================== > > Lars van Gemerden > > [email protected] > > +31 6 26 88 55 39 > > ==================================== > > _______________________________________________ > > Stackless mailing list > > [email protected] > > http://www.stackless.com/mailman/listinfo/stackless > > > > _______________________________________________ > Stackless mailing list > [email protected] > http://www.stackless.com/mailman/listinfo/stackless > > > > -- > ==================================== > Lars van Gemerden > [email protected] > +31 6 26 88 55 39 > ==================================== > _______________________________________________ > Stackless mailing list > [email protected] > http://www.stackless.com/mailman/listinfo/stackless
_______________________________________________ Stackless mailing list [email protected] http://www.stackless.com/mailman/listinfo/stackless
