Ok, i inspected the pickles with picketools.dis(), the import trick with vanilla python was too much work (too many files and classes) and i ran from the command line (no Wing) and at the moment i don't see a clear pattern anymore.
I'll have to get back to this later, Thanks for the input, pickletools.dis() is interesting. Cheers, Lars On Fri, Jan 10, 2014 at 4:49 PM, Anselm Kruis <[email protected]>wrote: > Ok, I didn't get it, that you didn't run tasklets. > > How to analyse a pickle: > > pickle = dumps(...) > from pickletools import optimize, dis > pickle = optimize(pickle) > dis(pickle) > > Now you can look at the opcodes. If your pickle really references the your > module, it will contain an "GLOBAL" opcode with the module name and the > name of a member of the module. > More likely your pickle contains a by value copy of the __dict__ of your > module or your class. Opcodes are documented in detail in the middle of > pickletools.py. > > This is a general problem: if the pickler finds a dictionary, it is > usually unable to tell, if the dictionary is the dictionary of a > module-object of a another dictionary. Usually you end up with a by-value > copy. Stackless adds some code to the pickler to recognise dictionaries of > imported modules (=module object in sys.modules.values()). See PyObject * > PyStackless_Pickle_ModuleDict(PyObject *pickler, PyObject *self) in > Stackless/pickling/prickelpit.c > > If you use the Python implementation of the Pickler (not cPickle) you can > look at the attribute "module_dict_ids" of the pickler-object after > pickling. It should be a dictionary and contain the a mapping from the ids > of module dictionaries to module objects. If the id of your modules > dictionary is not contained here, something got wrong and the pickle > probably contains a by value copy. > > I can't comment on Wing IDE as I never used it. > > > Cheers > Anselm > > > Am 10.01.2014 15:20, schrieb Christian Tismer: > >> Ah, >> >> ok, you did not mention Wing IDE, maybe they have got something wrong? >> >> You have your saved pickle, right? >> >> Write a little script that only does all the necessary imports and then >> loads >> the pickle. Run that in plain vanilla python and see if that works. >> If something stackless specific is there, it should crash. >> >> Try it with and without wing. >> >> Then you can also inspect the pickle using pickletools, and see what it >> is. >> Maybe your code mucks with something, but I cannot further guess >> without seeing your code. >> >> >>> import pickletools >> >>> help(pickletools) >> >> is pretty straightforward. >> >> cheers - chris >> >> >> On 10.01.14 10:54, lars van Gemerden wrote: >> >>> I think something else is going on: >>> >>> - at the moment i pickle the save, i have not created or run any >>> tasklets (apart from the main tasklet?). >>> - if i save an Model object and then add a class attribute to the >>> class of the model (class Model(object): _something_ = None), after i >>> reload the model object its class has no attribute '_something_ ' >>> (hasattr(type(model_object), '_something_') == True), however if i >>> create a new object of class Model, its class does have attribute >>> '_something_' >>> - if i save and load the model again, this behavior stays, probably >>> meaning that i now have two different class definitions in my save >>> file .... >>> >>> My conclusion is that modules are saved by value, without tasklets or >>> exceptions being pickled (i don't keep references to exceptions). >>> >>> Could it be that the Wing debugger forces the modules to be saved by >>> value? If so would that also happen in vanilla python? >>> >>> Cheers, Lars >>> >>> >>> On Fri, Jan 10, 2014 at 1:14 AM, lars van Gemerden >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> Thanks for the replies, >>> >>> the presentation i mentioned was at: >>> >>> https://ep2013.europython.eu/media/conference/slides/ >>> advanced-pickling-with-stackless-python-and-spickle.pdf >>> >>> >>> >>> I'll look into the other replies tomorrow, it's late .. >>> >>> Cheers, Lars >>> >>> >>> On Thu, Jan 9, 2014 at 12:11 PM, Christian Tismer >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> On 09.01.14 11:31, Anselm Kruis wrote: >>> >>> Hi Lars, >>> >>> ... >>> >>> You wrote "the loaded model seems to use a pre-change >>> class definition". Probably you observe the following: if >>> you pickle a tasklet, the tasklet contains the list of >>> frames on the stack. Each frame represents a running >>> function, but the frame does not reference the function >>> object but the code object of the function. Now code >>> objects are always pickled by value. Upon unpickling you >>> get the old code objects. With other words, you can't >>> magically modify the code, that currently executes. >>> Pickling code objects by value is required, because local >>> variables, cells, code and the instruction pointer must be >>> in an consistent state. >>> >>> >>> Well, Lars also asked >>> >>> >>> """ >>> So my questions are, is this caused by stackless (compared to >>> vanilla >>> python) and if so, what triggers the pickling of modules and >>> how can i >>> influence that or use the vanilla version of pickle instead. >>> """ >>> >>> Because he considered to use the vanilla python pickling, I >>> assumed >>> that no tasklets are involved. >>> That is a very special and known stackless feature. >>> >>> But maybe that is the bug, and there are unwanted references >>> to tasklets >>> in the model by chance. To avoid this, removing or running all >>> tasklets >>> to their end before pickling should help. >>> >>> cheers -- chris >>> >>> >>> -- Christian Tismer :^) >>> <mailto:[email protected] <mailto:[email protected]>> >>> >>> Software Consulting : Have a break! Take a ride >>> on Python's >>> Karl-Liebknecht-Str. 121 : *Starship* >>> http://starship.python.net/ >>> 14482 Potsdam : PGP key -> >>> http://pgp.uni-mainz.de >>> phone +49 173 24 18 776 <tel:%2B49%20173%2024%2018%20776> fax >>> +49 (30) 700143-0023 <tel:%2B49%20%2830%29%20700143-0023> >>> >>> PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B >>> 57F3 BF04 >>> whom do you want to sponsor today? >>> http://www.stackless.com/ >>> >>> >>> _______________________________________________ >>> Stackless mailing list >>> [email protected] <mailto:[email protected]> >>> >>> http://www.stackless.com/mailman/listinfo/stackless >>> >>> >>> >>> >>> -- ==================================== >>> Lars van Gemerden >>> [email protected] <mailto:[email protected]> >>> +31 6 26 88 55 39 <tel:%2B31%206%2026%2088%2055%2039> >>> >>> ==================================== >>> >>> >>> >>> >>> -- >>> ==================================== >>> Lars van Gemerden >>> [email protected] <mailto:[email protected]> >>> >>> +31 6 26 88 55 39 >>> ==================================== >>> >>> >>> _______________________________________________ >>> Stackless mailing list >>> [email protected] >>> http://www.stackless.com/mailman/listinfo/stackless >>> >> >> >> >> >> Ah, >> >> ok, you did not mention Wing IDE, maybe they have got something wrong? >> >> You have your saved pickle, right? >> >> Write a little script that only does all the necessary imports and then >> loads >> the pickle. Run that in plain vanilla python and see if that works. >> If something stackless specific is there, it should crash. >> >> Try it with and without wing. >> >> Then you can also inspect the pickle using pickletools, and see what it >> is. >> Maybe your code mucks with something, but I cannot further guess >> without seeing your code. >> >> >>> import pickletools >> >>> help(pickletools) >> >> is pretty straightforward. >> >> cheers - chris >> >> >> On 10.01.14 10:54, lars van Gemerden wrote: >> >>> I think something else is going on: >>> >>> - at the moment i pickle the save, i have not created or run any >>> tasklets (apart from the main tasklet?). >>> - if i save an Model object and then add a class attribute to the >>> class of the model (class Model(object): _something_ = None), after i >>> reload the model object its class has no attribute '_something_ ' >>> (hasattr(type(model_object), '_something_') == True), however if i >>> create a new object of class Model, its class does have attribute >>> '_something_' >>> - if i save and load the model again, this behavior stays, probably >>> meaning that i now have two different class definitions in my save >>> file .... >>> >>> My conclusion is that modules are saved by value, without tasklets or >>> exceptions being pickled (i don't keep references to exceptions). >>> >>> Could it be that the Wing debugger forces the modules to be saved by >>> value? If so would that also happen in vanilla python? >>> >>> Cheers, Lars >>> >>> >>> On Fri, Jan 10, 2014 at 1:14 AM, lars van Gemerden >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> Thanks for the replies, >>> >>> the presentation i mentioned was at: >>> https://ep2013.europython.eu/media/conference/slides/ >>> advanced-pickling-with-stackless-python-and-spickle.pdf >>> >>> >>> I'll look into the other replies tomorrow, it's late .. >>> >>> Cheers, Lars >>> >>> >>> On Thu, Jan 9, 2014 at 12:11 PM, Christian Tismer >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> On 09.01.14 11:31, Anselm Kruis wrote: >>> >>> Hi Lars, >>> >>> ... >>> >>> You wrote "the loaded model seems to use a pre-change >>> class definition". Probably you observe the following: if >>> you pickle a tasklet, the tasklet contains the list of >>> frames on the stack. Each frame represents a running >>> function, but the frame does not reference the function >>> object but the code object of the function. Now code >>> objects are always pickled by value. Upon unpickling you >>> get the old code objects. With other words, you can't >>> magically modify the code, that currently executes. >>> Pickling code objects by value is required, because local >>> variables, cells, code and the instruction pointer must be >>> in an consistent state. >>> >>> >>> Well, Lars also asked >>> >>> >>> """ >>> So my questions are, is this caused by stackless (compared to >>> vanilla >>> python) and if so, what triggers the pickling of modules and >>> how can i >>> influence that or use the vanilla version of pickle instead. >>> """ >>> >>> Because he considered to use the vanilla python pickling, I >>> assumed >>> that no tasklets are involved. >>> That is a very special and known stackless feature. >>> >>> But maybe that is the bug, and there are unwanted references >>> to tasklets >>> in the model by chance. To avoid this, removing or running all >>> tasklets >>> to their end before pickling should help. >>> >>> cheers -- chris >>> >>> >>> -- >>> Christian Tismer :^) >>> <mailto:[email protected] <mailto:[email protected]>> >>> >>> Software Consulting : Have a break! Take a ride >>> on Python's >>> Karl-Liebknecht-Str. 121 : *Starship* >>> http://starship.python.net/ >>> 14482 Potsdam : PGP key -> >>> http://pgp.uni-mainz.de >>> phone +49 173 24 18 776 <tel:%2B49%20173%2024%2018%20776> fax >>> +49 (30) 700143-0023 <tel:%2B49%20%2830%29%20700143-0023> >>> >>> PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B >>> 57F3 BF04 >>> whom do you want to sponsor today? >>> http://www.stackless.com/ >>> >>> >>> _______________________________________________ >>> Stackless mailing list >>> [email protected] <mailto:[email protected]> >>> >>> http://www.stackless.com/mailman/listinfo/stackless >>> >>> >>> >>> >>> -- >>> ==================================== >>> Lars van Gemerden >>> [email protected] <mailto:[email protected]> >>> +31 6 26 88 55 39 <tel:%2B31%206%2026%2088%2055%2039> >>> >>> ==================================== >>> >>> >>> >>> >>> -- >>> ==================================== >>> Lars van Gemerden >>> [email protected] <mailto:[email protected]> >>> >>> +31 6 26 88 55 39 >>> ==================================== >>> >>> >>> _______________________________________________ >>> Stackless mailing list >>> [email protected] >>> http://www.stackless.com/mailman/listinfo/stackless >>> >> >> >> -- >> Christian Tismer :^)<mailto:[email protected]> >> Software Consulting : Have a break! Take a ride on Python's >> Karl-Liebknecht-Str. 121 : *Starship*http://starship.python.net/ >> >> 14482 Potsdam : PGP key ->http://pgp.uni-mainz.de >> phone +49 173 24 18 776 fax +49 (30) 700143-0023 >> PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 >> whom do you want to sponsor today?http://www.stackless.com/ >> >> >> >> >> _______________________________________________ >> Stackless mailing list >> [email protected] >> http://www.stackless.com/mailman/listinfo/stackless >> >> > -- > Dipl. Phys. Anselm Kruis science + computing ag > Senior Solution Architect Ingolstädter Str. 22 > email [email protected] 80807 München, Germany > phone +49 89 356386 874 fax 737 www.science-computing.de > -- > Vorstandsvorsitzender/Chairman of the board of management: > Gerd-Lothar Leonhart > Vorstand/Board of Management: > Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr. Ingrid Zech > Vorsitzender des Aufsichtsrats/ > Chairman of the Supervisory Board: > Philippe Miltin > Sitz/Registered Office: Tuebingen > Registergericht/Registration Court: Stuttgart > Registernummer/Commercial Register No.: HRB 382196 > > > _______________________________________________ > Stackless mailing list > [email protected] > http://www.stackless.com/mailman/listinfo/stackless > -- ==================================== Lars van Gemerden [email protected] +31 6 26 88 55 39 ====================================
_______________________________________________ Stackless mailing list [email protected] http://www.stackless.com/mailman/listinfo/stackless
