Re: [Stackless] pickling modules & stackless

Fri, 10 Jan 2014 08:12:56 -0800 

Ok, I didn't get it, that you didn't run tasklets.

How to analyse a pickle:

pickle = dumps(...)
from pickletools import optimize, dis
pickle = optimize(pickle)
dis(pickle)
Now you can look at the opcodes. If your pickle really references the your module, it will contain an "GLOBAL" opcode with the module name and the name of a member of the module. More likely your pickle contains a by value copy of the __dict__ of your module or your class. Opcodes are documented in detail in the middle of pickletools.py.
This is a general problem: if the pickler finds a dictionary, it is usually unable to tell, if the dictionary is the dictionary of a module-object of a another dictionary. Usually you end up with a by-value copy. Stackless adds some code to the pickler to recognise dictionaries of imported modules (=module object in sys.modules.values()). See PyObject * PyStackless_Pickle_ModuleDict(PyObject *pickler, PyObject *self) in Stackless/pickling/prickelpit.c
If you use the Python implementation of the Pickler (not cPickle) you can look at the attribute "module_dict_ids" of the pickler-object after pickling. It should be a dictionary and contain the a mapping from the ids of module dictionaries to module objects. If the id of your modules dictionary is not contained here, something got wrong and the pickle probably contains a by value copy. 

I can't comment on Wing IDE as I never used it.


Cheers
  Anselm


Am 10.01.2014 15:20, schrieb Christian Tismer:

Ah,

ok, you did not mention Wing IDE, maybe they have got something wrong?

You have your saved pickle, right?

Write a little script that only does all the necessary imports and then
loads
the pickle. Run that in plain vanilla python and see if that works.
If something stackless specific is there, it should crash.

Try it with and without wing.

Then you can also inspect the pickle using pickletools, and see what it is.
Maybe your code mucks with something, but I cannot further guess
without seeing your code.

 >>> import pickletools
 >>> help(pickletools)

is pretty straightforward.

cheers - chris


On 10.01.14 10:54, lars van Gemerden wrote:

I think something else is going on:

- at the moment i pickle the save, i have not created or run any
tasklets (apart from the main tasklet?).
- if i save an Model object and then add a class attribute to the
class of the model (class Model(object): _something_ = None), after i
reload the model object its class has no attribute '_something_ '
(hasattr(type(model_object), '_something_') == True), however if i
create a new object of class Model, its class does have attribute
'_something_'
- if i save and load the model again, this behavior stays, probably
meaning that i now have two different class definitions in my save
file ....

My conclusion is that modules are saved by value, without tasklets or
exceptions being pickled (i don't keep references to exceptions).

Could it be that the Wing debugger forces the modules to be saved by
value? If so would that also happen in vanilla python?

Cheers, Lars


On Fri, Jan 10, 2014 at 1:14 AM, lars van Gemerden
<[email protected] <mailto:[email protected]>> wrote:

    Thanks for the replies,

    the presentation i mentioned was at:

https://ep2013.europython.eu/media/conference/slides/advanced-pickling-with-stackless-python-and-spickle.pdf



    I'll look into the other replies tomorrow, it's late ..

    Cheers, Lars


    On Thu, Jan 9, 2014 at 12:11 PM, Christian Tismer
    <[email protected] <mailto:[email protected]>> wrote:

        On 09.01.14 11:31, Anselm Kruis wrote:

            Hi Lars,

            ...

            You wrote "the loaded model seems to use a pre-change
            class definition". Probably you observe the following: if
            you pickle a tasklet, the tasklet contains the list of
            frames on the stack. Each frame represents a running
            function, but the frame does not reference the function
            object but the code object of the function. Now code
            objects are always pickled by value. Upon unpickling you
            get the old code objects. With other words, you can't
            magically modify the code, that currently executes.
            Pickling code objects by value is required, because local
            variables, cells, code and the instruction pointer must be
            in an consistent state.


        Well, Lars also asked


        """
        So my questions are, is this caused by stackless (compared to
        vanilla
        python) and if so, what triggers the pickling of modules and
        how can i
        influence that or use the vanilla version of pickle instead.
        """

        Because he considered to use the vanilla python pickling, I
        assumed
        that no tasklets are involved.
        That is a very special and known stackless feature.

        But maybe that is the bug, and there are unwanted references
        to tasklets
        in the model by chance. To avoid this, removing or running all
        tasklets
        to their end before pickling should help.

        cheers -- chris


        --         Christian Tismer             :^)
<mailto:[email protected] <mailto:[email protected]>>
        Software Consulting          :     Have a break! Take a ride
        on Python's
        Karl-Liebknecht-Str. 121     :    *Starship*
        http://starship.python.net/
        14482 Potsdam                :     PGP key ->
        http://pgp.uni-mainz.de
        phone +49 173 24 18 776 <tel:%2B49%20173%2024%2018%20776>  fax
        +49 (30) 700143-0023 <tel:%2B49%20%2830%29%20700143-0023>
        PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B
        57F3 BF04
              whom do you want to sponsor today?
http://www.stackless.com/


        _______________________________________________
        Stackless mailing list
        [email protected] <mailto:[email protected]>
        http://www.stackless.com/mailman/listinfo/stackless




    --     ====================================
    Lars van Gemerden
    [email protected] <mailto:[email protected]>
    +31 6 26 88 55 39 <tel:%2B31%206%2026%2088%2055%2039>
    ====================================




--
====================================
Lars van Gemerden
[email protected] <mailto:[email protected]>
+31 6 26 88 55 39
====================================


_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless





Ah,

ok, you did not mention Wing IDE, maybe they have got something wrong?

You have your saved pickle, right?

Write a little script that only does all the necessary imports and then
loads
the pickle. Run that in plain vanilla python and see if that works.
If something stackless specific is there, it should crash.

Try it with and without wing.

Then you can also inspect the pickle using pickletools, and see what it is.
Maybe your code mucks with something, but I cannot further guess
without seeing your code.

 >>> import pickletools
 >>> help(pickletools)

is pretty straightforward.

cheers - chris


On 10.01.14 10:54, lars van Gemerden wrote:

I think something else is going on:

- at the moment i pickle the save, i have not created or run any
tasklets (apart from the main tasklet?).
- if i save an Model object and then add a class attribute to the
class of the model (class Model(object): _something_ = None), after i
reload the model object its class has no attribute '_something_ '
(hasattr(type(model_object), '_something_') == True), however if i
create a new object of class Model, its class does have attribute
'_something_'
- if i save and load the model again, this behavior stays, probably
meaning that i now have two different class definitions in my save
file ....

My conclusion is that modules are saved by value, without tasklets or
exceptions being pickled (i don't keep references to exceptions).

Could it be that the Wing debugger forces the modules to be saved by
value? If so would that also happen in vanilla python?

Cheers, Lars


On Fri, Jan 10, 2014 at 1:14 AM, lars van Gemerden
<[email protected] <mailto:[email protected]>> wrote:

    Thanks for the replies,

    the presentation i mentioned was at:
    
https://ep2013.europython.eu/media/conference/slides/advanced-pickling-with-stackless-python-and-spickle.pdf


    I'll look into the other replies tomorrow, it's late ..

    Cheers, Lars


    On Thu, Jan 9, 2014 at 12:11 PM, Christian Tismer
    <[email protected] <mailto:[email protected]>> wrote:

        On 09.01.14 11:31, Anselm Kruis wrote:

            Hi Lars,

            ...

            You wrote "the loaded model seems to use a pre-change
            class definition". Probably you observe the following: if
            you pickle a tasklet, the tasklet contains the list of
            frames on the stack. Each frame represents a running
            function, but the frame does not reference the function
            object but the code object of the function. Now code
            objects are always pickled by value. Upon unpickling you
            get the old code objects. With other words, you can't
            magically modify the code, that currently executes.
            Pickling code objects by value is required, because local
            variables, cells, code and the instruction pointer must be
            in an consistent state.


        Well, Lars also asked


        """
        So my questions are, is this caused by stackless (compared to
        vanilla
        python) and if so, what triggers the pickling of modules and
        how can i
        influence that or use the vanilla version of pickle instead.
        """

        Because he considered to use the vanilla python pickling, I
        assumed
        that no tasklets are involved.
        That is a very special and known stackless feature.

        But maybe that is the bug, and there are unwanted references
        to tasklets
        in the model by chance. To avoid this, removing or running all
        tasklets
        to their end before pickling should help.

        cheers -- chris


        --
        Christian Tismer             :^)
        <mailto:[email protected] <mailto:[email protected]>>
        Software Consulting          :     Have a break! Take a ride
        on Python's
        Karl-Liebknecht-Str. 121     :    *Starship*
        http://starship.python.net/
        14482 Potsdam                :     PGP key ->
        http://pgp.uni-mainz.de
        phone +49 173 24 18 776 <tel:%2B49%20173%2024%2018%20776>  fax
        +49 (30) 700143-0023 <tel:%2B49%20%2830%29%20700143-0023>
        PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B
        57F3 BF04
              whom do you want to sponsor today? http://www.stackless.com/


        _______________________________________________
        Stackless mailing list
        [email protected] <mailto:[email protected]>
        http://www.stackless.com/mailman/listinfo/stackless




    --
    ====================================
    Lars van Gemerden
    [email protected] <mailto:[email protected]>
    +31 6 26 88 55 39 <tel:%2B31%206%2026%2088%2055%2039>
    ====================================




--
====================================
Lars van Gemerden
[email protected] <mailto:[email protected]>
+31 6 26 88 55 39
====================================


_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless



--
Christian Tismer             :^)<mailto:[email protected]>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship*http://starship.python.net/
14482 Potsdam                :     PGP key ->http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?http://www.stackless.com/



_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless



--
 Dipl. Phys. Anselm Kruis                       science + computing ag
 Senior Solution Architect                      Ingolstädter Str. 22
 email [email protected]             80807 München, Germany
 phone +49 89 356386 874  fax 737               www.science-computing.de
--
Vorstandsvorsitzender/Chairman of the board of management:
Gerd-Lothar Leonhart
Vorstand/Board of Management:
Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr. Ingrid Zech 
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196


_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless

Reply via email to