Re: [sage-devel] Re: Getting rid of the pickle jar

[Erik Bray]

On Mon, Oct 30, 2017 at 4:33 PM, Jeroen Demeyer  wrote:
> On 2017-10-30 16:28, William Stein wrote:
>>
>> Not necessarily.Pickle is *the* canonical extensible object
>> serialization system for Python.   It’s of course very extensible in
>> that users can define how objects pickle, eg by defining a dunder reduce
>> method.  As such they can of course make that method store output in any
>> binary format they want.
>
>
> The question is not what is possible, but what is the recommended way of
> doing things. Are we supposed to make our pickles hardware-independent or is
> that hopeless anyway?

I guess it depends on what you mean by "supposed to".  I believe that
the pickle formats for built-in types are hardware-independent.  E.g.
int values are stored as little-endian and loaded as little-endian in
a hardware-independent manner.  So that at least appears to be the
intent.


But I don't know if there's any *requirement* that a pickle be
hardware-independent when it comes to custom types, though in general
it's probably better that they are.  But for custom code that, say,
targets only one hardware platform anyways then that's up to the
developer

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-30 16:28, William Stein wrote:

Not necessarily.Pickle is *the* canonical extensible object
serialization system for Python.   It’s of course very extensible in
that users can define how objects pickle, eg by defining a dunder reduce
method.  As such they can of course make that method store output in any
binary format they want.


The question is not what is possible, but what is the recommended way of 
doing things. Are we supposed to make our pickles hardware-independent 
or is that hopeless anyway?


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[William Stein]

On Mon, Oct 30, 2017 at 8:17 AM Jeroen Demeyer 
wrote:

> Another very relevant question: are pickles supposed to be
> hardware/OS-independent? In other words: can I take a pickle from one
> machine and unpickle it on a different machine (assuming that the
> software version is the same)?



Not necessarily.Pickle is *the* canonical extensible object
serialization system for Python.   It’s of course very extensible in that
users can define how objects pickle, eg by defining a dunder reduce
method.  As such they can of course make that method store output in any
binary format they want.

Example: look at some of the cython matrix code I mentioned above.  I hope
it is architecture neutral as written, but I’m sure you could easily
imagine how to write something similar that isn’t.

- William


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
-- William Stein

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

Another very relevant question: are pickles supposed to be 
hardware/OS-independent? In other words: can I take a pickle from one 
machine and unpickle it on a different machine (assuming that the 
software version is the same)?


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Erik Bray]

On Mon, Oct 30, 2017 at 3:35 PM, Jeroen Demeyer  wrote:
> On 2017-10-30 15:12, Erik Bray wrote:
>>
>> My point, however, is baked directly into the file format--the pickle
>> format is very Python version-dependent (there are I think 5 different
>> pickle formats now)
>
>
> It's true that the format has changed, but always in a backward-compatible
> way. For basic Python objects (lists, numbers, strings, ...) I'm pretty sure
> that old pickles can still be unpickled correctly. The issue is with
> user-defined classes.

Right, I could have been more clear about that.  But for Sage it's
exactly those user-defined classes that matter most.  It would be good
to have some serious thought/discussion about portable serialization
options for Sage objects.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-30 15:12, Erik Bray wrote:

My point, however, is baked directly into the file format--the pickle
format is very Python version-dependent (there are I think 5 different
pickle formats now)


It's true that the format has changed, but always in a 
backward-compatible way. For basic Python objects (lists, numbers, 
strings, ...) I'm pretty sure that old pickles can still be unpickled 
correctly. The issue is with user-defined classes.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-29 23:14, Travis Scrimshaw wrote:

What about any object that does a TestSuite(foo).run()? This guarantees
that it pickles (assuming it is not skipping that test) and is an object
that someone would create.


Sounds like a good idea to me. We could have TestSuite(foo).run() create 
a special pickle which also stores how TestSuite.run() was called. Then 
we could re-run the TestSuite when testing the pickle jar.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-29 20:22, Simon King wrote:

The m*n-fold pickle jar should not pollute the SageMath sources.
In that way, a new pickle jar version wouldn't result in a new
to-be-merged git commit. Instead, the jar should only be stored
on some SageMath servers.


I don't like this part. It should be easily available to users for 
testing. Maybe make it an optional package and mark the test "# optional 
- pickle_jar".


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-30 08:41, Simon King wrote:

would you rather have no test at all than a
superficial consistency test on a wide range of objects, versions and
machines?


Yes. A test which doesn't actually test anything is worse than no test. 
That's exactly the situation that we currently have with the pickle jar.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Erik Bray]

On Sun, Oct 29, 2017 at 10:04 AM, Simon King  wrote:
> Hi Erik,
>
> On 2017-10-27, Erik Bray  wrote:
>> Plus, while pickling has many valid runtime use-cases, particularly
>> for IPC, and short-term preservation of objects between interpreter
>> sessions, it was *never* intended for long-term data storage,
>
> Seriously? Said who?
>
> I always thought of pickles as the default way to store the results
> of long computations, for later (potentially MUCH later) use. Of course,
> it always is possible to say "install an old software version to unpickle
> the results" or "write a routine that allows to read the old pickle in
> a new software version", but as part of user-friendliness, I would
> recommend that SageMath-developers keep considering it as their duty to
> make unpickling backward-compatible, to a reasonable extent.

I'll see if I can find an "authoritative" source making this claim,
but I believe it is conventional wisdom (along the lines of "goto
considered evil"--it's not as if there aren't valid uses for it but it
should be avoided unless you know what you're doing).

My point, however, is baked directly into the file format--the pickle
format is very Python version-dependent (there are I think 5 different
pickle formats now) and the way non-trivial objects are pickled is
highly tied to the version of the source code at which that pickled
object was saved.  It depends on the module layout, class names,
implementation of that object's __reduce__ at the time the pickle was
made, etc.  So while you can use pickle for long-term storage, it's
implicit in the format that if you want to unpickle an object saved at
a given time, you may need to be running the same version of the
software at which the pickle was created (it might be nice if there
were somehow a way to mark this explicitly when pickling Sage objects,
e.g. with the current git hash).

Anyways, as I wrote for short-term storage (e.g. results of long
computations that will be reused later in the same software) it's
fine. For long-term you want a more transparent, more interoperable
serialization format.  JSON isn't sufficient for all uses cases, but
there are other alternatives as well, even for binary data.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[David Roe]

I think tying it in with TestSuite(foo).run() is a good idea, but we
probably don't want to store pickles for every one (since test suites can
be run for multiple elements of the same type).  Maybe check to see if a
pickle has already been created for a given class, and if not, create one?
David

On Sun, Oct 29, 2017 at 6:14 PM, Travis Scrimshaw 
wrote:

>
>
> On Monday, October 30, 2017 at 5:46:49 AM UTC+10, Jeroen Demeyer wrote:
>>
>> On 2017-10-29 20:22, Simon King wrote:
>> > As part of the release process of a new version of SageMath,
>> > a new version of the pickle jar is created by some patchbots
>> > on m different machines and replaces the m oldest versions of
>> > the pickle jar.
>>
>> That's the easy part. The hard part is deciding what to put in the
>> pickle jar and how to test that the pickles still work. If we don't have
>> good ideas for the latter, the pickle jar is pointless.
>>
>
> What about any object that does a TestSuite(foo).run()? This guarantees
> that it pickles (assuming it is not skipping that test) and is an object
> that someone would create. Usually when those are marked as "# long time",
> it is the tests that take a long time (say, iterated over 1000 objects)
> rather than creation and (un)pickling.
>
> Best,
> Travis
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Travis Scrimshaw]

On Monday, October 30, 2017 at 5:46:49 AM UTC+10, Jeroen Demeyer wrote:
>
> On 2017-10-29 20:22, Simon King wrote: 
> > As part of the release process of a new version of SageMath, 
> > a new version of the pickle jar is created by some patchbots 
> > on m different machines and replaces the m oldest versions of 
> > the pickle jar. 
>
> That's the easy part. The hard part is deciding what to put in the 
> pickle jar and how to test that the pickles still work. If we don't have 
> good ideas for the latter, the pickle jar is pointless. 
>

What about any object that does a TestSuite(foo).run()? This guarantees 
that it pickles (assuming it is not skipping that test) and is an object 
that someone would create. Usually when those are marked as "# long time", 
it is the tests that take a long time (say, iterated over 1000 objects) 
rather than creation and (un)pickling.

Best,
Travis

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[William Stein]

On Sun, Oct 29, 2017 at 12:46 PM Jeroen Demeyer 
wrote:

> On 2017-10-29 20:22, Simon King wrote:
> > As part of the release process of a new version of SageMath,
> > a new version of the pickle jar is created by some patchbots
> > on m different machines and replaces the m oldest versions of
> > the pickle jar.
>
> That's the easy part. The hard part is deciding what to put in the
> pickle jar and how to test that the pickles still work. If we don't have
> good ideas for the latter, the pickle jar is pointless.


When I made the pickle jar in the first place, I automated putting the
output of all doctests in the pickle jar.


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
-- William Stein

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[Jeroen Demeyer]

On 2017-10-29 20:22, Simon King wrote:

As part of the release process of a new version of SageMath,
a new version of the pickle jar is created by some patchbots
on m different machines and replaces the m oldest versions of
the pickle jar.


That's the easy part. The hard part is deciding what to put in the 
pickle jar and how to test that the pickles still work. If we don't have 
good ideas for the latter, the pickle jar is pointless.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[William Stein]

On Sun, Oct 29, 2017 at 7:58 AM Dima Pasechnik  wrote:

> The conventional wisdom is to avoid pickles in favour of JSON (the latter
> is a platform-independent human-parseable text, the former is some
> Python-only binary stuff).


Whether this is a good idea depends a lot on the domain.  E.g., try storing
a 1x1 matrix over GF(2) in JSON versus as a sickle in Sage (using
the save command).  To use JSON, you'll have to start by writing your own
custom code to export (and import) to JSON, and whatever format you chose
it is going to be 100s of times less compact and slower than the fast
custom code we wrote for pickling.

A huge amount of work has been put into making various things (e.g.,
matrices) pickle and unpickle very efficiently (and stably over versions).
Basically 0 work has gone into converting sage objects to/from JSON.

 -- William


>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
-- William Stein

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Getting rid of the pickle jar

[David Roe]

I agree that removing pickles from 6+ years ago is a good idea.

I do think, however, that the idea of being able to save objects between
versions of Sage is valuable.  And we need some way to test it.  Maybe we
could move to some sort of rolling pickle jar, where we allow deprecations
after a certain amount of time?
David

On Sat, Oct 28, 2017 at 6:17 AM, Andrew  wrote:

> Definitely +1.
>
> On Saturday, 28 October 2017 10:17:59 UTC+2, Eric Gourgoulhon wrote:
>>
>> Hi,
>>
>> Le vendredi 27 octobre 2017 10:42:53 UTC+2, Jeroen Demeyer a écrit :
>>>
>>> Since the pickle jar doesn't do what it is meant to do, I suggest to be
>>> pragmatic and remove it completely.
>>>
>>>
>> +1
>>
>> Eric.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.