Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-07 Thread Pauli Virtanen
07.03.2015, 01:29, Julian Taylor kirjoitti: On 07.03.2015 00:20, Pauli Virtanen wrote: 06.03.2015, 22:43, Eric Firing kirjoitti: On 2015/03/06 10:23 AM, Pauli Virtanen wrote: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-07 Thread Sebastian Berg
On Sa, 2015-03-07 at 10:23 +, Robert Kern wrote: On Sat, Mar 7, 2015 at 9:54 AM, Pauli Virtanen p...@iki.fi wrote: How about an extra use_pickle=True kwarg that can be used to disable using pickle altogether in these routines? If we do, I'd vastly prefer `forbid_pickle=False`. The

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-07 Thread Robert Kern
On Sat, Mar 7, 2015 at 9:54 AM, Pauli Virtanen p...@iki.fi wrote: How about an extra use_pickle=True kwarg that can be used to disable using pickle altogether in these routines? If we do, I'd vastly prefer `forbid_pickle=False`. The use_pickle spelling suggests that you are asking it to use

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Julian Taylor
On 07.03.2015 00:20, Pauli Virtanen wrote: 06.03.2015, 22:43, Eric Firing kirjoitti: On 2015/03/06 10:23 AM, Pauli Virtanen wrote: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 22:23, Pauli Virtanen kirjoitti: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix of 2.7 and 3.3 systems, this problem makes

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 22:43, Eric Firing kirjoitti: On 2015/03/06 10:23 AM, Pauli Virtanen wrote: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Eric Firing
On 2015/03/06 1:29 PM, Julian Taylor wrote: I think the ship for a warning has long sailed. At this point its probably more an annoyance for python3 users and will not prevent many more python2 users from saving files that can't be loaded into python3. The point of a warning is that anything

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Ryan Nelson
Arnd, I can see where this is an issue. If you are trying to update your code for Py3, I still think that it would really help to add a version attribute of some sort to your new HDF files. You can then write a little check in your access code that looks for this variable. If it is not present,

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Arnd Baecker
On Fri, 6 Mar 2015, Pauli Virtanen wrote: Arnd Baecker arnd.baecker at web.de writes: [clip] Still I would have thought that this should be working out-of-the box, i.e. without the pickle.loads trick? Pickle files should be considered incompatible between Python 2 and Python 3. Python 3

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
Arnd Baecker arnd.baecker at web.de writes: [clip] Still I would have thought that this should be working out-of-the box, i.e. without the pickle.loads trick? Pickle files should be considered incompatible between Python 2 and Python 3. Python 3 interprets all bytes objects saved by Python 2

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Sebastian
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi all, As this also affects .npy files, which uses pickle internally, why can't this be done by Numpy itself? This breaks backwards compatibility in a very bad way in my opinion. The company I worked for uses Numpy and consorts a lot and also

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Benjamin Root
A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix of 2.7 and 3.3 systems, this problem makes npz format much less attractive. Ben Root On Fri, Mar 6, 2015 at 12:51 PM, Charles R Harris

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Charles R Harris
On Fri, Mar 6, 2015 at 10:34 AM, Sebastian se...@sebix.at wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi all, As this also affects .npy files, which uses pickle internally, why can't this be done by Numpy itself? This breaks backwards compatibility in a very bad way in my

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix of 2.7 and 3.3 systems, this problem makes npz format much less attractive. pickle is

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Eric Firing
On 2015/03/06 10:23 AM, Pauli Virtanen wrote: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix of 2.7 and 3.3 systems, this problem makes

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-05 Thread Ryan Nelson
This works if run from Py3. Don't know if it will *always* work. From that GH discussion you linked, it sounds like that is a bit of a hack. ## Illustrate problem with pytables data - python 2 to python 3. from __future__ import print_function import sys import numpy as np import

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-05 Thread Arnd Baecker
On Thu, 5 Mar 2015, Ryan Nelson wrote: This works if run from Py3. Don't know if it will *always* work. From that GH discussion you linked, it sounds like that is a bit of a hack. Great - based on your code I could modify my loader routine so that on python 3 it can load the files generated

[Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-04 Thread Anrd Baecker
Dear all, when preparing the transition of our repositories from python 2 to python 3, I encountered a problem loading pytables (.h5) files generated using python 2. I suspect that it is caused by a problem with pickling numpy arrays under python 3: The code appended at the end of this mail