Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3
07.03.2015, 01:29, Julian Taylor kirjoitti: On 07.03.2015 00:20, Pauli Virtanen wrote: 06.03.2015, 22:43, Eric Firing kirjoitti: On 2015/03/06 10:23 AM, Pauli Virtanen wrote: 06.03.2015, 20:00, Benjamin Root kirjoitti: A slightly different way to look at this is one of sharing data. If I am working on a system with 3.4 and I want to share data with others who may be using a mix of 2.7 and 3.3 systems, this problem makes npz format much less attractive. pickle is used in npy files only if there are object arrays in them. Of course, savez could just decline saving object arrays. Or issue a prominent warning. https://github.com/numpy/numpy/pull/5641 I think the ship for a warning has long sailed. At this point its probably more an annoyance for python3 users and will not prevent many more python2 users from saving files that can't be loaded into python3. How about an extra use_pickle=True kwarg that can be used to disable using pickle altogether in these routines? Another reason to do this is arbitrary code execution when loading pickles: https://www.cs.jhu.edu/~s/musings/pickle.html Easily demonstrated also with npy files (loading this file will only print something unexpected, nothing more malicious): http://pav.iki.fi/tmp/unexpected.npy ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3
On Sa, 2015-03-07 at 10:23 +, Robert Kern wrote: On Sat, Mar 7, 2015 at 9:54 AM, Pauli Virtanen p...@iki.fi wrote: How about an extra use_pickle=True kwarg that can be used to disable using pickle altogether in these routines? If we do, I'd vastly prefer `forbid_pickle=False`. The use_pickle spelling suggests that you are asking it to use pickle when it otherwise wouldn't, which is not the intention. I like the idea, at least for loading. Could also call it `allow_objects` with an explanation in the documentation. I would consider deprecating it and not allowing pickles as default, but I am not sure that is not going too far. However, I think we should be able to safely share data using npy. - Sebastian -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3
On Sat, Mar 7, 2015 at 9:54 AM, Pauli Virtanen p...@iki.fi wrote: How about an extra use_pickle=True kwarg that can be used to disable using pickle altogether in these routines? If we do, I'd vastly prefer `forbid_pickle=False`. The use_pickle spelling suggests that you are asking it to use pickle when it otherwise wouldn't, which is not the intention. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy array casting ruled not safe
This was originally posted on SO
(https://stackoverflow.com/questions/28853740/numpy-array-casting-ruled-not-safe)
and it was suggested it is probably a bug in numpy.take.
Python 2.7.8 |Anaconda 2.1.0 (32-bit)| (default, Jul 2 2014, 15:13:35) [MSC
v.1500 32 bit (Intel)] on win32
Type copyright, credits or license() for more information.
import numpy
numpy.__version__
'1.9.2'
a = numpy.array([9, 7, 5, 4, 3, 1], dtype=numpy.uint32)
b = numpy.array([1, 3], dtype=numpy.uint32)
c = a.take(b)
Traceback (most recent call last):
File pyshell#5, line 1, in module
c = a.take(b)
TypeError: Cannot cast array data from dtype('uint32') to dtype('int32')
according to the rule 'safe'
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array casting ruled not safe
On Sat, Mar 7, 2015 at 2:45 PM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Sat, Mar 7, 2015 at 2:02 PM, Dinesh Vadhia dineshbvad...@hotmail.com
wrote:
This was originally posted on SO (
https://stackoverflow.com/questions/28853740/numpy-array-casting-ruled-not-safe)
and it was suggested it is probably a bug in numpy.take.
Python 2.7.8 |Anaconda 2.1.0 (32-bit)| (default, Jul 2 2014, 15:13:35)
[MSC v.1500 32 bit (Intel)] on win32
Type copyright, credits or license() for more information.
import numpy
numpy.__version__
'1.9.2'
a = numpy.array([9, 7, 5, 4, 3, 1], dtype=numpy.uint32)
b = numpy.array([1, 3], dtype=numpy.uint32)
c = a.take(b)
Traceback (most recent call last):
File pyshell#5, line 1, in module
c = a.take(b)
TypeError: Cannot cast array data from dtype('uint32') to dtype('int32')
according to the rule 'safe'
This actually looks correct for 32-bit windows. Numpy indexes with a
signed type big enough to hold a pointer to void, which in this case is an
int32, and the uint32 cannot be safely cast to that type.
Chuck
I note that on SO Jaime made the suggestion that take use unsafe casting
and throw an error on out of bounds indexes. That sounds reasonable,
although for sufficiently large integer types an index could wrap around to
a good value. Maybe make it work only for npy_uintp.
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array casting ruled not safe
On Sat, Mar 7, 2015 at 2:02 PM, Dinesh Vadhia dineshbvad...@hotmail.com
wrote:
This was originally posted on SO (
https://stackoverflow.com/questions/28853740/numpy-array-casting-ruled-not-safe)
and it was suggested it is probably a bug in numpy.take.
Python 2.7.8 |Anaconda 2.1.0 (32-bit)| (default, Jul 2 2014, 15:13:35)
[MSC v.1500 32 bit (Intel)] on win32
Type copyright, credits or license() for more information.
import numpy
numpy.__version__
'1.9.2'
a = numpy.array([9, 7, 5, 4, 3, 1], dtype=numpy.uint32)
b = numpy.array([1, 3], dtype=numpy.uint32)
c = a.take(b)
Traceback (most recent call last):
File pyshell#5, line 1, in module
c = a.take(b)
TypeError: Cannot cast array data from dtype('uint32') to dtype('int32')
according to the rule 'safe'
This actually looks correct for 32-bit windows. Numpy indexes with a signed
type big enough to hold a pointer to void, which in this case is an int32,
and the uint32 cannot be safely cast to that type.
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array casting ruled not safe
On Sat, Mar 7, 2015 at 1:52 PM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Sat, Mar 7, 2015 at 2:45 PM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Sat, Mar 7, 2015 at 2:02 PM, Dinesh Vadhia dineshbvad...@hotmail.com
wrote:
This was originally posted on SO (
https://stackoverflow.com/questions/28853740/numpy-array-casting-ruled-not-safe)
and it was suggested it is probably a bug in numpy.take.
Python 2.7.8 |Anaconda 2.1.0 (32-bit)| (default, Jul 2 2014, 15:13:35)
[MSC v.1500 32 bit (Intel)] on win32
Type copyright, credits or license() for more information.
import numpy
numpy.__version__
'1.9.2'
a = numpy.array([9, 7, 5, 4, 3, 1], dtype=numpy.uint32)
b = numpy.array([1, 3], dtype=numpy.uint32)
c = a.take(b)
Traceback (most recent call last):
File pyshell#5, line 1, in module
c = a.take(b)
TypeError: Cannot cast array data from dtype('uint32') to dtype('int32')
according to the rule 'safe'
This actually looks correct for 32-bit windows. Numpy indexes with a
signed type big enough to hold a pointer to void, which in this case is an
int32, and the uint32 cannot be safely cast to that type.
Chuck
I note that on SO Jaime made the suggestion that take use unsafe casting
and throw an error on out of bounds indexes. That sounds reasonable,
although for sufficiently large integer types an index could wrap around to
a good value. Maybe make it work only for npy_uintp.
Chuck
It is mostly about consistency, and having take match what indexing already
does, which is to unsafely cast all integers:
In [11]: np.arange(10)[np.uint64(2**64-1)]
Out[11]: 9
I think no one has ever complained about that obviously wrong behavior, but
people do get annoyed if they cannot use their perfectly valid uint64 array
because we want to protect them from themselves. Sebastian has probably
given this more thought than anyone else, it would be interesting to hear
his thoughts on this.
Jaime
--
(\__/)
( O.o)
( ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
