Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 10:22 PM, Charles R Harris < charlesr.har...@gmail.com> wrote: > > On Tue, Oct 25, 2016 at 10:41 PM, Robert Kern wrote: >> >> On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris < charlesr.har...@gmail.com> wrote: >> > >> > Hi All, >> > >> > There is a proposed random number

Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Charles R Harris
On Tue, Oct 25, 2016 at 10:41 PM, Robert Kern wrote: > On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris < > charlesr.har...@gmail.com> wrote: > > > > Hi All, > > > > There is a proposed random number package PR now up on github: > https://github.com/numpy/numpy/pull/8209. It is from > > oleksand

Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris wrote: > > Hi All, > > There is a proposed random number package PR now up on github: https://github.com/numpy/numpy/pull/8209. It is from > oleksandr-pavlyk and implements the number random number package using MKL for increased speed. I think we

[Numpy-discussion] Intel random number package

2016-10-25 Thread Charles R Harris
Hi All, There is a proposed random number package PR now up on github: https://github.com/numpy/numpy/pull/8209. It is from oleksandr-pavlyk and implements the number random number package using MKL for increased speed. I think we are definitely interested in

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan wrote: > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) [[1,2]] Yes, but the problem is: suppose I have a 10 gigabyte array, and then

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 7:05 PM, Feng Yu wrote: > > Hi, > > Just another perspective. base' and 'data' in PyArrayObject are two > separate variables. > > base can point to any PyObject, but it is `data` that defines where > data is accessed in memory. > > 1. There is no clear way to pickle a point

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Feng Yu
Hi, Just another perspective. base' and 'data' in PyArrayObject are two separate variables. base can point to any PyObject, but it is `data` that defines where data is accessed in memory. 1. There is no clear way to pickle a pointer (`data`) in a meaningful way. In order for `data` member to mak

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan < harrigan.matt...@gmail.com> wrote: > > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) > >>> [[1,2]] > > Numpy arrays are different but refe

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Matthew Harrigan
It seems pickle keeps track of references for basic python types. x = [1] y = [x] x,y = pickle.loads(pickle.dumps((x,y))) x.append(2) print(y) >>> [[1,2]] Numpy arrays are different but references are forgotten after pickle/unpickle. Shared objects do not remain shared. Based on the quote below

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 3:07 PM, Stephan Hoyer wrote: > > On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: >> >> Concretely, what do would you suggest should happen with: >> >> base = np.zeros(1) >> view = base[:10] >> >> # case 1 >> pickle.dump(view, file) >> >> # case 2 >> pickle

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Stephan Hoyer
On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: > Concretely, what do would you suggest should happen with: > > base = np.zeros(1) > view = base[:10] > > # case 1 > pickle.dump(view, file) > > # case 2 > pickle.dump(base, file) > pickle.dump(view, file) > > # case 3 > pickle.dump(

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 12:38 PM, Stephan Hoyer wrote: > With a custom wrapper class, it's possible to preserve NumPy views when > pickling: > https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling > > This can result in significant time/space savings with pickling views

[Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Stephan Hoyer
With a custom wrapper class, it's possible to preserve NumPy views when pickling: https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling This can result in significant time/space savings with pickling views along with base arrays and brings the behavior of NumPy more in l

Re: [Numpy-discussion] padding options for diff

2016-10-25 Thread Peter Creasey
> Date: Mon, 24 Oct 2016 08:44:46 -0400 > From: Matthew Harrigan > > I posted a pull request which > adds optional padding kwargs "to_begin" and "to_end" to diff. Those > options are based on what's available in ediff1d. It closes this issue >

Re: [Numpy-discussion] Numpy integers to integer powers again again

2016-10-25 Thread Stephan Hoyer
I am also concerned about adding more special cases for NumPy scalars vs arrays. These cases are already confusing (e.g., making no distinction between 0d arrays and scalars) and poorly documented. On Mon, Oct 24, 2016 at 4:30 PM, Nathaniel Smith wrote: > On Mon, Oct 24, 2016 at 3:41 PM, Charles