Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 12:38 PM, Stephan Hoyer wrote: > With a custom wrapper class, it's possible to preserve NumPy views when > pickling: > https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling > > This can result in significant time/space savings

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Stephan Hoyer
On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: > Concretely, what do would you suggest should happen with: > > base = np.zeros(1) > view = base[:10] > > # case 1 > pickle.dump(view, file) > > # case 2 > pickle.dump(base, file) > pickle.dump(view, file) > > # case

[Numpy-discussion] Intel random number package

2016-10-25 Thread Charles R Harris
Hi All, There is a proposed random number package PR now up on github: https://github.com/numpy/numpy/pull/8209. It is from oleksandr-pavlyk and implements the number random number package using MKL for increased speed. I think we are definitely interested in

Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Charles R Harris
On Tue, Oct 25, 2016 at 10:41 PM, Robert Kern wrote: > On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris < > charlesr.har...@gmail.com> wrote: > > > > Hi All, > > > > There is a proposed random number package PR now up on github: > https://github.com/numpy/numpy/pull/8209.

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 3:07 PM, Stephan Hoyer wrote: > > On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: >> >> Concretely, what do would you suggest should happen with: >> >> base = np.zeros(1) >> view = base[:10] >> >> # case 1 >>

Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris wrote: > > Hi All, > > There is a proposed random number package PR now up on github: https://github.com/numpy/numpy/pull/8209. It is from > oleksandr-pavlyk and implements the number random number package using MKL for

Re: [Numpy-discussion] Intel random number package

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 10:22 PM, Charles R Harris < charlesr.har...@gmail.com> wrote: > > On Tue, Oct 25, 2016 at 10:41 PM, Robert Kern wrote: >> >> On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris < charlesr.har...@gmail.com> wrote: >> > >> > Hi All, >> > >> > There is a

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan wrote: > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) [[1,2]] Yes, but the problem is: suppose I have a

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan < harrigan.matt...@gmail.com> wrote: > > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) > >>> [[1,2]] > > Numpy arrays are different but

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 7:05 PM, Feng Yu wrote: > > Hi, > > Just another perspective. base' and 'data' in PyArrayObject are two > separate variables. > > base can point to any PyObject, but it is `data` that defines where > data is accessed in memory. > > 1. There is no

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Matthew Harrigan
It seems pickle keeps track of references for basic python types. x = [1] y = [x] x,y = pickle.loads(pickle.dumps((x,y))) x.append(2) print(y) >>> [[1,2]] Numpy arrays are different but references are forgotten after pickle/unpickle. Shared objects do not remain shared. Based on the quote

Re: [Numpy-discussion] padding options for diff

2016-10-25 Thread Peter Creasey
> Date: Mon, 24 Oct 2016 08:44:46 -0400 > From: Matthew Harrigan > > I posted a pull request which > adds optional padding kwargs "to_begin" and "to_end" to diff. Those > options are based on what's available in ediff1d. It

[Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Stephan Hoyer
With a custom wrapper class, it's possible to preserve NumPy views when pickling: https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling This can result in significant time/space savings with pickling views along with base arrays and brings the behavior of NumPy more in