Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan wrote: > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) [[1,2]] Yes, but the problem is: suppose I have a

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 7:05 PM, Feng Yu wrote: > > Hi, > > Just another perspective. base' and 'data' in PyArrayObject are two > separate variables. > > base can point to any PyObject, but it is `data` that defines where > data is accessed in memory. > > 1. There is no

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan < harrigan.matt...@gmail.com> wrote: > > It seems pickle keeps track of references for basic python types. > > x = [1] > y = [x] > x,y = pickle.loads(pickle.dumps((x,y))) > x.append(2) > print(y) > >>> [[1,2]] > > Numpy arrays are different but

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Matthew Harrigan
It seems pickle keeps track of references for basic python types. x = [1] y = [x] x,y = pickle.loads(pickle.dumps((x,y))) x.append(2) print(y) >>> [[1,2]] Numpy arrays are different but references are forgotten after pickle/unpickle. Shared objects do not remain shared. Based on the quote

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Robert Kern
On Tue, Oct 25, 2016 at 3:07 PM, Stephan Hoyer wrote: > > On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: >> >> Concretely, what do would you suggest should happen with: >> >> base = np.zeros(1) >> view = base[:10] >> >> # case 1 >>

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Stephan Hoyer
On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote: > Concretely, what do would you suggest should happen with: > > base = np.zeros(1) > view = base[:10] > > # case 1 > pickle.dump(view, file) > > # case 2 > pickle.dump(base, file) > pickle.dump(view, file) > > # case

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Nathaniel Smith
On Tue, Oct 25, 2016 at 12:38 PM, Stephan Hoyer wrote: > With a custom wrapper class, it's possible to preserve NumPy views when > pickling: > https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling > > This can result in significant time/space savings