On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan
wrote:
> It seems pickle keeps track of references for basic python types.
>
> x = [1]
> y = [x]
> x,y = pickle.loads(pickle.dumps((x,y)))
> x.append(2)
> print(y)
[[1,2]]
Yes, but the problem is: suppose I have a 10 gigabyte array, and then
On Tue, Oct 25, 2016 at 7:05 PM, Feng Yu wrote:
>
> Hi,
>
> Just another perspective. base' and 'data' in PyArrayObject are two
> separate variables.
>
> base can point to any PyObject, but it is `data` that defines where
> data is accessed in memory.
>
> 1. There is no clear way to pickle a point
Hi,
Just another perspective. base' and 'data' in PyArrayObject are two
separate variables.
base can point to any PyObject, but it is `data` that defines where
data is accessed in memory.
1. There is no clear way to pickle a pointer (`data`) in a meaningful
way. In order for `data` member to mak
On Tue, Oct 25, 2016 at 5:09 PM, Matthew Harrigan <
harrigan.matt...@gmail.com> wrote:
>
> It seems pickle keeps track of references for basic python types.
>
> x = [1]
> y = [x]
> x,y = pickle.loads(pickle.dumps((x,y)))
> x.append(2)
> print(y)
> >>> [[1,2]]
>
> Numpy arrays are different but refe
It seems pickle keeps track of references for basic python types.
x = [1]
y = [x]
x,y = pickle.loads(pickle.dumps((x,y)))
x.append(2)
print(y)
>>> [[1,2]]
Numpy arrays are different but references are forgotten after
pickle/unpickle. Shared objects do not remain shared. Based on the quote
below
On Tue, Oct 25, 2016 at 3:07 PM, Stephan Hoyer wrote:
>
> On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote:
>>
>> Concretely, what do would you suggest should happen with:
>>
>> base = np.zeros(1)
>> view = base[:10]
>>
>> # case 1
>> pickle.dump(view, file)
>>
>> # case 2
>> pickle
On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith wrote:
> Concretely, what do would you suggest should happen with:
>
> base = np.zeros(1)
> view = base[:10]
>
> # case 1
> pickle.dump(view, file)
>
> # case 2
> pickle.dump(base, file)
> pickle.dump(view, file)
>
> # case 3
> pickle.dump(
On Tue, Oct 25, 2016 at 12:38 PM, Stephan Hoyer wrote:
> With a custom wrapper class, it's possible to preserve NumPy views when
> pickling:
> https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling
>
> This can result in significant time/space savings with pickling views
With a custom wrapper class, it's possible to preserve NumPy views when
pickling:
https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling
This can result in significant time/space savings with pickling views along
with base arrays and brings the behavior of NumPy more in l