[Numpy-discussion] Re: Find location of slice in it's base

[Dom Grigonis]

Thank you, sounds just what I need. Will work on this in due time.

> On 1 Sep 2023, at 00:38, Robert Kern  wrote:
> 
> On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis  > wrote:
> Hi everyone,
> 
> I am working with shared arrays and their slices. And trying to subclass 
> ndarray in such way so that they remap to memory on unpickling. I am using 
> package SharedArray, which doesn’t micro-manage memory locations, but rather 
> stores the whole map via shm using unique name. One issue I ran into is that 
> when I pickle & unpickle slice of memory mapped array, I need to store the 
> spec of the subset so that I can retrieve the same portion. 
> 
> The issue I am working on has been briefly discussed on stack: 
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array
>  
> .
>  Although I am not sure if it is the exact same one.
> 
> So, in short, is there a way to determine location of a slice in original 
> array. Ideally, it would be indexing which works for any non-copy 
> slice/subset possible.
> 
> If you chase the `.base` chain all the way down, you get to a 
> `SharedArray.map_owner` object with its `.addr` attribute giving the memory 
> address that's assigned to it in the current process. This will be the same 
> address that's in the `'data'` key of that first `ndarray` returned from 
> `SharedArray.create()` that you are making your views from. In your view 
> `ndarray` (which may be a view of views, with slices, transposes, reshapes, 
> and arbitrary `stride_tricks` manipulations in between). If you store the 
> difference between the view `ndarray`'s `'data'` address from the 
> `map_owner.addr` address, I think that's all you need to recreate the array 
> in a different process which gets assigned a different memory address for the 
> same `shm` name. Just add that offset to the `map_owner.addr` and restore the 
> rest of the information in `__array_interface__`, and I think you should be 
> good to go. I don't think you'll need to infer or represent the precise path 
> of Python-level operations (slices, transposes, reshapes, etc.) to which it 
> got to that point.
> 
> -- 
> Robert Kern
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org 
> 
> To unsubscribe send an email to numpy-discussion-le...@python.org 
> 
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ 
> 
> Member address: dom.grigo...@gmail.com 
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Robert Kern]

On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis  wrote:

> Hi everyone,
>
> I am working with shared arrays and their slices. And trying to subclass
> ndarray in such way so that they remap to memory on unpickling. I am using
> package SharedArray, which doesn’t micro-manage memory locations, but
> rather stores the whole map via shm using unique name. One issue I ran into
> is that when I pickle & unpickle slice of memory mapped array, I need to
> store the spec of the subset so that I can retrieve the same portion.
>
> The issue I am working on has been briefly discussed on stack:
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
> Although I am not sure if it is the exact same one.
>
> So, in short, is there a way to determine location of a slice in original
> array. Ideally, it would be indexing which works for any non-copy
> slice/subset possible.
>

If you chase the `.base` chain all the way down, you get to a
`SharedArray.map_owner` object with its `.addr` attribute giving the memory
address that's assigned to it in the current process. This will be the same
address that's in the `'data'` key of that first `ndarray` returned from
`SharedArray.create()` that you are making your views from. In your view
`ndarray` (which may be a view of views, with slices, transposes, reshapes,
and arbitrary `stride_tricks` manipulations in between). If you store the
difference between the view `ndarray`'s `'data'` address from the
`map_owner.addr` address, I think that's all you need to recreate the array
in a different process which gets assigned a different memory address for
the same `shm` name. Just add that offset to the `map_owner.addr` and
restore the rest of the information in `__array_interface__`, and I think
you should be good to go. I don't think you'll need to infer or represent
the precise path of Python-level operations (slices, transposes, reshapes,
etc.) to which it got to that point.

-- 
Robert Kern
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Dom Grigonis]

I am looking for something that would work for stride tricks too. Essentially, 
any possible view.

> On 31 Aug 2023, at 23:03, Aaron Meurer  wrote:
> 
> In principle this can be reconstructed from the strides, shape, and
> base memory address (a.ctypes.data) of the view and base arrays.
> However, not all views can be reconstructed using slices alone, for
> example, views from reshape operations or stride tricks. I don't know
> if it's possible to just construct a view directly given a base,
> offset, shape, and strides, but I think ideally that's what you'd
> want.
> 
> Aaron Meurer
> 
> On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis  wrote:
>> 
>> Hi everyone,
>> 
>> I am working with shared arrays and their slices. And trying to subclass 
>> ndarray in such way so that they remap to memory on unpickling. I am using 
>> package SharedArray, which doesn’t micro-manage memory locations, but rather 
>> stores the whole map via shm using unique name. One issue I ran into is that 
>> when I pickle & unpickle slice of memory mapped array, I need to store the 
>> spec of the subset so that I can retrieve the same portion.
>> 
>> The issue I am working on has been briefly discussed on stack: 
>> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
>>  Although I am not sure if it is the exact same one.
>> 
>> So, in short, is there a way to determine location of a slice in original 
>> array. Ideally, it would be indexing which works for any non-copy 
>> slice/subset possible.
>> 
>> Kind regards,
>> Dg
>> 
>> —Nothing ever dies, just enters the state of deferred evaluation—
>> 
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: asmeu...@gmail.com
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: dom.grigo...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Aaron Meurer]

In principle this can be reconstructed from the strides, shape, and
base memory address (a.ctypes.data) of the view and base arrays.
However, not all views can be reconstructed using slices alone, for
example, views from reshape operations or stride tricks. I don't know
if it's possible to just construct a view directly given a base,
offset, shape, and strides, but I think ideally that's what you'd
want.

Aaron Meurer

On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis  wrote:
>
> Hi everyone,
>
> I am working with shared arrays and their slices. And trying to subclass 
> ndarray in such way so that they remap to memory on unpickling. I am using 
> package SharedArray, which doesn’t micro-manage memory locations, but rather 
> stores the whole map via shm using unique name. One issue I ran into is that 
> when I pickle & unpickle slice of memory mapped array, I need to store the 
> spec of the subset so that I can retrieve the same portion.
>
> The issue I am working on has been briefly discussed on stack: 
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
>  Although I am not sure if it is the exact same one.
>
> So, in short, is there a way to determine location of a slice in original 
> array. Ideally, it would be indexing which works for any non-copy 
> slice/subset possible.
>
> Kind regards,
> Dg
>
> —Nothing ever dies, just enters the state of deferred evaluation—
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com