Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy

[Pauli Virtanen]

Wed, 07 Sep 2016 09:22:24 -0700, Nathaniel Smith kirjoitti:
[clip]
> I wonder if there is any way we can avoid the flag, and just make this
> happen automatically when appropriate? nditer has too many "unbreak-me"
> flags already.
> 
> Are there any cases where we *don't* want the copy-if-overlap behavior?
> Traditionally overlap has triggered undefined behavior, so there's no
> backcompat issue, right?

I didn't put it on by default, because of backward compatibility and side 
effects that break things.

On side effects: there are some bugs in ufunc code that need fixing if 
the flag is turned on (wheremask code breaks, and ufuncs write to wrong 
output arrays). Moreover, copying write operands with updateifcopy marks 
the original arrays as read-only, until the copied array is decrefed. 
There may also be other side effects that are not so obvious.

The PR is not mergeable if the flag would be on by default --- that 
requires inspecting all the uses of the iterator in the numpy codebase 
and making sure there's no weird stuff done. I'm not sure how much 3rd 
party code is using the iterator, but I'm a bit worried also that copies 
break assumptions also there.

It might be possible to turn it on by default for operands with COPY or 
UPDATEIFCOPY flags --- but I'm not sure if that's helpful (now you'd need 
to set the flags to all input operands).

-- 
Pauli Virtanen

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy

[Sebastian Berg]

On Mi, 2016-09-07 at 09:22 -0700, Nathaniel Smith wrote:
> On Sep 7, 2016 9:03 AM, "Sebastian Berg" 
> wrote:
> >
> > Hi all,
> >
> > Pauli just opened a nice pull request [1] to add overlap detection
> to
> > the new iterator, this means adding a new iterator flag:
> >
> > `NPY_ITER_COPY_IF_OVERLAP`
> >
> > If passed to the iterator (also exposed in python), the iterator
> will
> > copy the operands such that reading and writing should only occur
> for
> > identical operands. For now this is implemented by always copying
> the
> > output/writable operand (this could be improved though, so I would
> not
> > say its fixed API).
> I wonder if there is any way we can avoid the flag, and just make
> this happen automatically when appropriate? nditer has too many
> "unbreak-me" flags already.
> Are there any cases where we *don't* want the copy-if-overlap
> behavior? Traditionally overlap has triggered undefined behavior, so
> there's no backcompat issue, right?

Puh, I remember weird abuses, that sometimes stopped working. Even just
adding it to ufuncs might destroy some weird cases in someones
script

Whether or not we can just make it default, might be worth thinking
about it. What do downstream projects that use the API think? My guess
is that would be projects such as numexpr, numba, or I think theano?

Maybe another approach is to think about some other way to make good
defaults to the iterator easier/saner. Heck, I wonder if we would
default to things like "zero size ok" and warned about it, anyone would
notice unless as in: Oh I should make it zero size ok ;).

- Sebastian


> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy

[Nathaniel Smith]

On Sep 7, 2016 9:03 AM, "Sebastian Berg"  wrote:
>
> Hi all,
>
> Pauli just opened a nice pull request [1] to add overlap detection to
> the new iterator, this means adding a new iterator flag:
>
> `NPY_ITER_COPY_IF_OVERLAP`
>
> If passed to the iterator (also exposed in python), the iterator will
> copy the operands such that reading and writing should only occur for
> identical operands. For now this is implemented by always copying the
> output/writable operand (this could be improved though, so I would not
> say its fixed API).

I wonder if there is any way we can avoid the flag, and just make this
happen automatically when appropriate? nditer has too many "unbreak-me"
flags already.

Are there any cases where we *don't* want the copy-if-overlap behavior?
Traditionally overlap has triggered undefined behavior, so there's no
backcompat issue, right?

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy

[Sebastian Berg]

Hi all,

Pauli just opened a nice pull request [1] to add overlap detection to
the new iterator, this means adding a new iterator flag:

`NPY_ITER_COPY_IF_OVERLAP`

If passed to the iterator (also exposed in python), the iterator will
copy the operands such that reading and writing should only occur for
identical operands. For now this is implemented by always copying the
output/writable operand (this could be improved though, so I would not
say its fixed API).

Since adding this flag is new API, please feel free to suggest other
names/approaches or even decline the change ;).


This is basically a first step, which should be easily followed by
adding overlap detection to ufuncs, removing traps such as the well (or
not so well known) `a += a.T`. Other parts of numpy may follow one by
one.

The work is based on his older awesome new memory overlap detection
implementation.

If there are no comments, I will probably merge it very soon, so we can
look at the follow up things.

- Sebastian


[1] https://github.com/numpy/numpy/pull/8026

signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion