Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy
Wed, 07 Sep 2016 09:22:24 -0700, Nathaniel Smith kirjoitti: [clip] > I wonder if there is any way we can avoid the flag, and just make this > happen automatically when appropriate? nditer has too many "unbreak-me" > flags already. > > Are there any cases where we *don't* want the copy-if-overlap behavior? > Traditionally overlap has triggered undefined behavior, so there's no > backcompat issue, right? I didn't put it on by default, because of backward compatibility and side effects that break things. On side effects: there are some bugs in ufunc code that need fixing if the flag is turned on (wheremask code breaks, and ufuncs write to wrong output arrays). Moreover, copying write operands with updateifcopy marks the original arrays as read-only, until the copied array is decrefed. There may also be other side effects that are not so obvious. The PR is not mergeable if the flag would be on by default --- that requires inspecting all the uses of the iterator in the numpy codebase and making sure there's no weird stuff done. I'm not sure how much 3rd party code is using the iterator, but I'm a bit worried also that copies break assumptions also there. It might be possible to turn it on by default for operands with COPY or UPDATEIFCOPY flags --- but I'm not sure if that's helpful (now you'd need to set the flags to all input operands). -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy
On Mi, 2016-09-07 at 09:22 -0700, Nathaniel Smith wrote: > On Sep 7, 2016 9:03 AM, "Sebastian Berg" > wrote: > > > > Hi all, > > > > Pauli just opened a nice pull request [1] to add overlap detection > to > > the new iterator, this means adding a new iterator flag: > > > > `NPY_ITER_COPY_IF_OVERLAP` > > > > If passed to the iterator (also exposed in python), the iterator > will > > copy the operands such that reading and writing should only occur > for > > identical operands. For now this is implemented by always copying > the > > output/writable operand (this could be improved though, so I would > not > > say its fixed API). > I wonder if there is any way we can avoid the flag, and just make > this happen automatically when appropriate? nditer has too many > "unbreak-me" flags already. > Are there any cases where we *don't* want the copy-if-overlap > behavior? Traditionally overlap has triggered undefined behavior, so > there's no backcompat issue, right? Puh, I remember weird abuses, that sometimes stopped working. Even just adding it to ufuncs might destroy some weird cases in someones script Whether or not we can just make it default, might be worth thinking about it. What do downstream projects that use the API think? My guess is that would be projects such as numexpr, numba, or I think theano? Maybe another approach is to think about some other way to make good defaults to the iterator easier/saner. Heck, I wonder if we would default to things like "zero size ok" and warned about it, anyone would notice unless as in: Oh I should make it zero size ok ;). - Sebastian > -n > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy
On Sep 7, 2016 9:03 AM, "Sebastian Berg" wrote: > > Hi all, > > Pauli just opened a nice pull request [1] to add overlap detection to > the new iterator, this means adding a new iterator flag: > > `NPY_ITER_COPY_IF_OVERLAP` > > If passed to the iterator (also exposed in python), the iterator will > copy the operands such that reading and writing should only occur for > identical operands. For now this is implemented by always copying the > output/writable operand (this could be improved though, so I would not > say its fixed API). I wonder if there is any way we can avoid the flag, and just make this happen automatically when appropriate? nditer has too many "unbreak-me" flags already. Are there any cases where we *don't* want the copy-if-overlap behavior? Traditionally overlap has triggered undefined behavior, so there's no backcompat issue, right? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] New iterator API (nditer): Overlap detection in NumPy
Hi all, Pauli just opened a nice pull request [1] to add overlap detection to the new iterator, this means adding a new iterator flag: `NPY_ITER_COPY_IF_OVERLAP` If passed to the iterator (also exposed in python), the iterator will copy the operands such that reading and writing should only occur for identical operands. For now this is implemented by always copying the output/writable operand (this could be improved though, so I would not say its fixed API). Since adding this flag is new API, please feel free to suggest other names/approaches or even decline the change ;). This is basically a first step, which should be easily followed by adding overlap detection to ufuncs, removing traps such as the well (or not so well known) `a += a.T`. Other parts of numpy may follow one by one. The work is based on his older awesome new memory overlap detection implementation. If there are no comments, I will probably merge it very soon, so we can look at the follow up things. - Sebastian [1] https://github.com/numpy/numpy/pull/8026 signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion