Re: [Numpy-discussion] __numpy_ufunc__

[Marten van Kerkwijk]

HI Stephan,

Indeed, `__array_ufunc__` is None would be for classes that interact
with arrays only as if they were any other numeric type, and thus have
no use for ufuncs, but may need normal operations (astropy's `Unit`
class is a reasonably good example).

Your example also makes clear that, indeed, setting __array__ or
__array_ufunc__ to None implies different things, so concretely here
the proposal is that if `__array_ufunc__` is None, ndarray methods
will return `NotImplemented`.

As an aside, I think that if we do not have something like that, we'll
be stuck with supporting `__array_priority__`. (Which is OK by me too,
but it might as well be a conscious choice.)

All the best,

Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Stephan Hoyer]

On Wed, Feb 22, 2017 at 6:31 AM, Marten van Kerkwijk <
m.h.vankerkw...@gmail.com> wrote:

> It seems to me entirely logical (but then it would, I suggested it
> before...) that we allow opting out by setting `__array_ufunc__` to
> None; in that case, binops return NotImplemented and ufuncs raise
> errors. (In addtion, or alternatively, one could allow setting
> `__array__` to None, which would generally disable something to be
> turned into an array object).
>

This is indeed appealing, but I recall this was still a point of contention
because it leaves intact two different ways to override arithmetic
involving numpy arrays. Mimicking all this logic on classes designed to
wrap well-behaved array-like classes (e.g., xarray, which can wrap NumPy or
Dask arrays) could be painful -- it's easier to just call np.add and let it
handle all the dispatching rather than also worrying about NotImplemented.
That said, I think the opt-out is probably OK, as long we make it clear
that defining __array_ufunc__ to do arithmetic overriding is the preferred
solution (and provide appropriate Mixin classes to make this easier).

Just to be clear: if __array__ = None but __array_ufunc__ is defined, this
would be a class that defines array-like operations but can't be directly
converted into a NumPy array? For example, a scipy.sparse matrix?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Marten van Kerkwijk]

Hi All,

I'd very much like to get `__array_ufunc__` in, and am willing to do
some work, but fear we need to get past the last sticking point. As I
noted in Chuck's PR [1], in python 3.6 there is now an explicit
language change [2], which I think is relevant:
```
It is now possible to set a special method to None to indicate that
the corresponding operation is not available. For example, if a class
sets __iter__() to None, the class is not iterable.
```

It seems to me entirely logical (but then it would, I suggested it
before...) that we allow opting out by setting `__array_ufunc__` to
None; in that case, binops return NotImplemented and ufuncs raise
errors. (In addtion, or alternatively, one could allow setting
`__array__` to None, which would generally disable something to be
turned into an array object).

But I should note that I much prefer to get something in over wait yet
another round! In astropy, there is now more and more clamouring to
offer options for pure ndarray functions where quantities are more
logical because quantities are twice as slow -- this would instantly
be solved with __array_ufunc__... If we can decide on this, then I'd
gladly help with remaining issues (e.g., the `ndarray.__array_ufunc__`
method, so super can be used).

All the best,

Marten

[1] https://github.com/numpy/numpy/pull/8247
[2] https://docs.python.org/3.6/whatsnew/3.6.html#other-language-changes
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Mon, Nov 7, 2016 at 1:08 AM, Ralf Gommers  wrote:

>
>
> On Mon, Nov 7, 2016 at 9:10 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>>> request enabling it . Likely
>>> we will want to make some changes up front before merging that, so some
>>> discussion is in order.
>>>
>>>
>> As a first order of business, let's decide whether to remove the index
>> and rename `__numpy_ufunc__`. The motivation for this is discussed in issue
>> #5986. 
>> If we decide positive on that (I'm in favor),
>>
>
> It seems like everyone on that issue is in favor or at least +0. So +1
> from me too.
>
>
>> I would be happy with the proposed name `__array_ufunc__`, although
>> something more descriptive like `__handle_ufunc__` might be better.
>>
>
>
>
>> This is a wonderful opportunity for bike shedding for those so inclined ;)
>>
>
Let me try to break things down an bit.


*Uncontroversial*


   - Rename __numpy_ufunc__ to __array_ufunc__
   - Remove index
   - Out is always a tuple


I think this much is useful even if nothing else is done.

*Goals*


   - Deprecate __array_priority__
   - Ufuncs should succeed or error, never return NotImplemented
   - Add __array_ufunc__ stub to ndarray.


I don't think these are controversial either, but they are longer term
except possibly the last. Note that never returning NotImplemented
disentangles ufuncs from ndarray binops, which I think is a good thing.

*Binops*


Here we come to the crux of the last arguments. The functions used for
binops can currently be set dynamically, the method that is currently used
to set them when the ufunc module is loaded. I think we want to do away
with that at some point and fix a set of ufuncs with which they are
implemented. This allows folks to overide the binop behavior using
__array_ufunc__. I think that is mostly of interest to people who are
subclassing ndarray and with that restriction doesn't bother me except that
it entangles ufuncs with binops. However, what I'd like to see as well is
an opt out for objects that don't care about ufuncs, but want to override
the python numeric operators, something simple like `__me_me_me__`, or,
more seriously, `__array_opt_out__`  that will only come into play if the
defining object is on the right hand side of an instance of ndarray. In
that case the binop would return NotImplemented so as to defer to the
Python machinery.  Note that  __array_priority__ is currently (ab)used for
this.

*Numpy scalars*

Numpy scalars default to the corresponding PyArray_Type or
PyGenericArrType_Type unless both arguments can be converted to the same c
type as the calling scalar, so I don't think there is a problem there. Note
that they do check _array_priority_ before trying to convert unknown
objects to array scalars in a fallback case.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Ralf Gommers]

On Mon, Nov 7, 2016 at 9:21 AM, Nathan Goldbaum 
wrote:

> I'm interested in this, but was never able to dive in to the lengthy
> discussions on this. I'm curious if there is a summary of the current
> state of things somewhere that I could read to catch up.


Old proposal, original implementation:
https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst

Main issue for discussion: https://github.com/numpy/numpy/issues/5844. That
really needs a summary, but I'm not sure there's an up-to-date one.

Ralf



> On Sunday, November 6, 2016, Charles R Harris 
> wrote:
>
>>
>>
>> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>>> request enabling it . Likely
>>> we will want to make some changes up front before merging that, so some
>>> discussion is in order.
>>>
>>>
>> As a first order of business, let's decide whether to remove the index
>> and rename `__numpy_ufunc__`. The motivation for this is discussed in issue
>> #5986. 
>> If we decide positive on that (I'm in favor), I would be happy with the
>> proposed name `__array_ufunc__`, although something more descriptive like
>> `__handle_ufunc__` might be better. This is a wonderful opportunity for
>> bike shedding for those so inclined ;)
>>
>> Chuck
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Ralf Gommers]

On Mon, Nov 7, 2016 at 9:10 AM, Charles R Harris 
wrote:

>
>
> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>> request enabling it . Likely
>> we will want to make some changes up front before merging that, so some
>> discussion is in order.
>>
>>
> As a first order of business, let's decide whether to remove the index and
> rename `__numpy_ufunc__`. The motivation for this is discussed in issue
> #5986. 
> If we decide positive on that (I'm in favor),
>

It seems like everyone on that issue is in favor or at least +0. So +1 from
me too.


> I would be happy with the proposed name `__array_ufunc__`, although
> something more descriptive like `__handle_ufunc__` might be better.
>



> This is a wonderful opportunity for bike shedding for those so inclined ;)
>

pass :)

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Nathan Goldbaum]

I'm interested in this, but was never able to dive in to the lengthy
discussions on this. I'm curious if there is a summary of the current state
of things somewhere that I could read to catch up.

On Sunday, November 6, 2016, Charles R Harris 
wrote:

>
>
> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
> charlesr.har...@gmail.com
> > wrote:
>
>> Hi All,
>>
>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>> request enabling it . Likely
>> we will want to make some changes up front before merging that, so some
>> discussion is in order.
>>
>>
> As a first order of business, let's decide whether to remove the index and
> rename `__numpy_ufunc__`. The motivation for this is discussed in issue
> #5986. 
> If we decide positive on that (I'm in favor), I would be happy with the
> proposed name `__array_ufunc__`, although something more descriptive like
> `__handle_ufunc__` might be better. This is a wonderful opportunity for
> bike shedding for those so inclined ;)
>
> Chuck
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris  wrote:

> Hi All,
>
> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
> request enabling it . Likely we
> will want to make some changes up front before merging that, so some
> discussion is in order.
>
>
As a first order of business, let's decide whether to remove the index and
rename `__numpy_ufunc__`. The motivation for this is discussed in issue
#5986. 
If we decide positive on that (I'm in favor), I would be happy with the
proposed name `__array_ufunc__`, although something more descriptive like
`__handle_ufunc__` might be better. This is a wonderful opportunity for
bike shedding for those so inclined ;)

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

Hi All,

For those interested in continuing the __numpy_ufunc__ saga, there is a pull
request enabling it . Likely we
will want to make some changes up front before merging that, so some
discussion is in order.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Marten van Kerkwijk]

Hi Chuck,

> We were pretty close. IIRC, the outstanding issue was some sort of override.

Correct. With a general sentiment of those downstream that it would be
great to merge in any form, as it will be really helpful! (Generic
speedup of factor of 2 for computationally expensive ufuncs (sin, cos,
etc.) that needs scaling in Quantity...)

> At the developer meeting at scipy 2015 it was agreed that it would be easy
> to finish things up under the rubric "make Pauli happy".

That would certainly make me happy too!

Other items that were brought up (trying to summarize from issues
linked above, and links therein):

1. Remove index argument
2. Out always a tuple
3. Let ndarray have a __numpy_ufunc__ stub, so one can super it.

Here, the first item implied a possible name change (to
__array_ufunc__); if that's too troublesome, I don't think it really
hurts to have the argument, though it is somewhat "unclean" for the
case that only the output has __numpy_ufunc__.

All the best,

Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Mon, Oct 31, 2016 at 11:39 AM, Stephan Hoyer  wrote:

> Recall that I think we wanted to rename this to __array_ufunc__, so we
> could change the function signature: https://github.com/numpy/
> numpy/issues/5986
>
> I'm still a little nervous about this. Chunk -- what is your proposal for
> resolving the outstanding issues from https://github.com/numpy/
> numpy/issues/5844?
>

We were pretty close. IIRC, the outstanding issue was some sort of
override. At the developer meeting at scipy 2015 it was agreed that it
would be easy to finish things up under the rubric "make Pauli happy". But
that wasn't happening which is why I asked Nathaniel to disable it for
1.10.0. It is now a year later, things have cooled, and, IMHO, it is time
to take another shot at it.



Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Stephan Hoyer]

Recall that I think we wanted to rename this to __array_ufunc__, so we
could change the function signature:
https://github.com/numpy/numpy/issues/5986

I'm still a little nervous about this. Chunk -- what is your proposal for
resolving the outstanding issues from
https://github.com/numpy/numpy/issues/5844?

On Mon, Oct 31, 2016 at 10:31 AM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Mon, Oct 31, 2016 at 11:08 AM, Marten van Kerkwijk <
> m.h.vankerkw...@gmail.com> wrote:
>
>> Hi Chuck,
>>
>> I've revived my Quantity PRs that use __numpy_ufunc__ but is it
>> correct that at present in *dev, one cannot use it?
>>
>
> It's not enabled yet.
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Mon, Oct 31, 2016 at 11:08 AM, Marten van Kerkwijk <
m.h.vankerkw...@gmail.com> wrote:

> Hi Chuck,
>
> I've revived my Quantity PRs that use __numpy_ufunc__ but is it
> correct that at present in *dev, one cannot use it?
>

It's not enabled yet.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Marten van Kerkwijk]

Hi Chuck,

I've revived my Quantity PRs that use __numpy_ufunc__ but is it
correct that at present in *dev, one cannot use it?

All the best,

Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Marten van Kerkwijk]

> The __numpy_ufunc__ functionality is the last bit I want for 1.12.0, the
> rest of the remaining changes I can kick forward to 1.13.0. I will start
> taking a look tomorrow, probably starting with Nathaniel's work.

Great! I'll revive the Quantity PRs that implement __numpy_ufunc__!

-- Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Sat, Oct 29, 2016 at 7:03 PM, Stephan Hoyer  wrote:

> I'm happy to revisit the __numpy_ufunc__ discussion (I still want to see
> it happen!), but I don't recall scalars being a point of contention.
>

The __numpy_ufunc__ functionality is the last bit I want for 1.12.0, the
rest of the remaining changes I can kick forward to 1.13.0. I will start
taking a look tomorrow, probably starting with Nathaniel's work.


>
> The obvious thing to do with scalars would be to treat them the same as
> 0-dimensional arrays, though I might be missing some nuance...
>

That's my thought. Currently they just look at __array_priority__ and call
the corresponding array method if needed, so that maybe needs some
improvement and a formal statement of intent.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Stephan Hoyer]

I'm happy to revisit the __numpy_ufunc__ discussion (I still want to see it
happen!), but I don't recall scalars being a point of contention.

The obvious thing to do with scalars would be to treat them the same as
0-dimensional arrays, though I might be missing some nuance...

On Sat, Oct 29, 2016 at 6:02 AM, Charles R Harris  wrote:

> Hi All,
>
> Does anyone remember discussion of numpy scalars apropos __numpy_ufunc__?
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

Hi All,

Does anyone remember discussion of numpy scalars apropos __numpy_ufunc__?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__ and 1.9 release

[Julian Taylor]

On 15.07.2014 20:06, Julian Taylor wrote:
 hi,
 as you may know we want to release numpy 1.9 soon. We should have solved
 most indexing regressions the first beta showed.
 
 The remaining blockers are finishing the new __numpy_ufunc__ feature.
 This feature should allow for alternative method to overriding the
 behavior of ufuncs from subclasses.
 It is described here:
 https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst
 
 The current blocker issues are:
 https://github.com/numpy/numpy/issues/4753
 https://github.com/numpy/numpy/pull/4815
 
 I'm not to familiar with all the complications of subclassing so I can't
 really say how hard this is to solve.
 My issue is that it there still seems to be debate on how to handle
 operator overriding correctly and I am opposed to releasing a numpy with
 yet another experimental feature that may or may not be finished
 sometime later. Having datetime in infinite experimental state is bad
 enough.
 I think nobody is served well if we release 1.9 with the feature
 prematurely based on a not representative set of users and the later
 after more users showed up see we have to change its behavior.
 
 So I'm wondering if we should delay the introduction of this feature to
 1.10 or is it important enough to wait until there is a consensus on the
 remaining issues?
 

So its been a week and we got a few answers and new issues.
To summarize:
- to my knowledge no progress was made on the issues
- scipy already has a released version using the current implementation
- no very loud objections to delaying the feature to 1.10
- I am still unfamiliar with the problematics of subclassing, but don't
want to release something new which has unsolved issues.

That scipy already uses it in a released version (0.14) is very
problematic. Can maybe someone give some insight if the potential
changes to resolve the remaining issues would break scipy?
If so we have following choices:
- declare what we have as final and close the remaining issues as 'won't
fix'. Any changes would have to have a new name __numpy_ufunc2__ or a
somehow versioned the interface
- delay the introduction, potentially breaking scipy 0.14 when numpy
1.10 is released.

I would like to get the next (and last) numpy 1.9 beta out soon, so I
would propose to make a decision until this Saturday the 26.02.2014
however misinformed it may be.

Please note that the numpy 1.10 release cycle is likely going to be a
very long one as we are currently planning to change a bunch of default
behaviours that currently raise deprecation warnings and possibly will
try to fix string types, text IO and datetime. Please see the future
changes notes in the current 1.9.x release notes.
If we delay numpy_ufunc it is not unlikely that it will take a year
until we release 1.10. Though we could still put it into a earlier 1.9.1.

Cheers,
Julian
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__ and 1.9 release

[Pauli Virtanen]

23.07.2014, 20:37, Julian Taylor kirjoitti:
[clip: __numpy_ufunc__]
 So its been a week and we got a few answers and new issues. To
 summarize: - to my knowledge no progress was made on the issues -
 scipy already has a released version using the current
 implementation - no very loud objections to delaying the feature to
 1.10 - I am still unfamiliar with the problematics of subclassing,
 but don't want to release something new which has unsolved issues.
 
 That scipy already uses it in a released version (0.14) is very 
 problematic. Can maybe someone give some insight if the potential 
 changes to resolve the remaining issues would break scipy?
 
 If so we have following choices:
 
 - declare what we have as final and close the remaining issues as
 'won't fix'. Any changes would have to have a new name
 __numpy_ufunc2__ or a somehow versioned the interface - delay the
 introduction, potentially breaking scipy 0.14 when numpy 1.10 is
 released.
 
 I would like to get the next (and last) numpy 1.9 beta out soon, so
 I would propose to make a decision until this Saturday the
 26.02.2014 however misinformed it may be.

It seems fairly unlikely to me that the `__numpy_ufunc__` interface
itself requires any changes. I believe the definition of the interface
is quite safe to consider as fixed --- it is a fairly straighforward
hook for Numpy ufuncs. (There are also no essential changes in it
since last year.)

For the binary operator overriding, Scipy sets the constraint that

ndarray * spmatrix

MUST call spmatrix.__rmul__ even if spmatrix.__numpy_ufunc__ is
defined. spmatrixes are not ndarray subclasses, and various
subclassing problems do not enter here.

Note that this binop discussion is somewhat separate from the
__numpy_ufunc__ interface itself. The only information available about
it at the binop stage is `hasattr(other, '__numpy_ufunc__')`.

   ***

Regarding the blockers:

(1) https://github.com/numpy/numpy/issues/4753

This is a bug in the argument normalization --- output arguments are
not checked for the presence of __numpy_ufunc__ if they are passed
as keyword arguments (as a positional argument it works). It's a bug
in the implementation, but I don't think it is really a blocker.

Scipy sparse matrices will in practice seldom be used as output args
for ufuncs.

   ***

(2) https://github.com/numpy/numpy/pull/4815

The is open question concerns semantics of `__numpy_ufunc__` versus
Python operator overrides. When should ndarray.__mul__(other) return
NotImplemented?

Scipy sparse matrices are not subclasses of ndarray, so the code in
question in Numpy gets to run only for

ndarray * spmatrix

This provides a constraint to what solution we can choose in Numpy to
deal with the issue:

ndarray.__mul__(spmatrix)  MUST  continue to return NotImplemented

This is the current behavior, and cannot be changed: it is not
possible to defer this to __numpy_ufunc__(ufunc=np.multiply), because
sparse matrices define `*` as the matrix multiply, and not the
elementwise multiply. (This settles one line of discussion in the
issues --- ndarray should defer.)

How Numpy currently determines whether to return NotImplemented in
this case or to call np.multiply(self, other) is by comparing
`__array_priority__` attributes of `self` and `other`. Scipy sparse
matrices define an `__array_priority__` larger than ndarrays, which
then makes a NotImplemented be returned.

The idea in the __numpy_ufunc__ NEP was to replace this with
`hasattr(other, '__numpy_ufunc__') and hasattr(other, '__rmul__')`.
However, when both self and other are ndarray subclasses in a certain
configuration, both end up returning NotImplemented, and Python raises
TypeError.

The `__array_priority__` mechanism is also broken in some of the
subclassing cases: https://github.com/numpy/numpy/issues/4766

As far as I see, the backward compatibility requirement from Scipy
only rules out the option that ndarray.__mul__(other) should
unconditionally call `np.add(self, other)`.

We have some freedom how to solve the binop vs. subclass issues. It's
possible to e.g. retain the __array_priority__ stuff as a backward
compatibility measure as we do currently.

-- 
Pauli Virtanen


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] __numpy_ufunc__ and 1.9 release

[Julian Taylor]

hi,
as you may know we want to release numpy 1.9 soon. We should have solved
most indexing regressions the first beta showed.

The remaining blockers are finishing the new __numpy_ufunc__ feature.
This feature should allow for alternative method to overriding the
behavior of ufuncs from subclasses.
It is described here:
https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst

The current blocker issues are:
https://github.com/numpy/numpy/issues/4753
https://github.com/numpy/numpy/pull/4815

I'm not to familiar with all the complications of subclassing so I can't
really say how hard this is to solve.
My issue is that it there still seems to be debate on how to handle
operator overriding correctly and I am opposed to releasing a numpy with
yet another experimental feature that may or may not be finished
sometime later. Having datetime in infinite experimental state is bad
enough.
I think nobody is served well if we release 1.9 with the feature
prematurely based on a not representative set of users and the later
after more users showed up see we have to change its behavior.

So I'm wondering if we should delay the introduction of this feature to
1.10 or is it important enough to wait until there is a consensus on the
remaining issues?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

On Wed, Jul 16, 2014 at 12:53 PM, Ralf Gommers ralf.gomm...@gmail.com
wrote:




 On Wed, Jul 16, 2014 at 10:07 AM, Nathaniel Smith n...@pobox.com wrote:

 Weirdly, I never received Chuck's original email in this thread. Should
 some list admin be informed?

 Also weirdly, my reply didn't show up on gmane. Not sure if it got
 through, so re-sending:

 It's already in, so do you mean not using? Would help to know what the
 issue is, because it's finished enough that it's already used in a released
 version of scipy (in sparse matrices).


My own feeling is that we should leave it in as it is fairly useable and
just needs to have some problematic case worked out.  The fact that scipy
already uses it is a strong argument to keep it in. I think Julian's
concern is that they won't be worked out.

Julian has started another thread on the topic and that is probably where
the conversation should continue.

snip

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__ and 1.9 release

[Charles R Harris]

On Wed, Jul 16, 2014 at 11:16 AM, Pauli Virtanen p...@iki.fi wrote:

 Hi,

 15.07.2014 21:06, Julian Taylor kirjoitti:
 [clip: __numpy_ufunc__]
  So I'm wondering if we should delay the introduction of this
  feature to 1.10 or is it important enough to wait until there is a
  consensus on the remaining issues?

 My 10c:

 The feature is not so much in hurry that it alone should delay 1.9.
 Moreover, it's best for everyone that it is bug-free on the first go,
 and it gets some real-world testing before the release. Better safe than
 sorry.

 I'd pull it out from 1.9.x branch, and iron out the remaining wrinkles
 before 1.10.


Thanks Pauli, your opinion on the matter is what I needed to see and I'll
take it as dispositive.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Sebastian Berg]

On Mi, 2014-07-16 at 09:07 +0100, Nathaniel Smith wrote:
 Weirdly, I never received Chuck's original email in this thread.
 Should some list admin be informed?
 

I send some mails yesterday and they never arrived... Not sure if it is
a problem on my side or not.

 I also am not sure what/where Julian's comments were, so I second the
 call for context :-). Putting it off until 1.10 doesn't seem like an
 obviously bad idea to me, but specifics would help...
 
 (__numpy_ufunc__ is the new system for allowing arbitrary third party
 objects to override how ufuncs are applied to them, i.e. it means
 np.sin(sparsemat) and np.sin(gpuarray) can be defined to do something
 sensible. Conceptually it replaces the old
 __array_prepare__/__array_wrap__ system, which was limited to ndarray
 subclasses and has major limits on what you can do. Of course
 __array_prepare/wrap__ will also continue to be supported for
 compatibility.)
 
 -n
 
 On 16 Jul 2014 00:10, Benjamin Root ben.r...@ou.edu wrote:
 Perhaps a bit of context might be useful? How is numpy_ufunc
 different from the ufuncs that we know and love? What are the
 known implications? What are the known shortcomings? Are there
 ABI and/or API concerns between 1.9 and 1.10?
 
 
 Ben Root
 
 
 
 On Mon, Jul 14, 2014 at 2:22 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:
 Hi All,
 
 Julian has raised the question of including
 numpy_ufunc in numpy 1.9. I don't feel strongly one
 way or the other, but it doesn't seem to be finished
 yet and 1.10 might be a better place to work out the
 remaining problems along with the astropy folks
 testing possible uses.
 
 
 Thoughts?
 
 
 Chuck 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__ and 1.9 release

[Pauli Virtanen]

Hi,

15.07.2014 21:06, Julian Taylor kirjoitti:
[clip: __numpy_ufunc__]
 So I'm wondering if we should delay the introduction of this
 feature to 1.10 or is it important enough to wait until there is a
 consensus on the remaining issues?

My 10c:

The feature is not so much in hurry that it alone should delay 1.9.
Moreover, it's best for everyone that it is bug-free on the first go,
and it gets some real-world testing before the release. Better safe than
sorry.

I'd pull it out from 1.9.x branch, and iron out the remaining wrinkles
before 1.10.

Pauli

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Nathaniel Smith]

Weirdly, I never received Chuck's original email in this thread. Should
some list admin be informed?

I also am not sure what/where Julian's comments were, so I second the call
for context :-). Putting it off until 1.10 doesn't seem like an obviously
bad idea to me, but specifics would help...

(__numpy_ufunc__ is the new system for allowing arbitrary third party
objects to override how ufuncs are applied to them, i.e. it means
np.sin(sparsemat) and np.sin(gpuarray) can be defined to do something
sensible. Conceptually it replaces the old __array_prepare__/__array_wrap__
system, which was limited to ndarray subclasses and has major limits on
what you can do. Of course __array_prepare/wrap__ will also continue to be
supported for compatibility.)

-n
On 16 Jul 2014 00:10, Benjamin Root ben.r...@ou.edu wrote:

 Perhaps a bit of context might be useful? How is numpy_ufunc different
 from the ufuncs that we know and love? What are the known implications?
 What are the known shortcomings? Are there ABI and/or API concerns between
 1.9 and 1.10?

 Ben Root


 On Mon, Jul 14, 2014 at 2:22 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Hi All,

 Julian has raised the question of including numpy_ufunc in numpy 1.9. I
 don't feel strongly one way or the other, but it doesn't seem to be
 finished yet and 1.10 might be a better place to work out the remaining
 problems along with the astropy folks testing possible uses.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Ralf Gommers]

On Mon, Jul 14, 2014 at 8:22 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,

 Julian has raised the question of including numpy_ufunc in numpy 1.9. I
 don't feel strongly one way or the other, but it doesn't seem to be
 finished yet and 1.10 might be a better place to work out the remaining
 problems along with the astropy folks testing possible uses.

 Thoughts?


It's already in, so do you mean not using? Would help to know what the
issue is, because it's finished enough that it's already used in a released
version of scipy (in sparse matrices).

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Benjamin Root]

Perhaps a bit of context might be useful? How is numpy_ufunc different from
the ufuncs that we know and love? What are the known implications? What are
the known shortcomings? Are there ABI and/or API concerns between 1.9 and
1.10?

Ben Root


On Mon, Jul 14, 2014 at 2:22 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,

 Julian has raised the question of including numpy_ufunc in numpy 1.9. I
 don't feel strongly one way or the other, but it doesn't seem to be
 finished yet and 1.10 might be a better place to work out the remaining
 problems along with the astropy folks testing possible uses.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

[Ralf Gommers]

On Wed, Jul 16, 2014 at 10:07 AM, Nathaniel Smith n...@pobox.com wrote:

 Weirdly, I never received Chuck's original email in this thread. Should
 some list admin be informed?

Also weirdly, my reply didn't show up on gmane. Not sure if it got through,
so re-sending:

It's already in, so do you mean not using? Would help to know what the
issue is, because it's finished enough that it's already used in a released
version of scipy (in sparse matrices).

Ralf

I also am not sure what/where Julian's comments were, so I second the call
 for context :-). Putting it off until 1.10 doesn't seem like an obviously
 bad idea to me, but specifics would help...

 (__numpy_ufunc__ is the new system for allowing arbitrary third party
 objects to override how ufuncs are applied to them, i.e. it means
 np.sin(sparsemat) and np.sin(gpuarray) can be defined to do something
 sensible. Conceptually it replaces the old __array_prepare__/__array_wrap__
 system, which was limited to ndarray subclasses and has major limits on
 what you can do. Of course __array_prepare/wrap__ will also continue to be
 supported for compatibility.)

-n
 On 16 Jul 2014 00:10, Benjamin Root ben.r...@ou.edu wrote:

 Perhaps a bit of context might be useful? How is numpy_ufunc different
 from the ufuncs that we know and love? What are the known implications?
 What are the known shortcomings? Are there ABI and/or API concerns between
 1.9 and 1.10?

 Ben Root


 On Mon, Jul 14, 2014 at 2:22 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Hi All,

 Julian has raised the question of including numpy_ufunc in numpy 1.9. I
 don't feel strongly one way or the other, but it doesn't seem to be
 finished yet and 1.10 might be a better place to work out the remaining
 problems along with the astropy folks testing possible uses.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] __numpy_ufunc__

[Charles R Harris]

Hi All,

Julian has raised the question of including numpy_ufunc in numpy 1.9. I
don't feel strongly one way or the other, but it doesn't seem to be
finished yet and 1.10 might be a better place to work out the remaining
problems along with the astropy folks testing possible uses.

Thoughts?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion