Re: [Numpy-discussion] Forcing gufunc to error with size zero input
On Sun, 2019-09-29 at 00:20 -0400, Warren Weckesser wrote:
> On 9/28/19, Eric Wieser wrote:
> > Can you just raise an exception in the gufuncs inner loop? Or is
> > there no
> > mechanism to do that today?
>
> Maybe? I don't know what is the idiomatic way to handle errors
> detected in an inner loop. And pushing this particular error
> detection into the inner loop doesn't feel right.
>
Basically, since you want to release the GIL, you can grab and set an
error right now. That will work, although grabbing the GIL from the
inner loop is not ideal, at least in the sense that it does not work
with subinterpreters (but numpy does not currently work with those in
any case). We do use this internally, I believe.
Well, even without dtypes, I think we probably want a few extra API
around UFuncs, and that is setup/teardown (not necessarily as such
functions), as well as a return value for the inner loop to signal
iteration stop.
There was a long discussion about that, for example here:
https://github.com/numpy/numpy/issues/12518
There is another use-case, that we probably want to allow optimized
loop selection (necessary/used in casting)..
Note that I believe all of this type of logic should be moved into a
UFuncImpl [0] object, so that it can be DType (and especially user
DType) specific without bloating up the current UFunc object too much.
Although that puts a lot of power out there, so may be good to limit it
a lot iniyially
Best,
Sebastian
[0] It was Erics suggestion/name, I do not know if it came up earlier.
>
> > I don't think you were proposing that core dimensions should
> > _never_ be
> > allowed to be 0,
>
> No, I'm not suggesting that. There are many cases where a length 0
> core dimension is fine.
>
> I'm interested in the case where there is not a meaningful definition
> of the operation on the empty set. The mean is an
> example. Currently
> `np.mean([])` generates two warnings (one useful, the other cryptic
> and apparently incidental), and returns nan. Returning nan is one
> way
> to handle such a case; another is to raise an error like
> `np.amax([])`
> does. I'd like to raise an error in the example that I'm working on
> ('peaktopeak' at https://github.com/WarrenWeckesser/npuff). The
> function is a gufunc, not a reduction of a binary operation, so the
> 'identity' argument of PyUFunc_FromFuncAndDataAndSignature has no
> effect.
>
> > but if you were I disagree. I spent a fair amount of work
> > enabling that for linalg because it provided some convenient base
> > cases.
> >
> > We could go down the route of augmenting the gufuncs signature
> > syntax to
> > support requiring non-empty dimensions, like we did for optional
> > ones -
> > although IMO we should consider switching from a string
> > minilanguage to a
> > structured object specification if we plan to go too much further
> > with
> > extending it.
>
> After only a quick glance at that code: one option is to add a '+'
> after the input names in the signature that must have a length that
> is
> at least 1. So the signature for functions like `mean` (if you were
> to reimplement it as a gufunc, and wanted an error instead of nan),
> `amax`, `ptp`, etc, would be '(i+)->()'.
>
> However, the only meaningful uses-cases of this enhancement that I've
> come up with are these simple reductions. So I don't know if making
> such a change to the signature is worthwhile. On the other hand,
> there are many examples of useful 1-d reductions that aren't the
> reduction of an associative binary operation. It might be worthwhile
> to have a new convenience function just for the case '(i)->()', maybe
> something like PyUFunc_OneDReduction_FromFuncAndData (ugh, that's
> ugly, but I think you get the idea), and that function can have an
> argument to specify that the length must be at least 1.
>
> I'll see if that is feasible, but I won't be surprised to learn that
> there are good reasons for *not* doing that.
>
> Warren
>
>
>
> > On Sat, Sep 28, 2019, 17:47 Warren Weckesser <
> > warren.weckes...@gmail.com>
> > wrote:
> >
> > > I'm experimenting with gufuncs, and I just created a simple one
> > > with
> > > signature '(i)->()'. Is there a way to configure the gufunc
> > > itself so
> > > that an empty array results in an error? Or would I have to
> > > create a
> > > Python wrapper around the gufunc that does the error checking?
> > > Currently, when passed an empty array, the ufunc loop is called
> > > with
> > > the core dimension associated with i set to 0. It would be nice
> > > if
> > > the code didn't get that far, and the ufunc machinery "knew" that
> > > this
> > > gufunc didn't accept a core dimension that is 0. I'd like to
> > > automatically get an error, something like the error produced by
> > > `np.max([])`.
> > >
> > > Warren
> > > ___
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion@python.org
> > >
Re: [Numpy-discussion] Forcing gufunc to error with size zero input
On 9/29/19, Warren Weckesser wrote:
> On 9/28/19, Eric Wieser wrote:
>> Can you just raise an exception in the gufuncs inner loop? Or is there no
>> mechanism to do that today?
>
> Maybe? I don't know what is the idiomatic way to handle errors
> detected in an inner loop. And pushing this particular error
> detection into the inner loop doesn't feel right.
>
>
>>
>> I don't think you were proposing that core dimensions should _never_ be
>> allowed to be 0,
>
>
> No, I'm not suggesting that. There are many cases where a length 0
> core dimension is fine.
>
> I'm interested in the case where there is not a meaningful definition
> of the operation on the empty set. The mean is an example. Currently
> `np.mean([])` generates two warnings (one useful, the other cryptic
> and apparently incidental), and returns nan. Returning nan is one way
> to handle such a case; another is to raise an error like `np.amax([])`
> does. I'd like to raise an error in the example that I'm working on
> ('peaktopeak' at https://github.com/WarrenWeckesser/npuff). The
> function is a gufunc, not a reduction of a binary operation, so the
> 'identity' argument of PyUFunc_FromFuncAndDataAndSignature has no
> effect.
>
>> but if you were I disagree. I spent a fair amount of work
>> enabling that for linalg because it provided some convenient base cases.
>>
>> We could go down the route of augmenting the gufuncs signature syntax to
>> support requiring non-empty dimensions, like we did for optional ones -
>> although IMO we should consider switching from a string minilanguage to a
>> structured object specification if we plan to go too much further with
>> extending it.
>
> After only a quick glance at that code: one option is to add a '+'
> after the input names in the signature that must have a length that is
> at least 1. So the signature for functions like `mean` (if you were
> to reimplement it as a gufunc, and wanted an error instead of nan),
> `amax`, `ptp`, etc, would be '(i+)->()'.
>
> However, the only meaningful uses-cases of this enhancement that I've
> come up with are these simple reductions.
Of course, just minutes after sending the email, I realized I *do*
know of other signatures that could benefit from a check on the core
dimension size. An implementation of Pearson's correlation
coefficient as a gufunc would have signature (i),(i)->(), and the core
dimension i must be at least *2* for the calculation to be well
defined. Other correlations would also likely require a nonzero core
dimension.
Warren
> So I don't know if making
> such a change to the signature is worthwhile. On the other hand,
> there are many examples of useful 1-d reductions that aren't the
> reduction of an associative binary operation. It might be worthwhile
> to have a new convenience function just for the case '(i)->()', maybe
> something like PyUFunc_OneDReduction_FromFuncAndData (ugh, that's
> ugly, but I think you get the idea), and that function can have an
> argument to specify that the length must be at least 1.
>
> I'll see if that is feasible, but I won't be surprised to learn that
> there are good reasons for *not* doing that.
>
> Warren
>
>
>
>>
>> On Sat, Sep 28, 2019, 17:47 Warren Weckesser
>> wrote:
>>
>>> I'm experimenting with gufuncs, and I just created a simple one with
>>> signature '(i)->()'. Is there a way to configure the gufunc itself so
>>> that an empty array results in an error? Or would I have to create a
>>> Python wrapper around the gufunc that does the error checking?
>>> Currently, when passed an empty array, the ufunc loop is called with
>>> the core dimension associated with i set to 0. It would be nice if
>>> the code didn't get that far, and the ufunc machinery "knew" that this
>>> gufunc didn't accept a core dimension that is 0. I'd like to
>>> automatically get an error, something like the error produced by
>>> `np.max([])`.
>>>
>>> Warren
>>> ___
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forcing gufunc to error with size zero input
On 9/28/19, Eric Wieser wrote:
> Can you just raise an exception in the gufuncs inner loop? Or is there no
> mechanism to do that today?
Maybe? I don't know what is the idiomatic way to handle errors
detected in an inner loop. And pushing this particular error
detection into the inner loop doesn't feel right.
>
> I don't think you were proposing that core dimensions should _never_ be
> allowed to be 0,
No, I'm not suggesting that. There are many cases where a length 0
core dimension is fine.
I'm interested in the case where there is not a meaningful definition
of the operation on the empty set. The mean is an example. Currently
`np.mean([])` generates two warnings (one useful, the other cryptic
and apparently incidental), and returns nan. Returning nan is one way
to handle such a case; another is to raise an error like `np.amax([])`
does. I'd like to raise an error in the example that I'm working on
('peaktopeak' at https://github.com/WarrenWeckesser/npuff). The
function is a gufunc, not a reduction of a binary operation, so the
'identity' argument of PyUFunc_FromFuncAndDataAndSignature has no
effect.
> but if you were I disagree. I spent a fair amount of work
> enabling that for linalg because it provided some convenient base cases.
>
> We could go down the route of augmenting the gufuncs signature syntax to
> support requiring non-empty dimensions, like we did for optional ones -
> although IMO we should consider switching from a string minilanguage to a
> structured object specification if we plan to go too much further with
> extending it.
After only a quick glance at that code: one option is to add a '+'
after the input names in the signature that must have a length that is
at least 1. So the signature for functions like `mean` (if you were
to reimplement it as a gufunc, and wanted an error instead of nan),
`amax`, `ptp`, etc, would be '(i+)->()'.
However, the only meaningful uses-cases of this enhancement that I've
come up with are these simple reductions. So I don't know if making
such a change to the signature is worthwhile. On the other hand,
there are many examples of useful 1-d reductions that aren't the
reduction of an associative binary operation. It might be worthwhile
to have a new convenience function just for the case '(i)->()', maybe
something like PyUFunc_OneDReduction_FromFuncAndData (ugh, that's
ugly, but I think you get the idea), and that function can have an
argument to specify that the length must be at least 1.
I'll see if that is feasible, but I won't be surprised to learn that
there are good reasons for *not* doing that.
Warren
>
> On Sat, Sep 28, 2019, 17:47 Warren Weckesser
> wrote:
>
>> I'm experimenting with gufuncs, and I just created a simple one with
>> signature '(i)->()'. Is there a way to configure the gufunc itself so
>> that an empty array results in an error? Or would I have to create a
>> Python wrapper around the gufunc that does the error checking?
>> Currently, when passed an empty array, the ufunc loop is called with
>> the core dimension associated with i set to 0. It would be nice if
>> the code didn't get that far, and the ufunc machinery "knew" that this
>> gufunc didn't accept a core dimension that is 0. I'd like to
>> automatically get an error, something like the error produced by
>> `np.max([])`.
>>
>> Warren
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NEP 32 is accepted. Now the work begins...
On 9/28/19, Sebastian Berg wrote: > On Sat, 2019-09-28 at 13:15 -0400, Warren Weckesser wrote: >> On 9/27/19, Warren Weckesser wrote: >> > NumPy devs, >> > >> > NEP 32 to remove the financial functions >> > (https://numpy.org/neps/nep-0032-remove-financial-functions.html) >> > has >> > been accepted. >> >> CI gurus: the web page containing the rendered NEPs, >> https://numpy.org/neps/, has not updated since the pull request that >> changed the status of NEP 32 to Accepted was merged >> (https://github.com/numpy/numpy/pull/14600). Does something else >> need >> to be done to get that page to regenerate? >> > > I pushed an empty commit to trigger deployment. That should happen > automatically (as it does OK for the devdocs). I do not know why it > does not work, and github did not yet answer my service request on it. > Thanks Sebastian. The NEPs web page is updated now. Warren > - Sebastian > > >> Warren >> >> >> The next step is to create the numpy-financial package >> > that will replace them. The repository for the new package is >> > https://github.com/numpy/numpy-financial. >> > >> > I have a work-in-progress pull request there to get the initial >> > structure set up. Reviews of the PR would be helpful, as would >> > contributions to set up Sphinx-based documentation, continuous >> > integration, PyPI packaging, and anything else that goes into >> > setting >> > up a "proper" package. Any help would be greatly appreciated! >> > >> > >> > Warren >> > >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NEP 32 is accepted. Now the work begins...
On Sat, 2019-09-28 at 13:15 -0400, Warren Weckesser wrote: > On 9/27/19, Warren Weckesser wrote: > > NumPy devs, > > > > NEP 32 to remove the financial functions > > (https://numpy.org/neps/nep-0032-remove-financial-functions.html) > > has > > been accepted. > > CI gurus: the web page containing the rendered NEPs, > https://numpy.org/neps/, has not updated since the pull request that > changed the status of NEP 32 to Accepted was merged > (https://github.com/numpy/numpy/pull/14600). Does something else > need > to be done to get that page to regenerate? > I pushed an empty commit to trigger deployment. That should happen automatically (as it does OK for the devdocs). I do not know why it does not work, and github did not yet answer my service request on it. - Sebastian > Warren > > > The next step is to create the numpy-financial package > > that will replace them. The repository for the new package is > > https://github.com/numpy/numpy-financial. > > > > I have a work-in-progress pull request there to get the initial > > structure set up. Reviews of the PR would be helpful, as would > > contributions to set up Sphinx-based documentation, continuous > > integration, PyPI packaging, and anything else that goes into > > setting > > up a "proper" package. Any help would be greatly appreciated! > > > > > > Warren > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forcing gufunc to error with size zero input
Can you just raise an exception in the gufuncs inner loop? Or is there no mechanism to do that today? I don't think you were proposing that core dimensions should _never_ be allowed to be 0, but if you were I disagree. I spent a fair amount of work enabling that for linalg because it provided some convenient base cases. We could go down the route of augmenting the gufuncs signature syntax to support requiring non-empty dimensions, like we did for optional ones - although IMO we should consider switching from a string minilanguage to a structured object specification if we plan to go too much further with extending it. On Sat, Sep 28, 2019, 17:47 Warren Weckesser wrote: > I'm experimenting with gufuncs, and I just created a simple one with > signature '(i)->()'. Is there a way to configure the gufunc itself so > that an empty array results in an error? Or would I have to create a > Python wrapper around the gufunc that does the error checking? > Currently, when passed an empty array, the ufunc loop is called with > the core dimension associated with i set to 0. It would be nice if > the code didn't get that far, and the ufunc machinery "knew" that this > gufunc didn't accept a core dimension that is 0. I'd like to > automatically get an error, something like the error produced by > `np.max([])`. > > Warren > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Forcing gufunc to error with size zero input
I'm experimenting with gufuncs, and I just created a simple one with signature '(i)->()'. Is there a way to configure the gufunc itself so that an empty array results in an error? Or would I have to create a Python wrapper around the gufunc that does the error checking? Currently, when passed an empty array, the ufunc loop is called with the core dimension associated with i set to 0. It would be nice if the code didn't get that far, and the ufunc machinery "knew" that this gufunc didn't accept a core dimension that is 0. I'd like to automatically get an error, something like the error produced by `np.max([])`. Warren ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error during pip install
On Sat, Sep 28, 2019 at 11:23 AM Alan Isaac wrote: > On 9/28/2019 12:12 PM, Charles R Harris wrote: > > I'm actually pleased that the install succeeded on Window, although you > won't have good BLAS/LAPACK, just the numpy C versions of lapack_lite. The > warning/error is a bit > > concerning though, it would be nice to know if it is from Python3.8 pip > or numpy. > > > Possibly relevant: > https://github.com/numpy/numpy/issues/11451 > > Yes, thanks, that looks to be the problem. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error during pip install
On 9/28/2019 12:12 PM, Charles R Harris wrote: I'm actually pleased that the install succeeded on Window, although you won't have good BLAS/LAPACK, just the numpy C versions of lapack_lite. The warning/error is a bit concerning though, it would be nice to know if it is from Python3.8 pip or numpy. Possibly relevant: https://github.com/numpy/numpy/issues/11451 Alan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NEP 32 is accepted. Now the work begins...
On 9/27/19, Warren Weckesser wrote: > NumPy devs, > > NEP 32 to remove the financial functions > (https://numpy.org/neps/nep-0032-remove-financial-functions.html) has > been accepted. CI gurus: the web page containing the rendered NEPs, https://numpy.org/neps/, has not updated since the pull request that changed the status of NEP 32 to Accepted was merged (https://github.com/numpy/numpy/pull/14600). Does something else need to be done to get that page to regenerate? Warren The next step is to create the numpy-financial package > that will replace them. The repository for the new package is > https://github.com/numpy/numpy-financial. > > I have a work-in-progress pull request there to get the initial > structure set up. Reviews of the PR would be helpful, as would > contributions to set up Sphinx-based documentation, continuous > integration, PyPI packaging, and anything else that goes into setting > up a "proper" package. Any help would be greatly appreciated! > > > Warren > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error during pip install
On Sat, Sep 28, 2019 at 9:45 AM Alan Isaac wrote: > >>> On Fri, Sep 27, 2019 at 10:12 AM Alan Isaac wrote: > >> Upgrading numpy with pip on Python 3.8b4 on Win 10 > >> produced: > >> ERROR: Could not install packages due to an > >> EnvironmentError: [WinError 123] The filename, > >> directory name, or volume label syntax is incorrect: > >> '"C:' > > >> However, the install appears to have been successful. > > >> On Fri, Sep 27, 2019 at 5:41 PM Charles R Harris wrote: > > I that the pip that comes with Python 3.8b4? > > Yes. > > On 9/27/2019 7:43 PM, Charles R Harris wrote: > > And where did you get NumPy, we don't have any compatible wheels. Was > this from source? > > > Umm, ... does `pip` automatically compile from source in > this case? (I just used `python38 -m pip install numpy`; > I'm afraid I did not specify a log file.) > Yes. I'm actually pleased that the install succeeded on Window, although you won't have good BLAS/LAPACK, just the numpy C versions of lapack_lite. The warning/error is a bit concerning though, it would be nice to know if it is from Python3.8 pip or numpy. > But I'll take the core message to be: wait for the wheels. > We will need to work on generating 3.8 wheels as soon as Python 3.8 is released. I'd like to try before then, but the simplest attempt failed and I didn't pursue it. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error during pip install
On Fri, Sep 27, 2019 at 10:12 AM Alan Isaac wrote: Upgrading numpy with pip on Python 3.8b4 on Win 10 produced: ERROR: Could not install packages due to an EnvironmentError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"C:' However, the install appears to have been successful. On Fri, Sep 27, 2019 at 5:41 PM Charles R Harris wrote: I that the pip that comes with Python 3.8b4? Yes. On 9/27/2019 7:43 PM, Charles R Harris wrote: And where did you get NumPy, we don't have any compatible wheels. Was this from source? Umm, ... does `pip` automatically compile from source in this case? (I just used `python38 -m pip install numpy`; I'm afraid I did not specify a log file.) But I'll take the core message to be: wait for the wheels. Cheers, Alan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
