[Numpy-discussion] Re: Switching default order to column-major

[Matthew Brett]

Hi,

On Mon, Nov 13, 2023 at 7:41 AM Aaron Meurer  wrote:
>
> High level abstractions like .flat or boolean indexing / np.nonzero()
> always use C ordering regardless of the underlying data.
>
> >>> list(np.asarray([[0, 1], [2, 3]]).flat)
> [0, 1, 2, 3]
> >>> list(np.asarray([[0, 1], [2, 3]], order='F').flat)
> [0, 1, 2, 3]

Just in case it caused others to pause as it did me, here's the
Boolean indexing demonstration:

>>> c_arr = np.asarray([[0, 1], [2, 3]])
>>> f_arr = np.asarray([[0, 1], [2, 3]], order='F')
>>> bool_arr = np.array([[False, True], [True, False]])
>>> c_arr[bool_arr]
array([1, 2])
>>> f_arr[bool_arr]
array([1, 2])
>>> c_arr[np.array(bool_arr, order='F')]. # Indexing array order is irrelevant
array([1, 2])

>>> np.nonzero(c_arr < 3)
(array([0, 0, 1]), array([0, 1, 0]))
>>> np.nonzero(f_arr < 3)
(array([0, 0, 1]), array([0, 1, 0]))

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Change in numpy.percentile

[Matthew Brett]

Hi,

On Wed, Oct 11, 2023 at 9:17 AM Peter Cock via NumPy-Discussion
 wrote:
>
>
>
> On Tue, Oct 10, 2023 at 6:32 PM Matthew Brett  wrote:
>>
>> Hi,
>>
>>
>> On Tue, 10 Oct 2023 at 00:55, Andrew Nelson  wrote:
>> >
>> >
>> > On Mon, 9 Oct 2023 at 23:50, Matthew Brett  wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Mon, Oct 9, 2023 at 11:49 AM Andrew Nelson  wrote:
>> >> Could you say more about why you consider:
>> >> np.mean(x, dropna=True)
>> >> to be less clear in intent than:
>> >> np.nanmean(x)
>> >> ?  Is it just that someone could accidentally forget that the default
>> >
>> >
>> > The discussion isn't a deal breaker for me, I just wanted to put out a 
>> > different POV.
>> > The name of the function encodes what it does. By putting them both in the 
>> > function name it's clear what the function does.
>> >
>> > ...
>> >
>> > Imagine that one has a large codebase and you have to find all the 
>> > locations where nans could affect a mean. There may be lots of prod, sum, 
>> > etc, also distributed within the codebase. You wouldn't want to search for 
>> > `dropna` because you get every function that handles a nan. If you search 
>> > for nanmean you only get the locations you want.
>>
>> So, is this the more or less the difference between:
>>
>> grep 'np\.nanmean' *.py
>>
>> and
>>
>> grep 'np\.mean(.*,\s*dropna\s*=\s*True' *.py
>>
>> ?
>>
>> Cheers,
>>
>> Matthew
>>
>
> Keep in mind that the dropna argument might very well be on a different
> line (especially with black formatting), so searches could be much harder
> than looking for the nanmean function.
>
> (I do not deal with enough NaN data to have a strong view either way here)

Sorry - yes - I had kept that in mind - it's just the regexp would
have been a bit more complex, and I was trying to get at the point
that:

a) We won't find ourselves wanting do such a search very often (I
can't remember eve needing it), and
b) When we do have to do such a search, it's well within the wit of
man to do it.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Change in numpy.percentile

[Matthew Brett]

Hi,


On Tue, 10 Oct 2023 at 00:55, Andrew Nelson  wrote:
>
>
> On Mon, 9 Oct 2023 at 23:50, Matthew Brett  wrote:
>>
>> Hi,
>>
>> On Mon, Oct 9, 2023 at 11:49 AM Andrew Nelson  wrote:
>> Could you say more about why you consider:
>> np.mean(x, dropna=True)
>> to be less clear in intent than:
>> np.nanmean(x)
>> ?  Is it just that someone could accidentally forget that the default
>
>
> The discussion isn't a deal breaker for me, I just wanted to put out a 
> different POV.
> The name of the function encodes what it does. By putting them both in the 
> function name it's clear what the function does.
>
> nanmean -> deals with nan when calculating a mean.
>
> -vs-
>
> mean -> calculates a mean
>   |
>   > oh, it has dropna as a keyword argument, that's how you deal with nan.
>
>
> Imagine that one has a large codebase and you have to find all the locations 
> where nans could affect a mean. There may be lots of prod, sum, etc, also 
> distributed within the codebase. You wouldn't want to search for `dropna` 
> because you get every function that handles a nan. If you search for nanmean 
> you only get the locations you want.

So, is this the more or less the difference between:

grep 'np\.nanmean' *.py

and

grep 'np\.mean(.*,\s*dropna\s*=\s*True' *.py

?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Change in numpy.percentile

[Matthew Brett]

Hi,

On Mon, Oct 9, 2023 at 11:49 AM Andrew Nelson  wrote:
>
> On Mon, 9 Oct 2023 at 20:34,  wrote:
>>
>> Surely you can do this for all functions of eg.nan*. Why separate them is 
>> the only thing that distinguishes them.  Setting the parameter seems to be 
>> more handy and user-friendly.  Well for me it's seems better to do it right 
>> away in NumPy 2.0
>
>
> I think I prefer the clearer intent of having nan* functions.

Could you say more about why you consider:

np.mean(x, dropna=True)

to be less clear in intent than:

np.nanmean(x)

?  Is it just that someone could accidentally forget that the default
for `np.mean` is not to drop NaNs?If so - is that a major problem?
  We would be introducing `dropna=True` as not-default, on a world
that is used to the default.

I must say I have several times found myself thinking - why is there a
separate function for means when dropping NaN?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Change in numpy.percentile

[Matthew Brett]

Hi,

Is there any reason to have separate functions - or to keep enforcing
that?I agree, an equivalent of R's rm.na argument seems like a
very reasonable and useful addition, such as (sorry for the
obviousness):

np.mean(x, dropna=True)

and so on,

Cheers,

Matthew

On Mon, Oct 9, 2023 at 9:17 AM Juan Nunez-Iglesias  wrote:
>
>
> On Mon, 9 Oct 2023, at 7:07 PM, Andrew Nelson wrote:
>
> On Mon, 9 Oct 2023 at 16:36, Jerome Kieffer  wrote:
> I'd be ambivalent on making this change. THere are a whole host of other 
> `np.nan*` functions, would they all need to be modified as well? e.g. 
> nanprod, nansum, nanargmin, ..
>
>
> I think obviously, either change all functions or none. The question is 
> whether such a change would fit into the overall NumPy 2.0 and array-API 
> plans. 路‍♂️
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matthew.br...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matthew Brett]

On Wed, May 31, 2023 at 3:04 PM Robert Kern  wrote:
>
> I would much, much rather have the special functions in the `np.*` namespace 
> be more accurate than fast on all platforms. These would not have been on my 
> list for general purpose speed optimization. How much time is actually spent 
> inside sin/cos even in a trig-heavy numpy program? And most numpy programs 
> aren't trig-heavy, but the precision cost would be paid and noticeable even 
> for those programs. I would want fast-and-inaccurate functions to be strictly 
> opt-in for those times that they really paid off. Probably by providing them 
> in their own module or package rather than a runtime switch, because it's 
> probably only a part of my program that needs that kind of speed and can 
> afford that precision loss while there will be other parts that need the 
> precision.
>

What Robert said :)

But I still think the ideal would be the runtime option, maybe via the
proposed context manager, for them as do need it, or want to try it
out.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matthew Brett]

Hi,

On Wed, May 31, 2023 at 11:52 AM Ralf Gommers  wrote:
>
>
>
> On Wed, May 31, 2023 at 12:28 PM Chris Sidebottom  
> wrote:
>>
>> Matthew Brett wrote:
>> > Hi,
>> > On Wed, May 31, 2023 at 8:40 AM Matti Picus matti.pi...@gmail.com wrote:
>> > > On 31/5/23 09:33, Jerome Kieffer wrote:
>> > > Hi Sebastian,
>> > > I had a quick look at the PR and it looks like you re-implemented the 
>> > > sin-cos
>> > > function using SIMD.
>> > > I wonder how it compares with SLEEF (header only library,
>> > > CPU-architecture agnostic SIMD implementation of transcendental
>> > > functions with precision validation). SLEEF is close to the Intel SVML
>> > > library in spirit  but extended to multi-architecture (tested on PowerPC
>> > > and ARM for example).
>> > > This is just curiosity ...
>> > > Like Juan, I am afraid of this change since my code, which depends on
>> > > numpy for sin/cos used for rotation is likely to see large change of
>> > > behavior.
>> > > Cheers,
>> > > Jerome
>> > > I think we should revert the changes. They have proved to be disruptive,
>> > > and I am not sure the improvement is worth the cost.
>> > > The reversion should add  a test that cements the current user 
>> > > expectations.
>> > > The path forward is a different discussion, but for the 1.25 release I
>> > > think we should revert.
>> > > Is there a way to make the changes opt-in for now, while we go back to
>> > see if we can improve the precision?
>>
>> This would be similar to the approach libmvec is taking 
>> (https://sourceware.org/glibc/wiki/libmvec), adding the `--disable-mathvec` 
>> option, although they favour the 4ULP variants rather than the higher 
>> accuracy ones by default. If someone can advise as to the most appropriate 
>> place for such a toggle I can look into adding it, I would prefer for the 
>> default to be 4ULP to match libc though.
>
>
> We have a build-time toggle for SVML (`disable-svml` in `meson_options.txt` 
> and an `NPY_DISABLE_SVML` environment variable for the distutils build). This 
> one should look similar I think - and definitely not separate Python API with 
> `np.fastmath` or similar. The flag can then default to the old 
> (higher-precision, slower) behavior for <2.0, and the fast version for >=2.0 
> somewhere halfway through the 2.0 development cycle - assuming the tweak in 
> precision that Sebastian suggests is possible will remove the worst accuracy 
> impacts that have now been identified.

The ideal would be a run-time toggle for people to experiment with,
with binary wheels.  Is that practical?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matthew Brett]

Hi,

On Wed, May 31, 2023 at 8:40 AM Matti Picus  wrote:
>
>
> On 31/5/23 09:33, Jerome Kieffer wrote:
> > Hi Sebastian,
> >
> > I had a quick look at the PR and it looks like you re-implemented the 
> > sin-cos
> > function using SIMD.
> > I wonder how it compares with SLEEF (header only library,
> > CPU-architecture agnostic SIMD implementation of transcendental
> > functions with precision validation). SLEEF is close to the Intel SVML
> > library in spirit  but extended to multi-architecture (tested on PowerPC
> > and ARM for example).
> > This is just curiosity ...
> >
> > Like Juan, I am afraid of this change since my code, which depends on
> > numpy for sin/cos used for rotation is likely to see large change of
> > behavior.
> >
> > Cheers,
> >
> > Jerome
>
>
> I think we should revert the changes. They have proved to be disruptive,
> and I am not sure the improvement is worth the cost.
>
> The reversion should add  a test that cements the current user expectations.
>
> The path forward is a different discussion, but for the 1.25 release I
> think we should revert.

Is there a way to make the changes opt-in for now, while we go back to
see if we can improve the precision?

If that's not practical - would it be reasonable to guess that there
will only be a very small proportion of users who will notice large
whole-code performance gains from the e.g. 5x performance gain for
transcendental functions?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Software Freedom Conservancy calls for move from Github

[Matthew Brett]

Hi,

On Mon, Jul 4, 2022 at 3:21 PM Charles R Harris
 wrote:
>
>
>
> On Mon, Jul 4, 2022 at 7:48 AM Matthew Brett  wrote:
>>
>> Hi,
>>
>> I just came across this:
>>
>> https://sfconservancy.org/GiveUpGitHub/
>>
>> I guess this is something we should review and consider - although it
>> would obviously have serious costs.
>>
>> Cheers,
>>
>
> I didn't see anything in the article that made me want to switch. I would 
> need to see actual cases of abuse. I also recall that Linus used Bitkeeper 
> because it was the best tool at the time and made that argument. The trouble 
> arose when Andrew Tridgell reversed engineered the protocol, which Larry 
> McVoy complained violated the terms of use. So on and so forth. I haven't 
> noted problems along that line with GitHub. It doesn't bother me that it is 
> proprietary as long as they are responsive and I do appreciate the work going 
> on to make it better. Someone has to pay for that and we are getting a free 
> ride. It might be nice if we could back up the issues and PRs so history 
> wouldn't be lost if we did need to move at some point, but that would be a 
> good idea on any site.

Yes, I felt the same - that the arguments there were not completely
compelling to me - yet - but I was wondering whether we should take
the opportunity to take stock, and make sure we have prepared, in case
we do have to move, or if there are a subset of developers who want to
work elsewhere, collaborating with the main repository on Github.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Software Freedom Conservancy calls for move from Github

[Matthew Brett]

Hi,

I just came across this:

https://sfconservancy.org/GiveUpGitHub/

I guess this is something we should review and consider - although it
would obviously have serious costs.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

On Thu, Jun 30, 2022 at 8:14 PM Matthew Brett  wrote:
>
> Hi,
>
> On Thu, Jun 30, 2022 at 10:00 AM Ralf Gommers  wrote:
> >
> >
> >
> > On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett  
> > wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers  
> >> wrote:
> >> >
> >> >
> >> >
> >> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett  
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard
> >> >>  wrote:
> >> >> >>
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> I am very sorry - I feel I should know this, or be able to work it
> >> >> >> out, but is there a way of setting the flags to the C compiler and 
> >> >> >> the
> >> >> >> linker, for the Numpy build, on Windows?
> >> >> >>
> >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but 
> >> >> >> I
> >> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - 
> >> >> >> and
> >> >> >> I can't see any way of setting these options from the command line.
> >> >> >> Am I missing something?
> >> >> >>
> >> >> >> Cheers,
> >> >> >>
> >> >> >> Matthew
> >> >>
> >> >> >> Member address: kevin.k.shepp...@gmail.com
> >> >> >
> >> >> >
> >> >> > I think these are all hard coded and non-changable.  This was the 
> >> >> > case when I got NumPy building with clang-cl.
> >> >>
> >> >> That was my impression too, but I was delayed by the text at:
> >> >>
> >> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags
> >> >>
> >> >> which says:
> >> >>
> >> >> """
> >> >> Additional compiler flags can be supplied by setting the OPT, FOPT
> >> >> (for Fortran), and CC environment variables. When providing options
> >> >> that should improve the performance of the code ensure that you also
> >> >> set -DNDEBUG so that debugging code is not executed.
> >> >> """
> >> >>
> >> >> I guess we should change that text to note these do not work on Windows.
> >> >>
> >> >> I think you can supply extra Fortran flags with the `config_gc` option
> >> >> to `setup.py`, but I don't think the others have any effect on
> >> >> Windows.
> >> >>
> >> >> I was also confused by these lines in `azure-steps-windows.yml`:
> >> >>
> >> >> """
> >> >> $env:CFLAGS = "-m32"
> >> >> $env:LDFLAGS = "-m32"
> >> >> """
> >> >>
> >> >> I assume these don't actually have any effect.
> >> >
> >> >
> >> > No, the above all seem wrong (unless there has been a major regression 
> >> > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, 
> >> > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want 
> >> > to remove flags that are already hardcoded in numpy.distutils, then 
> >> > that's a different story - you need to edit the numpy.distutils source 
> >> > code then.
> >> >
> >> > I just stumbled on a Windows problem with `$env:`, and the problem was 
> >> > Windows CI is hopelessly weird. It matters for example if you do 
> >> > something in a Powershell context (`ps |` in a .yml file) or outside of 
> >> > it. So it may be something like that.
> >>
> >> Are you sure though - that you can append flags to the compile and
> >> link step with CFLAGS and LDFLAGS?   It didn't work for me locally.
> >> And I can't see where they would have an effect, on Windows, in the
> >> code.
> >
> >
> > Yes I'm sure, and it's even one of the few things in numpy.distutils that 
> > are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you can 
> > make that test fail when setting env vars in a couple of different ways?
>
> I've put up a minimal repo:
>
> https://github.com/matthew-brett/minnpd
>
> I ran
>
> > $env:CFLAGS = "-mbad-opt"
> > $env:LDFLAGS="-bad-option"
> > python setup.py build
>
> You can see from the `build.log` file, that the build doesn't pick up
> either flag.

Windows also ignores the CC environment variable:

$ $env:CC = "cl -mbad-opt"
$ python setup.py build

(no error, no sign of option)

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

On Thu, Jun 30, 2022 at 10:00 AM Ralf Gommers  wrote:
>
>
>
> On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett  
> wrote:
>>
>> Hi,
>>
>> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers  wrote:
>> >
>> >
>> >
>> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett  
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard
>> >>  wrote:
>> >> >>
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am very sorry - I feel I should know this, or be able to work it
>> >> >> out, but is there a way of setting the flags to the C compiler and the
>> >> >> linker, for the Numpy build, on Windows?
>> >> >>
>> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> >> >> I can't see any way of setting these options from the command line.
>> >> >> Am I missing something?
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Matthew
>> >>
>> >> >> Member address: kevin.k.shepp...@gmail.com
>> >> >
>> >> >
>> >> > I think these are all hard coded and non-changable.  This was the case 
>> >> > when I got NumPy building with clang-cl.
>> >>
>> >> That was my impression too, but I was delayed by the text at:
>> >>
>> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags
>> >>
>> >> which says:
>> >>
>> >> """
>> >> Additional compiler flags can be supplied by setting the OPT, FOPT
>> >> (for Fortran), and CC environment variables. When providing options
>> >> that should improve the performance of the code ensure that you also
>> >> set -DNDEBUG so that debugging code is not executed.
>> >> """
>> >>
>> >> I guess we should change that text to note these do not work on Windows.
>> >>
>> >> I think you can supply extra Fortran flags with the `config_gc` option
>> >> to `setup.py`, but I don't think the others have any effect on
>> >> Windows.
>> >>
>> >> I was also confused by these lines in `azure-steps-windows.yml`:
>> >>
>> >> """
>> >> $env:CFLAGS = "-m32"
>> >> $env:LDFLAGS = "-m32"
>> >> """
>> >>
>> >> I assume these don't actually have any effect.
>> >
>> >
>> > No, the above all seem wrong (unless there has been a major regression 
>> > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, 
>> > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to 
>> > remove flags that are already hardcoded in numpy.distutils, then that's a 
>> > different story - you need to edit the numpy.distutils source code then.
>> >
>> > I just stumbled on a Windows problem with `$env:`, and the problem was 
>> > Windows CI is hopelessly weird. It matters for example if you do something 
>> > in a Powershell context (`ps |` in a .yml file) or outside of it. So it 
>> > may be something like that.
>>
>> Are you sure though - that you can append flags to the compile and
>> link step with CFLAGS and LDFLAGS?   It didn't work for me locally.
>> And I can't see where they would have an effect, on Windows, in the
>> code.
>
>
> Yes I'm sure, and it's even one of the few things in numpy.distutils that are 
> tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you can make 
> that test fail when setting env vars in a couple of different ways?

I've put up a minimal repo:

https://github.com/matthew-brett/minnpd

I ran

> $env:CFLAGS = "-mbad-opt"
> $env:LDFLAGS="-bad-option"
> python setup.py build

You can see from the `build.log` file, that the build doesn't pick up
either flag.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

On Thu, Jun 30, 2022 at 6:30 PM Charles R Harris
 wrote:
>
>
>
> On Thu, Jun 30, 2022 at 11:02 AM Matthew Brett  
> wrote:
>>
>> Hi Chuck,
>>
>> On Thu, Jun 30, 2022 at 2:13 PM Charles R Harris
>>  wrote:
>> >
>> >
>> >
>> > On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett  
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I am very sorry - I feel I should know this, or be able to work it
>> >> out, but is there a way of setting the flags to the C compiler and the
>> >> linker, for the Numpy build, on Windows?
>> >>
>> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> >> I can't see any way of setting these options from the command line.
>> >> Am I missing something?
>> >>
>> >> Cheers,
>> >>
>> >> Matthew
>> >
>> >
>> > I don't know how you are using env, but variables set that way are local 
>> > to the shell in which they are set. In PS that is every separate shell 
>> > invocation.
>>
>> Maybe you are thinking of local PS variables?  $env: variables do get
>> passed to subshells, like exported Bash variables.
>
>
> Yes. But my experience was azure-pipelines where every shell invocation was a 
> different subprocess (I think), so setting the environment variables in one 
> shell using env: didn't show up in the others. But I don't know what your 
> usage is.

Yes, for Azure pipelines or Github workflows, you have to do special
work to persist the environment variables between steps - but here I
was just running on my local machine.   It's on my local machine I
find that CFLAGS and LDFLAGS don't seem to affect compilation /
linking.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi Chuck,

On Thu, Jun 30, 2022 at 2:13 PM Charles R Harris
 wrote:
>
>
>
> On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett  wrote:
>>
>> Hi,
>>
>> I am very sorry - I feel I should know this, or be able to work it
>> out, but is there a way of setting the flags to the C compiler and the
>> linker, for the Numpy build, on Windows?
>>
>> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> I can't see any way of setting these options from the command line.
>> Am I missing something?
>>
>> Cheers,
>>
>> Matthew
>
>
> I don't know how you are using env, but variables set that way are local to 
> the shell in which they are set. In PS that is every separate shell 
> invocation.

Maybe you are thinking of local PS variables?  $env: variables do get
passed to subshells, like exported Bash variables.

"""
# show_env.py
import os

print('MY VAR', os.environ.get('MY_VAR'))
print('LOCAL VAR', os.environ.get('LOCAL_VAR'))
"""

Then:

"""
PS C:\tmp> $env:MY_VAR = 'foo'
PS C:\tmp> $LOCAL_VAR = 'bar'
PS C:\tmp> python .\show_env.py
"""

gives

"""
MY VAR foo
LOCAL VAR None
"""

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers  wrote:
>
>
>
> On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett  
> wrote:
>>
>> Hi,
>>
>> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard
>>  wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I am very sorry - I feel I should know this, or be able to work it
>> >> out, but is there a way of setting the flags to the C compiler and the
>> >> linker, for the Numpy build, on Windows?
>> >>
>> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> >> I can't see any way of setting these options from the command line.
>> >> Am I missing something?
>> >>
>> >> Cheers,
>> >>
>> >> Matthew
>>
>> >> Member address: kevin.k.shepp...@gmail.com
>> >
>> >
>> > I think these are all hard coded and non-changable.  This was the case 
>> > when I got NumPy building with clang-cl.
>>
>> That was my impression too, but I was delayed by the text at:
>>
>> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags
>>
>> which says:
>>
>> """
>> Additional compiler flags can be supplied by setting the OPT, FOPT
>> (for Fortran), and CC environment variables. When providing options
>> that should improve the performance of the code ensure that you also
>> set -DNDEBUG so that debugging code is not executed.
>> """
>>
>> I guess we should change that text to note these do not work on Windows.
>>
>> I think you can supply extra Fortran flags with the `config_gc` option
>> to `setup.py`, but I don't think the others have any effect on
>> Windows.
>>
>> I was also confused by these lines in `azure-steps-windows.yml`:
>>
>> """
>> $env:CFLAGS = "-m32"
>> $env:LDFLAGS = "-m32"
>> """
>>
>> I assume these don't actually have any effect.
>
>
> No, the above all seem wrong (unless there has been a major regression 
> recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, 
> CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to 
> remove flags that are already hardcoded in numpy.distutils, then that's a 
> different story - you need to edit the numpy.distutils source code then.
>
> I just stumbled on a Windows problem with `$env:`, and the problem was 
> Windows CI is hopelessly weird. It matters for example if you do something in 
> a Powershell context (`ps |` in a .yml file) or outside of it. So it may be 
> something like that.

Are you sure though - that you can append flags to the compile and
link step with CFLAGS and LDFLAGS?   It didn't work for me locally.
And I can't see where they would have an effect, on Windows, in the
code.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard
 wrote:
>>
>>
>> Hi,
>>
>> I am very sorry - I feel I should know this, or be able to work it
>> out, but is there a way of setting the flags to the C compiler and the
>> linker, for the Numpy build, on Windows?
>>
>> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> I can't see any way of setting these options from the command line.
>> Am I missing something?
>>
>> Cheers,
>>
>> Matthew

>> Member address: kevin.k.shepp...@gmail.com
>
>
> I think these are all hard coded and non-changable.  This was the case when I 
> got NumPy building with clang-cl.

That was my impression too, but I was delayed by the text at:

https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags

which says:

"""
Additional compiler flags can be supplied by setting the OPT, FOPT
(for Fortran), and CC environment variables. When providing options
that should improve the performance of the code ensure that you also
set -DNDEBUG so that debugging code is not executed.
"""

I guess we should change that text to note these do not work on Windows.

I think you can supply extra Fortran flags with the `config_gc` option
to `setup.py`, but I don't think the others have any effect on
Windows.

I was also confused by these lines in `azure-steps-windows.yml`:

"""
$env:CFLAGS = "-m32"
$env:LDFLAGS = "-m32"
"""

I assume these don't actually have any effect.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Setting C and linker flags for Windows build

[Matthew Brett]

Hi,

I am very sorry - I feel I should know this, or be able to work it
out, but is there a way of setting the flags to the C compiler and the
linker, for the Numpy build, on Windows?

I'm trying to set the flags for a build with Windows mingw-w64 - but I
believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
I can't see any way of setting these options from the command line.
Am I missing something?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Future of numpy.distutils

[Matthew Brett]

Hi,

On Fri, Jun 10, 2022 at 9:41 AM Jerome Kieffer  wrote:
>
> Dear Numpy developpers,
>
> We are developing a set of scientific tools
> (https://github.com/silx-kit) and all our build infrastructure is based
> on `numpy.distutils` which apparently is going to disappear in the
> coming years. Beside us, the `scipy` project was using it ...
>
> Ralf Gommers has ported `scipy` to build with `meson-python` and there
> are apparently some sharp edges remaining, especially under windows and
> macos. I wonder if you can comment on the sustainability of this
> approach or if you would advice us another build tool.

I am sure that Ralf will say more, but I did some of the work getting
the macOS and Windows builds working with Meson.   I am not sure what
sharp edges you are thinking of.  As you can imagine, the macOS
implementation was pretty straightforward, the Windows one less so,
but for the usual reasons, of differences between the MSVC compiler
and gcc toolchains.  But even there, it turned out that the modern
Windows gcc toolchains were up to the task, so we had to make
relatively few changes.   And, as I'm sure you know, Scipy has
relatively complex build requirements.

So my guess is that you won't have much trouble getting the Mac and
Windows builds working once you've ported your Linux builds to Meson.
 I'd be happy to help with any problems you do run into.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Importing Numpy when using libpython

[Matthew Brett]

Hi,

On Wed, Jun 8, 2022 at 7:34 PM  wrote:
>
> Hi All,
>
> Hope this is the right forum.
>
> I am working on using Numpy from the programming language Racket.
>
> My plan of attack is to use Python via `libpython`.
> That is, from Racket I use `ffilib` to load `libpython` and from there I use 
> the C API
> to control Python.
>
> Here is what works at the moment:
>
> 1. From Racket I can load `libpython` via `ffilib`.
> 2. It is possible to initialize a Python process and run Python programs in 
> it.
> 3. It is possible to import modules written in Python.
>
> What doesn't work is importing `numpy`.
>
> The error I get when I run `import numpy` is:
>
> ImportError: 
> dlopen(/usr/local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so,
>  0x0002):
>   symbol not found in flat namespace 
> '_PyBaseObject_Type'

There is some discussion of libpython, embedded interpreters and the
Python namespace symbols here:

https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html

and the following discussion, especially:

https://mail.python.org/pipermail/distutils-sig/2016-February/028286.html

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Deprecate np.MachAr?

[Matthew Brett]

Hi,

On Sat, Nov 6, 2021 at 4:54 PM Ralf Gommers  wrote:
>
>
>
> On Tue, Oct 26, 2021 at 9:20 PM bas van beek  wrote:
>>
>> Hi all,
>>
>>
>>
>> The subject of `MachAr` recently came up in 
>> https://github.com/numpy/numpy/pull/20132 and
>>
>> an earlier community meeting, more notably: it’s questionably role as public 
>> (and even private?) API.
>>
>>
>>
>> Ever since the introduction of hard-coded `finfo` parameters back in 
>> numpy/numpy#8504 there has
>>
>> been progressively less need for computing machine parameters during 
>> runtime, to the point where
>>
>> `MachAr` is effectively unused in the numpy codebase itself[1]. From a 
>> user-API point of view, the main
>>
>> difference between ` finfo` and `MachAr` these days is that the latter 
>> produces the same results roughly
>>
>> 4 orders of magnitude slower than the former…
>>
>>
>>
>> Are there any thoughts about deprecating it?
>
>
> For the record: we discussed this in the community meeting two weeks ago, and 
> everyone seemed fine with or in favor of deprecating MachAr.

I haven't looked at this code for a while - but is there any point in
keeping it somewhere outside the NumPy codebase, for use when we hit
an unusual e.g. longdouble type, and we want to infer `finfo`?   My
slight guess is no, that it's too unreliable in that case, and we'd be
better off annotating the remaining code with suggestions to get this
information.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Python 3.10 update

[Matthew Brett]

Hi,

On Mon, Oct 18, 2021 at 11:23 PM Andrew Nelson  wrote:
>
> I realised that I hadn't installed hypothesis. Now things are a little 
> different (the number of tests detected), but still not working:
>
> ```
> conda create -n test -c defaults python=3.10
> conda activate test
> pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
> pip install pytest hypothesis
> python -c 'import numpy as np; np.test()'
> ```
> gives:
>
>
> /Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/numpy/distutils/ccompiler.py:8:
>  DeprecationWarning: The distutils package is deprecated and slated for 
> removal in Python 3.12. Use setuptools or check PEP 632 for potential 
> alternatives
>
>   from distutils import ccompiler
>
> /Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/numpy/distutils/ccompiler.py:17:
>  DeprecationWarning: The distutils.sysconfig module is deprecated, use 
> sysconfig instead
>
>   from distutils.sysconfig import customize_compiler
>
> NumPy version 1.22.0.dev0+1435.gf4ef0fd5d
>
> NumPy relaxed strides checking option: True
>
> NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* 
> FMA3* AVX2* AVX512F? AVX512CD? AVX512_KNL? AVX512_SKX? AVX512_CLX? 
> AVX512_CNL? AVX512_ICL?
>
> INTERNALERROR> Traceback (most recent call last):
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py",
>  line 269, in wrap_session
>
> INTERNALERROR> session.exitstatus = doit(config, session) or 0
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/main.py",
>  line 322, in _main
>
> INTERNALERROR> config.hook.pytest_collection(session=session)
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py",
>  line 265, in __call__
>
> INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), 
> kwargs, firstresult)
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_manager.py",
>  line 80, in _hookexec
>
> INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, 
> firstresult)
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/pluggy/_callers.py",
>  line 55, in _multicall
>
> INTERNALERROR> gen.send(outcome)
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py",
>  line 1210, in pytest_collection
>
> INTERNALERROR> self._validate_config_options()
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py",
>  line 1233, in _validate_config_options
>
> INTERNALERROR> self._warn_or_fail_if_strict(f"Unknown config option: 
> {key}\n")
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py",
>  line 1269, in _warn_or_fail_if_strict
>
> INTERNALERROR> 
> self.issue_config_time_warning(PytestConfigWarning(message), stacklevel=3)
>
> INTERNALERROR>   File 
> "/Users/andrew/miniconda3/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py",
>  line 1321, in issue_config_time_warning
>
> INTERNALERROR> warnings.warn(warning, stacklevel=stacklevel)
>
> INTERNALERROR> pytest.PytestConfigWarning: Unknown config option: env
>
>
> 1287 deselected in 54.82s

Is it possible you have a funny option in a pytest configuration file somewhere?

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Python 3.10 update

[Matthew Brett]

Hi,

On Mon, Oct 18, 2021 at 2:28 PM Andrew Nelson  wrote:
>
>
>
> On Mon, 18 Oct 2021, 23:44 Charles R Harris,  wrote
>>>
>>> ```
>>
>>
>> Thanks for testing. Are you on Intel or Arm?
>
>
> Intel

Running the equivalent commands in a fresh macOS 11.6 Intel virtualenv
gives no errors or test failures:

mkvirtualenv --python=`which python3.10` py310  # Python.org Python
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
pip install pytest hypothesis
python -c 'import numpy as np; np.test()'

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Musllinux wheels

[Matthew Brett]

Hi Stefan,

If you haven't got to it, I can do it.

Cheers,

Matthew

On Tue, Oct 12, 2021 at 1:13 AM Stefan van der Walt
 wrote:
>
> On Mon, Oct 11, 2021, at 07:56, Matthew Brett wrote:
> > My guess is that that would be less than 90 minutes of work for
> > someone with some Docker experience.
>
> I'll give it a shot.
>
> Stéfan
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matthew.br...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Musllinux wheels

[Matthew Brett]

Hi,

On Mon, Oct 11, 2021 at 3:05 PM Matti Picus  wrote:
>
> On 11/10/21 4:38 pm, Matthew Brett wrote:
> > Hi,
> >
> > It will probably soon be trivial to do a manylinux Alpine build / test
> > on multibuild.  There's a work on progress PR here,
> >
> > https://github.com/matthew-brett/multibuild/pull/430
> >
> > that only requires someone to build a trivial test container, in order
> > to get merged.
> >
> > It's also generally very easy to test locally with a minimal Docker setup.
>
>
> A PR to https://github.com/multi-build/docker-images to add Apline
> images with a complement of python versions would be welcome. The
> motivation for not using the muslinux containers provided by
> pypa/manylinux is to check that the wheel built on the pypa/manylinux
> containers will work in the full distro.

My guess is that that would be less than 90 minutes of work for
someone with some Docker experience.

It basically involves some edits to the existing build script for
Pythons on the current Intel test images.  See this directory:

https://github.com/multi-build/docker-images/tree/focal/docker

and the PR linked above for details.

Please do ask for help if stuck.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Musllinux wheels

[Matthew Brett]

Hi,

It will probably soon be trivial to do a manylinux Alpine build / test
on multibuild.  There's a work on progress PR here,

https://github.com/matthew-brett/multibuild/pull/430

that only requires someone to build a trivial test container, in order
to get merged.

It's also generally very easy to test locally with a minimal Docker setup.

Cheers,

Matthew

On Mon, Oct 11, 2021 at 2:01 PM Ralf Gommers  wrote:
>
>
>
> On Mon, Oct 11, 2021 at 2:10 PM Matti Picus  wrote:
>>
>>
>> On 11/10/21 2:22 pm, Ralf Gommers wrote:
>> >
>> >
>> > On Mon, Oct 11, 2021 at 9:52 AM Laurie O > > <mailto:laurie_opper...@hotmail.com>> wrote:
>> >
>> > CI request: build PEP 656
>> > (https://www.python.org/dev/peps/pep-0656/
>> > <https://www.python.org/dev/peps/pep-0656/>) compliant "musllinux"
>> > wheels. This will allow easy installation in Alpine Docker images.
>> > GitHub issue: https://github.com/numpy/numpy/issues/20089
>> > <https://github.com/numpy/numpy/issues/20089>
>> >
>> >
>> > Thanks for asking Laurie. I'd say we may perhaps want to do this, but
>> > only if and when it becomes much easier - for example once we have
>> > migrated all our wheel build infra to GitHub Actions and cibuildwheel.
>> > For now I'm inclined to say that https://github.com/alpine-wheels
>> > <https://github.com/alpine-wheels> has NumPy wheels you can use, and
>> > that Alpine is a niche platform, so this is quite low-prio for us.
>> >
>> > In general, providing wheels for every platform under the sun is not
>> > sustainable - see
>> > https://mail.python.org/archives/list/numpy-discussion@python.org/thread/46HT2SYDHBNLOC6N5RTXI7CN32YWIJWR/#I4474NYSZQVZEIJOA7W43Q6QPTXTXDR5
>> > <https://mail.python.org/archives/list/numpy-discussion@python.org/thread/46HT2SYDHBNLOC6N5RTXI7CN32YWIJWR/#I4474NYSZQVZEIJOA7W43Q6QPTXTXDR5>
>> > for more context.
>> >
>> > Cheers,
>> > Ralf
>> >
>> I am a little worried about releasing wheels untested that will then
>> come back to use as issues with Nans, error states, or inaccurate
>> results. We should have some verification that the platform passes the
>> test suite, preferably a specific run in the main CI.
>
>
> I agree. Platforms where no maintainer is able to build/test locally to 
> resolve issues are already annoying, and not having regular CI is worse. 
> That's why I'm not sure we want to provide Alpine wheels even if our wheel 
> build machinery setup improves. In the end it becomes a maintainer bandwidth 
> issue.
>
> Cheers,
> Ralf
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matthew.br...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Drop 32 bit manylinux wheels

[Matthew Brett]

Hi,

Dropping 32-bit wheels seems very reasonable at this stage, as long as
we keep testing on 32-bit, for the Raspberry Pi folks,

Cheers,

Matthew

On Sun, Oct 3, 2021 at 6:27 PM Charles R Harris
 wrote:
>
> Hi All,
>
> Dropping 32 manylinux wheels was discussed in a triage meeting some time ago 
> and the general consensus was to do so. However, it has never been run past 
> the larger community on the discussion list, so this is that. Note that we 
> have already dropped 32 bit manylinux wheels for the upcoming Python 3.10 as 
> that Python version is only available on Ubuntu focal (20.04) and Ubuntu 
> dropped 32 bit support in 19.10 while Fedora dropped it in 31. Rasbian is 
> still largely 32 bit, but they have their own ppa source.  We will continue 
> to offer 32 wheels on Windows as it remains popular there.
>
> Thoughts?
>
> Chuck
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matthew.br...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[Matthew Brett]

Only to say that:

* I used to have a very firm preference for mail, because I'm pretty
happy with Gmail as a mail interface, and I didn't want to have
another channel I had to monitor, but
* I've spent more time on Discourse over the last year, mainly on
Jupyter, but I have also set up instances for my own projects.  I now
have a fairly strong preference for Discourse, because of its very
nice Markdown authoring, pleasant web interface for reviewing
discussions and reasonable mailing list mode.
* I have hardly used Github Discussions, so I can't comment on them.
Are there large projects that are happy with them?   How does that
compare to Discourse, for example?
* It will surely cause some harm if it is not clear where discussions
happen, mainly (mailing list, Discourse, Github Discussions) so it
seems to me better to decide on one standard place, and commit to
that.

Cheers,

Matthew

On Fri, Oct 1, 2021 at 4:39 PM Rohit Goswami  wrote:
>
> I’m firmly against GH discussions because of the upvoting mechanism. We don’t 
> need to be Reddit or SO. .NET had a bad experience with the discussions as 
> well [1].
>
> [1] https://github.com/dotnet/aspnetcore/issues/29935
>
> — Rohit
>
> On 1 Oct 2021, at 15:04, Andras Deak wrote:
>
> On Fri, Oct 1, 2021 at 4:27 PM Ilhan Polat  wrote:
>>
>> The reason why I mentioned GH discussions is that literally everybody who is 
>> engaged with the code, is familiar with the format, included in the codebase 
>> product and has replies in built unlike the Discourse (opinion is mine) 
>> useless flat discussion design where replies are all over the place just 
>> like the mailing list in case you are not using a tree view supporting 
>> client. Hence topic hijacking is one of the main usability difficulties of 
>> emails.
>>
>> The goal here is to have a coherent engagement with everyone not just within 
>> a small circle, such that there is indeed a discussion happening rather than 
>> a few people chiming in. It would be a nice analytics exercise to have how 
>> many active users using these lists. I'd say 20-25 max for contribs and team 
>> members which is really not much. I know some people are still using IRC and 
>> mailing lists but I wouldn't argue that these are the modern media to have 
>> proper engaging discussions. "Who said to whom" is the bread and butter of 
>> such discussions. And I do think that discourse is exactly the same thing 
>> with mailing lists with a slightly better UI while virtually everyone else 
>> in the world is doing replies.
>
>
> (There are probably a lot of users like myself who follow the mailing list 
> discussions but rarely feel the need to speak up themselves. Not that this 
> says much either way in the discussion, just pointing it out).
>
> I'm not intimately familiar with github discussions (I've only used it a few 
> times), but as far as I can tell it only has answers (or "comments") and 
> comments (or "replies") on answers, i.e. 2 levels of replies rather than a 
> flat single level of replies. If this is indeed the case then I'm not sure 
> it's that much better than a flat system, since when things really get hairy 
> then 2 levels are probably also insufficient to ensure "who said to whom". 
> The "clear replies" argument would hold stronger (in my peanut-gallery 
> opinion) for a medium that supports full reply trees like many comment 
> sections do on various websites.
>
> András
>
>>
>> I would be willing to help with the objections raised since I have been 
>> using GH discussions for quite a while now and there are many tools 
>> available for administration of the discussions. For example,
>>
>> https://github.blog/changelog/2021-09-14-notification-emails-for-discussions/
>>
>> is a recent feature. I don't work for GitHub obviously and have nothing to 
>> do with them but the reasons I'm willing to hear about.
>>
>>
>>
>>
>>
>>
>> On Fri, Oct 1, 2021 at 3:07 PM Matthew Brett  wrote:
>>>
>>> Hi,
>>>
>>> On Fri, Oct 1, 2021 at 1:57 PM Rohit Goswami  wrote:
>>> >
>>> > I guess then the approach overall would evolve to something like using 
>>> > the mailing list to announce discourse posts which need input. Though I 
>>> > would assume that the web interface essentially makes the mailing list 
>>> > almost like discourse, even for new users.
>>> >
>>> > The real issue IMO is still the moderation efforts and additional 
>>> > governance needed for maintaining discourse.
>>>
>>> Yes - that was what I meant.   I do see that maili

[Numpy-discussion] Re: spam on the mailing lists

[Matthew Brett]

Hi,

On Fri, Oct 1, 2021 at 1:57 PM Rohit Goswami  wrote:
>
> I guess then the approach overall would evolve to something like using the 
> mailing list to announce discourse posts which need input. Though I would 
> assume that the web interface essentially makes the mailing list almost like 
> discourse, even for new users.
>
> The real issue IMO is still the moderation efforts and additional governance 
> needed for maintaining discourse.

Yes - that was what I meant.   I do see that mailing lists are harder
to moderate, in that once the email has gone out, it is difficult to
revoke.  So is the argument just that you *can* moderate on Discourse,
therefore you need to think about it more?  Do we have any reason to
think that more moderation will in fact be needed?  We've needed very
little so far on the mailing list, as far as I can see.

Chers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[Matthew Brett]

Hi,

On Fri, Oct 1, 2021 at 1:54 AM Rohit Goswami  wrote:
>
> Although it is true that discourse is easier for newcomers in a lot of ways, 
> it is far worse for governance and consensus. The mailing list, by having 
> essentially sequential topics sent out to all subscribers is easier to keep 
> track of than a large number of forum topics.

Thanks for this - but - could you say more about what you mean here?
In what way is the mailing list less of a problem in terms of
governance and consensus?  Is it just that - it is not easy to
withdraw a mailing list post?   I did have a quick look at the links
you sent - but they seemed to be instances where the community took
the opportunity to formalize a process that was already happening.

I don't agree about it being easier to keep track of topics on a
mailing list. Surely the opposite is true, and one of the main
motivations for Discourse?   I personally use the mailing list option
to read posts in the first instance, but I don't remember feeling the
need to know whether all subscribers have seen a particular post.  In
practice, mailing list mechanics mean that many people skip read their
mail and miss discussions.

Cheers,

Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

Re: [Numpy-discussion] ENH: Proposal to add KML_BLAS support

[Matthew Brett]

Hi,

The only pages I could find on KML_BLAS were in Chinese.  Like Matti,
I'd be interested to know about the license for this library.  Which
particular ARM chips will it run on?

Cheers,

Matthew

On Mon, Feb 22, 2021 at 12:15 PM ChunLin Fang  wrote:
>
>   Hi all,
> Whether you're running apps on your phone or the world's fastest 
> supercomputer, you're most likely running ARM. Many major events have 
> occurred related to ARM archtecture:
>
> Apple may have done the most to make ARM relatively relevant in popular 
> culture with its new ARM-based M1 processor.
> Amazon Web Services launched its Graviton2 processors based on the Arm 
> architecture , which promise up to 40% better performance from comparable 
> x86-based instances for 20% less.
> Microsoft currently uses Arm-based chips from Qualcomm in some of its Surface 
> PCs.
> Huawei unveiled a new chipset called the Kunpeng based on ARM, designed to go 
> into its new TaiShan servers, in a bid to boost its nascent cloud business.
>
>  So It's obvious that ARM will become more and more popular in the 
> future, Since Intel MKL has provide good accelerate support for X86-based 
> chips, Huawei also published KML_BLAS(kunpeng math library blas) that can 
> make full advantage of ARM-based chips,  KML_BLAS is a mathematical library 
> for basic linear algebra operations. it provides three levels of 
> high-performance vector operations: vector-vector operations, vector-matrix 
> operations, and matrix-matrix operations. The performance advantage is shown 
> in the attachment compared with OpenBlas. Can we add KML_BLAS support to 
> numpy?
>
> Cheers,
> Chunlin Fang(github ID:Qiyu8)
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we transfer MacPython/numpy-wheels to the NumPy org?

[Matthew Brett]

Seems like a good idea.  It would need Matti or someone to set up the
various keys I think?
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping manylinux1 wheels for NumPy 1.20.

[Matthew Brett]

Hi,

Updated for Numpy wheels only - BigQuery [1], Notebook [2].

41% of 32-bit wheels need manylinux1, 30% of 64-bit wheels.

Ralf - agreed we shouldn't wait too long for old pip - but maybe we
need to think of some way of reminding people with old pip to upgrade?

Cheers,

Matthew

[1] https://gist.github.com/dc410698ca9e422aec08e4554eac6678
[2] https://gist.github.com/77879cb58b28b3d05c3c14b8a45687e8

On Fri, Aug 28, 2020 at 3:37 PM Matthew Brett  wrote:
>
> Hi,
>
> On Thu, Aug 27, 2020 at 10:51 PM Charles R Harris
>  wrote:
> >
> > Hi All,
> >
> > The 32 bit manylinux1 wheels are proving problematic, see 
> > https://github.com/numpy/numpy/issues/17174. One proposed solution is to 
> > only release manylinux2010 linux wheels for the NumPy 1.20 release. 
> > Thoughts?
>
> I think it may still be too early to discontinue manylinux1, sadly.
>
> Systems requiring manylinux1 are those with:
>
> pip < 19.0 (Jan 2019) [1]
> Linux distribution older than around 2010 (glibc < 2.12) [2]
>
> I did a PyPI BigQuery [3] just now, editing to give results for 32,
> and 64 bit (by changing the manylinux wheel name matching regexp).
>
> Then I processed a bit with Pandas [4].
>
> It looks like about 34% of PyPI manylinux*_i686 downloads are for
> systems that actually need manylinux1, and about 17% of
> manylinux*_x86_64.  See the table in [4] for a listing of the top 10
> entries.
>
> Cheers,
>
> Matthew
>
> [1] https://github.com/pypa/manylinux
> [2] https://www.python.org/dev/peps/pep-0571/
> [3] https://gist.github.com/e3901b344b8d81f5633908347b1b333e
> [4] https://gist.github.com/0f624ddbc34bc3db8bcae23e3eeb7b54
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping manylinux1 wheels for NumPy 1.20.

[Matthew Brett]

Hi,

On Thu, Aug 27, 2020 at 10:51 PM Charles R Harris
 wrote:
>
> Hi All,
>
> The 32 bit manylinux1 wheels are proving problematic, see 
> https://github.com/numpy/numpy/issues/17174. One proposed solution is to only 
> release manylinux2010 linux wheels for the NumPy 1.20 release. Thoughts?

I think it may still be too early to discontinue manylinux1, sadly.

Systems requiring manylinux1 are those with:

pip < 19.0 (Jan 2019) [1]
Linux distribution older than around 2010 (glibc < 2.12) [2]

I did a PyPI BigQuery [3] just now, editing to give results for 32,
and 64 bit (by changing the manylinux wheel name matching regexp).

Then I processed a bit with Pandas [4].

It looks like about 34% of PyPI manylinux*_i686 downloads are for
systems that actually need manylinux1, and about 17% of
manylinux*_x86_64.  See the table in [4] for a listing of the top 10
entries.

Cheers,

Matthew

[1] https://github.com/pypa/manylinux
[2] https://www.python.org/dev/peps/pep-0571/
[3] https://gist.github.com/e3901b344b8d81f5633908347b1b333e
[4] https://gist.github.com/0f624ddbc34bc3db8bcae23e3eeb7b54
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] manylinux upgrade for numpy wheels

[Matthew Brett]

Hi,

On Tue, Feb 4, 2020 at 10:38 PM Nathaniel Smith  wrote:
>
> Pretty sure the 2010 and 2014 images both have much newer compilers than that.
>
> There are still a lot of users on CentOS 6, so I'd still stick to 2010 for 
> now on x86_64 at least. We could potentially start adding 2014 wheels for the 
> other platforms where we currently don't ship wheels – gotta be better than 
> nothing, right?
>
> There probably still is some tail of end users whose pip is too old to know 
> about 2010 wheels. I don't know how big that tail is. If we wanted to be 
> really careful, we could ship both manylinux1 and manylinux2010 wheels for a 
> bit – pip will automatically pick the latest one it recognizes – and see what 
> the download numbers look like.

That all sounds right to me too.

Cheers,

Matthew

> On Tue, Feb 4, 2020, 13:18 Charles R Harris  wrote:
>>
>> Hi All,
>>
>> Thought now would be a good time to decide on upgrading manylinux for the 
>> 1.19 release so that we can make sure that everything works as expected. The 
>> choices are
>>
>> manylinux1 -- CentOS 5, currently used, gcc 4.2 (in practice 4.5), only 
>> supports i686, x86_64.
>> manylinux2010 -- CentOS 6, gcc 4.5, only supports i686, x86_64.
>> manylinux2014 -- CentOS 7, gcc 4.8, supports many more architectures.
>>
>> The main advantage of manylinux2014 is that it supports many new 
>> architectures, some of which we are already testing against. The main 
>> disadvantage is that it requires pip >= 19.x, which may not be much of a 
>> problem 4 months from now but will undoubtedly cause some installation 
>> problems. Unfortunately, the compiler remains archaic, but folks interested 
>> in performance should be using a performance oriented distribution or 
>> compiling for their native architecture.
>>
>> Chuck
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

[Matthew Brett]

Hi,

On Mon, Jan 20, 2020 at 3:00 PM YueCompl  wrote:
>
> Hello folks,
>
> Myself a long time Numpy user, and recently done some Haskell stuff,
>
> Please have a look at  
> https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods 
> and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to 
> port Numpy to Haskell,
> with http://hackage.haskell.org/package/accelerate  and/or  
> http://hackage.haskell.org/package/repa doing the number crunching?

Interesting - do you have any feeling for the differences in speed for
Haskell accelerate array processing as compared to Numpy with default
OpenBLAS?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Code review for adding axis argument to permutation and shuffle function

[Matthew Brett]

Hi,

Thanks - yes - I agree, an axis argument seems like a very sensible idea.

Cheers,

Matthew

On Fri, Sep 13, 2019 at 7:48 AM Juan Nunez-Iglesias  wrote:
>
> I don’t understand why the proposal would be controversial in any way. It’s 
> very natural to have `axis=` keyword arguments in NumPy, and it’s the lack of 
> them that is surprising. My only additional suggestion would be to allow 
> tuples of axes, but that can come later.
>
> Juan.
>
> > On 13 Sep 2019, at 4:25 pm, mattip  wrote:
> >
> > This proposal to add an axis argument to permutation and shuffle seems to
> > have garnered no reply. Are people OK with it (for the new random.Generator
> > only) ?
> >
> >
> >
> > --
> > Sent from: http://numpy-discussion.10968.n7.nabble.com/
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP 32: Remove the financial functions from NumPy

[Matthew Brett]

Hi,

Maybe worth asking over at the Pandas list?  I bet there are more
Python / finance people over there.

Cheers,

Matthew

On Wed, Sep 4, 2019 at 7:11 PM Ilhan Polat  wrote:
>
> +1 on removing them from NumPy. I think there are plenty of alternatives 
> already so many that we might even consider deprecating them just like SciPy 
> misc module by pointing to alternatives.
>
> On Tue, Sep 3, 2019 at 6:38 PM Sebastian Berg  
> wrote:
>>
>> On Tue, 2019-09-03 at 08:56 -0400, Warren Weckesser wrote:
>> > Github issue 2880 ("Get financial functions out of main namespace",
>>
>> Very briefly, I am absolutely in favor of this.
>>
>> Keeping the functions in numpy seems more of a liability than help
>> anyone. And this push is more likely to help users by spurring
>> development on a good replacement, than a practically unmaintained
>> corner of NumPy that may seem like it solves a problem, but probably
>> does so very poorly.
>>
>> Moving them into a separate pip installable package seems like the best
>> way forward until a better replacement, to which we can point users,
>> comes up.
>>
>> - Sebastian
>>
>>
>> > https://github.com/numpy/numpy/issues/2880) has been open since 2013.
>> > In a recent community meeting, it was suggested that we create a NEP
>> > to propose the removal of the financial functions from NumPy.  I have
>> > submitted "NEP 32:  Remove the financial functions from NumPy" in a
>> > pull request at https://github.com/numpy/numpy/pull/14399.  A copy of
>> > the latest version of the NEP is below.
>> >
>> > According to the NEP process document, "Once the PR is in place, the
>> > NEP should be announced on the mailing list for discussion (comments
>> > on the PR itself should be restricted to minor editorial and
>> > technical fixes)."  This email is the announcement for NEP 32.
>> >
>> > The NEP includes a brief summary of the history of the financial
>> > functions, and has links to several relevant mailing list threads,
>> > dating back to when the functions were added to NumPy in 2008.  I
>> > recommend reviewing those threads before commenting here.
>> >
>> > Warren
>> >
>> > -
>> >
>> > ==
>> > NEP 32 — Remove the financial functions from NumPy
>> > ==
>> >
>> > :Author: Warren Weckesser 
>> > :Status: Draft
>> > :Type: Standards Track
>> > :Created: 2019-08-30
>> >
>> >
>> > Abstract
>> > 
>> >
>> > We propose deprecating and ultimately removing the financial
>> > functions [1]_
>> > from NumPy.  The functions will be moved to an independent
>> > repository,
>> > and provided to the community as a separate package with the name
>> > ``numpy_financial``.
>> >
>> >
>> > Motivation and scope
>> > 
>> >
>> > The NumPy financial functions [1]_ are the 10 functions ``fv``,
>> > ``ipmt``,
>> > ``irr``, ``mirr``, ``nper``, ``npv``, ``pmt``, ``ppmt``, ``pv`` and
>> > ``rate``.
>> > The functions provide elementary financial calculations such as
>> > future value,
>> > net present value, etc. These functions were added to NumPy in 2008
>> > [2]_.
>> >
>> > In May, 2009, a request by Joe Harrington to add a function called
>> > ``xirr`` to
>> > the financial functions triggered a long thread about these functions
>> > [3]_.
>> > One important point that came up in that thread is that a "real"
>> > financial
>> > library must be able to handle real dates.  The NumPy financial
>> > functions do
>> > not work with actual dates or calendars.  The preference for a more
>> > capable
>> > library independent of NumPy was expressed several times in that
>> > thread.
>> >
>> > In June, 2009, D. L. Goldsmith expressed concerns about the
>> > correctness of the
>> > implementations of some of the financial functions [4]_.  It was
>> > suggested then
>> > to move the financial functions out of NumPy to an independent
>> > package.
>> >
>> > In a GitHub issue in 2013 [5]_, Nathaniel Smith suggested moving the
>> > financial
>> > functions from the top-level namespace to ``numpy.financial``.  He
>> > also
>> > suggested giving the functions better names.  Responses at that time
>> > included
>> > the suggestion to deprecate them and move them from NumPy to a
>> > separate
>> > package.  This issue is still open.
>> >
>> > Later in 2013 [6]_, it was suggested on the mailing list that these
>> > functions
>> > be removed from NumPy.
>> >
>> > The arguments for the removal of these functions from NumPy:
>> >
>> > * They are too specialized for NumPy.
>> > * They are not actually useful for "real world" financial
>> > calculations, because
>> >   they do not handle real dates and calendars.
>> > * The definition of "correctness" for some of these functions seems
>> > to be a
>> >   matter of convention, and the current NumPy developers do not have
>> > the
>> >   background to judge their correctness.
>> > * There has been little interest among past and present NumPy
>> > developers

Re: [Numpy-discussion] Regarding adding new function in Numpy

[Matthew Brett]

Hi,

On Sun, Jul 7, 2019 at 11:41 PM ANURAG ANURAG  wrote:
>
> Sir,
> I was recently converting some Matlab codes into python, where I recently 
> stumble upon bitget(n,x) function of Matlab,  b = bitget(A,bit) returns the 
> bit value at the position bit in integer array A. I was unable to find any 
> Numpy equivalent of this, so should I create a pull request in Numpy for the 
> same!

Thanks for the suggestion.  I think Eric Wieser is right in his
comment on the related pull request:

https://github.com/numpy/numpy/pull/13937#issuecomment-509091992

The function is a one-liner in Python - albeit a slightly fancy
one-liner.  Also - I have never had need of this function in Matlab or
Numpy, and had no idea it existed.   So, I'd vote against, unless it
turned out it was much more common in use than my experience would
suggest,

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with absolute value

[Matthew Brett]

Hi,

On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris
 wrote:
>
>
>
> On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi  
> wrote:
>>
>> Hi,
>>
>> It turns out you're running into a bit-error. In general, the two's 
>> complement of -2 ** (n-1) with the bit-length being limited to n bits is 
>> itself... No way around that. And integers don't set hardware exceptions so 
>> checking for errors like these is hard as well.
>>
>> TL;DR: It's an error with how the integer is stored in memory and how you're 
>> running out of space.
>>
>> Regards,
>> Hameer Abbasi
>
>
> More like the eight bit twos complement of -128 is -128, bytes cannot 
> represent 128. Matlab used to (still does?) solve this problem by returning 
> 127 instead :) Basically, the data needs more precision. Returning an 
> unsigned type would lead to it's own problems with unexpected promotions when 
> the result was used. We could, I suppose, raise a warning, although that 
> might be considered noisy. If you use `absolute` instead, you can specify the 
> dtype.

I still think this is a major wart in numpy's abs.

I'd really like to add a new function, `uabs` which would return an
unsigned int for integer inputs.  I'm happy to do a pull request if
that also seems sensible to y'all,

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how much does binary size matter?

[Matthew Brett]

Hi,

Obviously this is a trade-off; if we can increase binary size we can
add more optimizations, for more platforms, and development will be a
little faster, because we are not having to spend time optimizing for
binary size.

If people on slow internet connections had to download numpy multiple
times, I guess this would be an issue, but I've lived behind
excruciatingly slow and unreliable connections, in Cuba, and, for a
download you do a few times a year, I very much doubt there would be
practical difference between 16M and say, 20M.  If it's 16M vs 50M,
then I think it's worth having the discussion, with the relevant
trade-offs.

Cheers,

Matthew

On Fri, Apr 26, 2019 at 1:30 PM Éric Depagne  wrote:
>
> Le vendredi 26 avril 2019, 12:49:39 SAST Ilhan Polat a écrit :
> Hi Ihlan,
>
> That's an interesting link, but they provide the average, which is not a very
> good indicator. I have myself a 100 Mb/s link where I live, which means that
> as Akamai ranks my country with an average speed of 6.7 Mb/s, a lot of person
> have a connection that does not reach 1 Mb/s. Of course, many of those will
> not be interested in downloading numpy, so that might not be an issue.
>
> Éric.
>
> > here is a baseline
> > https://en.wikipedia.org/wiki/List_of_countries_by_Internet_connection_speed
> > s . Probably a good idea to throttle values at 60% of the bandwidth and you
> > get a crude average delay it would cause per 1MB worldwide.
> >
> > On Fri, Apr 26, 2019 at 11:49 AM Éric Depagne  wrote:
> > > Le vendredi 26 avril 2019, 11:10:56 SAST Ralf Gommers a écrit :
> > > Hi Ralf,
> > >
> > > > Right now a wheel is 16 MB. If we increase that by 10%/50%/100% - are we
> > > > causing a real problem for someone?
> > >
> > > Access to large bandwidth is not universal at all, and in many countries
> > > (I'd
> > > even say in most of the countries around the world), 16 Mb is a
> > > significant
> > > amount of data so increasing it is a burden.
> > >
> > > Cheers,
> > > Éric.
> > >
> > > > Thanks,
> > > > Ralf
> > >
> > > --
> > > Un clavier azerty en vaut deux
> > > --
> > > Éric Depagne
> > >
> > >
> > >
> > > ___
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion@python.org
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
> --
> Un clavier azerty en vaut deux
> --
> Éric Depagne
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] random.choice(replace=False) very slow

[Matthew Brett]

Hi,

I noticed that numpy.random.choice was very slow, with the
replace=False option, and then I noticed it can (for most cases) be
made many hundreds of times faster in Python code:

In [18]: sample = np.random.uniform(size=100)
In [19]: timeit np.random.choice(sample, 500, replace=False)
42.1 ms ± 214 µs per loop (mean ± std. dev. of 7 runs, 10
loops each)
IIn [22]: def rc(x, size):
...: n = np.prod(size)
...: n_plus = n * 2
...: inds = np.unique(np.random.randint(0, n_plus+1, size=n_plus))[:n]
...: return x[inds].reshape(size)
In [23]: timeit rc(sample, 500)
86.5 µs ± 421 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)each)

Is there a reason why it's so slow in C?  Could something more
intelligent than the above be used to speed it up?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] count_nonzero axis argument?

[Matthew Brett]

On Mon, Sep 17, 2018 at 2:24 PM, Warren Weckesser
 wrote:
>
>
> On Mon, Sep 17, 2018 at 7:38 AM Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> Is there any reason that np.count_nonzero should not take an axis
>> argument?  As in:
>>
>> >>> np.better_count_nonzero([[10, 11], [0, 3]], axis=1)
>> array([2, 1])
>>
>
>
> It already does (since version 1.12.0):
> https://docs.scipy.org/doc/numpy/reference/generated/numpy.count_nonzero.html

Oops - sorry - I am so used to upgrading all the time, it didn't occur
to me I had an old version in my standard setup.  Thanks, and sorry
for the distraction.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] count_nonzero axis argument?

[Matthew Brett]

Hi,

Is there any reason that np.count_nonzero should not take an axis
argument?  As in:

>>> np.better_count_nonzero([[10, 11], [0, 3]], axis=1)
array([2, 1])

It would be much more useful if it did...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

[Matthew Brett]

Hi,

On Wed, Aug 15, 2018 at 5:36 PM, Hameer Abbasi
 wrote:
> On 15. Aug 2018, at 18:25, Matthew Brett  wrote:
>
> Hi,
>
> Thanks Nathaniel for this thoughtful response.
>
> On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith  wrote:
> ...
>
> The other approach would be to incrementally add clean, well-defined
> dunder methods like __array_ufunc__, __array_concatenate__, etc. This
> way we end up putting some thought into each interface, making sure
> that it's something we can support, protecting downstream libraries
> from unnecessary complexity (e.g. they can implement
> __array_concatenate__ instead of hstack, vstack, row_stack,
> column_stack, ...), or avoiding adding new APIs entirely (e.g., by
> converting existing functions into ufuncs so __array_ufunc__ starts
> automagically working). And in the end we get a clean list of dunder
> methods that new array container implementations have to define. It's
> plausible to imagine a generic test suite for array containers. (I
> suspect that every library that tries to implement __array_function__
> will end up with accidental behavioral differences, just because the
> numpy API is so vast and contains so many corner cases.) So the
> clean-well-defined-dunders approach has lots of upsides. The big
> downside is that this is a much longer road to go down.
>
>
> Does everyone agree that, if we had infinite time and resources, this
> would be the better solution?
>
>
> More resources means (given NumPy’s consensus system), more people have to
> agree on the overall design, so in my mind, it might even be slower.

I don't think that's likely.  As far as I can see, past discussions
have been slow because several people need to get deep down into the
details in order to understand the problem, and then stay focused
through the discussion.  When there is no-one working on that
full-time, it's easy for the discussion to drift into the background,
and the shared understanding is lost.  My suspicion is, to the extent
that Matti and Tyler can devote time and energy to shepherding the
discussion, these will become quicker and more productive.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Dropping 32-bit from macOS wheels

[Matthew Brett]

Hi,

Short version:

We are planning to drop 32-bit compatibility from the numpy macOS
wheels.  This email is to ask for feedback.

Long version:

macOS has tooling to allow distribution of binaries that have objects
for more than one architecture.  These are called "fat" binaries, and
they were useful when Macs could be PPC, 32-bit or 64-bit intel.  Now
of course, they are effectively all 64-bit Intel.

Amusingly enough, you can use the "lipo" command to operate on fat
binaries. For example, system Python (at /usr/bin/python) is a fat
binary, containing objects for 32 and 64-bit.

$ lipo -info /usr/bin/python
Architectures in the fat file: /usr/bin/python are: i386 x86_64

It is possible to run Python in 32-bit mode with:

$ arch -i386 /usr/bin/python

Up until the last release of numpy, we have build fat (32 and 64 bit)
binaries for numpy, so, if you do run Python in 32-bit mode, you can
still import and use numpy.   In the last release, I accidentally
broke the 32-bit part of the build - see
https://github.com/numpy/numpy/issues/11625 .

We could fix this, but we suspect that there are very few people still
using 32-bit on macOS.   If you do use it - please let us know, and
we'll consider the options.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adoption of a Code of Conduct

[Matthew Brett]

Hi,

On Fri, Aug 3, 2018 at 9:35 AM, Stefan van der Walt
 wrote:
> On August 3, 2018 09:50:38 Robert Kern  wrote:
>>
>> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern  wrote:
>>>
>>>  Nope, concision is definitely not my strength. But I hope I
>>> made the argument clear, at least.
>>
>>
>> No, wait. I got it:
>>
>> Bad actors use "diversity of political beliefs" in bad faith as cover for
>> undermining the goals of the diversity statement. Marginalized groups want
>> more assurance that our community (1) isn't one of those bad actors and (2)
>> is willing and capable of resisting those bad actors when they come.
>
>
> That's a very useful summary; thank you.
>
> I think we can fairly easily add a sentence that encourages participation
> from a wide diversity of people, while making it clear that including
> someone in the conversation does not give them free reigns in contradiction
> with the rest of the guidelines.
>
> Ralf, if you agree, shall we do this for SciPy, and use the new version for
> NumPy too?

I must say, I disagree.  I think we're already treading close to the
edge with the current document, and it's more likely we'd get closer
still with virtually any addition on this line.   I'm in favor of
keeping the political beliefs in there, on the basis it's really not
too hard to distinguish good-faith political beliefs, and the current
atmosphere is so repellent to people who would not identify as
progressive, that I would like them to feel they have some protection.
If you will not allow me "no change" and you offered me a) paragraph
by group of the not-discriminated trying to imagine something
comforting to imagined extremely sensitive and progressive (name your
other group here) or b) no stated defense for not-progressive persons,
I'd take b).

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adoption of a Code of Conduct

[Matthew Brett]

Hi,

On Wed, Aug 1, 2018 at 4:12 PM, Nathan Goldbaum  wrote:
>
>
> On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers  wrote:
>>
>>
>>
>> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum 
>> wrote:
>>>
>>> I realize this was probably brought up in the discussions about the scipy
>>> code of conduct which I have not looked at, but I’m troubled by the
>>> inclusion of “political beliefs” in the document.
>>
>>
>> It was not brought up explicitly as far as I remember.
>>
>>>
>>> See e.g.
>>> https://github.com/jupyter/governance/pull/5
>>
>>
>> That's about moving names around. I don't see any mention of political
>> beliefs?
>
>
> Sorry about that, I elided the 6. This is the correct link:
>
> https://github.com/jupyter/governance/pull/56
>
>>
>>
>>>
>>> As a thought experiment, what if someone’s political beliefs imply that
>>> other contributors are not deserving of human rights? Increasingly ideas
>>> like this are coming into the mainstream worldwide and I think this is a
>>> real concern that should be considered.
>>
>>
>> There is a difference between having beliefs, and expressing those beliefs
>> in ways that offends others. I don't see any problem with saying that we
>> welcome anyone, irrespective of political belief. However, if someone starts
>> expressing things that are intolerant (like someone else not deserving human
>> rights) on any of our communication forums or in an in-person meeting, that
>> would be a clear violation of the CoC. Which can be dealt with via the
>> reporting and enforcement mechanism in the CoC.
>>
>> I don't see a problem here, but I would see a real problem with removing
>> the "political beliefs" phrase.
>
>
> For another perspective on this issue see
> https://where.coraline.codes/blog/oscon/, where Coraline Ada describes her
> reasons for not speaking at OSCON this year due to a similar clause in the
> code of conduct.

I agree with Ralf.  From your link:

"""
But the inclusion of this language, making political affiliation a
protected class, leads me to believe that alt-right technologists
would be as welcome at the conference as I would be. Including
alt-right technologists who display on their clothing, for example,
neo-Nazi insignia. Or t-shirts printed with anti-trans or anti-Black
slogans. These could easily be interpreted as protected political
speech.
"""

That's the point.  If you wear a t-shirt with anti-trans or anti-Black
slogans to a Scipy event covered by the code of conduct, that would
qualify as 'expressing things that are intolerant', as Ralf put it.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

On Sat, Jun 30, 2018 at 12:36 AM, Charles R Harris
 wrote:
>
>
> On Fri, Jun 29, 2018 at 4:35 PM, Matthew Brett 
> wrote:
>>
>> On Fri, Jun 29, 2018 at 11:31 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Tue, Jun 26, 2018 at 3:55 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Tue, Jun 26, 2018 at 10:43 PM, Matti Picus 
>> >> wrote:
>> >> > On 19/06/18 10:57, Matthew Brett wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Tue, Jun 19, 2018 at 6:27 PM, Matti Picus 
>> >> >> wrote:
>> >> >>>
>> >> >>> On 19/06/18 09:58, Charles R Harris wrote:
>> >> >>>>>
>> >> >>>>> What I was curious about is that there were no more "daily"
>> >> >>>>> builds
>> >> >>>>> of
>> >> >>>>> master.
>> >> >>>>
>> >> >>>> Is that right?  That there were daily builds of master, on
>> >> >>>> Appveyor?
>> >> >>>> I don't know how those worked, I only recently got cron permission
>> >> >>>> ...
>> >> >>>
>> >> >>>
>> >> >>> No, but there used to be daily builds on travis. They stopped 8
>> >> >>> days
>> >> >>> ago,
>> >> >>> https://travis-ci.org/MacPython/numpy-wheels/builds.
>> >> >>
>> >> >> Oops - yes - sorry - I retired the 'daily' branch, in favor of
>> >> >> 'master', but forgot to update the Travis-CI settings.
>> >> >>
>> >> >> Done now.
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Matthew
>> >> >>
>> >> > FWIW, still no daily builds at
>> >> > https://travis-ci.org/MacPython/numpy-wheels/builds
>> >>
>> >> You mean, some days there appears to be no build?  The build matrix
>> >> does show Cron-triggered jobs, the last of which was a few hours ago:
>> >> https://travis-ci.org/MacPython/numpy-wheels/builds/397008012
>> >>
>> >> Cheers,
>> >>
>> >> Matthew
>> >
>> >
>> > The cron wheels are getting built and tested, but they aren't uploading
>> > to
>> > rackspace.
>>
>> The cron wheels go to the "pre" container at
>>
>> https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com
>>
>
> Ah, there they are ... except ... you cancelled the builds I was waiting for
> :) I was building wheels so we could have folks test the DLL load problem,
> which I'm pretty sure if fixed anyway, so I suppose waiting on the daily
> isn't a big a deal.

Oh - sorry - I was rushing to get 1.14.5 wheels built.  Can you
retrigger the builds?  Do you want me to?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

On Fri, Jun 29, 2018 at 11:31 PM, Charles R Harris
 wrote:
>
>
> On Tue, Jun 26, 2018 at 3:55 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Tue, Jun 26, 2018 at 10:43 PM, Matti Picus 
>> wrote:
>> > On 19/06/18 10:57, Matthew Brett wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Tue, Jun 19, 2018 at 6:27 PM, Matti Picus 
>> >> wrote:
>> >>>
>> >>> On 19/06/18 09:58, Charles R Harris wrote:
>> >>>>>
>> >>>>> What I was curious about is that there were no more "daily" builds
>> >>>>> of
>> >>>>> master.
>> >>>>
>> >>>> Is that right?  That there were daily builds of master, on Appveyor?
>> >>>> I don't know how those worked, I only recently got cron permission
>> >>>> ...
>> >>>
>> >>>
>> >>> No, but there used to be daily builds on travis. They stopped 8 days
>> >>> ago,
>> >>> https://travis-ci.org/MacPython/numpy-wheels/builds.
>> >>
>> >> Oops - yes - sorry - I retired the 'daily' branch, in favor of
>> >> 'master', but forgot to update the Travis-CI settings.
>> >>
>> >> Done now.
>> >>
>> >> Cheers,
>> >>
>> >> Matthew
>> >>
>> > FWIW, still no daily builds at
>> > https://travis-ci.org/MacPython/numpy-wheels/builds
>>
>> You mean, some days there appears to be no build?  The build matrix
>> does show Cron-triggered jobs, the last of which was a few hours ago:
>> https://travis-ci.org/MacPython/numpy-wheels/builds/397008012
>>
>> Cheers,
>>
>> Matthew
>
>
> The cron wheels are getting built and tested, but they aren't uploading to
> rackspace.

The cron wheels go to the "pre" container at
https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

Hi,

On Tue, Jun 26, 2018 at 10:43 PM, Matti Picus  wrote:
> On 19/06/18 10:57, Matthew Brett wrote:
>>
>> Hi,
>>
>> On Tue, Jun 19, 2018 at 6:27 PM, Matti Picus 
>> wrote:
>>>
>>> On 19/06/18 09:58, Charles R Harris wrote:
>>>>>
>>>>> What I was curious about is that there were no more "daily" builds of
>>>>> master.
>>>>
>>>> Is that right?  That there were daily builds of master, on Appveyor?
>>>> I don't know how those worked, I only recently got cron permission ...
>>>
>>>
>>> No, but there used to be daily builds on travis. They stopped 8 days ago,
>>> https://travis-ci.org/MacPython/numpy-wheels/builds.
>>
>> Oops - yes - sorry - I retired the 'daily' branch, in favor of
>> 'master', but forgot to update the Travis-CI settings.
>>
>> Done now.
>>
>> Cheers,
>>
>> Matthew
>>
> FWIW, still no daily builds at
> https://travis-ci.org/MacPython/numpy-wheels/builds

You mean, some days there appears to be no build?  The build matrix
does show Cron-triggered jobs, the last of which was a few hours ago:
https://travis-ci.org/MacPython/numpy-wheels/builds/397008012

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

Hi,

On Tue, Jun 19, 2018 at 6:27 PM, Matti Picus  wrote:
> On 19/06/18 09:58, Charles R Harris wrote:
>>
>> > What I was curious about is that there were no more "daily" builds of
>> > master.
>>
>> Is that right?  That there were daily builds of master, on Appveyor?
>> I don't know how those worked, I only recently got cron permission ...
>
>
> No, but there used to be daily builds on travis. They stopped 8 days ago,
> https://travis-ci.org/MacPython/numpy-wheels/builds.

Oops - yes - sorry - I retired the 'daily' branch, in favor of
'master', but forgot to update the Travis-CI settings.

Done now.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

On Tue, Jun 19, 2018 at 2:46 PM, Charles R Harris
 wrote:
>
>
> On Tue, Jun 19, 2018 at 4:57 AM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Tue, Jun 19, 2018 at 2:44 AM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Mon, Jun 18, 2018 at 5:58 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> On Tue, Jun 19, 2018 at 12:24 AM, Charles R Harris
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Mon, Jun 18, 2018 at 3:13 PM, Matthew Brett
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Mon, Jun 18, 2018 at 9:42 PM, Charles R Harris
>> >> >>  wrote:
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum
>> >> >> > 
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> I think Matthew Brett needs to fix this.
>> >> >> >
>> >> >> >
>> >> >> > That would be nice, but I'm not convinced it is helpful :) I note
>> >> >> > that
>> >> >> > latest `apache-libcloud` does not install directly on windows,
>> >> >> > there
>> >> >> > seem to
>> >> >> > be some missing dependencies.>
>> >> >>
>> >> >> I'm happy to give it a go - Chuck - can I cancel the various builds
>> >> >> running on my account, so I can do some debugging.
>> >> >
>> >> >
>> >> > Absolutely! Nuke those suckers ...
>> >>
>> >> Hmm - I just tried installing certifi to get the SSL certificates, and
>> >> removed --no-ssl-check.  I wonder if something changed in the
>> >> Rackspace protocols, or something.
>> >>
>> >> In case it's useful, I'm using a little repo that runs an Appveyor job
>> >> then drops into an RDP server for me to log into, with the relevant
>> >> bit here:
>> >>
>> >> https://github.com/matthew-brett/appvfutz/blob/master/appveyor.yml#L24
>> >>
>> >> See:
>> >>
>> >> https://www.gep13.co.uk/blog/how-to-use-appveyor-remote-desktop-connection
>> >>
>> >> That said, maybe the fix doesn't work, let's wait on the builds.
>> >>
>> >
>> > Looks like that fixes the problem. Probably scipy-wheels will need that
>> > fix
>> > also.
>>
>> I put it in.
>>
>> > Do you know if new wheels with the same name will overwrite the old
>> > ones? ISTR that that is the case.
>>
>> Right - they overwrite the old ones.
>>
>> > BTW, there don't seem to be any nightly builds, does something need
>> > reconfiguration?
>>
>> For Appveyor?  You need a cron-enabled account.  My account is
>> enabled, I just emailed the appveyor support with my username, and an
>> explanation.  Maybe worth doing the same for the numpy account?
>> Thereafter, you can just enter the cron time string in the settings,
>> to enable daily builds.
>>
>
> What I was curious about is that there were no more "daily" builds of
> master.

Is that right?  That there were daily builds of master, on Appveyor?
I don't know how those worked, I only recently got cron permission ...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

Hi,

On Tue, Jun 19, 2018 at 2:44 AM, Charles R Harris
 wrote:
>
>
> On Mon, Jun 18, 2018 at 5:58 PM, Matthew Brett 
> wrote:
>>
>> On Tue, Jun 19, 2018 at 12:24 AM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Mon, Jun 18, 2018 at 3:13 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Mon, Jun 18, 2018 at 9:42 PM, Charles R Harris
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> I think Matthew Brett needs to fix this.
>> >> >
>> >> >
>> >> > That would be nice, but I'm not convinced it is helpful :) I note
>> >> > that
>> >> > latest `apache-libcloud` does not install directly on windows, there
>> >> > seem to
>> >> > be some missing dependencies.>
>> >>
>> >> I'm happy to give it a go - Chuck - can I cancel the various builds
>> >> running on my account, so I can do some debugging.
>> >
>> >
>> > Absolutely! Nuke those suckers ...
>>
>> Hmm - I just tried installing certifi to get the SSL certificates, and
>> removed --no-ssl-check.  I wonder if something changed in the
>> Rackspace protocols, or something.
>>
>> In case it's useful, I'm using a little repo that runs an Appveyor job
>> then drops into an RDP server for me to log into, with the relevant
>> bit here:
>>
>> https://github.com/matthew-brett/appvfutz/blob/master/appveyor.yml#L24
>>
>> See:
>> https://www.gep13.co.uk/blog/how-to-use-appveyor-remote-desktop-connection
>>
>> That said, maybe the fix doesn't work, let's wait on the builds.
>>
>
> Looks like that fixes the problem. Probably scipy-wheels will need that fix
> also.

I put it in.

> Do you know if new wheels with the same name will overwrite the old
> ones? ISTR that that is the case.

Right - they overwrite the old ones.

> BTW, there don't seem to be any nightly builds, does something need
> reconfiguration?

For Appveyor?  You need a cron-enabled account.  My account is
enabled, I just emailed the appveyor support with my username, and an
explanation.  Maybe worth doing the same for the numpy account?
Thereafter, you can just enter the cron time string in the settings,
to enable daily builds.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

On Tue, Jun 19, 2018 at 12:24 AM, Charles R Harris
 wrote:
>
>
> On Mon, Jun 18, 2018 at 3:13 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Mon, Jun 18, 2018 at 9:42 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum 
>> > wrote:
>> >>
>> >> I think Matthew Brett needs to fix this.
>> >
>> >
>> > That would be nice, but I'm not convinced it is helpful :) I note that
>> > latest `apache-libcloud` does not install directly on windows, there
>> > seem to
>> > be some missing dependencies.>
>>
>> I'm happy to give it a go - Chuck - can I cancel the various builds
>> running on my account, so I can do some debugging.
>
>
> Absolutely! Nuke those suckers ...

Hmm - I just tried installing certifi to get the SSL certificates, and
removed --no-ssl-check.  I wonder if something changed in the
Rackspace protocols, or something.

In case it's useful, I'm using a little repo that runs an Appveyor job
then drops into an RDP server for me to log into, with the relevant
bit here:

https://github.com/matthew-brett/appvfutz/blob/master/appveyor.yml#L24

See: https://www.gep13.co.uk/blog/how-to-use-appveyor-remote-desktop-connection

That said, maybe the fix doesn't work, let's wait on the builds.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matthew Brett]

Hi,

On Mon, Jun 18, 2018 at 9:42 PM, Charles R Harris
 wrote:
>
>
> On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum 
> wrote:
>>
>> I think Matthew Brett needs to fix this.
>
>
> That would be nice, but I'm not convinced it is helpful :) I note that
> latest `apache-libcloud` does not install directly on windows, there seem to
> be some missing dependencies.>

I'm happy to give it a go - Chuck - can I cancel the various builds
running on my account, so I can do some debugging.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Updated 1.15.0 release notes

[Matthew Brett]

Hi Nathan,

One very helpful think you could do, is add a Travis-CI matrix entry
where you are testing against the latest numpy nightly builds.

I got a bit lost in your tox setup, but the basic idea is that, for
one test entry, you add the following flags to pip:

-f 
https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com
--pre

In that case, you'll pull in the latest nightly build of Numpy.  See
the Scipy .travis.yml setup for an example.

Cheers,

Matthew

On Thu, Jun 14, 2018 at 2:16 AM, Nathan Goldbaum  wrote:
> OK I guess I missed that announcement.
>
> I wouldn’t mind more than one email with a reminder to test.
>
> On Wed, Jun 13, 2018 at 7:42 PM Charles R Harris 
> wrote:
>>
>> On Wed, Jun 13, 2018 at 6:28 PM, Nathan Goldbaum 
>> wrote:
>>>
>>> Hi Chuck,
>>>
>>> Are you planning on doing an rc release this time? I think the NumPy 1.14
>>> release was unusually bumpy and part of that was the lack of an rc. One
>>> example: importing h5py caused a warning under numpy 1.14 and an h5py
>>> release didn’t come out with a workaround or fix for a couple months. There
>>> was also an issue with array printing that caused problems in yt (although
>>> both yt and NumPy quickly did bugfix releases that fixed that).
>>>
>>> I guess 1.14 was particularly noisy, but still I’d really appreciate
>>> having a prerelease version to test against and some time to report issues
>>> with the prerelease so numpy and other projects can implement workarounds as
>>> needed without doing a release that might potentially break real users who
>>> happen to install right after numpy 1.x.0 comes out.
>>
>>
>> There was a 1.14.0rc1. I was too quick for the full release, just waited
>> three weeks, so maybe four this time. Too few people actually test the
>> candidates and give feedback, so I tend to regard the *.*.0 releases as the
>> true rc :)
>>
>> Chuck
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Efficiency of Numpy wheels and simple way to benchmark Numpy installation?

[Matthew Brett]

Hi,

On Sun, May 27, 2018 at 9:12 PM, Nathaniel Smith  wrote:
> Performance is an incredibly multi-dimensional thing. Modern computers are
> incredibly complex, with layers of interacting caches, different
> microarchitectural features (do you have AVX2? does your cpu's branch
> predictor interact in a funny way with your workload?), compiler
> optimizations that vary from version to version, ... and different parts of
> numpy are affected differently by an these things.
>
> So, the only really reliable answer to a question like this is, always, that
> you need to benchmark the application you actually care about in the
> contexts where it will actually run (or as close as you can get to that).
>
> That said, as a general rule of thumb, the main difference between different
> numpy builds is which BLAS library they use, which primarily affects the
> speed of numpy's linear algebra routines. The wheels on pypi use either
> OpenBLAS (on Windows and Linux), or Accelerate (in MacOS. The conda packages
> provided as part of the Anaconda distribution normally use Intel's MKL.
>
> All three of these libraries are generally pretty good. They're all serious
> attempts to make a blazing fast linear algebra library, and much much faster
> than naive implementations. Generally MKL has a reputation for being
> somewhat faster than the others, when there's a difference. But again,
> whether this happens, or is significant, for *your* app is impossible to say
> without trying it.

Yes - I'd be surprised if you find a significant difference in
performance for real usage between pip / OpenBLAS and conda / MKL -
but if you do, please let us know, and we'll investigate.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Splitting MaskedArray into a separate package

[Matthew Brett]

Hi,

On Wed, May 23, 2018 at 10:42 PM, Stefan van der Walt
<stef...@berkeley.edu> wrote:
> On May 23, 2018 14:28:05 Matthew Brett <matthew.br...@gmail.com> wrote:
>>
>>
>> Can I ask what the plans are for supporting missing values, inside or
>> outside numpy?  Is there are successor to MaskedArray - and is this
>> part of the succession plan?
>
>
> I am not aware of any concrete plans, maybe others can chime in?
>
> It's a bit strange, the words that are used in this thread: "succession",
> "purification", "elimination", and "purge". I don't have my knife out for
> MaskedArrays; I merged a lot of Pierre's work myself. I simply suspect there
> may be a better and more supporting home/project configuration for it,
> perhaps still under the NumPy umbrella.

The NEP notes that MaskedArray imposes a significant maintenance
burden, as a motivation for removing it.  I'm sure you'd predict that
the Numpy developers are likely to spend less time on it, if it moves
to its own package.  I guess the hope would be that others would take
over, but is that likely?  What if they don't?

Would it be reasonable to develop an alternative plan for missing
arrays in concert with this NEP, maybe along the lines that Allan
mentioned, above?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Splitting MaskedArray into a separate package

[Matthew Brett]

Hi,


On Wed, May 23, 2018 at 9:51 PM, Stefan van der Walt
 wrote:
> Hi Eric,
>
> On May 23, 2018 13:25:44 Eric Firing  wrote:
>
>> On 2018/05/23 9:06 AM, Matti Picus wrote:
>> I understand at least some of the motivation and potential advantages,
>> but as it stands, I find this NEP highly alarming.
>
>
> I am not at my computer right now, so I will respond in more detail later.
> But I wanted to address your statement above:
>
> I see a NEP as an opportunity to discuss and flesh out an idea, and I
> certainly hope that you there's no reason for alarm.
>
> I do not expect to know whether this is a good idea before discussions
> conclude, so I appreciate your feedback. If we cannot find good support for
> the idea, with very specific benefits, it should simply be dropped.
>
> But, I think there's a lot to learn from the conversation in the meantime
> w.r.t. exactly how streamlined people want NumPy to be, how core
> functionality can perhaps be strengthened by becoming a customer of our own
> API, how to optimally maintain sub-components, etc.

Can I ask what the plans are for supporting missing values, inside or
outside numpy?  Is there are successor to MaskedArray - and is this
part of the succession plan?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introduction: NumPy developers at BIDS

[Matthew Brett]

Hi Stéfan,

On Tue, Apr 17, 2018 at 10:07 PM, Stefan van der Walt
<stef...@berkeley.edu> wrote:
> On Tue, 10 Apr 2018 10:03:06 -0700, Nathan Goldbaum wrote:
>> You may want to explore other venues for this sort of feedback, e.g. a
>> SciPy BoF session, which will capture a different subset of the
>> community.
>
> Thanks for the suggestion, Nathan.  We are coordinating with SciPy2018
> to have both a BoF and sprint at the end of the conference.
>
> On Tue, 10 Apr 2018 18:05:14 +0100, Matthew Brett wrote:
>> How about weekly open developer hangouts, recorded, to keep it all
>> public?
>
> Thanks for that idea, Matthew.  While we are ramping up, there's a lot
> of noise in sorting things out.  So how about we do dedicated monthly
> hangouts, where everyone can weigh in on the discussion, and where the
> signal-to-noise ratio is higher for interested parties?
>
> We are tracking all work items here on Trello:
>
>   https://trello.com/b/Azg4fYZH/numpy-at-bids
>
> (Of course, a lot happens directly on NumPy issues too, but this board
> is for publicly tracking "work to support the work").

Hum - I see the Trello board is for bite- to meal- size practical
issues, but not for the general process of how to engage the community
in guiding the the project - is that fair?

I was thinking about the engage community part, because it seems to me
it would be good to spend time on that first, and if it was me, I
think I'd go for more regular public meetings / discussions at this
stage rather than less.

I'm thinking now of Jarrod's / Brian's story about "more typing", I'm
sure you know the one I mean :)

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introduction: NumPy developers at BIDS

[Matthew Brett]

Yo,

How about weekly open developer hangouts, recorded, to keep it all public?

Cheers,

Matthew

On Tue, Apr 10, 2018 at 5:59 PM, Stefan van der Walt
 wrote:
> Hi Eric,
>
> On Sun, 08 Apr 2018 08:02:19 -1000, Eric Firing wrote:
>> On 2018/04/07 9:19 PM, Stefan van der Walt wrote:
>> > We would love community input on identifying the best areas & issues to
>> > pay attention to,
>>
>> What is the best way to provide this, and how will the decisions be
>> made?
>
> These are good questions.  We are also new at this, so while we have
> some ideas on how things could work, we may have to refine the process
> along the way.
>
> We want to operate as openly as we can, so discussing ideas on the
> mailing list is a preferred first option.  But we're also open to
> inchoate ideas and recommendations (including on how we run things on
> our end) via email.  Unless instructed explicitly otherwise, those ideas
> will likely bubble up into posts here anyway.
>
> Since we're learning the ropes, we'd like to expose the team to a wide
> variety of ideas.  Visitors to the team are most welcome---please reach
> out to me if you want to talk to us, either in person or via video chat.
>
> Can you help us think of good ways to learn "community priorities"?
> E.g., for GitHub issues, should we take monthly polls, count the number
> of "thumbs up"s, consider issues with the most comments, or tally the
> number of explicit mentions of team members?
>
> Best regards,
> Stéfan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] f2py bug in numpy v1.12.0 and above?

[Matthew Brett]

Hi,

On Mon, Jan 29, 2018 at 1:02 PM, Andrew Nelson  wrote:
> Something similar was mentioned at
> https://github.com/scipy/scipy/issues/8325.

Yes, that one was also for Anaconda...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] f2py bug in numpy v1.12.0 and above?

[Matthew Brett]

Hi,

On Mon, Jan 29, 2018 at 10:16 AM, Solbrig,Jeremy
 wrote:
> I have a suite of fortran code that I compile with f2py and use as a plugin
> to a python package.  I am using Python v2.7 from Anaconda.  When compiled
> using numpy v1.11.3 or lower, everything works fine, but if I upgrade to any
> more recent version I begin running into a runtime error.  Presumably I am
> just not linking something that needs to be linked, but I'm not sure what
> library that would be.
>
>
> The error:
>
> ImportError: .../libstddev_kernel.so: undefined symbol:
> _gfortran_stop_numeric_f08
>
> Where libestddev_kernel.so is built using this command:
> f2py --fcompiler=gnu95 --quiet --opt="-O3" -L. -L/usr/lib  -I. -I.../include
> -I/usr/include  -m libstddev_kernel -c stddev_kernel/stddev_kernel.f90
> config.o
>
>
> Is there something additional I should be linking to within numpy or my
> anaconda installation?

Do you get the same problem with a pip install of numpy?   If not,
this may be an Anaconda packaging bug rather than a numpy bug ...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] with np.testing.assert_raises still requires nose.

[Matthew Brett]

Hi,

On Tue, Jan 23, 2018 at 10:04 AM, Andrew Nelson  wrote:
> Dear np list,
> from a recent scipy PR (https://github.com/scipy/scipy/pull/8322) it appears
> that the `np.testing.assert_raises` context manager still requires nose to
> be installed:
> https://ci.appveyor.com/project/scipy/scipy/build/1.0.1550/job/s5nq8xqd0n72feqf
>
> The appveyor test is using numpy-1.14.0
>
> This doesn't seem to be an issue on our travis tests, only the windows
> tests. Is this a bug?

I noticed that too.  For my own project I ended up adding a replacement stub:

"""
from unittest import TestCase

assert_raises = TestCase().assertRaises
"""

I think that's all nose is doing, but I couldn't find the code path in
a quick grep.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of rint?

[Matthew Brett]

Hi,

On Fri, Jan 19, 2018 at 3:24 PM, Charles R Harris
<charlesr.har...@gmail.com> wrote:
>
>
> On Fri, Jan 19, 2018 at 7:48 AM, Matthew Brett <matthew.br...@gmail.com>
> wrote:
>>
>> Hi Chuck,
>>
>> Thanks for the replies, they are very helpful.
>>
>> On Fri, Jan 19, 2018 at 1:51 PM, Charles R Harris
>> <charlesr.har...@gmail.com> wrote:
>> >
>> >
>> > On Fri, Jan 19, 2018 at 6:41 AM, Charles R Harris
>> > <charlesr.har...@gmail.com> wrote:
>> >>
>> >>
>> >>
>> >> On Fri, Jan 19, 2018 at 3:30 AM, Matthew Brett
>> >> <matthew.br...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Sorry for my confusion, but I noticed (as a result of the discussion
>> >>> here [1]) that np.rint and the fallback C function [2] seem to round
>> >>> to even.  But - my impression was that C rint, by default, rounds down
>> >>> [3].   Is numpy rint not behaving the same way as the GNU C library
>> >>> rint?
>> >>>
>> >>> In [4]: np.rint(np.arange(0.5, 11))
>> >>> Out[4]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])
>> >>>
>> >>> In [5]: np.round(np.arange(0.5, 11))
>> >>> Out[5]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])
>> >>
>> >>
>> >> The GNU C documentation says that rint "round(s) x to an integer value
>> >> according to the current rounding mode." The rounding mode is
>> >> determined by
>> >> settings in the FPU control word. Numpy runs with it set to round to
>> >> even,
>> >> although, IIRC, there is a bug on windows where the library is not
>> >> setting
>> >> those  bits correctly.
>> >
>> >
>> > Round to even is also the Python default rounding mode.
>>
>> Do you mean that it is Python setting the FPU control word?  Or do we
>> set it?  Do you happen to know where that is in the source?  I did a
>> quick grep just now without anything obvious.
>
>
> I can't find official (PEP) documentation, but googling indicates that in
> Python 3, `round` rounds to even, and in Python 2 it rounds up. See also
> https://docs.python.org/3/whatsnew/3.0.html.

But I guess this could be the Python implementation of round, rather
than rint and the FPU control word?  I'm asking because the question
arose about npy_rint at the C level ...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of rint?

[Matthew Brett]

Hi Chuck,

Thanks for the replies, they are very helpful.

On Fri, Jan 19, 2018 at 1:51 PM, Charles R Harris
<charlesr.har...@gmail.com> wrote:
>
>
> On Fri, Jan 19, 2018 at 6:41 AM, Charles R Harris
> <charlesr.har...@gmail.com> wrote:
>>
>>
>>
>> On Fri, Jan 19, 2018 at 3:30 AM, Matthew Brett <matthew.br...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Sorry for my confusion, but I noticed (as a result of the discussion
>>> here [1]) that np.rint and the fallback C function [2] seem to round
>>> to even.  But - my impression was that C rint, by default, rounds down
>>> [3].   Is numpy rint not behaving the same way as the GNU C library
>>> rint?
>>>
>>> In [4]: np.rint(np.arange(0.5, 11))
>>> Out[4]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])
>>>
>>> In [5]: np.round(np.arange(0.5, 11))
>>> Out[5]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])
>>
>>
>> The GNU C documentation says that rint "round(s) x to an integer value
>> according to the current rounding mode." The rounding mode is determined by
>> settings in the FPU control word. Numpy runs with it set to round to even,
>> although, IIRC, there is a bug on windows where the library is not setting
>> those  bits correctly.
>
>
> Round to even is also the Python default rounding mode.

Do you mean that it is Python setting the FPU control word?  Or do we
set it?  Do you happen to know where that is in the source?  I did a
quick grep just now without anything obvious.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Behavior of rint?

[Matthew Brett]

Hi,

Sorry for my confusion, but I noticed (as a result of the discussion
here [1]) that np.rint and the fallback C function [2] seem to round
to even.  But - my impression was that C rint, by default, rounds down
[3].   Is numpy rint not behaving the same way as the GNU C library
rint?

In [4]: np.rint(np.arange(0.5, 11))
Out[4]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])

In [5]: np.round(np.arange(0.5, 11))
Out[5]: array([ 0.,  2.,  2.,  4.,  4.,  6.,  6.,  8.,  8., 10., 10.])

Cheers,

Matthew

[1] https://github.com/nipy/dipy/issues/1402
[2] 
https://github.com/numpy/numpy/blob/master/numpy/core/src/npymath/npy_math_internal.h.src#L290
[3] https://www.gnu.org/software/libc/manual/html_node/Rounding-Functions.html
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.14.0 release

[Matthew Brett]

Hi,

On Sun, Jan 14, 2018 at 3:35 AM, Eric Wieser
 wrote:
> Did recarrays change? I didn’t see anything in the release notes.
>
> Not directly, but structured arrays did, for which recarrays are really just
> a thin and somewhat buggy wrapper.

Oh dear oh dear - for some reason I had completely missed these
changes, and the justification for them.

They do exactly the kind of thing that Konrad Hinsen was complaining
about before, with justification, which is to change the behavior of
previous code, without an intervening (long) period of raising an
error.  In this case, the benefits of these changes seem small,
compared to the inevitable breakage and silently changed results they
will cause.

Is there any chance of reversing them?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Proposal - change to OpenBLAS for Windows wheels

[Matthew Brett]

Hi,

I suggest we switch from ATLAS to OpenBLAS for our Windows wheels:

* OpenBLAS is much faster, at least when Tony Kelman tested it last year [1];
* We now have an automated Appveyor build for OpenBLAS [2, 3];
* Tests are passing with 32-bit and 64-bit wheels [4];
* The next Scipy release will have OpenBLAS wheels;

Any objections / questions / alternatives?

Cheers,

Matthew

[1] https://github.com/numpy/numpy/issues/5479#issuecomment-185033668
[2] https://github.com/matthew-brett/build-openblas
[3] https://ci.appveyor.com/project/matthew-brett/build-openblas
[4] https://ci.appveyor.com/project/matthew-brett/numpy-wheels/build/1.0.50
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping support for Accelerate

[Matthew Brett]

On Tue, Jul 25, 2017 at 3:00 PM, Nathaniel Smith <n...@pobox.com> wrote:
> On Tue, Jul 25, 2017 at 6:48 AM, Matthew Brett <matthew.br...@gmail.com> 
> wrote:
>> On Tue, Jul 25, 2017 at 2:19 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>> I updated the bit about OpenBLAS wheel with some more information on
>>> the status of that work. It's not super important, but FYI.
>>
>> Maybe remove the bit (of my text) that you crossed out, or removed the
>> strikethrough and qualify?  At the moment it's confusing, because I
>> believe what I wrote is correct, so leaving in there and crossed out
>> looks kinda weird.
>
> Eh, it's a little weird because there's no specification needed
> really, we can implement it any time we want to. It was stalled for a
> long time because I ran into arcane technical problems dealing with
> the MacOS linker, but that's solved and now it's just stalled due to
> lack of attention.
>
> I deleted the text but feel free to qualify further if you think it's useful.

Are you saying that we should consider this specification approved
already?  Or that we should go ahead without waiting for approval?  I
guess the latter.  I guess you're saying you think there would be no
bad consequences for doing this if the spec subsequently changed
before being approved?  It might be worth adding something like that
to the text, in case there's somebody who wants to do some work on
that.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping support for Accelerate

[Matthew Brett]

On Tue, Jul 25, 2017 at 2:19 PM, Nathaniel Smith  wrote:
> I updated the bit about OpenBLAS wheel with some more information on
> the status of that work. It's not super important, but FYI.

Maybe remove the bit (of my text) that you crossed out, or removed the
strikethrough and qualify?  At the moment it's confusing, because I
believe what I wrote is correct, so leaving in there and crossed out
looks kinda weird.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping support for Accelerate

[Matthew Brett]

Hi,

On Sun, Jul 23, 2017 at 5:07 PM, Ilhan Polat  wrote:
> Ouch, that's from 2012 :(  I'll add this thread as a reference to the wiki
> list.
>
>
> On Sun, Jul 23, 2017 at 5:22 PM, Nathan Goldbaum 
> wrote:
>>
>> See
>> https://mail.scipy.org/pipermail/numpy-discussion/2012-August/063589.html
>> and replies in that thread.
>>
>> Quote from an Apple engineer in that thread:
>>
>> "For API outside of POSIX, including GCD and technologies like Accelerate,
>> we do not support usage on both sides of a fork(). For this reason among
>> others, use of fork() without exec is discouraged in general in processes
>> that use layers above POSIX."
>>
>> On Sun, Jul 23, 2017 at 10:16 AM, Ilhan Polat 
>> wrote:
>>>
>>> That's probably because I know nothing about the issue, is there any
>>> reference I can read about?
>>>
>>> But in general, please feel free populate new items in the wiki page.
>>>
>>> On Sun, Jul 23, 2017 at 11:15 AM, Nathaniel Smith  wrote:

 I've been wishing we'd stop shipping Accelerate for years, because of
 how it breaks multiprocessing – that doesn't seem to be on your list
 yet.

 On Sat, Jul 22, 2017 at 3:50 AM, Ilhan Polat 
 wrote:
 > A few months ago, I had the innocent intention to wrap LDLt
 > decomposition
 > routines of LAPACK into SciPy but then I am made aware that the
 > minimum
 > required version of LAPACK/BLAS was due to Accelerate framework. Since
 > then
 > I've been following the core SciPy team and others' discussion on this
 > issue.
 >
 > We have been exchanging opinions for quite a while now within various
 > SciPy
 > issues and PRs about the ever-increasing Accelerate-related issues and
 > I've
 > compiled a brief summary about the ongoing discussions to reduce the
 > clutter.
 >
 > First, I would like to kindly invite everyone to contribute and
 > sharpen the
 > cases presented here
 >
 > https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate
 >
 > The reason I specifically wanted to post this also in NumPy mailing
 > list is
 > to probe for the situation from the NumPy-Accelerate perspective. Is
 > there
 > any NumPy specific problem that would indirectly effect SciPy should
 > the
 > support for Accelerate is dropped?
 >
 >
 >
 >
 > ___
 > NumPy-Discussion mailing list
 > NumPy-Discussion@python.org
 > https://mail.python.org/mailman/listinfo/numpy-discussion
 >



 --
 Nathaniel J. Smith -- https://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@python.org
 https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>>
>>> ___
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion

I added some more discussion, and some links to previous discussion on
the mailing list.

I also pointed to this PR :
https://github.com/MacPython/numpy-wheels/pull/1 - which builds
OpenBLAS wheels for numpy.  The same kind of thing would work fine for
Scipy.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Polynomial silent breakage with 1.13

[Matthew Brett]

Hi,

On Fri, Jul 7, 2017 at 6:14 PM, Eric Wieser  wrote:
> That’s a regression, and it’s on me, in 8762.
>
> That was a side effect of a fix for the weird behaviour here.
>
> I think we need to fix this in 1.13.2, so we should file an issue about it.

Thanks for the feedback.  Do you want to file an issue, or should I?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Polynomial silent breakage with 1.13

[Matthew Brett]

Hi,

Our (nipy's) test suite just failed with the upgrade to numpy 1.13,
and the cause boiled down to this:

```
import numpy as np

poly = np.poly1d([1])
poly.c[0] *= 2
print(poly.c)
```

Numpy 1.12 gives (to me) expected output:

[2]

Numpy 1.13 gives (to me) unexpected output:

[1]

The problem is caused by the fact that the coefficients are now a
*copy* of the actual coefficient array - I think in an attempt to stop
us modifying the coefficients directly.

I can't see any deprecation warnings with `-W always`.

The pain point here is that code that used to give the right answer
has now (I believe silently) switched to giving the wrong answer.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making a 1.13.2 release

[Matthew Brett]

On Thu, Jul 6, 2017 at 3:37 PM, Charles R Harris
<charlesr.har...@gmail.com> wrote:
>
>
> On Thu, Jul 6, 2017 at 7:15 AM, Matthew Brett <matthew.br...@gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Thu, Jul 6, 2017 at 2:10 PM, Charles R Harris
>> <charlesr.har...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I've delayed the NumPy 1.13.2 release hoping for Python 3.6.2 to show up
>> > fixing #29943  so we can close #9272, but the Python release has been
>> > delayed to July 11 (expected). The Python problem means that NumPy
>> > compiled
>> > with Python 3.6.1 will not run in Python 3.6.0. However, I've also been
>> > asked to have a bugfixed version of 1.13 available for Scipy 2017 next
>> > week.
>> > At this point it looks like the best thing to do is release 1.13.1
>> > compiled
>> > with Python 3.6.1 and ask folks to upgrade Python if they have a
>> > problem,
>> > and then release 1.13.2 as soon as 3.6.2 is released.
>>
>> I think this problem only applies to Windows.  We might be able to
>> downgrade the Appveyor Python 3.6.1 to 3.6.0 for that - I can look
>> into it today if it would help.
>>
>> While I'm at it - how about switching to OpenBLAS wheels on Windows
>> for this release?
>>
>> Cheers,
>>
>> Matthew
>
>
> Haste makes waste ;) I'd rather put off the move to OpenBlas to 1.14 to
> allow more time for it to settle,

I'd only say that I don't know of any settling that is likely to
happen.  I suspect that not many people have tried the experimental
wheels.   I've automated the build process both for OpenBLAS and the
OpenBLAS wheels, and I believe those are solid now.

> and compiling against Python 3.6.0 seems
> like more work than it is worth,

Probably about two hours of futzing on Appveyor - your call - I'm
happy not to do it :)

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making a 1.13.2 release

[Matthew Brett]

Hi,

On Thu, Jul 6, 2017 at 2:10 PM, Charles R Harris
 wrote:
> Hi All,
>
> I've delayed the NumPy 1.13.2 release hoping for Python 3.6.2 to show up
> fixing #29943  so we can close #9272, but the Python release has been
> delayed to July 11 (expected). The Python problem means that NumPy compiled
> with Python 3.6.1 will not run in Python 3.6.0. However, I've also been
> asked to have a bugfixed version of 1.13 available for Scipy 2017 next week.
> At this point it looks like the best thing to do is release 1.13.1 compiled
> with Python 3.6.1 and ask folks to upgrade Python if they have a problem,
> and then release 1.13.2 as soon as 3.6.2 is released.

I think this problem only applies to Windows.  We might be able to
downgrade the Appveyor Python 3.6.1 to 3.6.0 for that - I can look
into it today if it would help.

While I'm at it - how about switching to OpenBLAS wheels on Windows
for this release?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Scipy 2017 NumPy sprint

[Matthew Brett]

On Wed, Jul 5, 2017 at 11:31 AM, Peter Cock  wrote:
> On Wed, Jul 5, 2017 at 11:25 AM, Ralf Gommers  wrote:
>>
>>
>> On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock 
>> wrote:
>>>
>>> Note that TravisCI does not yet have official Python support on Mac OS X,
>>>
>>> https://github.com/travis-ci/travis-ci/issues/2312
>>>
>>> I believe it is possible to do anyway by faking it under another setting
>>> (e.g. pretend to be a generic language build, and use the system Python
>>> or install your own specific version of Python as needed), so that may be
>>> worth trying during a sprint.
>>
>>
>> That approach has worked reliably for
>> https://github.com/MacPython/numpy-wheels for a while now, so should be
>> straightforward.
>>
>> Ralf
>
> Thanks for that link - I'm going off topic but the MacPython wiki page goes
> into more background about how they build wheels for PyPI which I'm
> very interested to read up on:
>
> https://github.com/MacPython/wiki/wiki/Spinning-wheels

Yes, you'll see that the multibuild framework that numpy and scipy
uses, includes utilities to download Python.org Python and build
against that, in Spinning-wheels fashion,

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Scipy 2017 NumPy sprint

[Matthew Brett]

On Wed, Jul 5, 2017 at 11:25 AM, Ralf Gommers <ralf.gomm...@gmail.com> wrote:
>
>
> On Wed, Jul 5, 2017 at 10:14 PM, Peter Cock <p.j.a.c...@googlemail.com>
> wrote:
>>
>> Note that TravisCI does not yet have official Python support on Mac OS X,
>>
>> https://github.com/travis-ci/travis-ci/issues/2312
>>
>> I believe it is possible to do anyway by faking it under another setting
>> (e.g. pretend to be a generic language build, and use the system Python
>> or install your own specific version of Python as needed), so that may be
>> worth trying during a sprint.
>
>
> That approach has worked reliably for
> https://github.com/MacPython/numpy-wheels for a while now, so should be
> straightforward.

And https://travis-ci.org/MacPython/scipy-wheels where we are testing
OSX, 64 and 32 bit manylinux builds daily.  That didn't catch the
recent ndimage error because I'd disabled the 32-bit builds there.

Numpy, scipy, and a fairly large number of other projects use
https://github.com/matthew-brett/multibuild to set up builds in this
way for manylinux, OSX and (with a bit more effort) Windows.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] UC Berkeley hiring developers to work on NumPy

[Matthew Brett]

Hi,

On Fri, May 19, 2017 at 10:06 PM, Nathaniel Smith <n...@pobox.com> wrote:
> On Mon, May 15, 2017 at 1:43 AM, Matthew Brett <matthew.br...@gmail.com> 
> wrote:
>> Hi,
>>
>> On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
>> <charlesr.har...@gmail.com> wrote:
>>>
>>>
>>> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> As some of you know, I've been working for... quite some time now to
>>>> try to secure funding for NumPy. So I'm excited that I can now
>>>> officially announce that BIDS [1] is planning to hire several folks
>>>> specifically to work on NumPy. These will full time positions at UC
>>>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>>>> and the general goal will be to work on some of the major priorities
>>>> we identified at the last dev meeting: more flexible dtypes, better
>>>> interoperation with other array libraries, paying down technical debt,
>>>> and so forth. Though I'm sure the details will change as we start to
>>>> dig into things and engage with the community.
>>>>
>>>> More details soon; universities move slowly, so nothing's going to
>>>> happen immediately. But this is definitely happening and I wanted to
>>>> get something out publicly before the conference season starts – so if
>>>> you're someone who might be interested in coming to work with me and
>>>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>>>> line and we can chat! I'll be at PyCon next week if anyone happens to
>>>> be there. And feel free to spread the word.
>>>
>>>
>>> Excellent news. Do you have any sort of timeline in mind?
>>>
>>> It will be interesting to see what changes this leads to, both in the code
>>> and in the project sociology.
>>
>> I was thinking the same thing - if this does come about, it would
>> likely have a big impact on practical governance.  It could also mean
>> that more important development conversations happen off-list.   It
>> seems to me it would be good to plan for this consciously.
>
> Yeah, definitely. Being able to handle changes like this was one of
> the major motivations for all the governance discussions we started a
> few years ago, and it's something we'll need to keep an eye on going
> forward. To state it explicitly though: the idea is to fund folks so
> that they can contribute to numpy within our existing process of open
> community review, and preserving and growing that community is very
> much one of the grant's goals; no-one should get special privileges
> because of where their paycheck is coming from. If at some point you
> (or anyone) feel like we're deviating from that please speak up.

I think Chuck's term 'sociology' is a good one; although it's good to
have the governance document, I don't think it covers all the issues
that your initiative brings up.  At the moment, the people doing the
most work on numpy (by commits at least) are Chuck, Eric and Julian.
As far as I know, we don't have any full-time developers.   When
you've got up and running it sounds like you'll have at least two
full-time developers.  I guess you'll be their manager, and that they
will be physically housed in the BIDS.   So, this will represent a big
shift in practical influence, from a more or less completely
distributed pool of developer hours, to something much closer to an
institution-owned project.   Of course, we can hope that this doesn't
have any negative consequences in terms of project dynamics, but it
seems to me that it would be sensible to think of the risks, and plan
for them, rather than waiting for the expected problems to arise, when
they may be too late to fix.

We had this kind of discussion with Travis a while ago, and he pushed
back about conflicts of interests for people in the BIDS.  At the
time, that wasn't a serious issue, because you were the only BIDS
member actively committing to numpy.  That changes when it's you, and
two full-time developers.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] UC Berkeley hiring developers to work on NumPy

[Matthew Brett]

Hi,

On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
 wrote:
>
>
> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith  wrote:
>>
>> Hi all,
>>
>> As some of you know, I've been working for... quite some time now to
>> try to secure funding for NumPy. So I'm excited that I can now
>> officially announce that BIDS [1] is planning to hire several folks
>> specifically to work on NumPy. These will full time positions at UC
>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>> and the general goal will be to work on some of the major priorities
>> we identified at the last dev meeting: more flexible dtypes, better
>> interoperation with other array libraries, paying down technical debt,
>> and so forth. Though I'm sure the details will change as we start to
>> dig into things and engage with the community.
>>
>> More details soon; universities move slowly, so nothing's going to
>> happen immediately. But this is definitely happening and I wanted to
>> get something out publicly before the conference season starts – so if
>> you're someone who might be interested in coming to work with me and
>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>> line and we can chat! I'll be at PyCon next week if anyone happens to
>> be there. And feel free to spread the word.
>
>
> Excellent news. Do you have any sort of timeline in mind?
>
> It will be interesting to see what changes this leads to, both in the code
> and in the project sociology.

I was thinking the same thing - if this does come about, it would
likely have a big impact on practical governance.  It could also mean
that more important development conversations happen off-list.   It
seems to me it would be good to plan for this consciously.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion