[Numpy-discussion] Re: Generate documentation for the 2.0.0 release

[Matti Picus]

I will make a PR to https://github.com/numpy/doc

Matti


On 16/06/2024 19:04, Charles R Harris wrote:
PS, you will need to check out `.spin/cmds.py` from the main branch in 
order to build the documentation.


Chuck

On Sun, Jun 16, 2024 at 9:59 AM Charles R Harris 
 wrote:


Hi All,

Could someone running Python < 3.12 generate the documentation for
the NumPy 2.0.0 release. My native Python is 3.12, which works
fine except that there is no distutils. Trying to run the build
under 3.11, after hard coding the Python version in the
doc/Makefile, runs into various problems, and there are probably
some subtle Python version mismatches buried in the scripts
somewhere that I don't want to spend time chasing down at the moment.

Chuck


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Unexpected return values for np.mod with x2=np.inf and similar

[Matti Picus]

What operating system?


If I recall correctly, NumPy tries to be compatible with CPython for 
these edge cases.



The actual implementation is a bit scattered. I think it would be nice 
if we could have an "explain" decorator to ufuncs that would return the 
name of the inner loop used in practice to aid in debugging and 
teaching. Until then your best bet is to build NumPy locally with debug 
information and use a debugger, but even that can be challenging at times.


Matti


On 07/06/2024 21:10, jesse.live...@gmail.com wrote:

Hi all,

I ran into an odd edge-case with np.mod and was wondering if this is the 
expected behavior, and if so why. This is on a fresh install of python 3.10.14 
with numpy 1.26.4 from conda-forge.
...
Any ideas why these are the return values? I had a hard time tracking down 
where in the numpy source np.mod was coming from.
Jesse
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com
https://github.com/numpy/numpy/blob/main/numpy/_core/src/umath/loops_modulo.dispatch.c.src#L557

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Build NumPy with Debugging Symbols with Meson

[matti picus]

On 15/05/2024 23:55, Robert McLeod wrote:
|
| Hi everyone,
|
| Is there a gist or similar guide anywhere on the steps required to
build NumPy with debugging symbols on the Windows platform using the
new Meson build system?


In general, I would try to avoid using python_d.exe. It requires the
entire software stack to be rebuilt with that python, which can be
quite painful (as you discovered). Instead, you can build only the
project you wish to debug with release-with-debug-symbols (on windows,
these are `*.pdb` files), which will allow you to run the exe under a
debugger.

Another tip for debugging: add the line `import pdb;pdb.set_trace()`
to your python code, which will stop the process under test. Then you
can fire up visual studio, and in the debugger menu choose
attach-to-process and connect to the correct PID to start debugging
compiled code. If you do this after `import numpy`, you should be able
to navigate to the code you wish to debug in an editor window and set
a breakpoint. Then go back to your python terminal and continue the
run.

Matti
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New feature

[Matti Picus]

For a start you could share the link for the stand-alone package here, 
with a short description of what it does.


Additional steps would be to write a blog post about it, write it up on 
reddit's python subreddit, give a talk about it at a conference.


Matti


On 25/04/2024 23:36, Alexei Lisitsa wrote:

Ok, thanks package stand alone I did but how I advertise it?
Maybe you as expert can say some thing that is needed , something to 
develop I want some new experience




___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Matti Picus]

On 05/05/2024 11:32, Mark Harfouche wrote:

I know this is somewhat the 11th hour for the numpy 2.0 release but as 
downstream packager and user of numpy, I would really like to ask that 
Numpy strives to adhere to NEP29, and SPEC0


SEPC0 makes it pretty clear that Python 3.9 should not be included in 
any "future" versions.


Leading by example in this case will help downstream make the hard 
decision to adhere to the compatibility guidelines from SPEC0.


Thank you for considering this last minute request. I know it adds 
work at this stage.


Mark



I think NumPy should not be the leader in dropping versions, rather 
should be one of the more conservative packages since other packages 
depend on it. We have indeed dropped 3.9 on HEAD, and will not be 
supporting it for 2.1, but to me it makes sense to support it for the 
large 2.0 release.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New feature

[Matti Picus]

On 25/04/2024 23:16, Alexei Lisitsa wrote:


Waiting for answer



What kind of answer would you like? I took a look at numpy_list[0] and 
if it serves your needs, that is great, but I don't think such ndarray 
generation routines should be added to NumPy until they become more 
commonly known, used, and "voted" popular. I would suggest you


- add some examples to the README[1] to demonstrate its utility. I found 
I had to drill down to the code[1] to see what it actually does.


- make it a stand-alone package and not part of NumPy's code base

Matti


[0] 
https://github.com/alessionuovo/mionumpy/tree/list_operations/numpy_extensions


[1] 
https://github.com/alessionuovo/mionumpy/blob/list_operations/numpy_extensions/numpy_list/list_op.py


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Making show_runtime and show_config enable file output

[Matti Picus]

On 09/04/2024 2:42, Matan Addam wrote:


Hello,

I find the information printed by the above mentioned functions to be useful 
for understanding performance context on installed machines, as well as 
variability across machines when troubleshooting. How would the maintainers 
view a pull request adding to those functions the option of directing their 
output to a file?

These functions, at least as built on my machine using numpy 1.24.4, using 
python's print and pprint for their outputs, both of which functions allowing 
arguments for redirecting their output to a file. Adding this option may enable 
recording the information to files without redirecting all of stdout.

What would your position be?

Or are they actually a facade built upon installation by dynamically generated 
code, which yields different function implementations on different platforms?

It could be otherwise nice to provide access to a dict of the data for the more 
general purpose, which would enable all desiderata of interest leveraging this 
information.

Kind regards,
Matan



Makes sense to me to output a json-formatted file

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Moving the weekly traige/community meetings

[Matti Picus]

Could we move the weekly community/triage meetings one hour later? Some 
participants have a permanent conflict, and the current time is 
inconvenient for my current time zone.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Accelerated NumPy with Weld compiler

[Matti Picus]

On 20/3/24 15:13, Hemant Singh via NumPy-Discussion wrote:

Stanford and MIT jointly developed the Weld language and compiler to run NumPy 
and Pandas significantly faster, 10x-250x.
My company has changed the compiler to be production ready. Anyone interested 
in trialing this compiler, please contact me with your work email.

Best,

Hemant



Hi Hemant. Weld seems like a cool technology. It seems that the 
opensource weld repo [0] has been dormant for the past couple of years. 
Will you be contributing back to that? We would like to encourage free 
access to technology, but understand people need to make a living. Do 
you have a model for licensing or sharing your compiler?


Matti


[0] https://github.com/weld-project/weld

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Arrays of variable itemsize

[Matti Picus]

I am not sure what kind of a scheme would support various-sized native 
ints. Any scheme that puts pointers in the array is going to be worse: 
the pointers will be 64-bit. You could store offsets to data, but then 
you would need to store both the offsets and the contiguous data, nearly 
doubling your storage. What shape are your arrays, that would be the 
minimum size of the offsets?


Matti


On 13/3/24 18:15, Dom Grigonis wrote:
By the way, I think I am referring to integer arrays. (Or integer part 
of floats.)


I don’t think what I am saying sensibly applies to floats as they are.

Although, new float type could base its integer part on such concept.

—

Where I am coming from is that I started to hit maximum bounds on 
integer arrays, where most of values are very small and some become 
very large. And I am hitting memory limits. And I don’t have many 
zeros, so sparse arrays aren’t an option.


Approximately:
90% of my arrays could fit into `np.uint8`
1% requires `np.uint64`
the rest 9% are in between.

And there is no predictable order where is what, so splitting is not 
an option either.




On 13 Mar 2024, at 17:53, Nathan  wrote:

Yes, an array of references still has a fixed size width in the array 
buffer. You can think of each entry in the array as a pointer to some 
other memory on the heap, which can be a dynamic memory allocation.


There's no way in NumPy to support variable-sized array elements in 
the array buffer, since that assumption is key to how numpy 
implements strided ufuncs and broadcasting.,


On Wed, Mar 13, 2024 at 9:34 AM Dom Grigonis  
wrote:


Thank you for this.

I am just starting to think about these things, so I appreciate
your patience.

But isn’t it still true that all elements of an array are still
of the same size in memory?

I am thinking along the lines of per-element dynamic memory
management. Such that if I had array [0, 1e1], the first
element would default to reasonably small size in memory.


On 13 Mar 2024, at 16:29, Nathan  wrote:

It is possible to do this using the new DType system.

Sebastian wrote a sketch for a DType backed by the GNU
multiprecision float library:
https://github.com/numpy/numpy-user-dtypes/tree/main/mpfdtype

It adds a significant amount of complexity to store data outside
the array buffer and introduces the possibility of
use-after-free and dangling reference errors that are impossible
if the array does not use embedded references, so that’s the
main reason it hasn’t been done much.

On Wed, Mar 13, 2024 at 8:17 AM Dom Grigonis
 wrote:

Hi all,

Say python’s builtin `int` type. It can be as large as
memory allows.

np.ndarray on the other hand is optimized for vectorization
via strides, memory structure and many things that I
probably don’t know. Well the point is that it is convenient
and efficient to use for many things in comparison to
python’s built-in list of integers.

So, I am thinking whether something in between exists? (And
obviously something more clever than np.array(dtype=object))

Probably something similar to `StringDType`, but for
integers and floats. (It’s just my guess. I don’t know
anything about `StringDType`, but just guessing it must be
better than np.array(dtype=object) in combination with
np.vectorize)

Regards,
dgpb

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to
numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: nathan12...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: dom.grigo...@gmail.com


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: nathan12...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: dom.grigo...@gmail.com



___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

[Numpy-discussion] Change in behaviour: broadcasting rules for np.solve

[Matti Picus]

API changes should hit the mailing list, so I am copying from PR 
https://github.com/numpy/numpy/pull/25914 by Aaron Meurer:





Previously the np.linalg.solve documentation stated:

|a : (..., M, M) array_like Coefficient matrix. b : {(..., M,), (..., M, 
K)}, array_like |


however, this is inherently ambiguous. For example, if a has shape (2, 
2, 2) and b has shape (2, 2), b could be treated as a (2,) stack of (2,) 
column vectors, in which case the result should have shape (2, 2), or as 
a single 2x2 matrix, in which case, the result should have shape (2, 2, 2).



NumPy resolved this ambiguity in a confusing way, which was to treat b 
as (..., M) whenever b.ndim == a.ndim - 1, and as (..., M, K) otherwise.



A much more consistent way to handle this ambiguity is to treat b as a 
single vector if and only if it is 1-dimensional, i.e., use


|b : {(M,), (..., M, K)}, array_like |

This is consistent with, for instance, the matmul operator, which only 
uses the special 1-D vector logic if an operand is exactly 
1-dimensional, and treats the operands as (stacks of) 2-D matrices 
otherwise. The PR  updates np.linalg.solve() to use this behavior.



This is a backwards compatibility break, as any instance where the b 
array has more than one dimension and exactly one fewer dimension than 
the a array will now use the matrix logic, potentially returning a 
different result with a different shape. The previous behavior can be 
manually emulated with something like


np.solve(a,b[...,None])[...,0]

since b as a (M,) vector is equivalent to b as a (M, 1) matrix (or the 
user could manually import and use the internal solve1() gufunc which 
implements the b-as-vector logic).



This change aligns the solve() function with the array API, which 
resolves this broadcasting ambiguity in this way. See

https://data-apis.org/array-api/latest/extensions/generated/array_api.linalg.solve.html#array_api.linalg.solve





Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: builing numpy on windows

[Matti Picus]

On 27/2/24 06:09, Ganesh Rajora via NumPy-Discussion wrote:

Hi Team,

I am Ganesh working for an MNC here in India and I am working on 
customised Python where I build set of python modules with python.


I do not install them directly from web but I build each and 
everything from source code.  this is because security concerns at the 
organisation.


In similar line I want to get some help from you on building numpy on 
windows, as I did not find any direct reference on how to do that on 
any of numpy's official web. I am facing lots of issues to build that, 
If you could help me out with it or could give me a right point of 
contact to discuss the issue would be great help.


I have posted the issue here on stackoverflow -

Issues in buildingnumpy from source on Windows 




Thanks,
Ganesh



I posted on the stack overflow link. My guess is you are not building an 
OpenBLAS MSVC import library, but using MSVC to build NumPy.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: JSON format for multi-dimensional data

[Matti Picus]

On 20/2/24 01:24, phili...@loco-labs.io wrote:


Hi community,

This memo is a proposal to implement a compact and reversible (lossless 
round-trip) JSON interface for multi-dimensional data and in particular for 
Numpy (see issue #12481). The links to the documents are at the end of the memo.

The JSON-NTV (Named and Typed value) format is a JSON format which integrates a 
notion of type. This format has also been implemented for tabular data (see 
NTV-pandas package available in the pandas ecosystem and the PDEP12 
specification). .

The use of this format has the following advantages:
- Taking into account data types not known to Numpy,
- Reversible format (lossless round-trip)
- Interoperability with other tools for tabular or multi-dimensional data (e.g. 
pandas, Xarray)
- Ease of sharing Json format
- Binary coding possible (e.g. CBOR format)
- Format integrating data of different nature

The associated Jupyter Notebook presents some key points of this proposal 
(first draft):

Summary:
   - introduction
   - benefits
   - multi-dimensionnal data
   - Multi-dimensional types
   - Format JSON
   - Using the NTV format
   - Equivalence of tabular format and multidimensional format
   - Astropy specific points
   - Units and quantities
   - Coordinates
   - Tables
   - Other structures

This subject seems important to me (in particular for interoperability issues) 
and I would like to have your feedback before working on the implementation.
Especially,
- do you think this “semantic” format is interesting to use?
- do you have any particular expectations or subjects that I need to study 
beforehand?
- do you have any examples or test cases to offer me?
And of course, any type of remark and comment is welcome.

Thanks in advance !

links:
- Jupyter notebook : 
https://nbviewer.org/github/loco-philippe/Environmental-Sensing/blob/main/python/Tests/numpy_tests.ipynb
- JSON-NTV format : https://www.ietf.org/archive/id/draft-thomy-json-ntv-02.html
- JSON-NTV overview : 
https://nbviewer.org/github/loco-philippe/NTV/blob/main/example/example_ntv.ipynb
- NTV tabular format : 
https://www.ietf.org/archive/id/draft-thomy-ntv-tab-00.html#name-tabular-structure
- NTV-pandas package : 
https://github.com/loco-philippe/ntv-pandas/blob/main/README.md
- NTV-pandas examples : 
https://nbviewer.org/github/loco-philippe/ntv-pandas/blob/main/example/example_ntv_pandas.ipynb
- Pandas specification - PDEP12 : 
https://pandas.pydata.org/pdeps/0012-compact-and-reversible-JSON-interface.html


There is an open issue [1] about such a format, is this the same or 
different?



We discussed this at the latest triage meeting. While interoperability 
is one of NumPy's goals, and something we care deeply about, we were not 
sure how this initiative will play out. Perhaps, like the Pandas 
package, it should live outside NumPy for a while until some wider 
consensus could emerge. We did have a few questions about the standard:


- How does it handle sharing data? NumPy can handle very large ndarrays, 
and a read-only container with a shared memory location, like in DLPack 
[0] seems more natural than a format that precludes sharing data.


- Is there a size limitation either on the data or on the number of 
dimensions? Could this format represent, for instance, data with more 
than 100 dimensions, which could not be mapped back to NumPy.



Matti


[0] https://dmlc.github.io/dlpack/latest/

[1] https://github.com/numpy/numpy/issues/12481

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: How does Numpy model non-float arrays?

[Matti Picus]

On 22/8/23 02:25, Dylon Edwards wrote:


It is my understanding that Numpy accelerates array operations with BLAS where 
possible, but BLAS does not support all the dtypes that Numpy does. How does 
Numpy model non-float arrays like arrays of dtype=bool or dtype=object?


Numpy only uses BLAS where it can. Where it cannot it doesn't use BLAS. 
Is there a particular operation you are interested in?


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: mixed mode arithmetic

[Matti Picus]

On 10/7/23 16:13, Jens Glaser via NumPy-Discussion wrote:

Hi Matti,

The documentation for numpy.dot currently states

"""
out
ndarray, optional
Output argument. This must have the exact kind that would be returned if it was 
not used. In particular, it must have the right type, must be C-contiguous, and 
its dtype must be the dtype that would be returned for dot(a,b). This is a 
performance feature. Therefore, if these conditions are not met, an exception 
is raised, instead of attempting to be flexible.
"""

I think this means that if dot(a,b) returned FP32 for FP16 inputs, it would be 
consistent with this API to supply a full precision output array. All that 
would be needed in an actual implementation is a mixed_precision flag (or 
output_dtype option) for this op to override the usual type promotion rules. Do 
you agree?

Jens
___



`np.dot` is strange. Could you use `np.matmul` instead, which is a real 
ufunc and (I think) already does this?


Matti.

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: mixed mode arithmetic

[Matti Picus]

On 9/7/23 23:34, glaserj--- via NumPy-Discussion wrote:

Reviving this old thread - I note that numpy.dot supports in-place 
computation for performance reasons like this

c = np.empty_like(a, order='C')
np.dot(a, b, out=c)

However, the data type of the pre-allocated c array must match the result 
datatype of a times b. Now, with some accelerator hardware (i.e. tensor cores 
or matrix multiplication engines in GPUs), mixed precision arithmetics with 
relaxed floating point precision (i.e.., which are not necessarily IEEE754 
conformant) but with faster performance are possible, which could be supported 
in downstream libraries such as cupy.

Case in point, a mixed precision calculation may take half precision inputs, 
but accumulate in and return full precision outputs. Due to the above mentioned 
type consistency, the outputs would be unnecessarily demoted (truncated) to 
half precision again. The current API of numpy does not expose mixed precision 
concepts. Therefore, it would be nice if it was possible to build in support 
for hardware accelerated linear algebra, even if that may not be available on 
the standard (CPU) platforms numpy is typically compiled for.

I'd be happy to flesh out some API concepts, but would be curious to first get 
an opinion from others. It may be necessary to weigh the complexity of adding 
such support explicitly against providing minimal hooks for add-on libraries in 
the style of JMP (for jax.numpy), or AMP (for torch).

Jens


If your goal is "accumulate in and return full precision outputs", then 
you can allocate C as the full precision type, and NumPy should do the 
right thing. Note it may convert the entire input array to the final 
dtype, rather than doing it "on the fly" which could be expensive in 
terms of memory.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New user dtypes and the buffer protocol

[Matti Picus]

On 6/7/23 20:44, Evgeni Burovski wrote:


On Thu, Jul 6, 2023 at 7:56 PM Nathan  wrote:
>
> Hi all,
>
> As you may know, I'm currently working on a variable-width string 
dtype using the new experimental user dtype API. As part of this work 
I'm running into papercuts that future dtype authors will likely hit 
and I've been trying to fix them as I go.

>
> One issue I'd like to raise with the list is that the Python buffer 
protocol and the `__array_interface__` protocol support a limited set 
of data types.

>
> This leads to three concrete issues I'm working around:
>
>    * The `npy` file format uses the type strings defined by the 
`__array_interface__` protocol, so any type that doesn't have a type 
string defined in that protocol cannot currently be saved [1].

>
>     * Cython uses the buffer protocol in its support for numpy 
arrays and in the typed memoryview interface so that means any array 
with a dtype that doesn't support the buffer protocol cannot be 
accessed using idiomatic cython code [2]. The same issue means cython 
can't easily support float16 or datetime dtypes [3].

>
>     * Currently new dtypes don't have a way to export a string 
version of themselves that numpy can subsequently load (implicitly 
importing the dtype). This makes it more awkward to update downstream 
libraries that currently treat dtypes as strings.

>
> One way to fix this is to define an ad-hoc extension to the buffer 
protocol. Officially, the buffer protocol only supports the format 
codes used in the struct module [4]. Unofficially, memoryview doesn't 
raise a NotImplementedError if you pass it an invalid format code, 
only raising an error when it tries to access the data. This means we 
can stuff an arbitrary string into the format code. See the proposal 
from Sebastian on the Python Discuss forum [5] and his 
proof-of-concept [6]. The hardest issue with this approach is that 
it's a social problem, requiring cross-project coordination with at 
least Cython, and possibly a PEP to standardize whatever extension to 
the buffer protocol we come up with.

>
> Another option would be to exchange data using the arrow data format 
[7], which already supports many of the kinds of memory layouts custom 
dtype authors might want to use and supports defining custom data 
types [8]. The big issue here is that NumPy probably can't depend on 
the arrow C++ library (I think?) so we would need to write a bunch of 
code to support arrow data layouts and data types, but then we would 
also need to do the same thing on the Cython side.

>
> Implementing either of these approaches fixes the issues I 
enumerated above at the cost of some added complexity. We don't 
necessarily have to make an immediate decision for my work to be 
viable, I can work around most of these issues, but I think now is 
probably the time to raise this as an issue and see if anyone has 
strong opinions about what NumPy should ultimately do.

>
> I've raised this on the Cython mailing list to get their take as 
well [9].

>
> [1] https://github.com/numpy/numpy/issues/24110
> [2] https://github.com/numpy/numpy/issues/18442
> [3] https://github.com/numpy/numpy/issues/4983
> [4] https://docs.python.org/3/library/struct.html#format-strings
> [5] 
https://discuss.python.org/t/buffer-protocol-and-arbitrary-data-types/26256

> [6] https://github.com/numpy/numpy/issues/23500#issuecomment-1525103546
> [7] https://arrow.apache.org/docs/format/Columnar.html
> [8] https://arrow.apache.org/docs/format/Columnar.html#extension-types
> [9] https://mail.python.org/pipermail/cython-devel/2023-July/005434.html

I wonder if the dlpack protocol can be helpful for these kinds of dtypes?



No. DLPack has an enum for a fixed number of known dtypes [0], and 
adding new ones is non-trivial.


[0] 
https://github.com/dmlc/dlpack/blob/ca4d00ad3e2e0f410eeab3264d21b8a39397f362/include/dlpack/dlpack.h#L158


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matti Picus]

  
  
In PR 23925 the tests for allowed ULP error for sin and cos were
  tightened for float32 and float64 to 1.49 and 1.00 respectively.
  The use of SVML/SIMD routines for sin/cos on float64 was disabled
  across the board, and the checks all pass. People using the latest
  HEAD version should be able to try out the code. People using the
  wheels from
  https://anaconda.org/scientific-python-nightly-wheels/numpy will
  have to wait a few days for a new upload. Please try it out and
  let us know if the slowdown is noticeable, and if the
  accuracy/precision is now as expected.
Matti
  

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Numpy with eigen c++ binding

[Matti Picus]

On 6/6/23 06:46, darshan patel wrote:


it seems like numpy is moving toward c++ implementation, so is there any plan 
to have eigen c++ library also inline with numpy to get better performance?
is there any ongoing work happening around this?


NumPy does not currently use eigen and I am not aware of any plans to do so.

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: mean_std function returning both mean and std

[Matti Picus]

On 2/6/23 13:41, Matti Picus wrote:

On 2/6/23 13:09, Ronald van Elburg wrote:

Mean_var, mean_std and tests are now ready. 
(https://github.com/soundappraisal/numpy/tree/stdmean-dev-001)


Some decisions made during implementation:
   - the output shape of mean follows the output shape of the 
variance or the standard deviation. So it responds in the same way to 
the keepdims flag as the variance and the standard deviation.
   - the resizing of the mean is placed in _mean_var the overhead on 
the old functions std and var is minimal as they set mean_out to None.
   - the intermediate mean used can not be replaced with the mean 
produced by _mean as the output of the latter can not be broadcast to 
the incoming data.

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com



For a previous discussion of a performant solution, see 
https://github.com/numpy/numpy/issues/13199. That issue is more about 
var but also touches on a paper that has a two-pass solution for 
calculating mean and var together


Matti

Ahh, I see that issue is mentioned in your issue 
https://github.com/numpy/numpy/issues/23741


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: mean_std function returning both mean and std

[Matti Picus]

On 2/6/23 13:09, Ronald van Elburg wrote:


Mean_var, mean_std and tests are now ready. 
(https://github.com/soundappraisal/numpy/tree/stdmean-dev-001)

Some decisions made during implementation:
   - the output shape of mean follows the output shape of the variance or the 
standard deviation. So it responds in the same way to the keepdims flag as the 
variance and the standard deviation.
   - the resizing of the mean is placed in _mean_var the overhead on the old 
functions std and var is minimal as they set mean_out to None.
   - the intermediate mean used can not be replaced with the mean produced by 
_mean as the output of the latter can not be broadcast to the incoming data.
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com



For a previous discussion of a performant solution, see 
https://github.com/numpy/numpy/issues/13199. That issue is more about 
var but also touches on a paper that has a two-pass solution for 
calculating mean and var together


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matti Picus]

On 31/5/23 14:12, Matthew Brett wrote:


The ideal would be a run-time toggle for people to experiment with,
with binary wheels.  Is that practical?

Cheers,

Matthew



There is a discussion about a runtime context variable/manager that 
would extend errorstate to have a precision flag as well in 
https://github.com/numpy/numpy/issues/23362.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Precision changes to sin/cos in the next release?

[Matti Picus]

On 31/5/23 09:33, Jerome Kieffer wrote:

Hi Sebastian,

I had a quick look at the PR and it looks like you re-implemented the sin-cos
function using SIMD.
I wonder how it compares with SLEEF (header only library,
CPU-architecture agnostic SIMD implementation of transcendental
functions with precision validation). SLEEF is close to the Intel SVML
library in spirit  but extended to multi-architecture (tested on PowerPC
and ARM for example).
This is just curiosity ...

Like Juan, I am afraid of this change since my code, which depends on
numpy for sin/cos used for rotation is likely to see large change of
behavior.

Cheers,

Jerome



I think we should revert the changes. They have proved to be disruptive, 
and I am not sure the improvement is worth the cost.


The reversion should add  a test that cements the current user expectations.

The path forward is a different discussion, but for the 1.25 release I 
think we should revert.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Google Season of Docs 2023 Project Roadmap

[Matti Picus]

On 27/4/23 23:40, Mars Lee wrote:


Hi everyone,
I'm Mars, the selected technical writer for NumPy’s Google Summer of Docs 
project proposal, the ‘NumPy Contributor Comics'.

The project roadmap and timeline can be read here: 
https://medium.com/@marsbarlee/gsod-numpy-contributor-comics-project-roadmap-521280503fbd.
 In a few weeks, I'll host short brainstorming sessions to gather community 
input during the NumPy Community and Docs meeting. It will also be open for 
review soon.

Looking forward to shaping and sharing these comics with everyone!
Best,
Mars



Thanks Mars for taking this on, and congratulations on being selected.

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: How to fix this problem? np.arange(0,20.01,0.01)

[Matti Picus]

On 3/4/23 18:11, sideri...@gmail.com wrote:


The last element of "np.arange(0,20.01,0.01)" is 20.01 in my python result.

Is this numpy error ?
___



This is due to inaccurate float value representation. The documentation 
for arange states:


"When using a non-integer step, such as 0.1, it is often better to use 
|numpy.linspace| 
.


See the Warning sections below for more information."


Please read the warning section

https://numpy.org/devdocs/reference/generated/numpy.arange.html#numpy-arange.


Matti


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Congratulation to our newest maintainer Mukulika Pahari

[Matti Picus]

The NumPy steering council recently granted maintainer rights to 
Mukulika Pahari (https://github.com/Mukulikaa/)



Mukulika was recently elected to co-lead the numpy documentation team, 
and has been active in reviewing documentation issues and PRs. Welcome 
and thanks for all you do.

Matti
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: CI: testing of musllinux

[Matti Picus]

On 21/12/22 06:41, Andrew Nelson wrote:


On Wed, 21 Dec 2022 at 07:29, Andrew Nelson  wrote:

In https://github.com/scipy/scipy/issues/17630 we're seeing some
issues (amongst others) with the scipy test suite that uses
numpy==1.24.0 on musllinux.

I was wondering if numpy would like to add a musllinux CI entry?
This would help prepare for a musllinux wheel.



From local testing in a docker container there are currently several 
test fails and at least one segfault on numpy main with 
`musllinux_x86_64`.


Maybe we should have a scientific-python wide discussion of what 
platforms we wish to support, like NEP 29 for python versions. The NEP 
should include some mechanism for adding new platforms. It doesn't make 
sense to me to be testing something on SciPy's CI if we are not testing 
it in NumPy.



As for musllinux: if we decided to go ahead I think we should proceed 
cautiously with CirrusCI. We are still learning its limitations. Maybe 
we should add the CI run to github actions instead?



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: ANN: NumPy Fellowship Program & Sayed Adel as our first Developer in Residence

[matti . picus]

This is an important milestone and a great initiative. I am looking forward to 
working with Sayed.
Matti
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Expanding support in testing for external collections.Sequence objects

[Matti Picus]

On 30/11/22 05:47, Stefan van der Walt wrote:

On Tue, Nov 29, 2022, at 07:21, i...@markopacak.com wrote:

The debate is whether np.testing.asset_equal should support
collections.Sequence objects.

assert list(sequence1) == list(sequence2)

should do the trick, and also handles int vs float and nan comparisons.


What is the general view on adding support in testing for Sequence objects?

If it's a straightforward change that doesn't impact the NumPy test suite, it's 
probably fine. Let's see what the others say.

Stéfan



Apparently nothing is really straightforward. On the issue there is 
discussion of problems this causes when comparing strings (and how the 
naive implementation breaks lots of tests), and this comment [0] from a 
Pandas maintainer


> If you do go down this path, take a look at 
pandas._libs.lib.is_list_like. I've found it to be a minor PITA to 
accommodate every corner case that users ask to consider listlike



Matti

[0] https://github.com/numpy/numpy/issues/22569#issuecomment-1331586018

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Add a new CI provider for aarch64, macos arm64, musl

[Matti Picus]

I am writing to the list for visibility: I opened an issue [0] about 
adding the Cirrus CI provider to take over some of the CI task runners 
from travis.com. We have been experiencing strange timeouts on the 
travis runs, and in general the platform does not feel stable. SciPy 
made the transition about two months ago and seem to be satisfied.



Thoughts?

Matti


[0] https://github.com/numpy/numpy/issues/22640

[1] https://github.com/numpy/numpy/pull/22636#issuecomment-1321682645

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Introducing Arm Optimized Routines

[Matti Picus]

On 8/11/22 19:45, Chris Sidebottom wrote:

Hi Matti,

Thanks for your questions :-)


This seems like it would improve performance on aarch64. Would the routines 
also work with the Apple silicon?

Yip, I can't see a reason why that wouldn't be the case.


If these are new routines, it would be better to implement them in terms of the 
numpy universal intrinsics rather than adding a new submodule.

These would be the same routines as seen in SVML (integrated here: 
https://github.com/numpy/numpy/blob/main/numpy/core/src/umath/loops_umath_fp.dispatch.c.src#L67),
 which use the universal intrinsics before using the SVML library, the actual 
surface area is minimal so I'd propose we follow a similar path with our 
existing routines and then aim to apply universal intrinsics if that's possible 
in the future - does that sound like a good approach?

Cheers,
Chris



Yes, if the routines already exist then it would seem an additional 
submodule of code would be the best path forward, as long as the license 
is compatible.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Introducing Arm Optimized Routines

[Matti Picus]

Thanks, this seems like it would improve performance on aarch64. Would 
the routines also work with the Apple silicon arm64?


If these are new routines, it would be better to implement them in terms 
of the numpy universal intrinsics rather than adding a new submodule.


Matti



On 8/11/22 13:30, Chris Sidebottom wrote:


Hello,

Here at Arm, we've been investigating how we can improve performance 
on AArch64. One way in which we can improve performance is by 
integrating some existing optimized routines 
(https://github.com/ARM-software/optimized-routines), similar to the 
SVML methods for AVX512 that are currently included as a git 
submodule. Our intent is to include the optimized routines repository 
as an additional submodule which we can then use to provide routines 
on AArch64 for ASIMD, SVE and beyond.


Currently, we're targeting 4-ULP as this aligns with libmvec 
(https://sourceware.org/glibc/wiki/libmvec) and the SVML integration 
(https://github.com/numpy/numpy/pull/19478). This is alongside adding 
sufficient error handling to pass the Numpy test suite, meeting the 
test requirements highlighted in the SVML integration 
(https://github.com/numpy/numpy/pull/19478#issuecomment-893001722).


We've already started curating the necessary functions, let us know if 
you have any feedback.


Cheers,

Chris

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended 
recipient, please notify the sender immediately and do not disclose 
the contents to any other person, use it for any purpose, or store or 
copy the information in any medium. Thank you.


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: PR hasn't been responded to in 2 weeks

[Matti Picus]

The idea of using fft in polynomial.polymul in the PR got a cool 
reception. It is not clear if we should add this to NumPy or refer users 
to other packages.  Maybe the mailing list would be a forum to discuss 
the need for the enhancement. It should have at least one other champion 
in order to move forward.



The reviewers also asked you to consider a different interface 
(method=), I did not see a response to that suggestion.


Matti


On 28/9/22 08:23, matteo_luigi_raso wrote:
I made a PR (#22255) that received some attention, but has been 
ignored for 2 weeks now. I commented on the PR 5 days ago, but there 
was still no response.


Sent with Proton Mail  secure email.

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Enhancement for AArch64 SVE instruction set

[matti picus]

It seems cirrus-ci offers AWS EKS Graviton2 instances [0] and this is
free for open source projects. Do you know if that offering has
SVE-enabled CPUs?

Matti
[0] https://cirrus-ci.org/guide/linux/

On Fri, Sep 16, 2022 at 5:54 AM kawakam...@fujitsu.com
 wrote:
>
> Hi,
>
> It's been a long time since I first contacted here,
> but I submitted my pull request about handling Arm64 SVE architecture 
> yesterday.
> https://github.com/numpy/numpy/pull/22265
>
> Since there may be no public CI environment that runs the SVE instruction set,
> I tested my source code on an inhouse server (Fujitsu FX700 with A64FX).
> - A64FX is one of the Armv8.2-a + SVE compliant CPUs.
> I think the regression test was successfully completed.
> - python3 runtests.py --cpu-baseline=sve --cpu-dispatch=none
> The result shows "21354 passed, 203 skipped, 1302 deselected, 30 xfailed, 7 
> xpassed".
>
> My implementation is similar to those of AVX/AVX2/ASIMD.
> - SVE intrinsics are defined in numpy/core/src/common/simd/sve/*.h files.
>
> Travis CI reported errors
> https://github.com/numpy/numpy/pull/22265/checks?check_run_id=8384699529
> , but it seems that the job exceeded the maximum log length, and has been 
> terminated.
>
> I would appreciate your review of my pull request, as well as your comments 
> and advice
> on this mailing list.
>
>
> Thanks,
> Kentaro
>
>
>
>
>
>
>
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Representation of NumPy scalars

[Matti Picus]

On 9/9/22 04:15, Warren Weckesser wrote:

...
To quote from https://docs.python.org/3/library/functions.html#repr:


For many types, this function makes an attempt to return a string
that would yield an object with the same value when passed to eval();

Sebastian, is this an explicit goal of the change?  (Personally, I've
gotten used to not taking this too seriously, but my world view is
biased by the long-term use of NumPy, which has never followed this
guideline.)

If that is a goal, than the floating point types with precision
greater than double precision will need to display the argument of the
type as a string.  For example, the following is run on a platform
where numpy.longdouble is extended precision (80 bits):

```
In [161]: longpi = np.longdouble('3.14159265358979323846')

In [162]: longpi
Out[162]: 3.1415926535897932385

In [163]: np.longdouble(3.1415926535897932385)  # Argument is parsed
as 64 bit float
Out[163]: 3.141592653589793116

In [164]: np.longdouble('3.1415926535897932385')  # Correctly
reproduces the longdouble
Out[164]: 3.1415926535897932385
```

Warren



As others have mentioned, the change will greatly enhance UX at the cost 
of documentation cleanups. While the representation may not be perfectly 
roundtrip-able, I think it still is an improvement and worthwhile. 
Elsewhere I have suggested we need more documentation around 
array/scalar printing, perhaps that would be a place to mention the 
limitations of string representations.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: An extension of the .npy file format

[Matti Picus]

Sorry for the late reply. Adding a new "*.npy" format feature to allow 
writing to the file in chunks is nice but seems a bit limited. As I 
understand the proposal, reading the file back can only be done in the 
chunks that were originally written. I think other libraries like zar or 
h5py have solved this problem in a more flexible way. Is there a reason 
you cannot use a third-party library to solve this? I would think if you 
have an array too large to write in one chunk you will need third-party 
support to process it anyway.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Support for Multiple Interpreters (Subinterpreters) in numpy

[Matti Picus]

On 23/8/22 03:16, Matti Picus wrote:


...

One thing CPython could do is to provide clear documentation how to 
port a small c-extension module [1]



Matti
[1] https://github.com/python/cpython/issues/79601



I should have searched the documentation, there is now a quite extensive 
guide [2] including all the different interfaces provided for getting 
per-interpreter module state.



Matti


[2] https://docs.python.org/3.11/howto/isolating-extensions.html

[3] https://docs.python.org/3.11/c-api/type.html#c.PyType_GetModuleState

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Support for Multiple Interpreters (Subinterpreters) in numpy

[Matti Picus]

On 22/8/22 18:59, Eric Snow wrote:

Hi all,

CPython has supported multiple interpreters (in the same process) for
a long time, but only through the C-API.  I'm working on exposing that
functionality to Python code (see PEP 554), aiming for 3.12.  I expect
that users will find the feature useful (particularly with a
per-interpreter GIL--see PEP 684) and that it will be used a lot more
over the coming years.  This has the potential to impact extension
module projects, especially large ones like numpy, which is why I'm
reaching out to you.

Use of multiple interpreters depends on isolation between them.  When
an extension module is imported in multiple interpreters, it is loaded
separately into a new module object in each.  Extensions often store
module data/state in C globals, which means the the multiple instances
end up sharing data.  This causes problems, more so once we have one
GIL per interpreter.

Over the years we have added machinery to help extensions get the
necessary isolation, moving away from global variables.  This includes
PEPs 384, 3121, and 489.  This has culminated in the guide you can
find in PEP 630.

Note that nothing should change when only a single interpreter is in
use (basically the status quo).  With PEP 684, importing an
incompatible extension outside the main (initial) interpreter will now
be an ImportError.  (Currently the behavior is undefined and too often
results in hard-to-debug failures and crashes.)

Thus extension module maintainers do have the option to *not* support
multiple interpreters.  Unfortunately, that doesn't mean their users
won't pester them about adding support.  We all recognize how that
dynamic can be draining on a project.  The potential burden on
maintainers is a serious factor for these upcoming changes.  numpy is
likely to be affected more than any other project.  That's why I'm
starting this thread.

PEP 684 discusses all of the above.  What I'm after with this thread is:

* to make sure the numpy maintainers are clear on what interpreter
isolation requires of the project
* a clear picture of what changes numpy would need (and how much work
that would be)
* feedback on what the CPython team can do to minimize that work
(incl. adding new C APIs)

I'm fine with having the discussion here, but I will probably create a
new category on discuss.python.org for a variety of similar threads
related to multiple interpreters and supporting them.  Having our
discussion there may lead to more participation from more CPython core
devs than just me.  Do you have any preference for or against any
particular venue?

Thanks!

-eric
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com


Thanks for starting the conversation. I would personally prefer the 
discussion about NumPy be here, general discussions could be elsewhere.



Please correct me if I am wrong: I understand that multiple interpreters 
would require us to (at least):


- refactor all the static module global state in NumPy and make it 
re-entrant or immortal including converting stack-allocated 
PyTypeObjects to heap types.


- find a mechanism to access the per-interpreter module state

- carefully consider places in the code that we steal references either 
intentionally or because that is the CPython C-API we are using


- measure the performance implications of the necessary changes

- plan forward/backward compatibility


This seems like a significant undertaking, and is why we have rejected 
casual calls for supporting multiple interpreters in the past [2], [3], 
[4]. Supporting multiple interpreters is currently not on the NumPy 
roadmap [0]. Priorities can be changed, through dialog with the NumPy 
community, and others can propose changes to NumPy via NEPs, PRs, and 
issues, but we are unlikely to engage directly in the work if it is not 
an agreed upon goal. There are other initiatives around NumPy that may 
dovetail with multiple interpreters. For instance the HPy group hit many 
of the issues above when creating a  port of NumPy [5]. It would be good 
to get like-minded people talking about this and to pool resources, 
maybe someone on this list has a strong opinion and would be willing to 
put in some work on the subject.



One thing CPython could do is to provide clear documentation how to port 
a small c-extension module [1]



Matti


[0] https://numpy.org/neps/roadmap.html

[1] https://github.com/python/cpython/issues/79601

[2] https://github.com/numpy/numpy/issues/665

[3] https://github.com/numpy/numpy/issues/14384

[4] https://github.com/numpy/numpy/issues/16963

[5] https://github.com/hpyproject/numpy-hpy/tree/graal-team/hpy#readme

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe 

[Numpy-discussion] Re: Display runtime information through a new function: `show_runtime`

[Matti Picus]

  
  
The PR to implement this was merged
Matti

  

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: I need help on the issue of replacing the original C math library with an external C math library

[matti picus]

Sorry for the late reply, this ended up in my SPAM filter. I think
google does not like your qq.com domain.

The best way to manage blas/lapack implementations right now is via
the site.cfg file, the example site.cfg.example has extensive
instructions. If it is not working as expected, please share what you
have so far so we can help you, it may be a simple typo.
Matti

On Tue, Aug 2, 2022 at 3:42 PM <454626...@qq.com> wrote:
>
> Hello developers,
> The external math library contains some dynamic link libraries of C language 
> (.so files) and some C header files (.h files), I am tring to link my custom 
> library to NumPy and replace the original bottom math library. I need to add 
> several compile options to the C compiler (gcc etc.), but I have not found 
> the corrosponding part of NumPy source code.
> Merely modifying 'site.cfg' does not work at all, and then adding lib and 
> include directory to '~/numpy/distutils/system_info.py' still does not work. 
> When compiling NumPy source code, it gives a fatal error: myBLAS.h: No such 
> file or directory.
> My doubts: Which code file should I focus on? How to modify configurations 
> and add compile options like '-I /path/to/includes -L /path/to/libs -lmyBLAS'?
> Could you please give me a hint? Looking forward to your reply.
> Thanks!
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Additional aliases for MaskedArray

[Matti Picus]

PR 21998 [0] adds methods "ones_like", "zeros_like", "dsplit","split", 
and "vsplit" for MaskedArrays. The motivation was to clean up broken 
documentation links, and it seemed logical to add the interfaces. As far 
as I can tell we do not have users asking for these methods. Should we 
add them (expanding the np.ma namespace) or find another way to fix the 
broken documentation links?


Matti


[0] https://github.com/numpy/numpy/pull/21998

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Display runtime information through a new function: `show_runtime`

[Matti Picus]

My assumption in advising Ganesh to create a new function was that there 
may be some people who find show_config useful. Personally I do not find 
it useful. If there is agreement that what matters is runtime and not 
build-time then we should replace the current show_config. Otherwise we 
should have two functions, since the lengthy output of build-time values 
will confuse people looking for only runtime values.


Matti


On 7/8/22 07:08, Stefan van der Walt wrote:

Hi Ganesh,

If the aim is to merge these functions (show_config/runtime) 
eventually, why not merge them now and avoid introducing another 
function into the root namespace?


Stéfan

On Sat, Aug 6, 2022, at 20:55, Ganesh Kathiresan wrote:

Hi Team,

We are planning on adding a new function called `np.show_runtime` to 
display the runtime information of the machine in addition to 
`np.show_config` which displays the build-related information.
Relevant PR: ENH: Use threadpoolctl in show_runtime (a new function) 
#21468 


Reasons for new function:
1. Will be helpful in root-causing issuing with information from the 
acceleration frameworks.

2. Build time information alone is not always helpful [1]

Sample Output of new function
>>> np.show_runtime() [{'simd_extensions': {'baseline': ['SSE', 'SSE2', 
'SSE3'], 'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 
'F16C', 'FMA3', 'AVX2'], 'not_found': ['AVX512F', 'AVX512CD', 
'AVX512_KNL', 'AVX512_KNM', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 
'AVX512_ICL']}}, {'architecture': 'Zen', 'filepath': 
'/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so 
', 'internal_api': 'openblas', 
'num_threads': 12, 'prefix': 'libopenblas', 'threading_layer': 
'pthreads', 'user_api': 'blas', 'version': '0.3.20'}]

Under the hood
`np.show_runtime` will use `threadpoolctl` [2] to get the runtime 
information of the frameworks. In addition to this, we will also 
display useful SIMD extensions that are supported.


Future Scope
1. When we move towards meson, we can leverage the additional 
information and revamp `np.show_config` to utilize it.
2. If needed, we can gradually merge `show_runtime` and `show_config` 
to display information in sections and various other formats.


Please do let us know if you have any suggestions on improving this 
function or concerns about the information provided by the new function.


Thanks,
Ganesh

[1] show_config displays useless info when installed through 
conda-forge channel  
(#14492)

[2] threadpoolctl 


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: aarch64 wheels on github actions

[Matti Picus]

I personally prefer using the travis runs since they get tested, even 
with the additional maintenance burden of requesting travis credits when 
they run out. It is good to know there is another option.


Matti


On 7/8/22 09:15, Andrew Nelson wrote:
At the moment Linux aarch64 wheels are being made in a TravisCI job. 
I've been experimenting recently about making scipy wheels for that 
target on Github Actions. cibuildwheel + github actions does quite 
well for that purpose. It takes around 2 hours for scipy wheels to be 
created (but not tested). I would expect numpy wheels to take less 
time to build.


I think it would be a relatively minor modification of the 
.github/workflows/wheels script.


cheers,
Andrew.
--
_
Dr. Andrew Nelson


_

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: generate a random sample based on independent priors in `random.choice` #22082

[Matti Picus]

Every new feature, method, function or keyword argument adds cognitive 
load, maintainer burden, makes the package larger, and can confuse users 
so we reject much more than we accept.


The first step would be to present a convincing case _why_ Poisson 
sampling should be added to NumPy, before discussing the _how_.


Matti


On 4/8/22 21:11, Robert Kern wrote:
As discussed on the corresponding Github issue[1], this is not going 
to be functionality that we add onto `choice()` nor is it likely 
something that we will add as a separate `Generator` method. The 
desired computation is straightforward to do by composing existing 
functionality.


[1] https://github.com/numpy/numpy/issues/22082

On Thu, Aug 4, 2022 at 2:07 PM Rodo-Singh  
wrote:


Proposed new feature or change:
Objective: Sample elements from the given iterator (a list, a
numpy array, etc.) based upon pre-defined probabilities associated
with each element which may not sums upto 1.

Overview
* In the numpy.random.choice function the cases where we're
explicitly providing the list of probabilistic values for an input
sample, the requirement expects the sum of the whole list to be 1.
This makes sense when all the elements are possible observation
for a single random variable whose pmf(probability mass function)
is nothing but the p list.
* But when every element in that list can be regarded as
observation of separate independent Bernoulli r.v. (random
variable), then the problem falls into the category of
multi-label. Intuitively speaking, for each element in the list or
array given to us, we'll toss a coin (may be biased or unbiased)
~B(p) (i.e., follows Bernoulli with p as success probability for
head).
* The output array or list would probably be a proper subset or
can be a full list or can be an empty one.
* Plus, here the argument size should automatically get shut down
as we just want which all elements got selected into the output
list after n coin tossess (where len(list) = n). Also it may
happen that each element is independent as argued above, but the
sum(p) = 1. Then we can probably put an extra argument
independence = True or independence = False.
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: robert.k...@gmail.com



--
Robert Kern

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Need help extending a NumPy array in C

[Matti Picus]

On 8/7/22 01:37, lpsm...@uw.edu wrote:


Hello!  I am maintaining a C++ codebase with extensive ties to Python bindings 
(via SWIG).  One of the features of the code is that we define (in C) a 
subclass of a NumPy Array.  Everything worked until we started getting this 
message with numpy 1.23:

RuntimeError: Object of type  appears to be C 
subclassed NumPy array, void scalar, or allocated in a non-standard way.NumPy 
reserves the right to change the size of these structures. Projects are required to 
take this into account by either recompiling against a specific NumPy version or 
padding the struct and enforcing a maximum NumPy version.

My problem is that I don't know how do to either of those things.  I would have 
assumed that whatever I compiled against, it would always be compiled against a 
specific NumPy version, and I also assumed that 'enforcing a maximum NumPy 
version' would happen in requirements.txt for the Python package, but that also 
seems to not be the case.  Any hints?  Thank you!

-Lucian Smith



Did you rerun SWIG to regenerate the bindings before the 1.23 build?

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: assert_array_equal scalar broadcast behaviour

[Matti Picus]

On 16/6/22 19:28, Jon Morris wrote:


Hello all,

I was recently tripped up by issue #9542 
, where a call to 
assert_array_equal unexpectedly passed because a single scalar can be 
declared equal to an array if every value in the array is the same. 
I’ve created pull #21595  
to address the problem – what does everyone think? Is this an 
acceptable solution or is there a better way to resolve this issue?


Many thanks,

Jon

Jon Morris

Software Developer



The proposed solution, to add a "strict" kwarg to assert_array_equal, 
looks good to me. Thanks!


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: numpy build question

[Matti Picus]

On 14/6/22 18:14, Jesse Peet wrote:
I am trying to build numpy 1.22.3 in container using a rhel8 host  and 
the container being rhel7.

My version of gcc is 4.8.5-44.   Here is the error I get

RuntimeError: Broken toolchain: cannot link a simple C++ program. 
note: A compiler with support for C++11 language features is required.




Jesse S. Peet
Configuration Manager Specialist/Engineer
Email: jesse.p...@raytheon.com
Cell: 719-322-4257



We require support for c++11. Please use a newer compiler or an older 
version of NumPy.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Njumpy-stl

[Matti Picus]

Hmm. Do you mean this project [0] on PyPI? It is developed by a team 
entirely separate from NumPy. The README should have an appropriate 
disclaimer. Frank you can find more information about the library in 
their documentation [1]


Matti

[0] https://pypi.org/project/numpy-stl/

[1]  https://numpy-stl.readthedocs.io/en/latest/

On 9/6/22 17:27, frank.underdown@quaise.energy wrote:

Hello,

I recently discovered numpy-stl.  I am looking for documentations and tutorials 
to learn how to use this library.
Also, if you know of any books where this library is discussed, I would 
appreciate you making a recommendation.

Thank you in advance for your help.

Cheers!

Frank
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Fuzzing integration of Numpy into OSS-Fuzz

[Matti Picus]

On 7/6/22 14:02, david korczynski wrote:

Hi Numpy maintainers,

Would you be interested in integrating continuous fuzzing by way of
OSS-Fuzz? Fuzzing is a way to automate test-case generation and has been
heavily used for memory unsafe languages. Recently efforts have been put
into fuzzing memory safe languages and Python is one of the languages
where it would be great to use fuzzing.

...

Let me know your thoughts on this and if you have any questions as I’m
happy to clarify or go more into details with fuzzing.

Kind regards,
David



Could you compare and contrast this to hypothesis [0], which we are 
already using in our testing?


I don't understand what you mean by "Python is one of the languages 
where it would be great to use fuzzing". Why?


Matti


[0] https://hypothesis.readthedocs.io/en/latest/index.html

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Numpy binary wheels and CI for win/arm64 platform

[Matti Picus]

When you say "we could request access to new Volterra machines": how do 
you see that playing out? Who would provide them, and who would maintain 
them? Would they be available to the more general scientific python 
community? Who would pay the bill? If the people supporting this niche 
platform stop doing so, can we drop the wheels? CPython has PEP 11 [0], 
maybe this effort should start by writing a similar NEP for the 
scientific python community.


Matti


[0] https://peps.python.org/pep-0011


On 6/6/22 13:35, Niyas Sait wrote:

Hello,

It has been a while since we discussed adding CI/CD support for 
Windows on Arm. Let me share a few updates and see if we can find a 
way forward.


First of all, Thanks for suggesting cibuildwheel. We have couple of 
patches in progress to add native [1] and cross-compilation support 
[2] to cibuildwheel. They could solve the binary wheel creation and 
releases. However, we may still lack support for CI.


I would like to check if there is any interest in using Windows/Arm64 
VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) 
[4] to add CI support for WoA platform.


In order to use Azure VMs with GitHub Actions or Azure Pipeline, we 
must set up self-hosted runners. I am not sure if that would be an 
acceptable workflow for the numpy community. If that could work we 
could potentially find ways to fund the required VMs for NumPy project.


Also, we could request access to new Volterra machines for Numpy 
CI/CD. I know it is not easy to maintain them as part of CI/CD 
workflow, but if any maintainers are interested in using them for 
CI/CD purposes, we can progress on that discussion.


Let me know what you think.

Thanks
Niyas

[1] https://github.com/pypa/cibuildwheel/pull/920
[2] https://github.com/pypa/cibuildwheel/pull/1108
[3] 
https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines-with-ampere-altra-armbased-processors/
[4] 
https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation-experiences-at-scale-with-windows/

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Matti Picus]

On 26/5/22 05:40, Aaron Meurer wrote:


We cannot do that (yet, at least). Failing to publish wheels for a supported 
Python version on a major OS is far worse than dropping support completely. 
This will remain true until the time that Pip starts defaulting to wheels-only 
and never picks the sdist if there's an older release for that platform + 
Python combo.

sounds like if even the latest version doesn't support a given CPython
version (and has CPython-versioned wheels), then pip installing it
will fail. Is that correct?



No. The problem is that if we do not provide binary wheels, pip 
installing Numpy will not fail. It will try to build from source. Even 
if this painfully and slowly succeeds, it means the user probably did 
not get the BLAS support they thought they were going to get. The 
situation for SciPy is similar but worse: it takes forever to build, and 
will probably likewise fail to build properly.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Dropping the pdf documentation.

[Matti Picus]

On 23/5/22 01:51, Rohit Goswami wrote:


Being very hard to read should not be reason enough to stop generating 
them. In places with little to no internet connectivity often the PDF 
documentation is invaluable.


I personally use the PDF documentation both on my phone and e-reader 
when I travel simply because it is more accessible and has better 
search capabilities.


It is true that SciPy has removed them, but that doesn't necessarily 
mean we need to follow suit. Especially relevant (IMO) is that large 
parts of the NumPy documentation still make sense when read 
sequentially (going back to when it was at some point partially kanged 
from Travis' book).


I'd be happy to spend time (and plan to) working on fixing concrete 
issues other than straw-man and subjective arguments.


Personally I'd like to see the NumPy documentation have PDFs in a 
fashion where each page / chapter can be downloaded individually.


-- Rohit

P.S.: If we have CI timeout issues, for the PDF docs we could also 
have a dedicated repo and only build for releases.


P.P.S: FWIW the Python docs are also still distributed in PDF form.

On 22 May 2022, at 21:41, Stephan Hoyer wrote:

+1 let’s drop the PDF docs. They are already very hard to read.

On Sun, May 22, 2022 at 1:06 PM Charles R Harris
 wrote:

Hi All,

This is a proposal to drop the generation of pdf documentation
and only generate the html version. This is a one way change
due to the difficulty maintaining/fixing the pdf versions. See
minimal discussion here
.

Chuck



Thanks Rohit for the offer to take on this project.

I don't think we should block the release on the existence of PDF 
documentation. It is a "nice to have", not a hard requirement.



One strategy to discover problems with the PDF builds in CI would be to 
add a weekly build of PDF.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New stable documentation

[matti picus]

On Sat, 21 May 2022 at 16:26, Charles R Harris 
wrote:

>
>
> On Sat, May 21, 2022 at 5:38 AM Sebastian Berg 
> wrote:
>
>> On Sat, 2022-05-21 at 10:33 +0300, matti picus wrote:
>> > On Sat, 21 May 2022 at 08:46, Ralf Gommers 
>> > wrote:
>> >
>> > >
>> > >
>> > > On Sat, May 21, 2022 at 3:40 AM Charles R Harris <
>> > > charlesr.har...@gmail.com> wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > I've put up new stable documentation for NumPy 1.22.4. I'd
>> > > > appreciate it
>> > > > if those familiar with how they want the documentation to look
>> > > > could take a
>> > > > look at it so that fixes can be made while I'm still in the
>> > > > documentation
>> > > > state.
>> > > >
>> > >
>> > > Thanks Chuck. It all looks fine to me.
>> > >
>> > > Cheers,
>> > > Ralf
>> > >
>> >
>> > I don’t see the version pull down, is that my browser?
>>
>> I was a bit surprised by that as well but I/we forgot to mark the PR as
>> backport candidate, so it slipped through:
>> https://github.com/numpy/numpy/pull/21426
>>
>> If backport+pushing docs again is very easy that could be nice?  But
>> otherwise we will have it on the "stable" docs with 1.23 anyway.
>>
>> Pamphile still has a styling PR open:
>> https://github.com/numpy/numpy/pull/21451
>>
>> But maybe it doesn't matter (compared to just having the pulldown).
>>
>>
> I can do that without a new release (I use Python 3.10 and modify the
> procedure and doc/Makefile a bit), so mark them for backport if you want
> new docs. But it is probably easier to wait for 1.23.
>
> Chuck
>

Then the 1.22 docs would not have the pull down.
Not the end of the world, but if it is easy to add it would be nice.
Matti

>
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New stable documentation

[matti picus]

On Sat, 21 May 2022 at 08:46, Ralf Gommers  wrote:

>
>
> On Sat, May 21, 2022 at 3:40 AM Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> I've put up new stable documentation for NumPy 1.22.4. I'd appreciate it
>> if those familiar with how they want the documentation to look could take a
>> look at it so that fixes can be made while I'm still in the documentation
>> state.
>>
>
> Thanks Chuck. It all looks fine to me.
>
> Cheers,
> Ralf
>

I don’t see the version pull down, is that my browser?
Matti

>
>
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Enhancement for AArch64 SVE instruction set

[Matti Picus]

On 19/5/22 16:50, kawakam...@fujitsu.com wrote:


Hello,

I am working on speeding up NumPy with the AArch64 SVE instruction set.
I could not find a numpy implementation for SVE. Is there already a test 
implementation or discussion about SVE support?


OpenBLAS apparently has SVE support from 0.3.20 [0], which was merged to 
NumPy yesterday.



So far we have only support for NEON and ASIMD in NumPy, see [1] for a 
description of the way we use intrinsics. Contributions to improve the 
implementations and the documentation would be welcome. We currently use 
travis CI to run our aarch64 tests, if there was a way to get access to 
more advanced machines that also would be good. Do you know of 
commercially available machines with SVE or SVE2 support?



Matti


[0] 
https://github.com/xianyi/OpenBLAS/blob/faf58d2b3ffb20fd334cab080700be564ef74ec2/Changelog.txt#L28


[1] https://numpy.org/devdocs/reference/simd/build-options.html



___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [Feature Request] Add alias of np.concatenate as np.concat

[Matti Picus]

  
  
The array API defines concat [0] so that is a powerful reason to
  implement an alias as part of bringing NumPy into compliance.


Matti


[0]
https://data-apis.org/array-api/latest/API_specification/manipulation_functions.html

  

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Automatic formatters for only changed lines

[Matti Picus]

On 6/5/22 09:09, Stefan van der Walt wrote:

Hi Matti,

On Thu, May 5, 2022, at 22:08, Matti Picus wrote:

This somehow runs when I do "git commit"? Do I need to add anything to
my git environment to pick up the hook? How do I turn it off if I find
it prevents progress or breaks? Do we need to add anything to the
documentation to warn developers about possible complications or errors?

The hook is optional, and is set up  by running pre-commit (which itself is a 
pip-installable python package):

...

Stéfan



I work under various virtualenvs and conda environments. So if I run 
"git commit" when the proper virtualenv is not activated, the pre-commit 
hook cannot run.


Will I get a notice that the hook has not run?

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Automatic formatters for only changed lines

[Matti Picus]

On 6/5/22 01:32, Stefan van der Walt wrote:

On Thu, May 5, 2022, at 12:00, Trevor Gross wrote:
I don't necessarily know that this should be enforced on CI off the 
bat (though in the future it could be), but at least having the tool 
available and easily usable would help to incrementally improve 
codebase style, and remove the need for formatting by hand.


On all projects we've set up pre-commit with style formatting, it has 
significantly reduced noise and nitpicks in PR discussions.  It makes 
for a better contributor experience, thus I am in favor.  Black isn't 
perfect by any means, but never having to talk about formatting again 
makes it worth it!


Stéfan



I (and maybe some others) am missing some context.


This somehow runs when I do "git commit"? Do I need to add anything to 
my git environment to pick up the hook? How do I turn it off if I find 
it prevents progress or breaks? Do we need to add anything to the 
documentation to warn developers about possible complications or errors?



If it runs locally before committing, what does "enforce on CI" mean and 
how is that different from the current linting CI job?



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NumPy 1.21.6 released

[Matti Picus]

On 12/4/22 19:46, Charles R Harris wrote:


Hi All,

On behalf of the NumPy team I am pleased to announce the release of 
NumPy 1.21.6. NumPy 1.21.6 is a very small release that achieves two 
things:


  * Backs out the mistaken backport of C++ code into 1.21.5.
  * Provides a 32 bit Windows wheel for Python 3.10.

The provision of the 32 bit wheel is intended to make life easier for 
oldest-supported-numpy.


The Python versions supported in this release are 3.7-3.10. If you 
want to compile your own version using gcc-11 you will need to use 
gcc-11.2+ to avoid problems. Wheels can be downloaded from PyPI 
; source archives, release 
notes, and wheel hashes are available on Github 
. Linux users 
will need pip >= 0.19.3 in order to install manylinux2010 and 
manylinux2014 wheels. A recent version of pip is needed to install the 
universal2 macos wheels.


Cheers,

Charles Harris


Thanks!

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Unpacking 0-D object arrays when setting item/filling

[Matti Picus]

On 15/2/22 09:53, Stefan van der Walt wrote:

On Mon, Feb 14, 2022, at 12:45, Sebastian Berg wrote:

But this is currently *not* consistently the case.  I wish to make this
consistent.  The confusion is around object arrays, though:

 value = np.array(None, dtype=object)
 arr[0] = value

Stores `value` without unpacking it currently.

 arr.fill(value)

Stores the `None` (unpacking `value`) if and only if `value` is 0-D.

That last behavior doesn't look right to me.  An object array should be thought 
of as a collection of pointers, and if you happen to want to point to a NumPy 
array, so be it.
I think we should strive for consistency and code simplicity. In the 
non-object case, it is clear that assignment will try to unpack an 
ndarray. So we should do the same thing with object arrays, and document 
the change in behaviour. Could we suggest a backward compatible 
alternative (would using a record dtype fit better with Stefan's mental 
model)?


Matti


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NumPy 1.22.1 has been released.

[Matti Picus]

On 18/1/22 5:37 pm, Peter Cock wrote:


Dear Charles,

Thank you for your work on the numpy releases, including v1.22.1.

I noticed that for Windows Python 3.10, there is only a 64-bit wheel:


...

Is this expected? My interest is in building Windows Wheels for 
packages compiling against NumPy.


Thank you,

Peter




Hi Peter. This is intentional, as our wheel building system is based on 
Azure CI, and they did not provide 32-bit python for 3.10 for windows. 
We decided to "solve" this and other wheel building issues by moving to 
cibuildwheel in github actions in the main repo (building on travis-ci 
for the aarch64 wheels) [1] for the 1.23 release. Digging around in the 
azure image repo issues, I see they may have restored 32-bit python [2] 
in mid Nov 2021, when we were already well into the 1.22 release cycle. 
I guess we could try to see if it works now. A PR to modify the azure 
pipelines yml would move us in this direction. I don't know that even if 
the PR is successful if we would put the wheels up on PyPI, that 
decision is up to the release manager.


Matti


[1] https://github.com/numpy/numpy/issues/20277

[2] https://github.com/actions/virtual-environments/issues/4400

[3] https://github.com/MacPython/numpy-wheels/blob/main/azure-pipelines.yml


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: MacPython travis funds have run out (again)

[Matti Picus]

On 14/1/22 10:51 am, Matti Picus wrote:


On 14/1/22 3:04 am, Charles R Harris wrote:

Hi All,

Travis wheel build problems again. There is still a positive balance, 
but apparently it is insufficiently positive. Which is strange if 
ARM64 is free. Wheel builds have been failing since last weekend.


Matti, could you ask for more credits? I'm thinking we really need to 
move off travis for ARM builds.


Chuck



I sent a request to supp...@travis-ci.com.


If we want to move off travis for aarch64, I think the only option 
right now is to use emulation, conda-forge does this on Azure machines 
[0] and the build + tests takes ~40 minutes. I can only imagine how 
long it would take to do move SciPy to a QEMU.



Do we have a way to reach out to an industry contact in the ARM 
foundation or to AWS (who have aarch 64 cloud machines) or NVidia (who 
also manufacture aarch64 machines) and discuss the problem with them?


Matti


[0] 
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=434709=logs=e9afaa34-1a0f-534e-78fc-fae528ccd915=e9afaa34-1a0f-534e-78fc-fae528ccd915=48395e60-bc5f-5e45-7455-d404f0c95f3a




The builds are running again [1].

Matti


[1] https://app.travis-ci.com/github/MacPython/numpy-wheels

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: MacPython travis funds have run out (again)

[Matti Picus]

On Fri, Jan 14, 2022, 08:53 Matti Picus <mailto:matti.pi...@gmail.com>> wrote:



...
If we want to move off travis for aarch64, I think the only option
right
now is to use emulation, conda-forge does this on Azure machines
[0] and
the build + tests takes ~40 minutes. I can only imagine how long it
would take to do move SciPy to a QEMU.


...
Matti


[0]

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=434709=logs=e9afaa34-1a0f-534e-78fc-fae528ccd915=e9afaa34-1a0f-534e-78fc-fae528ccd915=48395e60-bc5f-5e45-7455-d404f0c95f3a

<https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=434709=logs=e9afaa34-1a0f-534e-78fc-fae528ccd915=e9afaa34-1a0f-534e-78fc-fae528ccd915=48395e60-bc5f-5e45-7455-d404f0c95f3a>

On 14/1/22 12:08 pm, Kevin Sheppard wrote:
As a data point against relying on QEMU: statsmodels times out (5+ 
hours) when trying to build with QEMU rising cibuildwheel and GitHub 
actions.


Kevin



Interesting. It seems to work on the azure builds from conda-forge, I 
wonder what is different. Are the time out build logs still around?


Matti


[1] https://github.com/conda-forge/statsmodels-feedstock/runs/4196021348

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: MacPython travis funds have run out (again)

[Matti Picus]

On 14/1/22 3:04 am, Charles R Harris wrote:

Hi All,

Travis wheel build problems again. There is still a positive balance, 
but apparently it is insufficiently positive. Which is strange if 
ARM64 is free. Wheel builds have been failing since last weekend.


Matti, could you ask for more credits? I'm thinking we really need to 
move off travis for ARM builds.


Chuck



I sent a request to supp...@travis-ci.com.


If we want to move off travis for aarch64, I think the only option right 
now is to use emulation, conda-forge does this on Azure machines [0] and 
the build + tests takes ~40 minutes. I can only imagine how long it 
would take to do move SciPy to a QEMU.



Do we have a way to reach out to an industry contact in the ARM 
foundation or to AWS (who have aarch 64 cloud machines) or NVidia (who 
also manufacture aarch64 machines) and discuss the problem with them?


Matti


[0] 
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=434709=logs=e9afaa34-1a0f-534e-78fc-fae528ccd915=e9afaa34-1a0f-534e-78fc-fae528ccd915=48395e60-bc5f-5e45-7455-d404f0c95f3a


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: No credits left in travis for 1.22.0 release

[matti picus]

We are back in business with 100k credits.
Matti

On Fri, 31 Dec 2021 at 08:43, James Tocknell 
wrote:

> I'll need to check with the other maintainers (see
> https://github.com/h5py/h5py/issues/2026), but I believe h5py
> shouldn't need the x86 builds any more (I'm less sure about the other
> systems)?
>
> James
>
> On Fri, 31 Dec 2021 at 17:03, Matti Picus  wrote:
> >
> >
> > On 31/12/21 3:50 am, Charles R Harris wrote:
> > >
> > >
> > > On Thu, Dec 30, 2021 at 6:26 PM Charles R Harris
> > > mailto:charlesr.har...@gmail.com>> wrote:
> > >
> > > Hi All,
> > >
> > > We need to get more credits in order to run MacPython/numpy-wheels
> > > builds on travis. Anyone recall how to go about this?
> > >
> > > Chuck
> > >
> > >
> > > MacPython has used 800550 of 80 credits.  NumPy has used 625860 of
> > > 80 credits, so should be good for a couple of months. SciPy has
> > > one credit left, but doesn't use travis. I think the MacPython credits
> > > covers everyone who uses multibuild and travis, so NumPy is likely not
> > > the only one affected.
> > >
> > > Chuck
> > >
> > I requested more OpenSource credits for MacPython. The MacPython credits
> > cover many of the wheel-building projects, the main repos (numpy/numpy)
> > are under different accounts. I last reqeusted credits for MacPython in
> > October.
> >
> >
> > A bit more information:
> >
> > Travis-ci.com bills differently for each platform.
> >
> > For x86, they bill
> >
> > 10 credits per minute on linux
> >
> > 20 credits per minute on windows
> >
> > 50 credits per minute on macos
> >
> > aarch64 (arm linux), ppc, and s390x are **not billed** against credits.
> > I think this is because other companies provide the actual hardware
> runners.
> >
> >
> > Some of the projects under MacPython like matplotlib have migrated off
> > travis and use cibuildwheel in their main repo [0], using qemu for arm64.
> >
> >
> > Others  like numpy-wheels and openblas-libs moved their x86 builds to
> > other providers, their travis builds actually consume 0 credits.
> >
> >
> > h5py-wheels, scipy-wheels, pandas-wheels and others are still consuming
> > credits for their x86 builds. If all the projects moved to other
> > providers it would save opening requests for more credits, admittedly it
> > is easier to request credits than to move the build :). I could also be
> > more proactive in tracking the credit situation and request a renewal
> > before they run out, I will set a reminder to do so 2 1/2 months from
> now.
> >
> >
> > Matti
> >
> >
> > [0]
> >
> https://github.com/matplotlib/matplotlib/blob/main/.github/workflows/cibuildwheel.yml
> >
> > ___
> > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > To unsubscribe send an email to numpy-discussion-le...@python.org
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > Member address: aragilar+nu...@gmail.com
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Who is the tallest and strongest Chinese basketball player?

[Matti Picus]

On 31/12/21 6:14 am, Just allstar wrote:

Yao Ming ...



Sorry, I hit the wrong button when filtering spam off the list.

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: No credits left in travis for 1.22.0 release

[Matti Picus]

On 31/12/21 3:50 am, Charles R Harris wrote:



On Thu, Dec 30, 2021 at 6:26 PM Charles R Harris 
mailto:charlesr.har...@gmail.com>> wrote:


Hi All,

We need to get more credits in order to run MacPython/numpy-wheels
builds on travis. Anyone recall how to go about this?

Chuck


MacPython has used 800550 of 80 credits.  NumPy has used 625860 of 
80 credits, so should be good for a couple of months. SciPy has 
one credit left, but doesn't use travis. I think the MacPython credits 
covers everyone who uses multibuild and travis, so NumPy is likely not 
the only one affected.


Chuck

I requested more OpenSource credits for MacPython. The MacPython credits 
cover many of the wheel-building projects, the main repos (numpy/numpy) 
are under different accounts. I last reqeusted credits for MacPython in 
October.



A bit more information:

Travis-ci.com bills differently for each platform.

For x86, they bill

10 credits per minute on linux

20 credits per minute on windows

50 credits per minute on macos

aarch64 (arm linux), ppc, and s390x are **not billed** against credits. 
I think this is because other companies provide the actual hardware runners.



Some of the projects under MacPython like matplotlib have migrated off 
travis and use cibuildwheel in their main repo [0], using qemu for arm64.



Others  like numpy-wheels and openblas-libs moved their x86 builds to 
other providers, their travis builds actually consume 0 credits.



h5py-wheels, scipy-wheels, pandas-wheels and others are still consuming 
credits for their x86 builds. If all the projects moved to other 
providers it would save opening requests for more credits, admittedly it 
is easier to request credits than to move the build :). I could also be 
more proactive in tracking the credit situation and request a renewal 
before they run out, I will set a reminder to do so 2 1/2 months from now.



Matti


[0] 
https://github.com/matplotlib/matplotlib/blob/main/.github/workflows/cibuildwheel.yml


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: An article on numpy data types

[Matti Picus]

On 26/12/21 3:44 pm, Michael Siebert wrote:

Hey Lev,

I‘ve forgotten to mention my MacBook M1,
it‘s also int64 there.

Python on Windows is and is supposed to be, as far as I get it, a 
dying platform.



Your statement is the first time I have heard this. Of those who 
answered the 2020 Python Developers Survey[0], 68% use linux, 48% 
Windows and 29% macOS (yes, the total is more than 100%: users could 
tick more than one box), which was up slightly from 2018 [1] where 
windows was 47%. I couldn't find a line in the survey about WSL, but the 
people I know still want to work directly on Windows.



Matti


[0] https://www.jetbrains.com/lp/python-developers-survey-2020/, search 
for "Operating system"


[1] https://www.jetbrains.com/research/python-developers-survey-2018/ 
search for "Operating system"


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: help with PyArrayInterface

[Matti Picus]

On 21/12/21 11:27 pm, blor...@lbl.gov wrote:


... but I don't know what the typekind value should be. Can you point to a list 
of vlaues?
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com



>>> pprint.pprint([(k,v) for k,v in np.sctypeDict.items() if 
isinstance(k, str) and len(k) == 1])

[('?', ),
 ('b', ),
 ('B', ),
 ('h', ),
 ('H', ),
 ('i', ),
 ('I', ),
 ('p', ),
 ('P', ),
 ('l', ),
 ('L', ),
 ('q', ),
 ('Q', ),
 ('e', ),
 ('f', ),
 ('d', ),
 ('g', ),
 ('F', ),
 ('D', ),
 ('G', ),
 ('O', ),
 ('S', ),
 ('U', ),
 ('V', ),
 ('M', ),
 ('m', ),
 ('a', )]


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: computation of transcendentals

[matti picus]

Many of the ufuncs like exp and log are hightly optimized kernels
which use SIMD intrinsics where possible. The code can be found in
various files in tthe numpy/core/src/umath directory, for instance
here is the beginning of the SIMD exp function
https://github.com/numpy/numpy/blob/f521ee5f9a9868f38f7ca46d3c6a01aff8bea382/numpy/core/src/umath/loops_exponent_log.dispatch.c.src#L413
Matti


On Fri, Dec 10, 2021 at 3:53 PM  wrote:
>
> Hello,
>
> Can someone tell me (or point me to the relevant source code) what algorithms 
> NumPy uses to compute transcendentals, such as exp, log, etc?
>
> Thanks,
> Jeremiah
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Deprecating PyDataMem_SetEventHook

[Matti Picus]

NumPy has had a way to add a call to a user-supplied function whenever 
allocating via PyDataMem_SetEventHook [0]. Python added the tracemalloc 
[1] functionality which make this redundant. NumPy also added memory 
alloc/free overriding in NEP 49 [2] which will be part of the 1.22 
release. I propose deprecating the PyDataMem_SetEventHook function, and 
have submitted a PR to do so for 1.23 [3]. Thoughts? Is anyone using the 
hook for something that tracemalloc cannot do?



Matti


[0] 
https://numpy.org/devdocs/reference/c-api/data_memory.html#c.PyDataMem_SetEventHook


[1] https://docs.python.org/3/library/tracemalloc.html

[2] https://numpy.org/neps/nep-0049.html

[3] https://github.com/numpy/numpy/pull/20394

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: installing numpy on OpenIndiana

[Matti Picus]

  
  
Your mail ended up in my spam bucket, maybe that is why no-one
  responded.



On 11/11/21 8:58 pm, asyropou...@aol.com wrote:


  ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64


There is something seriously off with the way the compiler is
  being called. 



We don't support OpenIndiana. Are there python user groups you
  could reach out to?



This is the first error you need to solve. The usual way to build
  numpy from source is via `pip install`, not using the github
  tarballs. This should solve the versioneer problem. There are
  various options to `pip install` that will allow you more
  flexibility when trying to recover from a broken build. I do see
  you have `-m64` in the gcc command line, is that normal for
  building on this platform?


Matti

  

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[matti . picus]

Upate: since there is a clear delineation of spammers vs legitimate posts, I 
would like to tweak the policy [0] and from now on a single spam message will 
move email addresses down a category: from authorized to moderated and from 
moderated to unsubscribed. I have changed the policy statement accordingly, 
pending disagreement here. It makes the bookkeeping simpler since we don't have 
to track spam messages (we get on average 1 or 2 a day)

Matti 

[0] https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: [RFC] - numpy/SVML appears to be poorly optimized

[Matti Picus]

On 6/11/21 6:56 pm, Sayed Adel wrote:


> appears to be poorly optimized.

It should perform well, not poor neither heavily optimized.

> this also makes it quite difficult to improve (with either a better 
compiler or by hand).


We can put the blame on Intel for not sharing their source code but 
honestly, it seems we had no other option except accept what they provide.


> Some of the glaring issues are:
> 1. register allocation / spilling
> 2. rodata layouts / const-propagation of the values.
> 3. Very odd use of internal functions that really ought to be inlined.

let me add to your list another two points:
- It only works on Linux.
- It only works with AVX512.

> If so, are people open to patches that optimize them (either with 
new C implementations are in the current assembly

implementations).

Hopefully, we will able to convert them to universal intrinsics 
(nep-38) one day. As one of the team, I will try to push more time for it.


Thanks, Sayed.



Note the benchmarks on Sayed's PR [0] to move tanh to universal 
intrinsics. It not only supplies the routines for all 
universal-intrinsics-supported platforms, it even slightly increased 
performance on AVX512 (usual disclaimers about dangers of comparing 
benchmarks apply).



Matti


[0] https://github.com/numpy/numpy/pull/20363

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[matti . picus]

Update: we now have a team of moderators for the mailing list. The policy for 
moderation is in the list summary [0]. 

Since moderation began on Oct 26, we have rejected 15 spam messages, 
unsubscribed 2 email addresses that spammed more than 3 each, and released 4 
messages from moderated users to the mailing list. 

Matti

[0] https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] spam on the mailing lists

[Matti Picus]

The list default for new subscribers is to hold posts for approval 
before sending to the mailing list. Hopefully that will cut down the 
number of spam messages, at the cost of delay of maybe a day for new 
subscriber posts. In order to make the policy clear, I submitted a PR to 
numpy.org [0]. Comments and suggestions welcome there.


Matti

[0] https://github.com/numpy/numpy.org/pull/487

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[Matti Picus]

It seems the spammers have figured out how to get onto discourse as 
well. The last two messages on 
https://discuss.python.org/t/pypi-user-feedback/11125/4 (on the python 
discourse instance) are the first I have seen there.


Matti

On 10/10/21 7:37 pm, Hameer Abbasi wrote:

Hello everyone,

Just my 2 cents: I marked a few of the actual spam e-mails on this and the 
SciPy-user list as spam on my client, and it seems many random e-mails get sent 
to spam now, from both the NumPy and SciPy lists.

I’ve tried to pull as many out as possible by moving them to inbox, hoping 
it’ll get better, but it hasn’t so far.

I’d very much prefer Discourse or GitHub discussions, of the two I prefer 
Discussions, but Discourse isn’t too bad either.

With best regards,
Hameer Abbasi


Am 06.10.2021 um 22:57 schrieb Aaron Meurer :

Coming back to earth on the original discussion, is there really no
way to moderate new users on the mailing list platform used by this
list (mailman?)? On the SymPy list, which uses Google Groups, we
moderate all new users, so that first time posts have to be moderated
but after that people can post directly. We have had very minimal spam
even in the queue (to be fair, I'm sure Google applies its own
filtering), and only one in recent memory that got through, due to the
moderators not reading the message contents carefully enough.

Aaron Meurer

On Tue, Oct 5, 2021 at 7:23 AM Robert Kern  wrote:

On Tue, Oct 5, 2021 at 4:07 AM  wrote:
[spam]

Okay, now, they're just messing with us.

--
Robert Kern
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: asmeu...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: einstein.edi...@gmail.com


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Intel's Short Vector Math Library has been merged as a git submodule

[Matti Picus]

On 11/10/21 11:05 pm, Jerome Kieffer wrote:

On Mon, 11 Oct 2021 18:04:58 +0300
Matti Picus  wrote:


As SciPy already found out, some downstream libraries may need to tweak
their tolerances for some functions as a result of this PR. We wanted to
put it in early enough in the release cycle so that we can back it out
fully or partially if the accuracy degradation is too large, so please
speak up if you notice anything strange.

Thanks for warning in advance... now, we need find some computers to
test those versions. Do you know if it works "the same" with AVX2 ?
since most computers have AVX2 and for now you need the latest servers
to test AVX512.

Cheers,

Jerome


Short answer: the code path should be exactly the same on machines 
without AVX512 before and after this PR.


Long answer:

The use of intrinsics for ufunc loops is mostly described in the docs 
[0] . When calling a ufunc loop, a dispatch mechanism chooses the 
appropriate compiled loop for the available intrinsics on the system. 
You can see which intrinsics are supported on your installation of NumPy 
(new for 1.22) by using numpy.show_config(). The last few rows show 
which intrinsics are built into numpy and can possibly be used, and 
which subset is detected and will be used. This means we ship multiple 
variants of loops, and only one set will be used on each machine. So a 
machine without AVX512 will continue to use whatever loop it used before 
this PR.


Matti

[0] https://numpy.org/devdocs/reference/simd/simd-optimizations.html

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Intel's Short Vector Math Library has been merged as a git submodule

[Matti Picus]

Yesterday we merged PR 19478 [0] to add a open source version of Intel's 
Short Vector Math library into NumPy. The original PR was by Raghuveer 
Devulapalli. There was a lot of discussion on the PR, but now that I 
look back it seems the mailing list may not have been involved. The code 
lives in https://github.com/numpy/SVML, and in order to pull it in you 
must now do `git submodule update --init` before building NumPy. The 
library provides AVX-512 implementations of the following math 
functions: exp, exp2, log, log2, log10, expm1, log1p, cbrt, pow, sin, 
cos, tan, asin, acos, atan, atan2, sinh, cosh, tanh, asinh, acosh and 
atanh, speeding them up significantly where AVX-512 is available. Before 
this PR was merged, Raghuveer added a set of tests for accuracy and 
compliance for these functions [1]. The accuracy is now defined as up to 
4 ULP in the worst case for tan, cos, sin, asin, atan, expm1, but 
typically the error is up to 2 ULP.


As SciPy already found out, some downstream libraries may need to tweak 
their tolerances for some functions as a result of this PR. We wanted to 
put it in early enough in the release cycle so that we can back it out 
fully or partially if the accuracy degradation is too large, so please 
speak up if you notice anything strange.


Matti


[0] https://github.com/numpy/numpy/pull/19478

[1] https://github.com/numpy/numpy/pull/19485

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Musllinux wheels

[Matti Picus]

On 11/10/21 2:22 pm, Ralf Gommers wrote:



On Mon, Oct 11, 2021 at 9:52 AM Laurie O > wrote:


CI request: build PEP 656
(https://www.python.org/dev/peps/pep-0656/
) compliant "musllinux"
wheels. This will allow easy installation in Alpine Docker images.
GitHub issue: https://github.com/numpy/numpy/issues/20089



Thanks for asking Laurie. I'd say we may perhaps want to do this, but 
only if and when it becomes much easier - for example once we have 
migrated all our wheel build infra to GitHub Actions and cibuildwheel. 
For now I'm inclined to say that https://github.com/alpine-wheels 
 has NumPy wheels you can use, and 
that Alpine is a niche platform, so this is quite low-prio for us.


In general, providing wheels for every platform under the sun is not 
sustainable - see 
https://mail.python.org/archives/list/numpy-discussion@python.org/thread/46HT2SYDHBNLOC6N5RTXI7CN32YWIJWR/#I4474NYSZQVZEIJOA7W43Q6QPTXTXDR5 
 
for more context.


Cheers,
Ralf

I am a little worried about releasing wheels untested that will then 
come back to use as issues with Nans, error states, or inaccurate 
results. We should have some verification that the platform passes the 
test suite, preferably a specific run in the main CI.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: spam on the mailing lists

[Matti Picus]

On 29/9/21 9:07 pm, Stefan van der Walt wrote:

On Wed, Sep 29, 2021, at 03:02, Ralf Gommers wrote:
We don't have admin access to the python.org  
lists, so this is a bit of a problem. We have never had a spam 
problem, so we can ask to block this user first. If it continues to 
happen, we may be able to moderate new subscriber emails, but we do 
need to ask for permissions first and I'm not sure we'll get them.


A better solution longer term is migrating to Discourse, which has 
far better moderation tools than Mailman and is also more 
approachable for people not used to mailing lists (which is most 
newcomers to open source). Migrating is a bit of a pain, but with the 
new CZI grant having a focus on improving the contributor experience, 
we should be able to do this.


I would like to offer the use of https://discuss.scientific-python.org 
. I would be happy to handle 
email list migration, and have created the following two categories 
for NumPy discussion:


User discussion: https://discuss.scientific-python.org/c/user/numpy 

Contributor discussion: 
https://discuss.scientific-python.org/c/contributor/numpy 



We're happy to support this as part of the Scientific Python ecosystem 
grant, and will give admin rights to anyone on the NumPy developer 
team who wants to help manage / moderate discussions.


Of course, we can also just delete these if the team prefers to have 
their discussions somewhere else.  But I think there is a benefit to 
bringing community discussions together in one place.


Best regards,
Stéfan



Thanks for the offer to host the discussions on discourse. Personally, I 
find the email interface to discourse very clunky. I would prefer we 
exhaust the possibilities to stay on e-mail only before moving to discourse.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Moving community meeting to 16:00 UTC

[Matti Picus]

On Tue, Sep 28, 2021 at 7:20 PM Sebastian Berg 
mailto:sebast...@sipsolutions.net>> wrote:


Sorry about that, lets go with 16:30 UTC then, at least for now.  I
will announce that later for tomorrow.  If that ends up bad for anyone
(or the situation just changes) we can still move it.


Could someone modify the community calendar appropriately?
Thanks,
Matti
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

Re: [Numpy-discussion] Help Regarding Debugging Cython Code

[Matti Picus]

On 5/9/21 12:55 pm, Amardeep Singh wrote:

Hi Team

If some one has setup to debug cython code can  you please share 
the details.

I am using ubuntu 20.4 .
Not sure why gdb not picking up breakpoints.

I am able to debug c extensions but not cython code.

Thx



There are a number of ways to set up for debugging. The one that works 
for me is to do



git clean -xfdd

CFLAGS='-O0 -g3' python runtests.py -b

gdb --args python runtests.py -t file/I/with/the/test/I/want.py


The "git clean" should make sure you are not using pre-compiled 
c-extensions.


There are variants of this if you are using an IDE or like to use 
setup.py directly



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding New Feature

[Matti Picus]

On 19/8/21 6:15 pm, Bhavay Malhotra wrote:

Dear Team,

I’m thinking of adding a new feature in response to the issue no. #19039.

The feature is basically a function to check whether the data type of 
both the numpy arrays are same or not.


If the numpy arrays have different data types function return a False 
else it returns a True.



Thanking You.

Regards,

Bhavay


As we discussed on the issue https://github.com/numpy/numpy/issues/19039,


Is there a use-case where |"b.dtype == c.dtype"| would not suffice?


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Drop LGTM testing.

[Matti Picus]

On 15/8/21 12:15 am, Charles R Harris wrote:



On Sat, Aug 14, 2021 at 2:35 PM Eric Wieser 
mailto:wieser.eric%2bnu...@gmail.com>> 
wrote:


This might be worth creating a github issue for simply so we can
tag someone working at LGTM; they've been helpful in the past, and
it's possible we just need to fiddle with some configuration to
make it work.

It's also worth noting that LGTM runs against C code too; so even
if we disable it for python, it might be worth keeping around for C.


It's the C code that causes problems, LGTM builds the code with 
`python3 setup.py` and setup.py has a check for the Python version. 
There is no method to disable the C checks from the Github app and no 
method to specify Python version beyond 2 or 3.


I'd be happy to tag someone at LGTM, but I don't know who that would be.

Chuck

Personally, I would prefer we drop it. I do not recall an instance when 
we related to its results in a PR review in the last year, so for me it 
is just one more thing that breaks randomly. As it is our PR review and 
CI support bandwidth is limited, so I would prefer to optimize for that 
bottleneck.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reducing effort spent on wheel builds?

[Matti Picus]

On 16/7/21 9:11 pm, Chris Barker wrote:

Just a note on:

> For the record, I am +1 on removing sdists from PyPI until pip changes
its default to --only-binary :all: [1]

I agree that the defaults for pip are unfortunate (and indeed the 
legacy of pip doing, well, a lot, (i.e. building and installing and 
package managing and dependencies, and ...) with one interface.


However, There's a long tradition of sdists on PyPi -- and PyPi is 
used, for the most part, as the source of sdists for other systems 
(conda-forge for example). I did just check, and numpy is an exception 
-- it's pointing to gitHub:


source:
    url: https://github.com/numpy/numpy/releases/download/v{{ 
 version 
}}/numpy-{{ version }}.tar.gz


But others may be counting on sdists on PyPi.

Also, an sdist is not always the same as a gitHub release -- there is 
some "magic" in building it -- it's not just a copy of the repo. 
Again, numpy may be building its releases as an sdist (or it just 
doesn't. matter), but something to keep in mind.


Another thought is to only support platforms that have a 
committed maintainer -- I think that's how Python itself does it. The 
more obscure platforms are only supported if someone steps up to 
support them (I suppose that's technically true for all platforms, but 
not hard to find someone on the existing core dev team to support the 
majors). This can be a bit tricky, as the users of a platform may not 
have the skills to maintain the builds, but it seems fair enough to 
only support platforms that someone cares enough about to do the work.


-CHB


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

chris.bar...@noaa.gov 



Just an empty response since this ended up in my spam filter, and I am 
probably not the only one.


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Revert the return of a single NaN for `np.unique` with floating point numbers?

[Matti Picus]

On 2/8/21 8:49 pm, Ralf Gommers wrote:



On Mon, Aug 2, 2021 at 7:04 PM Sebastian Berg 
mailto:sebast...@sipsolutions.net>> wrote:


Hi all,

In NumPy 1.21, the output of `np.unique` changed in the presence of
multiple NaNs.  Previously, all NaNs were returned when we now only
return one (all NaNs were considered unique):

    a = np.array([1, 1, np.nan, np.nan, np.nan])

Before 1.21:

    >>> np.unique(a)
    array([ 1., nan, nan, nan])

After 1.21:

    array([ 1., nan])


This change was requested in an old issue:

https://github.com/numpy/numpy/issues/2111


And happened here:

https://github.com/numpy/numpy/pull/18070


While, it has a release note.  I am not sure the change got the
attention it deserved.  This would be especially worrying if it is a
regression for anyone?


I think it's now the expected answer, not a regression. `unique` is 
not an elementwise function that needs to adhere to IEEE-754 where nan 
!= nan. I can't remember reviewing this change, but it makes perfect 
sense to me.


Cheers,
Ralf



We were discussing this today (me and Matthew) and came up with an edge 
case when using set(a), it will return the old value. We should add this 
as a documented "feature"


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

[Matti Picus]

The NEP [0] and the corresponding PR [1] have gone through another round 
of editing. I would like to restart the discussion here if anyone has 
more to add. Things that have changed since the last round:


- The functions now accept a context argument

- The code has been cleaned up for consistency

- The language of the NEP has been tightened


Thanks to all who have contributed to the discussion so far.


Matti


[0] https://numpy.org/neps/nep-0049.html

[1] https://github.com/numpy/numpy/pull/17582

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reducing effort spent on wheel builds?

[Matti Picus]

On 15/7/21 4:11 pm, Ralf Gommers wrote:



On Thu, Jul 15, 2021 at 2:41 PM Matti Picus <mailto:matti.pi...@gmail.com>> wrote:


On the other hand, packaging is made harder as more technologies
go into
a wheel build. ... Perhaps there should be some discussion
of the cost of adding new build dependencies and somehow making the
dependency conditional ...


Not quite the right mailing list, but: it *is* an optional dependency 
in scipy 1.7.0, and there has been no proposal so far to make it 
required.



Cheers,
Ralf


Thanks for the clarification, sorry for the misunderstanding

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reducing effort spent on wheel builds?

[Matti Picus]

On 15/7/21 1:21 pm, Ralf Gommers wrote:


Hey all,

I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo 
again recently, and I've done the same for SciPy. ...


Cheers,
Ralf



Since my name was mentioned, the things I have spent time on for wheel 
packaging in order of time spent (not scientifically measured rather 
looking back at the closed PRs and thinking "oh yeah, that was painful") 
have been


- Updating manylnux to move away from 10 year old glibc on linux (still 
stuck and not clear how to finish it [0])


- Moving from travis-ci.org to travis-ci.com (with the panic around 
build credits) and from Appveyor to github actions/azure pipelines


- Moving from rackspace's wheel hosting to anaconda.org

- Working around CI failures with aarch64 for linux, mainly due to 
shortcomings in the free CI providers


- Bugs in dependencies: avoiding buggy Accelerate when building NumPy 
and debugging Windows/aarch64 problems with OpenBLAS


- Updating OpenBLAS versions

- Shepherding Apple M1 hardware through the manylinux/multibuild/wheel 
pipeline (all the hard work was done by others)


- Trying to make 64-bit interfaces for OpenBLAS work (99.9% done by Chuck)

- Updating PyPy versions


Only the last one, which was actually the least painful, would be helped 
by Ralf's list.



On the other hand, packaging is made harder as more technologies go into 
a wheel build. The twitter thread started with "SciPy added a required 
dependency on a technology which broke things, but people stepped up to 
fix the problem quickly" and morphed into "lets drop wheels for lesser 
used platforms". I am not sure the discussion should have moved away 
from the first point so quickly. Perhaps there should be some discussion 
of the cost of adding new build dependencies and somehow making the 
dependency conditional for a release or two until all the worst kinks 
are worked out.



For the record, I am +1 on removing sdists from PyPI until pip changes 
its default to --only-binary :all: [1]


Matti


[0] https://github.com/pypa/manylinux/issues/1012

[1] https://github.com/pypa/pip/issues/9140

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy's BLAS library on macOS?

[Matti Picus]

On 8/7/21 2:23 am, Jerry Morrison wrote:

Got it!

*Summary:*
* Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its 
embedded OpenBLAS on every platform that has a wheel.

  That OpenBLAS is always compiled with gcc/gfortran.
  In this case, `np.show_config()` reports `library_dirs = 
['/usr/local/lib']` even though there's no libblas in that directory.


* Installing numpy from source (e.g. `pip install numpy==1.21.0 
--no-binary numpy)` looks for BLAS & LAPACK libraries at build time as 
influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or 
by the file ~/.numpy-site.cfg.

  On macOS, 'accelerate' is in the default search order after 'openblas'.
  On macOS < 11.3, importing numpy that's linked to Accelerate will 
detect an Accelerate bug and raise a RuntimeError.



That seems correct, although admittedly show_config could do a better 
job. The problem is that not every BLAS implementation provides a 
convenient method to self-report.


It might be nice to document all this somewhere more permanent, the 
docstring for show_config might be a good place to start.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] `keepdims=True` for argmin/argmx and C-API `PyArray_ArgMaxWithKeepdims`

[Matti Picus]

On 1/7/21 7:49 pm, Gagandeep Singh wrote:

Hi,

So should I remove these new functions from public C-API? Let me know. 
I will do that.





Yes please. If needed we can add them, but once in we cannot remove them.

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Acceptance of NEP 36 - fair play

[Matti Picus]

Stefan has submitted a PR [0] to accept "NEP 36 -fair play" as accepted.

It was discussed recently on the mailing list [1] and in the original PR 
[2] . If there is no opposition in a week (27 June) the PR will be merged.



[0]https://github.com/numpy/numpy/pull/19284

[1] https://mail.python.org/pipermail/numpy-discussion/2021-June/081873.html

[2] https://github.com/numpy/numpy/pull/14779



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] copy="never" discussion and no deprecation cycle?

[Matti Picus]

  
  


On 18/6/21 2:23 am, Stephan Hoyer
  wrote:


  

  
Overall, I think using an enum is the happiest
  situation. It's a slightly awkward API, to be sure, but
  not very awkward in the scheme of things, and it's better
  than needing to wait a long time for a final resolution. 

  

  



Thanks for the clarification, I hadn't considered the use of new
  numpy code with older versions of numpy.


The enum values would be IF_NEEDED=0, (corresponding to False),
  ALWAYS=1 (corresponding to True), NEVER=2, right? In which
  namespace should they live:
 
  1) `np.Copymode.NEVER`? 
  2) `np.NEVER`? 
  3) `np.enums.Copymode.NEVER`?


Matti

  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] copy="never" discussion and no deprecation cycle?

[Matti Picus]

On 16/6/21 11:00 pm, Sebastian Berg wrote:


Hi all,

(sorry for the length, details/discussion below)

On the triage call, there seemed a preference to just try to skip the
deprecation and introduce `copy="never"`, `copy="if_needed"`, and
`copy="always"` (i.e. string options for the `copy` keyword argument).



Why this may be controversial: today someone could be calling 
|'||np.array(..., copy="never")', which would call| '|bool("never")', 
which would evaluate to 1, and would end up| doing the exact opposite of 
never. So their code is wrong, and they do not know it but are used to 
the error. If we change this, it would silently fix their code to do 
what they intended.



Is that a correct reading of the problem?

If so, I am in favor of the proposal to use string options in addition 
to boolean options.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP 36- fair play (was: request to remove the numpy-aarch64 package from PyPI)

[Matti Picus]

On 14/6/21 11:03 pm, Stefan van der Walt wrote:

On Sun, Jun 13, 2021, at 18:21, Charles R Harris wrote:



On Sun, Jun 13, 2021 at 10:47 AM Ralf Gommers > wrote:


FYI, I noticed this package that claimed to be maintained by us:
https://pypi.org/project/numpy-aarch64/
. That's not ours, so I
tried to contact the author (no email provided, but guessed the
same username on GitHub) and asked to remove it:
https://github.com/tomasriv/DNA_Sequence/issues/1
.

There are a very large number of packages with "numpy" in the
name on PyPI, and there's no way we can audit/police that
effectively, but if it's a rebuild that pretends like it's
official then I think it's worth doing something about. It could
contain malicious code for all we know.


That is a pretty misleading package description, would have fooled me 
if I didn't know better. I didn't get the impression it was 
malicious, but still . . .


Maybe now is a good time to move to accept:

https://numpy.org/neps/nep-0036-fair-play.html 



Stéfan



Having just re-read the NEP, I think the Motivation section should 
mention name re-use: "Additionally, we wish to reduce confusion when 
package names imply they are sanctioned or maintained by NumPy". Other 
than that it looks good to me. Do you want to make a PR to add the 
discussion and change the status, and notify the list of your intention 
to accept it?



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PR for keywordlabeler

[Matti Picus]

On 10/6/21 10:20 pm, Charles R Harris wrote:

Hi All,

There is a pending PR  that 
uses keywordlabeler  to 
automatically label PRs and issues. Installing the app requires giving 
it write permissions to PRs and issues. This isn't different than with 
the current labeler, but I note that we don't have a procedure for 
deciding such issues that may have security implications. So I am 
posting here before proceeding with app installation.


Thoughts?

Chuck



If there was a way to pin these actions to a hash tag that would be 
better, at the least it should pin to a version. We already use third 
party actions in the github workflow: checkout and setup-python (both 
specifying a "version" via "@v2"), 
larsoner/circleci-artifacts-redirector-action@master (hmm, that should 
pin to a version).



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] EHN: Discusions about 'add numpy.topk'

[Matti Picus]

On 29/5/21 5:28 pm, Ralf Gommers wrote:



On Fri, May 28, 2021 at 4:58 PM > wrote:


Hi all,

Finding topk elements is widely used in several fields, but missed
in NumPy.
I implement this functionality named as  numpy.topk using core numpy
functions and open a PR:

https://github.com/numpy/numpy/pull/19117


Any discussion are welcome.


Thanks for the proposal Kang. I think this functionality is indeed a 
fairly obvious gap in what Numpy offers, and would make sense to add. 
A detailed comparison with other libraries would be very helpful here. 
TensorFlow and JAX call this function `top_k`, while PyTorch, Dask and 
MXNet call it `topk`.


Two things to look at in more detail here are:
1. complete signatures of the function in each of those libraries, and 
what the commonality is there.
2. the argument Eric made on your PR about consistency with 
sort/argsort, and if we want topk/argtopk? Also, do other libraries 
have `argtopk`?


Cheers,
Ralf


Best wishes,

Kang Kai



Did this function come up at all in the array-API consortium dicussions?

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

[Matti Picus]

On 10/5/21 8:43 pm, Sebastian Berg wrote:


But right now the proposal says this is static, and I honestly don't
see much reason for it to be freeable?



I think this is the crux of the issue. The current design is for a 
singly-allocated struct to be passed around since it is just an 
aggregate of functions. If someone wants a different strategy (i.e. 
different alignment) they create a new policy: there are no additional 
parameters or data associated with the struct. I don't really see an ask 
from possible users for anything more, and so would prefer to remain 
with the simplest possible design. If the need arises in the future for 
additional data, which is doubtful, I am confident we can expand this as 
needed, and do not want to burden the current design with unneeded 
optional features.



It would be nice to hear from some actual users if they need the 
flexibility.



In any case I would like to resolve this quickly and get it into the 
next release, so if Eric is adamant that the advanced design is needed I 
will accept his proposal, since that seems easier than any of the 
alternatives so far.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

[Matti Picus]

On 6/5/21 2:07 pm, Eric Wieser wrote:
The NEP looks good, but I worry the API isn't flexible enough. My two 
main concerns are:


### Stateful allocators

Consider an allocator that aligns to `N` bytes, where `N` is 
configurable from a python call in someone else's extension module.

...

### Thread and async-local allocators

For tracing purposes, I expect it to be valuable to be able to 
configure the allocator within a single thread / coroutine.
If we want to support this, we'd most likely want to work with the 
PEP567 ContextVar API rather than a half-baked thread_local solution 
that doesn't work for async code.


This problem isn't as pressing as the statefulness problem.
Fixing it would amount to extending the `PyDataMem_SetHandler` API, 
and would be unlikely to break any code written against the current 
version of the NEP; meaning it would be fine to leave as a follow-up.
It might still be worth remarking upon as future work of some kind in 
the NEP.



I would prefer to leave both of these to a future extension for the NEP. 
Setting the alignment from a python-level call seems to be asking for 
trouble, and I would need to be convinced that the extra layer of 
flexibility is worth it.



It might be worth mentioning that this NEP may be extended in the 
future, but truthfully I think that is the case for all NEPs.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion