[Numpy-discussion] ANN: DataLab v0.15

[Pierre Raybaut]

Hello everyone,

We're excited to announce the release of DataLab v0.15!
https://datalab-platform.com

Note that the v0.15.1 release fixes a very annoying issue on macOS (screen
flickering when switching from a signal/image object to another).

DataLab is an open-source platform for scientific and technical data
processing and visualization, based on NumPy, SciPy, ... and Qt.
It's extendable with plugins, and works seamlessly as a complementary tool
with your IDE (e.g. Spyder, VSCode, ...) and Project Jupyter notebooks.

Once again, this release is the result of the feedback and contributions of
our early adopters. We're grateful for your support and look forward to
your continued insights.

 New Features:

- Added support for large CSV/text files (when opening or importing data in
Signal Panel, for example). Need to work on a GB file ? No worry, DataLab
won't freeze
- New auto downsampling feature for visualizing large signals -- handling
100Mpts signals without slowing down the user experience!
- New MSI installer (replacing the old EXE installer for the stand-alone
version on Windows) to avoid false positive detection by antiviral software

 You have a project or a use case in mind? (we can even write a tutorial
for you...) You want to contribute? We'd love to hear from you!
So please, don't hesitate to contact us or to open an issue (
https://github.com/DataLab-Platform/DataLab/issues/new/choose) on our
GitHub project for any request or question.

Thank you for taking the time to read my announcement. Looking forward to
your thoughts.

Best,
Pierre Raybaut
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Should we add a NumPy default protocol version for np.save?

[Chunqing Shan via NumPy-Discussion]

Currently, when NumPy saves data using pickle, it hard-coded the protocol
version to 3, which was the default value from Python 3.0 to Python 3.7.
However, since Python 3.7 has reached its end-of-life (EOL), there are no
actively maintained Python versions that default to using pickle protocol 3.  

  

By allowing NumPy to use the default pickle protocol version, objects larger
than 4GB can be pickled, resolving the issue described in #26224 [1]. Although
this new protocol version is incompatible with Python 3.3 and earlier
versions, Python 3.3 has long reached its EOL. Therefore, there is no need to
force pickle protocol 3 when saving data using pickle.  

  

One of the reasons for having a hard-coded version is that upstream Python may
bump the default version to a new version that will break the compatibility
between two supported versions of NumPy. This scenario won't happen as Python
core developers promised that it's a policy to only use the protocol supported
by all currently supported versions of Python as default.[2][3]  

  

Another reason for having a different NumPy default version of pickle protocol
is that because NumPy does not support every old Python version, we can
actually bump the default version earlier than upstream Python to get the
performance gains.  

  

I have done some experiments[4] that show while pickle protocol 4 improved
about 4% for overall np.save performance, protocol 5 did not have that kind of
performance improvement. Therefore, I'm not sure whether it is worth it to add
a NumPy default version of pickle protocol. Protocol 5 use out-of-band buffer
which may increase performance for lots of small pickles, but since np.save
typical usage is saving to a file, filesystem overhead on creating lots of
small files is the main bottleneck.  

  

There are two possible solutions to this problem. One is to just follow the
upstream protocol version(which is currently 4) and expect them to bump it
wisely when they should. The second is to add a pickle_protocol keyword to the
numpy.save function, defaulting to the highest version that the oldest Python
version NumPy supports (which is 5). Since this introduces some complexity
into a pretty base interface for NumPy, I believe it needs members of NumPy to
decide which way to go. I can finish the documentation (with an explanation of
performance differences on different protocol versions of pickle) and add this
to the release checklist of NumPy and code if we decide to add this keyword.  

  

I can also see if we can utilize pickle protocol 5's improvement with reuse of
out-of-band buffers to keep things in cache and improve performance in another
PR, if we decide to add NumPy's own default version of pickle protocol and set
it to 5.  

  

Best regards,  

Chunqing Shan  

  

1\.   

2\.

  

3\.   

4\. With  

a = np.array(np.zeros((1024, 1024, 24, 7)), dtype=object) # 1.5GB  

and time the following statement(/dev/shm is configured as a tmpfs)  

np.save('/dev/shm/a.npy', a)  

For protocol 3, on average 4.25s is required; for protocol 4, on average
4.07s; for protocol 5, on average 4.06s. This result is acquired on a Xeon
E-2286G bare metal with 2 16GB 2667 MT/s DDR4 ECC, with Python 3.11.2 and
NumPy 1.26.4.  

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: PR - can I get a new review?

[Nathan]

I think most of the build failures you’re seeing would be fixed by merging
with or rebasing on the latest main branch.

Note that there is currently an issue with some of the windows CI runners,
so you’ll see failures related to our spin configuration failing to handle
a gcov argument that was added in spin 0.9 released a couple days ago.

On Mon, May 6, 2024 at 8:48 PM Jake S.  wrote:

> Hi community,
>
> PR 26081  is about making
> numpy's ShapeType covariant and bound to a tuple of ints.  The community
> has requested this occasionally in issue 16544
> .  I'm reaching out via the
> listserv because it's been a few months, and I don't want it to get too
> stale.  I could really use some help pushing it over the finish line.
>
> Summary:
> Two numpy reviewers and one interested community member reviewed the PR
> and asked for a type alias akin to npt.NDArray that allowed shape.  I
> worked through the issues with TypeVarTuple and made npt.Array, and it was
> fragile, but passing CI.  After a few months passed, I returned to fix the
> fragility in the hopes of getting some more attention, but now it fails CI
> in some odd builds (passes the mypy bit).  I have no idea how to get these
> to pass, as they appear unrelated to anything I've worked on (OpenBLAS on
> windows, freeBSD...?).
>
> Thanks,
> Jake
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: nathan12...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Mark Harfouche]

I'm sorry to have used scikit-image as an example as it is an active
project (with Juan and other core dev even replying to this thread)
and we can try to release 0.22.1 just for this purpose.
But, what I'm trying to do is to avoid the churn that many other developers
will have when this gets rolled out to a wide audience when they have to
backport similar fixes. (Truthfully, i'm haunted by the LTS of scikit-image
where we kept 0.14.X alive and backported fixes for it.)

However, it seems that many are in favor of keeping support because enough
safeguards have put in place.

I'm excited to see the rollout whatever may happen!
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Fastpathing lexsort for integers

[ml]

Hi all (my first message in this list),

I have written some (Python) code to make lexsort on (lists of arrays of) 
integers more efficient, by "merging" all levels in a single array. I'm asking 
for some feedback between doing a PR.

Compared to the current code, there is an improvement as long as:
- the arrays (levels) are long at least 10^3 - the improvement becomes 
significant (~3x) above 10^4
- the sum of bits required for the representation of each level is no larger 
than the platform uint's bits; otherwise, my code falls back to Python int, and 
becomes way less performant.

My rough guess it that the second condition holds in the vast majority of use 
cases (including a planned use in scipy - see 
https://github.com/scipy/scipy/issues/20151 for context).

The code is here: https://github.com/toobaz/numpy/commit/aa5c6948

I know that the current form is not well written - at least _int_lexsort seems 
out of place, and imports placed inside the function - but I didn't know in 
which file it would belong (and anyway, this version makes a quick glance 
easy). In any case, feedback is welcome as to what should go where.

For the records, the approach is similar to that employed in pandas 
MultiIndexes: https://github.com/pandas-dev/pandas/pull/19074

API-wise, there are various possibilities to do this:

- the current status is that a new parameter "int_path" is added, and this 
determines which path to take; so nothing changes unless the user wants to use 
the new path

- alternatively, I could check if all arrays are int (O(1)) and their 
maxima/minima (O(n)) and use the optimized code path only if it relies on uint, 
not Python int. The downside is that I would (slightly?) slow down those cases 
which do fallback to the old code.

- moreover, in the current version "int_path" can take as value either a bool 
or a sorting algorithm, as understood by np.argsort. But an alternative is that 
we add a new argument "kind" to lexsort (consistently with argsort), which at 
the moment would raise NotImplementedError in the "old" path. By the way, my C 
is too rusty to really understand what goes on here:
https://github.com/numpy/numpy/blob/main/numpy/_core/src/multiarray/item_selection.c#L1796
... but maybe it wouldn't be so difficult to support switching "kind" also in 
the "general" lexsort (old code path).

- note that lexsort is guaranteed to be stable, while the new code path is 
stable only if "kind" is set to e.g. 'stable' (the default for argsort being 
'quicksort')

The current code misses tests (I'll be happy to add some once someone confirms 
that there is interest for this contribution), but some (performance) tests I 
ran are here:
https://github.com/toobaz/int_lexsort/blob/main/Show%20benchmarks.ipynb

Thanks for your attention,

Pietro Battiston
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Henry Schreiner]

This will be messier for projects building wheels and wanting to support 
non-EoL Python versions. To build a wheel with anything other than pybind11, 
you now need the oldest supported NumPy for Python < 3.9, the latest NumPy 1 
for Python 3.9, and NumPy 2 for Python 3.10+. I don't know if that's important 
in the decision, but thought I'd point it out. Also, according to NEP 29, 3.10+ 
only became the requirement a couple of weeks ago, while it has been months 
since SPEC 0 dropped it. I don't think either document really details what to 
do when there's a really long development cycle that spans a cutoff date.

If you drop 3.9 from the metadata, I don't think there's any need to secretly 
keep support. It's too hard to actually use it, and it's not guaranteed to 
work; it would be better to just tell anyone needing 3.9 to use a beta version 
when it was still supported.

(Rant below)

To be fair, I've never understood NEP 29's need to limit Python versions to 42 
months after the 2.7 issue was resolved with official Python EoL. Now there's a 
standard (60 months, exactly 5 versions), and almost all the rest of the 
ecosystem supports it. This just wedges a divide in the ecosystem between 
"scientific" and "everyone else". It makes me have to think "is this a 
scientific Python project? Or a general Python project?" when I really 
shouldn't have to on every project.

I really didn't understand SPEC 0's _tightening_ it to 36 months (and I was at 
the developer summit where this was decided, and stated I was interested in 
being involved in this, but was never included in any discussion on it, so not 
sure how this was even decided). Dropping Python doesn't hurt projects that are 
mostly stable, but ones that are not are really hurt by it. Python 3.8 is still 
heavily used; people don't mind that NumPy dropped 3.8 support because an older 
version works fine. But if there's a major change, then it makes smaller or new 
projects have to do extra work.

Current numbers (as of May 4th) for downloads of manylinux wheels:
* 2.7: 2%
* 3.5: 0.3%
* 3.6: 7.4%
* 3.7: 20.4%
* 3.8: 23.0%
* 3.9: 15.3%
* 3.10: 20.8%
* 3.11: 8.4%
* 3.12: 2.3%

So only 30% of users have Python 3.10+ or newer. Most smaller or newer projects 
can more than double their user base by supporting  3.8+. I could even argue 
that 3.7+ is still helpful for a new project. Once a library is large and 
stable, then it can go higher, even 3.13+ and not hurt anyone unless there's a 
major development.

Little rant finished. :)
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Tue, 2024-05-07 at 11:41 +0200, Gael Varoquaux wrote:
> On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> > make `pip install scikit-image==0.22` work if that version of
> > scikit-image depends on an unconstrained numpy version.
> 
> Would an option be for the scikit-image maintainers to release a
> version of scikit-image 0.22 (like 0.22.1) with a constraint numpy
> version?


I don't think it helps, pip will just skip that version and pick the
prevous one.  IIUC, the one thing you could do is release a new version
without a constraint that raises a detailed/informative error message
at runtime.
I.e. "work around" pip by telling users exactly what they should do.

- Sebastian


> 
> Gaël
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net
> 


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ilhan Polat]

I guess this is also a mandatory read after Henry's blog post appeared that
we had an extensive discussion with Python devs
https://discuss.python.org/t/requires-python-upper-limits/12663

On Tue, May 7, 2024 at 11:35 AM Sebastian Berg 
wrote:

> On Tue, 2024-05-07 at 15:46 +1000, Juan Nunez-Iglesias wrote:
> > On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
> > > This problem could have been avoided by proper use of upper bounds.
> > > Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug
> > > in scikit-image (definitely for ABI reasons, and arguably also for
> > > API reasons). It would really be useful if downstream packages
> > > started to take adding upper bounds correctly more seriously. E.g.,
> > > SciPy has always done it right, so the failure mode that this
> > > thread is about doesn't exist for SciPy. That said, this ship has
> > > sailed for 2.0 - most packages don't have upper bounds in some or
> > > all of their recent releases.
> >
> > I don't think this is a downstream problem, I think this is a "PyPI
> > has immutable metadata" problem. I'm a big fan of Henry Schreiner's
> > "Should You Use Upper Bound Version Constraints" <
> > https://iscinumpy.dev/post/bound-version-constraints/>, where he
> > argues convincingly that the answer is almost always no. This
> > highlighted bit contains the gist:
>
>
>
> Yes, that is all because of `pip` limitations, but those limitations
> are a given.  And I think it is unfortunate/odd that it effectively
> argues that the lower in the stack you are,  the fewer version you
> should support.
>
> But, with the clarification we have that there may be a lot of packages
> that never support both Python 3.9 and NumPy 2.
> That means not publishing for 3.9 may end up helping quite a lot of
> users who would have to downgrade NumPy explicitly.
>
> If that seems the case, that is an unfortunate, but good, argument for
> dropping 3.9.
>
> I don't have an idea for how many users we'll effectively help, or if
> we do the opposite because an application (more than library) wants to
> just use NumPy 2 always but still support Python 3.9.
> But it seems to me that is what the decision comes down to, and I can
> believe that it'll be a lot of hassle saved for `pip` installing users.
> (Note that skimage users will hit cython, so should get a relatively
> clear printout that includes a "please downgrade NumPy" suggestion.)
>
> - Sebastian
>
>
>
> >
> > > A library that requires a manual version intervention is not
> > > broken, it’s just irritating. A library that can’t be installed due
> > > to a version conflict is broken. If that version conflict is fake,
> > > then you’ve created an unsolvable problem where one didn’t exist.
> >
> > Dropping Py 3.9 will fix the issue for a subset of users, but
> > certainly not all users. Someone who pip installs scikit-image==0.22
> > on Py 3.10 will have a broken install. But importantly, they will be
> > able to fix it in user space.
> >
> > At any rate, it's not like NumPy (or SciPy, or scikit-image) don't
> > change APIs over several minor versions. Quoting Henry again:
> >
> > > Quite ironically, the better a package follows SemVer, the smaller
> > > the change will trigger a major version, and therefore the less
> > > likely a major version will break a particular downstream code.
> >
> > In short, and independent of the Py3.9 issue, I don't think we should
> > advocate for upper caps in general, because in general it is
> > impossible to know whether an update is going to break your library,
> > regardless of their SemVer practices, and a fake upper pin is worse
> > than no upper pin.
> >
> > Juan.
> > ___
> > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > To unsubscribe send an email to numpy-discussion-le...@python.org
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > Member address: sebast...@sipsolutions.net
>
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ilhanpo...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Tue, May 7, 2024 at 11:44 AM Gael Varoquaux <
gael.varoqu...@normalesup.org> wrote:

> On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> > make `pip install scikit-image==0.22` work if that version of
> scikit-image depends on an unconstrained numpy version.
>
> Would an option be for the scikit-image maintainers to release a version
> of scikit-image 0.22 (like 0.22.1) with a constraint numpy version?
>

Yes indeed, that would fix this (not for 0.21 of course, but there's only
so much that can be done here).

Doing exactly that is also the first point in the guidance in our 2.0
announcements issue: https://github.com/numpy/numpy/issues/24300. And I
asked specifically for scikit-image to do this months ago:
https://github.com/scikit-image/scikit-image/issues/7282#issuecomment-1885659412
.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Gael Varoquaux]

On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> make `pip install scikit-image==0.22` work if that version of scikit-image 
> depends on an unconstrained numpy version.

Would an option be for the scikit-image maintainers to release a version of 
scikit-image 0.22 (like 0.22.1) with a constraint numpy version?

Gaël
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Tue, 2024-05-07 at 15:46 +1000, Juan Nunez-Iglesias wrote:
> On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
> > This problem could have been avoided by proper use of upper bounds.
> > Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug
> > in scikit-image (definitely for ABI reasons, and arguably also for
> > API reasons). It would really be useful if downstream packages
> > started to take adding upper bounds correctly more seriously. E.g.,
> > SciPy has always done it right, so the failure mode that this
> > thread is about doesn't exist for SciPy. That said, this ship has
> > sailed for 2.0 - most packages don't have upper bounds in some or
> > all of their recent releases.
> 
> I don't think this is a downstream problem, I think this is a "PyPI
> has immutable metadata" problem. I'm a big fan of Henry Schreiner's
> "Should You Use Upper Bound Version Constraints" <
> https://iscinumpy.dev/post/bound-version-constraints/>, where he
> argues convincingly that the answer is almost always no. This
> highlighted bit contains the gist:



Yes, that is all because of `pip` limitations, but those limitations
are a given.  And I think it is unfortunate/odd that it effectively
argues that the lower in the stack you are,  the fewer version you
should support.

But, with the clarification we have that there may be a lot of packages
that never support both Python 3.9 and NumPy 2.
That means not publishing for 3.9 may end up helping quite a lot of
users who would have to downgrade NumPy explicitly.

If that seems the case, that is an unfortunate, but good, argument for
dropping 3.9.

I don't have an idea for how many users we'll effectively help, or if
we do the opposite because an application (more than library) wants to
just use NumPy 2 always but still support Python 3.9.
But it seems to me that is what the decision comes down to, and I can
believe that it'll be a lot of hassle saved for `pip` installing users.
(Note that skimage users will hit cython, so should get a relatively
clear printout that includes a "please downgrade NumPy" suggestion.)

- Sebastian



> 
> > A library that requires a manual version intervention is not
> > broken, it’s just irritating. A library that can’t be installed due
> > to a version conflict is broken. If that version conflict is fake,
> > then you’ve created an unsolvable problem where one didn’t exist.
> 
> Dropping Py 3.9 will fix the issue for a subset of users, but
> certainly not all users. Someone who pip installs scikit-image==0.22
> on Py 3.10 will have a broken install. But importantly, they will be
> able to fix it in user space.
> 
> At any rate, it's not like NumPy (or SciPy, or scikit-image) don't
> change APIs over several minor versions. Quoting Henry again:
> 
> > Quite ironically, the better a package follows SemVer, the smaller
> > the change will trigger a major version, and therefore the less
> > likely a major version will break a particular downstream code.
> 
> In short, and independent of the Py3.9 issue, I don't think we should
> advocate for upper caps in general, because in general it is
> impossible to know whether an update is going to break your library,
> regardless of their SemVer practices, and a fake upper pin is worse
> than no upper pin.
> 
> Juan.
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Tue, May 7, 2024 at 7:48 AM Juan Nunez-Iglesias  wrote:

> On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
>
> This problem could have been avoided by proper use of upper bounds.
> Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in
> scikit-image (definitely for ABI reasons, and arguably also for API
> reasons). It would really be useful if downstream packages started to take
> adding upper bounds correctly more seriously. E.g., SciPy has always done
> it right, so the failure mode that this thread is about doesn't exist for
> SciPy. That said, this ship has sailed for 2.0 - most packages don't have
> upper bounds in some or all of their recent releases.
>
>
> I don't think this is a downstream problem,
>

This is definitely a bug in scikit-image (see below), that is really not up
for discussion here. If you use the NumPy C API, you MUST add a constraint
to stay below the next major version.


> I think this is a "PyPI has immutable metadata" problem.
>

Yes, I'm very much aware:
https://pypackaging-native.github.io/key-issues/pypi_metadata_handling/#impact-of-immutable-metadata.
This just is what it is though, we have to live within the constraints of
how PyPI is designed.


> I'm a big fan of Henry Schreiner's "Should You Use Upper Bound Version
> Constraints" ,
> where he argues convincingly that the answer is almost always no.
>

Emphasis on "almost". Unfortunately, too many maintainers took away the
wrong message from that blog post - probably because it was written in
frustration at Poetry *defaulting* to adding upper bounds, and hence is a
bit too strong on the arguments against. The point was "default to adding
no bounds, only add them if you must" (because on average they do more harm
than good). Translating that to "never add any upper bounds" is wrong and
harmful.

To quote Henry himself from another recent discussion where a maintainer
incorrectly used this blog post to argue against upper bounds (
https://github.com/sympy/sympy/issues/26273#issuecomment-2028192560):

*"You are correct, the post is not meant to say you can't upper cap (except
the Python version, which doesn't even work correctly if you upper cap) if
you've thought about it and have a reason to. It does increase friction,
but if there's a high chance the next version(s) will break you and you are
willing to make releases shortly after the dependency does & keep the cap
up to date, go ahead and cap."*

In this case, it's not just a high chance that things will break, it's
guaranteed to break. NumPy does an import check for ABI mismatch, and will
always raise an exception if you build against 1.x and run with 2.x (or 2.x
and 3.x, etc.). So you must add `<2` when building against numpy 1.x, and
`<3` when building against 2.x. Leaving out the upper bound achieves
exactly nothing except for making things break for the users of your
package.

Dropping Py 3.9 will fix the issue for a subset of users, but certainly not
> all users. Someone who pip installs scikit-image==0.22 on Py 3.10 will have
> a broken install. But importantly, they will be able to fix it in user
> space.
>

Yes, this cannot be fixed on the NumPy side either way. But that's a
separate issue. There are three cases we have to think about:

1. Everything unconstrained: `pip install scikit-image` or `pip install
scikit-image numpy`
2. Everything constrained: `pip install scikit-image==0.22 numpy==1.26`  (+
everything else, typically used through a lock file)
3. A mix of constrained and unconstrained: `pip install scikit-image==0.22`
or `pip install scikit-image numpy=2.0` or `pip install scikit-image==0.22
numpy`

(2) is typically robust since lock files are usually generated from
already-working envs. Unless the list of packages is incomplete, and then
it may turn into (3).

(3) is up to the authors of each package, there's nothing we can do in
numpy to make `pip install scikit-image==0.22` work if that version of
scikit-image depends on an unconstrained numpy version.

(1) is what this thread, and the rollout plans for NumPy 2.0, are concerned
about. If we drop Python 3.9, we make that work more reliably. If we don't,
then on Python 3.9 a plain `pip install scikit-image` will be broken.


>
> At any rate, it's not like NumPy (or SciPy, or scikit-image) don't change
> APIs over several minor versions.
>

We have extensive docs on this, and a documented backwards compatibility
policy that we try hard to adhere to:
https://numpy.org/neps/nep-0023-backwards-compatibility.html
https://numpy.org/devdocs/dev/depending_on_numpy.html#runtime-dependency-version-ranges

I'm honestly a little tired of maintainers just ignoring that and
misunderstanding the problem.

In short, and independent of the Py3.9 issue, I don't think we should
> advocate for upper caps in general
>

No one is doing that "in general". It is specific to NumPy's situation, and
cannot be avoided. It's a price to pay of dealing 

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Mon, May 6, 2024 at 11:43 PM Aaron Meurer  wrote:

> On Mon, May 6, 2024 at 3:05 PM Ralf Gommers 
> wrote:
> >
>
> > So, I think I'm in favor of dropping Python 3.9 support after all to
> prevent problems. It is late in the game, but I do see that we're going to
> cause problems for packages that have already dropped 3.9, and I don't yet
> see an issue with dropping 3.9 for numpy 2.0 now. Does anyone see a
> potential failure mode if we go that way?
>
> Presumably dropping 3.9 support at this point would not mean removing
> anything that would actually break NumPy in Python 3.9. It would just
> mean adding the python_requires metadata and not building a 3.9 wheel.
> So if someone really needs a 3.9-compatible NumPy 2.0 they could build
> a wheel manually.
>
> I'm assuming the python_requires metadata is required though since
> otherwise pip would try to build a wheel from source.
>

Yes indeed, correct on all points.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com