[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Sun, May 12, 2024 at 8:39 PM Gael Varoquaux <
gael.varoqu...@normalesup.org> wrote:

> On Fri, May 10, 2024 at 04:42:49PM +0200, Ralf Gommers wrote:
> > It gets ever-easier to install new Python versions, with
> pyenv/conda/etc. The "my single Python install comes from python.org and
> I'm using the same one because I am afraid to upgrade" is much less of an
> issue than it was 10 years ago. And it's caused mostly by users having
> knowledge gaps. So yes it can be a pain for them, but they'll have to learn
> at some point anyway. Same for "my old HPC cluster uses ..." - it's often
> an even older Python anyway, and you'll have tons of reasons why you don't
> want your cluster-installed stack - learn to use Spack or Conda, and you'll
> be much happier in the long run.
>
> IMHO the view that its a tooling/knowledge gap problem is a bit
> disconnected of users. I meet many users who either
>
> 1. cannot control the Python version they run, or even the base
> environment, because of company culture (decisions at company level on
> these constraints). Maybe upper management in completely misguided here,
> but then upper management must be targeted, and it is not clear at all that
> constraints on packages is the right way to do it, as they typically never
> run any code.
>
> 2. have environments with many dependencies that get in gridlocks of
> dependencies that cannot be mutually satisfied. For my own work it becomes
> increasingly frequent for me to spawn multiple environments that then talk
> to each others (eg via files) to work around these problems.
>
>
> Problem 1 is probably something that user organizations should change, and
> we need to understand why they lock Python versions.


I don't think the problem is what you think it is. It's typically general
IT policies, meaning you cannot do things like download and install random
executables in places like C:\. This is quite common, and it's mostly some
Python installers (python.org ones in particular) falling under a general
policy. It's very rare to have a Python-tooling-specific block - I don't
think I have ever seen something like that. If you can run `pip` and
install to for example locations under your home dir, then you almost
always can also run pyenv/conda/mamba & co to install different Python
versions. And if you cannot run `pip`, there is nothing to discuss w.r.t.
release policy, new package versions don't matter.

It could be a QA issue, and this might reveal both a lack of good practices
> for QA (ie testing) but also the instability of the ecosystems that creates
> a fear of upgrade. We should not be too fast in dismissing these
> organizations as strife with bad practices that could easily be changed, as
> even tech-savy organizations (such as Google, I believe) run into these
> problems.
>

Google (along with many other large tech companies) uses a monorepo. That
means that there is only a single version of any package. This can be a
great strategy once you're large enough; it has costs too, but certainly
shouldn't be qualified as a bad or uninformed strategy. A common support
window like SPEC 0 is actually helpful there.


> Problem 2 is not a problem solvable by users: it comes from the fact that
> dependency version windows are too narrow. Without broad windows on
> dependencies, the more dependencies one has, the more one gets into an
> impossible situation. For this last reason, I strongly advocate that
> packages, in particular core packages, try hard to be as permissible as
> reasonably possible on dependencies.
>

You don't give any details, but the one common place I'm aware of where
narrow support windows are common is when deep learning / CUDA are used.
Many deep learning packages use something like a 3-6 month deprecation
policy, and a 6-12 month support window. And then often depend on 1 or max
2 versions of CUDA and cuDNN. This makes it very challenging to work with
more than one deep learning framework or use them in combination with other
complex packages like Ray. So yes, when you have lots of dependencies like
that, you will have a hard time.

Given you work with deep learning, I'll assume that that's the issue you
are seeing. I don't think we have evidence that a 3 year support window is
too narrow, and we do understand the cost of longer windows. I'll note also
that SPEC 0 says a minimum of 2 years after initial release (so back in
time), but due to backwards compatibility policies the support also goes at
least 1-1.5 years forward in time. So there's a >=3 year window of support
for feature releases of core packages in the scientific Python ecosystem.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Fri, May 10, 2024 at 11:28 PM Brigitta Sipőcz <
b.sipocz+numpyl...@gmail.com> wrote:

> Hi,
>
> Is there any way to know if other large libraries hasn't set an upper pin
> in their last release but since then dropped python version support?
>

This should be doable with either the PyPI JSON API or BigQuery:
https://warehouse.pypa.io/api-reference/json.html
https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/

If someone would want to attempt this, it may be a useful exercise to try
and catch potential issues and contact the maintainers of potentially
affected packages. Help is very much welcome.

This may also be a good place for a thank you to Sebastian, John Kirkham,
Clément Robert and @h-vetinari, who have been following up with many
packages in addition to the ones they maintain themselves, filing/resolving
issues, and helping with the 2.0 migration.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Gael Varoquaux]

On Fri, May 10, 2024 at 04:42:49PM +0200, Ralf Gommers wrote:
> It gets ever-easier to install new Python versions, with pyenv/conda/etc. The 
> "my single Python install comes from python.org and I'm using the same one 
> because I am afraid to upgrade" is much less of an issue than it was 10 years 
> ago. And it's caused mostly by users having knowledge gaps. So yes it can be 
> a pain for them, but they'll have to learn at some point anyway. Same for "my 
> old HPC cluster uses ..." - it's often an even older Python anyway, and 
> you'll have tons of reasons why you don't want your cluster-installed stack - 
> learn to use Spack or Conda, and you'll be much happier in the long run.

IMHO the view that its a tooling/knowledge gap problem is a bit disconnected of 
users. I meet many users who either 

1. cannot control the Python version they run, or even the base environment, 
because of company culture (decisions at company level on these constraints). 
Maybe upper management in completely misguided here, but then upper management 
must be targeted, and it is not clear at all that constraints on packages is 
the right way to do it, as they typically never run any code.

2. have environments with many dependencies that get in gridlocks of 
dependencies that cannot be mutually satisfied. For my own work it becomes 
increasingly frequent for me to spawn multiple environments that then talk to 
each others (eg via files) to work around these problems.



Problem 1 is probably something that user organizations should change, and we 
need to understand why they lock Python versions. It could be a QA issue, and 
this might reveal both a lack of good practices for QA (ie testing) but also 
the instability of the ecosystems that creates a fear of upgrade. We should not 
be too fast in dismissing these organizations as strife with bad practices that 
could easily be changed, as even tech-savy organizations (such as Google, I 
believe) run into these problems.

Problem 2 is not a problem solvable by users: it comes from the fact that 
dependency version windows are too narrow. Without broad windows on 
dependencies, the more dependencies one has, the more one gets into an 
impossible situation. For this last reason, I strongly advocate that packages, 
in particular core packages, try hard to be as permissible as reasonably 
possible on dependencies.

Cheers,

Gaël
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Brigitta Sipőcz]

Hi,

Is there any way to know if other large libraries hasn't set an upper pin
in their last release but since then dropped python version support?

Brigitta

On Fri, 10 May 2024 at 07:47, Ralf Gommers  wrote:

>
>
> On Thu, May 9, 2024 at 12:28 AM Thomas Caswell  wrote:
>
>> I think the spirit of NEP 29 is to pick your supported Python's when you
>> pick a target release date and you should then stick to it (to avoid "we
>> delayed so long we are over a cliff" decisions like this one).
>>
>
> That's true I believe.
>
>
>> We did NEP29 around the same time that Python went from 18 to 12 month
>> releases (my memory is that the cadence change was being considered but not
>> set).  If Python is targeting an 18mo release cycle, then a 36mo window +
>> 2mo delayed Python release could land you in only supporting one version of
>> Python which seems too few (again, my memory could be a bit off).  I was
>> not at the meeting where SPEC 0 was discussed,
>>
>
> Same here, I wasn't at that meeting so am unsure. I only got added to SPEC
> 0 as a co-author at the end because it's mostly a copy of NEP 29. "change
> in release cycle" is probably right as the motivation though.
>
>
>> but I suspect that the narrowing is to better align with an integer
>> number of Python releases while always hitting at least 2 versions of
>> Python.
>>
>
> At least 3, right? 36 months is always 3 minor versions.
>
>
>
>> 
>>
>> It is worth considering that CPython has both a concept of "bugfix"
>> (which is 18mo) and "security" (which is 42mo and source-only) support.  It
>> is arguable that by having "new feature" support and binary release
>> targeting a given version of Python for 36mo we are supporting a given
>> minor version of Python longer than upstream [speaking for myself I would
>> sign onto a proposal to do security release against the last version of our
>> libraries for all versions of Python that still have security support].
>>
>> 
>>
>> > So only 30% of users have Python 3.10+ or newer. Most smaller or newer
>> projects can more than double their user base by supporting  3.8+. I could
>> even argue that 3.7+ is still helpful for a new project. Once a library is
>> large and stable, then it can go higher, even 3.13+ and not hurt anyone
>> unless there's a major development.
>>
>> I have the exact opposite view of Henry here, I think for new projects
>> you should be very aggressive and only support the newest Python version as
>> you don't have any users (so do not have to be responsible yet)!
>>
>
> I think what you do as a new project author totally depends on your goals
> and priorities. Either way, the argument that only so few users "have a new
> Python" seems a bit off-target. That's not how people think - they look for
> new packages to use when they need functionality, and once they find
> something that fits their needs, they make it work. Nor are the statistics
> reflective of usage needs, especially for manylinux - it's more that
> production deployments stay on the same version of Python for their own
> lifetime I suspect. But such deployments also pin all their dependencies,
> so they're irrelevant to new versions/projects.
>
> It gets ever-easier to install new Python versions, with pyenv/conda/etc.
> The "my single Python install comes from python.org and I'm using the
> same one because I am afraid to upgrade" is much less of an issue than it
> was 10 years ago. And it's caused mostly by users having knowledge gaps. So
> yes it can be a pain for them, but they'll have to learn at some point
> anyway. Same for "my old HPC cluster uses ..." - it's often an even older
> Python anyway, and you'll have tons of reasons why you don't want your
> cluster-installed stack - learn to use Spack or Conda, and you'll be much
> happier in the long run.
>
> ---
>
> Back to the topic of dropping 3.9 here: there seem to be some minor
> concerns. Tom is right about the spirit of NEP29/SPEC0. And Sebastian is
> partially right I think: not about `pip install scipy-image==0.22`, because
> that should be picking 0.22.1 plus an older numpy; but `==0.22.0` instead
> cannot be fixed anyway, and that's perhaps a more common way of spelling an
> `==` constraint. So the last-minute dropping will at most have a marginally
> useful impact.
>
> Since Mark has started working on doing a single release of scikit-image
> supporting Python 3.9 (
> https://github.com/scikit-image/scikit-image/pull/7412), perhaps we can
> close the book on this request now, and decide to not change anything?
> I.e., we do release with 3.9 support as planned.
>
> Cheers,
> Ralf
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: b.sipocz+numpyl...@gmail.com
>
___
NumPy-Discussion mailing 

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Charles R Harris]

On Fri, May 10, 2024 at 8:47 AM Ralf Gommers  wrote:

>
>
> On Thu, May 9, 2024 at 12:28 AM Thomas Caswell  wrote:
>
>> I think the spirit of NEP 29 is to pick your supported Python's when you
>> pick a target release date and you should then stick to it (to avoid "we
>> delayed so long we are over a cliff" decisions like this one).
>>
>
> That's true I believe.
>
>
>> We did NEP29 around the same time that Python went from 18 to 12 month
>> releases (my memory is that the cadence change was being considered but not
>> set).  If Python is targeting an 18mo release cycle, then a 36mo window +
>> 2mo delayed Python release could land you in only supporting one version of
>> Python which seems too few (again, my memory could be a bit off).  I was
>> not at the meeting where SPEC 0 was discussed,
>>
>
> Same here, I wasn't at that meeting so am unsure. I only got added to SPEC
> 0 as a co-author at the end because it's mostly a copy of NEP 29. "change
> in release cycle" is probably right as the motivation though.
>
>
>> but I suspect that the narrowing is to better align with an integer
>> number of Python releases while always hitting at least 2 versions of
>> Python.
>>
>
> At least 3, right? 36 months is always 3 minor versions.
>
>
>
>> 
>>
>> It is worth considering that CPython has both a concept of "bugfix"
>> (which is 18mo) and "security" (which is 42mo and source-only) support.  It
>> is arguable that by having "new feature" support and binary release
>> targeting a given version of Python for 36mo we are supporting a given
>> minor version of Python longer than upstream [speaking for myself I would
>> sign onto a proposal to do security release against the last version of our
>> libraries for all versions of Python that still have security support].
>>
>> 
>>
>> > So only 30% of users have Python 3.10+ or newer. Most smaller or newer
>> projects can more than double their user base by supporting  3.8+. I could
>> even argue that 3.7+ is still helpful for a new project. Once a library is
>> large and stable, then it can go higher, even 3.13+ and not hurt anyone
>> unless there's a major development.
>>
>> I have the exact opposite view of Henry here, I think for new projects
>> you should be very aggressive and only support the newest Python version as
>> you don't have any users (so do not have to be responsible yet)!
>>
>
> I think what you do as a new project author totally depends on your goals
> and priorities. Either way, the argument that only so few users "have a new
> Python" seems a bit off-target. That's not how people think - they look for
> new packages to use when they need functionality, and once they find
> something that fits their needs, they make it work. Nor are the statistics
> reflective of usage needs, especially for manylinux - it's more that
> production deployments stay on the same version of Python for their own
> lifetime I suspect. But such deployments also pin all their dependencies,
> so they're irrelevant to new versions/projects.
>
> It gets ever-easier to install new Python versions, with pyenv/conda/etc.
> The "my single Python install comes from python.org and I'm using the
> same one because I am afraid to upgrade" is much less of an issue than it
> was 10 years ago. And it's caused mostly by users having knowledge gaps. So
> yes it can be a pain for them, but they'll have to learn at some point
> anyway. Same for "my old HPC cluster uses ..." - it's often an even older
> Python anyway, and you'll have tons of reasons why you don't want your
> cluster-installed stack - learn to use Spack or Conda, and you'll be much
> happier in the long run.
>
> ---
>
> Back to the topic of dropping 3.9 here: there seem to be some minor
> concerns. Tom is right about the spirit of NEP29/SPEC0. And Sebastian is
> partially right I think: not about `pip install scipy-image==0.22`, because
> that should be picking 0.22.1 plus an older numpy; but `==0.22.0` instead
> cannot be fixed anyway, and that's perhaps a more common way of spelling an
> `==` constraint. So the last-minute dropping will at most have a marginally
> useful impact.
>
> Since Mark has started working on doing a single release of scikit-image
> supporting Python 3.9 (
> https://github.com/scikit-image/scikit-image/pull/7412), perhaps we can
> close the book on this request now, and decide to not change anything?
> I.e., we do release with 3.9 support as planned.
>
>
I'll add that numpy 2.1.0 should follow a few months after 2.0.0. Normally,
it would be released in June without Python 3.9 support, but due to the
extended release cycle needed to get 2.0.0 out, it will be delayed.
Current policy is that we always support at least three Python versions,
but because we are not synced with Python, we end up with four. For
instance, 2.1.0 will support three versions on release (3.10-3.12), but
will finish with four after Python 3.13 is released. The current difficulty
is unique to the breaking changes in 2.0.0 

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Thu, May 9, 2024 at 12:28 AM Thomas Caswell  wrote:

> I think the spirit of NEP 29 is to pick your supported Python's when you
> pick a target release date and you should then stick to it (to avoid "we
> delayed so long we are over a cliff" decisions like this one).
>

That's true I believe.


> We did NEP29 around the same time that Python went from 18 to 12 month
> releases (my memory is that the cadence change was being considered but not
> set).  If Python is targeting an 18mo release cycle, then a 36mo window +
> 2mo delayed Python release could land you in only supporting one version of
> Python which seems too few (again, my memory could be a bit off).  I was
> not at the meeting where SPEC 0 was discussed,
>

Same here, I wasn't at that meeting so am unsure. I only got added to SPEC
0 as a co-author at the end because it's mostly a copy of NEP 29. "change
in release cycle" is probably right as the motivation though.


> but I suspect that the narrowing is to better align with an integer number
> of Python releases while always hitting at least 2 versions of Python.
>

At least 3, right? 36 months is always 3 minor versions.



> 
>
> It is worth considering that CPython has both a concept of "bugfix" (which
> is 18mo) and "security" (which is 42mo and source-only) support.  It is
> arguable that by having "new feature" support and binary release
> targeting a given version of Python for 36mo we are supporting a given
> minor version of Python longer than upstream [speaking for myself I would
> sign onto a proposal to do security release against the last version of our
> libraries for all versions of Python that still have security support].
>
> 
>
> > So only 30% of users have Python 3.10+ or newer. Most smaller or newer
> projects can more than double their user base by supporting  3.8+. I could
> even argue that 3.7+ is still helpful for a new project. Once a library is
> large and stable, then it can go higher, even 3.13+ and not hurt anyone
> unless there's a major development.
>
> I have the exact opposite view of Henry here, I think for new projects you
> should be very aggressive and only support the newest Python version as you
> don't have any users (so do not have to be responsible yet)!
>

I think what you do as a new project author totally depends on your goals
and priorities. Either way, the argument that only so few users "have a new
Python" seems a bit off-target. That's not how people think - they look for
new packages to use when they need functionality, and once they find
something that fits their needs, they make it work. Nor are the statistics
reflective of usage needs, especially for manylinux - it's more that
production deployments stay on the same version of Python for their own
lifetime I suspect. But such deployments also pin all their dependencies,
so they're irrelevant to new versions/projects.

It gets ever-easier to install new Python versions, with pyenv/conda/etc.
The "my single Python install comes from python.org and I'm using the same
one because I am afraid to upgrade" is much less of an issue than it was 10
years ago. And it's caused mostly by users having knowledge gaps. So yes it
can be a pain for them, but they'll have to learn at some point anyway.
Same for "my old HPC cluster uses ..." - it's often an even older Python
anyway, and you'll have tons of reasons why you don't want your
cluster-installed stack - learn to use Spack or Conda, and you'll be much
happier in the long run.

---

Back to the topic of dropping 3.9 here: there seem to be some minor
concerns. Tom is right about the spirit of NEP29/SPEC0. And Sebastian is
partially right I think: not about `pip install scipy-image==0.22`, because
that should be picking 0.22.1 plus an older numpy; but `==0.22.0` instead
cannot be fixed anyway, and that's perhaps a more common way of spelling an
`==` constraint. So the last-minute dropping will at most have a marginally
useful impact.

Since Mark has started working on doing a single release of scikit-image
supporting Python 3.9 (
https://github.com/scikit-image/scikit-image/pull/7412), perhaps we can
close the book on this request now, and decide to not change anything?
I.e., we do release with 3.9 support as planned.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Thomas Caswell]

I think the spirit of NEP 29 is to pick your supported Python's when you
pick a target release date and you should then stick to it (to avoid "we
delayed so long we are over a cliff" decisions like this one).

We did NEP29 around the same time that Python went from 18 to 12 month
releases (my memory is that the cadence change was being considered but not
set).  If Python is targeting an 18mo release cycle, then a 36mo window +
2mo delayed Python release could land you in only supporting one version of
Python which seems too few (again, my memory could be a bit off).  I was
not at the meeting where SPEC 0 was discussed, but I suspect that the
narrowing is to better align with an integer number of Python releases
while always hitting at least 2 versions of Python.



It is worth considering that CPython has both a concept of "bugfix" (which
is 18mo) and "security" (which is 42mo and source-only) support.  It is
arguable that by having "new feature" support and binary release
targeting a given version of Python for 36mo we are supporting a given
minor version of Python longer than upstream [speaking for myself I would
sign onto a proposal to do security release against the last version of our
libraries for all versions of Python that still have security support].



> So only 30% of users have Python 3.10+ or newer. Most smaller or newer
projects can more than double their user base by supporting  3.8+. I could
even argue that 3.7+ is still helpful for a new project. Once a library is
large and stable, then it can go higher, even 3.13+ and not hurt anyone
unless there's a major development.

I have the exact opposite view of Henry here, I think for new projects you
should be very aggressive and only support the newest Python version as you
don't have any users (so do not have to be responsible yet)!



A very bad idea: numpy2 should require py312 to buy us a bunch more time to
practically work through these issues.

Tom

On Wed, May 8, 2024 at 5:27 PM Aaron Meurer  wrote:

> On Tue, May 7, 2024 at 6:03 AM Henry Schreiner 
> wrote:
> >
> > This will be messier for projects building wheels and wanting to support
> non-EoL Python versions. To build a wheel with anything other than
> pybind11, you now need the oldest supported NumPy for Python < 3.9, the
> latest NumPy 1 for Python 3.9, and NumPy 2 for Python 3.10+. I don't know
> if that's important in the decision, but thought I'd point it out. Also,
> according to NEP 29, 3.10+ only became the requirement a couple of weeks
> ago, while it has been months since SPEC 0 dropped it. I don't think either
> document really details what to do when there's a really long development
> cycle that spans a cutoff date.
> >
> > If you drop 3.9 from the metadata, I don't think there's any need to
> secretly keep support. It's too hard to actually use it, and it's not
> guaranteed to work; it would be better to just tell anyone needing 3.9 to
> use a beta version when it was still supported.
> >
> > (Rant below)
> >
> > To be fair, I've never understood NEP 29's need to limit Python versions
> to 42 months after the 2.7 issue was resolved with official Python EoL. Now
> there's a standard (60 months, exactly 5 versions), and almost all the rest
> of the ecosystem supports it. This just wedges a divide in the ecosystem
> between "scientific" and "everyone else". It makes me have to think "is
> this a scientific Python project? Or a general Python project?" when I
> really shouldn't have to on every project.
> >
> > I really didn't understand SPEC 0's _tightening_ it to 36 months (and I
> was at the developer summit where this was decided, and stated I was
> interested in being involved in this, but was never included in any
> discussion on it, so not sure how this was even decided). Dropping Python
> doesn't hurt projects that are mostly stable, but ones that are not are
> really hurt by it. Python 3.8 is still heavily used; people don't mind that
> NumPy dropped 3.8 support because an older version works fine. But if
> there's a major change, then it makes smaller or new projects have to do
> extra work.
>
> I was never part of this discussion, but I totally support tightening
> the support window. Supporting more Python versions is not free for
> projects. Every version supported is another multiple in the CI
> matrix, and if your support window is X years that means you have to
> wait X years to use any new Python feature.
>
> The real issue here is Python's ridiculous faster release cadence. If
> package maintainers don't want to support 5 Python versions at once,
> then maybe that's a sign that CPython shouldn't either.
>
> I'll make the same argument now that I made in 2016 about dropping 2.7
> support https://www.asmeurer.com/blog/posts/moving-away-from-python-2/.
> If users want to use older versions of Python, that's their
> prerogative, but they then have no reason to also expect to be able to
> use the latest versions of libraries.
>
> Aaron Meurer
>
> 

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Aaron Meurer]

On Tue, May 7, 2024 at 6:03 AM Henry Schreiner  wrote:
>
> This will be messier for projects building wheels and wanting to support 
> non-EoL Python versions. To build a wheel with anything other than pybind11, 
> you now need the oldest supported NumPy for Python < 3.9, the latest NumPy 1 
> for Python 3.9, and NumPy 2 for Python 3.10+. I don't know if that's 
> important in the decision, but thought I'd point it out. Also, according to 
> NEP 29, 3.10+ only became the requirement a couple of weeks ago, while it has 
> been months since SPEC 0 dropped it. I don't think either document really 
> details what to do when there's a really long development cycle that spans a 
> cutoff date.
>
> If you drop 3.9 from the metadata, I don't think there's any need to secretly 
> keep support. It's too hard to actually use it, and it's not guaranteed to 
> work; it would be better to just tell anyone needing 3.9 to use a beta 
> version when it was still supported.
>
> (Rant below)
>
> To be fair, I've never understood NEP 29's need to limit Python versions to 
> 42 months after the 2.7 issue was resolved with official Python EoL. Now 
> there's a standard (60 months, exactly 5 versions), and almost all the rest 
> of the ecosystem supports it. This just wedges a divide in the ecosystem 
> between "scientific" and "everyone else". It makes me have to think "is this 
> a scientific Python project? Or a general Python project?" when I really 
> shouldn't have to on every project.
>
> I really didn't understand SPEC 0's _tightening_ it to 36 months (and I was 
> at the developer summit where this was decided, and stated I was interested 
> in being involved in this, but was never included in any discussion on it, so 
> not sure how this was even decided). Dropping Python doesn't hurt projects 
> that are mostly stable, but ones that are not are really hurt by it. Python 
> 3.8 is still heavily used; people don't mind that NumPy dropped 3.8 support 
> because an older version works fine. But if there's a major change, then it 
> makes smaller or new projects have to do extra work.

I was never part of this discussion, but I totally support tightening
the support window. Supporting more Python versions is not free for
projects. Every version supported is another multiple in the CI
matrix, and if your support window is X years that means you have to
wait X years to use any new Python feature.

The real issue here is Python's ridiculous faster release cadence. If
package maintainers don't want to support 5 Python versions at once,
then maybe that's a sign that CPython shouldn't either.

I'll make the same argument now that I made in 2016 about dropping 2.7
support https://www.asmeurer.com/blog/posts/moving-away-from-python-2/.
If users want to use older versions of Python, that's their
prerogative, but they then have no reason to also expect to be able to
use the latest versions of libraries.

Aaron Meurer

>
> Current numbers (as of May 4th) for downloads of manylinux wheels:
> * 2.7: 2%
> * 3.5: 0.3%
> * 3.6: 7.4%
> * 3.7: 20.4%
> * 3.8: 23.0%
> * 3.9: 15.3%
> * 3.10: 20.8%
> * 3.11: 8.4%
> * 3.12: 2.3%
>
> So only 30% of users have Python 3.10+ or newer. Most smaller or newer 
> projects can more than double their user base by supporting  3.8+. I could 
> even argue that 3.7+ is still helpful for a new project. Once a library is 
> large and stable, then it can go higher, even 3.13+ and not hurt anyone 
> unless there's a major development.
>
> Little rant finished. :)
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Mon, 2024-05-06 at 22:39 +, Henry Schreiner wrote:
> This will be messier for projects building wheels and wanting to
> support non-EoL Python versions. To build a wheel with anything other
> than pybind11, you now need the oldest supported NumPy for Python <
> 3.9, the latest NumPy 1 for Python 3.9, and NumPy 2 for Python 3.10+.
> I don't know if that's important in the decision, but thought I'd
> point it out. Also, according to NEP 29, 3.10+ only became the
> requirement a couple of weeks ago, while it has been months since
> SPEC 0 dropped it. I don't think either document really details what
> to do when there's a really long development cycle that spans a
> cutoff date.


FWIW, I have heard similar opinions now.  Supporting a stack of
libraries for all non-EoL Python versions is harder if NumPy must be
different.
The biggest problem would be if you end up with full support for only
NumPy 1 or NumPy 2 (i.e. similar to what numba has to do due to
promotion).  I hope that is rare enough that it doesn't matter, but I
can't say I am sure.
(And yeah, if that happens, we might see the ask of downstream to
support 3.9 and NumPy 2 in a release.  And trying to avoid that was
part of why the discussion started I think.)

- Sebastian


> 
> If you drop 3.9 from the metadata, I don't think there's any need to
> secretly keep support. It's too hard to actually use it, and it's not
> guaranteed to work; it would be better to just tell anyone needing
> 3.9 to use a beta version when it was still supported.
> 
> (Rant below)
> 
> To be fair, I've never understood NEP 29's need to limit Python
> versions to 42 months after the 2.7 issue was resolved with official
> Python EoL. Now there's a standard (60 months, exactly 5 versions),
> and almost all the rest of the ecosystem supports it. This just
> wedges a divide in the ecosystem between "scientific" and "everyone
> else". It makes me have to think "is this a scientific Python
> project? Or a general Python project?" when I really shouldn't have
> to on every project.
> 
> I really didn't understand SPEC 0's _tightening_ it to 36 months (and
> I was at the developer summit where this was decided, and stated I
> was interested in being involved in this, but was never included in
> any discussion on it, so not sure how this was even decided).
> Dropping Python doesn't hurt projects that are mostly stable, but
> ones that are not are really hurt by it. Python 3.8 is still heavily
> used; people don't mind that NumPy dropped 3.8 support because an
> older version works fine. But if there's a major change, then it
> makes smaller or new projects have to do extra work.
> 
> Current numbers (as of May 4th) for downloads of manylinux wheels:
> * 2.7: 2%
> * 3.5: 0.3%
> * 3.6: 7.4%
> * 3.7: 20.4%
> * 3.8: 23.0%
> * 3.9: 15.3%
> * 3.10: 20.8%
> * 3.11: 8.4%
> * 3.12: 2.3%
> 
> So only 30% of users have Python 3.10+ or newer. Most smaller or
> newer projects can more than double their user base by supporting 
> 3.8+. I could even argue that 3.7+ is still helpful for a new
> project. Once a library is large and stable, then it can go higher,
> even 3.13+ and not hurt anyone unless there's a major development.
> 
> Little rant finished. :)
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net
> 


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Mark Harfouche]

I'm sorry to have used scikit-image as an example as it is an active
project (with Juan and other core dev even replying to this thread)
and we can try to release 0.22.1 just for this purpose.
But, what I'm trying to do is to avoid the churn that many other developers
will have when this gets rolled out to a wide audience when they have to
backport similar fixes. (Truthfully, i'm haunted by the LTS of scikit-image
where we kept 0.14.X alive and backported fixes for it.)

However, it seems that many are in favor of keeping support because enough
safeguards have put in place.

I'm excited to see the rollout whatever may happen!
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Henry Schreiner]

This will be messier for projects building wheels and wanting to support 
non-EoL Python versions. To build a wheel with anything other than pybind11, 
you now need the oldest supported NumPy for Python < 3.9, the latest NumPy 1 
for Python 3.9, and NumPy 2 for Python 3.10+. I don't know if that's important 
in the decision, but thought I'd point it out. Also, according to NEP 29, 3.10+ 
only became the requirement a couple of weeks ago, while it has been months 
since SPEC 0 dropped it. I don't think either document really details what to 
do when there's a really long development cycle that spans a cutoff date.

If you drop 3.9 from the metadata, I don't think there's any need to secretly 
keep support. It's too hard to actually use it, and it's not guaranteed to 
work; it would be better to just tell anyone needing 3.9 to use a beta version 
when it was still supported.

(Rant below)

To be fair, I've never understood NEP 29's need to limit Python versions to 42 
months after the 2.7 issue was resolved with official Python EoL. Now there's a 
standard (60 months, exactly 5 versions), and almost all the rest of the 
ecosystem supports it. This just wedges a divide in the ecosystem between 
"scientific" and "everyone else". It makes me have to think "is this a 
scientific Python project? Or a general Python project?" when I really 
shouldn't have to on every project.

I really didn't understand SPEC 0's _tightening_ it to 36 months (and I was at 
the developer summit where this was decided, and stated I was interested in 
being involved in this, but was never included in any discussion on it, so not 
sure how this was even decided). Dropping Python doesn't hurt projects that are 
mostly stable, but ones that are not are really hurt by it. Python 3.8 is still 
heavily used; people don't mind that NumPy dropped 3.8 support because an older 
version works fine. But if there's a major change, then it makes smaller or new 
projects have to do extra work.

Current numbers (as of May 4th) for downloads of manylinux wheels:
* 2.7: 2%
* 3.5: 0.3%
* 3.6: 7.4%
* 3.7: 20.4%
* 3.8: 23.0%
* 3.9: 15.3%
* 3.10: 20.8%
* 3.11: 8.4%
* 3.12: 2.3%

So only 30% of users have Python 3.10+ or newer. Most smaller or newer projects 
can more than double their user base by supporting  3.8+. I could even argue 
that 3.7+ is still helpful for a new project. Once a library is large and 
stable, then it can go higher, even 3.13+ and not hurt anyone unless there's a 
major development.

Little rant finished. :)
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Tue, 2024-05-07 at 11:41 +0200, Gael Varoquaux wrote:
> On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> > make `pip install scikit-image==0.22` work if that version of
> > scikit-image depends on an unconstrained numpy version.
> 
> Would an option be for the scikit-image maintainers to release a
> version of scikit-image 0.22 (like 0.22.1) with a constraint numpy
> version?


I don't think it helps, pip will just skip that version and pick the
prevous one.  IIUC, the one thing you could do is release a new version
without a constraint that raises a detailed/informative error message
at runtime.
I.e. "work around" pip by telling users exactly what they should do.

- Sebastian


> 
> Gaël
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net
> 


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ilhan Polat]

I guess this is also a mandatory read after Henry's blog post appeared that
we had an extensive discussion with Python devs
https://discuss.python.org/t/requires-python-upper-limits/12663

On Tue, May 7, 2024 at 11:35 AM Sebastian Berg 
wrote:

> On Tue, 2024-05-07 at 15:46 +1000, Juan Nunez-Iglesias wrote:
> > On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
> > > This problem could have been avoided by proper use of upper bounds.
> > > Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug
> > > in scikit-image (definitely for ABI reasons, and arguably also for
> > > API reasons). It would really be useful if downstream packages
> > > started to take adding upper bounds correctly more seriously. E.g.,
> > > SciPy has always done it right, so the failure mode that this
> > > thread is about doesn't exist for SciPy. That said, this ship has
> > > sailed for 2.0 - most packages don't have upper bounds in some or
> > > all of their recent releases.
> >
> > I don't think this is a downstream problem, I think this is a "PyPI
> > has immutable metadata" problem. I'm a big fan of Henry Schreiner's
> > "Should You Use Upper Bound Version Constraints" <
> > https://iscinumpy.dev/post/bound-version-constraints/>, where he
> > argues convincingly that the answer is almost always no. This
> > highlighted bit contains the gist:
>
>
>
> Yes, that is all because of `pip` limitations, but those limitations
> are a given.  And I think it is unfortunate/odd that it effectively
> argues that the lower in the stack you are,  the fewer version you
> should support.
>
> But, with the clarification we have that there may be a lot of packages
> that never support both Python 3.9 and NumPy 2.
> That means not publishing for 3.9 may end up helping quite a lot of
> users who would have to downgrade NumPy explicitly.
>
> If that seems the case, that is an unfortunate, but good, argument for
> dropping 3.9.
>
> I don't have an idea for how many users we'll effectively help, or if
> we do the opposite because an application (more than library) wants to
> just use NumPy 2 always but still support Python 3.9.
> But it seems to me that is what the decision comes down to, and I can
> believe that it'll be a lot of hassle saved for `pip` installing users.
> (Note that skimage users will hit cython, so should get a relatively
> clear printout that includes a "please downgrade NumPy" suggestion.)
>
> - Sebastian
>
>
>
> >
> > > A library that requires a manual version intervention is not
> > > broken, it’s just irritating. A library that can’t be installed due
> > > to a version conflict is broken. If that version conflict is fake,
> > > then you’ve created an unsolvable problem where one didn’t exist.
> >
> > Dropping Py 3.9 will fix the issue for a subset of users, but
> > certainly not all users. Someone who pip installs scikit-image==0.22
> > on Py 3.10 will have a broken install. But importantly, they will be
> > able to fix it in user space.
> >
> > At any rate, it's not like NumPy (or SciPy, or scikit-image) don't
> > change APIs over several minor versions. Quoting Henry again:
> >
> > > Quite ironically, the better a package follows SemVer, the smaller
> > > the change will trigger a major version, and therefore the less
> > > likely a major version will break a particular downstream code.
> >
> > In short, and independent of the Py3.9 issue, I don't think we should
> > advocate for upper caps in general, because in general it is
> > impossible to know whether an update is going to break your library,
> > regardless of their SemVer practices, and a fake upper pin is worse
> > than no upper pin.
> >
> > Juan.
> > ___
> > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > To unsubscribe send an email to numpy-discussion-le...@python.org
> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > Member address: sebast...@sipsolutions.net
>
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ilhanpo...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Tue, May 7, 2024 at 11:44 AM Gael Varoquaux <
gael.varoqu...@normalesup.org> wrote:

> On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> > make `pip install scikit-image==0.22` work if that version of
> scikit-image depends on an unconstrained numpy version.
>
> Would an option be for the scikit-image maintainers to release a version
> of scikit-image 0.22 (like 0.22.1) with a constraint numpy version?
>

Yes indeed, that would fix this (not for 0.21 of course, but there's only
so much that can be done here).

Doing exactly that is also the first point in the guidance in our 2.0
announcements issue: https://github.com/numpy/numpy/issues/24300. And I
asked specifically for scikit-image to do this months ago:
https://github.com/scikit-image/scikit-image/issues/7282#issuecomment-1885659412
.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Gael Varoquaux]

On Tue, May 07, 2024 at 11:31:02AM +0200, Ralf Gommers wrote:
> make `pip install scikit-image==0.22` work if that version of scikit-image 
> depends on an unconstrained numpy version.

Would an option be for the scikit-image maintainers to release a version of 
scikit-image 0.22 (like 0.22.1) with a constraint numpy version?

Gaël
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Tue, 2024-05-07 at 15:46 +1000, Juan Nunez-Iglesias wrote:
> On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
> > This problem could have been avoided by proper use of upper bounds.
> > Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug
> > in scikit-image (definitely for ABI reasons, and arguably also for
> > API reasons). It would really be useful if downstream packages
> > started to take adding upper bounds correctly more seriously. E.g.,
> > SciPy has always done it right, so the failure mode that this
> > thread is about doesn't exist for SciPy. That said, this ship has
> > sailed for 2.0 - most packages don't have upper bounds in some or
> > all of their recent releases.
> 
> I don't think this is a downstream problem, I think this is a "PyPI
> has immutable metadata" problem. I'm a big fan of Henry Schreiner's
> "Should You Use Upper Bound Version Constraints" <
> https://iscinumpy.dev/post/bound-version-constraints/>, where he
> argues convincingly that the answer is almost always no. This
> highlighted bit contains the gist:



Yes, that is all because of `pip` limitations, but those limitations
are a given.  And I think it is unfortunate/odd that it effectively
argues that the lower in the stack you are,  the fewer version you
should support.

But, with the clarification we have that there may be a lot of packages
that never support both Python 3.9 and NumPy 2.
That means not publishing for 3.9 may end up helping quite a lot of
users who would have to downgrade NumPy explicitly.

If that seems the case, that is an unfortunate, but good, argument for
dropping 3.9.

I don't have an idea for how many users we'll effectively help, or if
we do the opposite because an application (more than library) wants to
just use NumPy 2 always but still support Python 3.9.
But it seems to me that is what the decision comes down to, and I can
believe that it'll be a lot of hassle saved for `pip` installing users.
(Note that skimage users will hit cython, so should get a relatively
clear printout that includes a "please downgrade NumPy" suggestion.)

- Sebastian



> 
> > A library that requires a manual version intervention is not
> > broken, it’s just irritating. A library that can’t be installed due
> > to a version conflict is broken. If that version conflict is fake,
> > then you’ve created an unsolvable problem where one didn’t exist.
> 
> Dropping Py 3.9 will fix the issue for a subset of users, but
> certainly not all users. Someone who pip installs scikit-image==0.22
> on Py 3.10 will have a broken install. But importantly, they will be
> able to fix it in user space.
> 
> At any rate, it's not like NumPy (or SciPy, or scikit-image) don't
> change APIs over several minor versions. Quoting Henry again:
> 
> > Quite ironically, the better a package follows SemVer, the smaller
> > the change will trigger a major version, and therefore the less
> > likely a major version will break a particular downstream code.
> 
> In short, and independent of the Py3.9 issue, I don't think we should
> advocate for upper caps in general, because in general it is
> impossible to know whether an update is going to break your library,
> regardless of their SemVer practices, and a fake upper pin is worse
> than no upper pin.
> 
> Juan.
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Tue, May 7, 2024 at 7:48 AM Juan Nunez-Iglesias  wrote:

> On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
>
> This problem could have been avoided by proper use of upper bounds.
> Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in
> scikit-image (definitely for ABI reasons, and arguably also for API
> reasons). It would really be useful if downstream packages started to take
> adding upper bounds correctly more seriously. E.g., SciPy has always done
> it right, so the failure mode that this thread is about doesn't exist for
> SciPy. That said, this ship has sailed for 2.0 - most packages don't have
> upper bounds in some or all of their recent releases.
>
>
> I don't think this is a downstream problem,
>

This is definitely a bug in scikit-image (see below), that is really not up
for discussion here. If you use the NumPy C API, you MUST add a constraint
to stay below the next major version.


> I think this is a "PyPI has immutable metadata" problem.
>

Yes, I'm very much aware:
https://pypackaging-native.github.io/key-issues/pypi_metadata_handling/#impact-of-immutable-metadata.
This just is what it is though, we have to live within the constraints of
how PyPI is designed.


> I'm a big fan of Henry Schreiner's "Should You Use Upper Bound Version
> Constraints" ,
> where he argues convincingly that the answer is almost always no.
>

Emphasis on "almost". Unfortunately, too many maintainers took away the
wrong message from that blog post - probably because it was written in
frustration at Poetry *defaulting* to adding upper bounds, and hence is a
bit too strong on the arguments against. The point was "default to adding
no bounds, only add them if you must" (because on average they do more harm
than good). Translating that to "never add any upper bounds" is wrong and
harmful.

To quote Henry himself from another recent discussion where a maintainer
incorrectly used this blog post to argue against upper bounds (
https://github.com/sympy/sympy/issues/26273#issuecomment-2028192560):

*"You are correct, the post is not meant to say you can't upper cap (except
the Python version, which doesn't even work correctly if you upper cap) if
you've thought about it and have a reason to. It does increase friction,
but if there's a high chance the next version(s) will break you and you are
willing to make releases shortly after the dependency does & keep the cap
up to date, go ahead and cap."*

In this case, it's not just a high chance that things will break, it's
guaranteed to break. NumPy does an import check for ABI mismatch, and will
always raise an exception if you build against 1.x and run with 2.x (or 2.x
and 3.x, etc.). So you must add `<2` when building against numpy 1.x, and
`<3` when building against 2.x. Leaving out the upper bound achieves
exactly nothing except for making things break for the users of your
package.

Dropping Py 3.9 will fix the issue for a subset of users, but certainly not
> all users. Someone who pip installs scikit-image==0.22 on Py 3.10 will have
> a broken install. But importantly, they will be able to fix it in user
> space.
>

Yes, this cannot be fixed on the NumPy side either way. But that's a
separate issue. There are three cases we have to think about:

1. Everything unconstrained: `pip install scikit-image` or `pip install
scikit-image numpy`
2. Everything constrained: `pip install scikit-image==0.22 numpy==1.26`  (+
everything else, typically used through a lock file)
3. A mix of constrained and unconstrained: `pip install scikit-image==0.22`
or `pip install scikit-image numpy=2.0` or `pip install scikit-image==0.22
numpy`

(2) is typically robust since lock files are usually generated from
already-working envs. Unless the list of packages is incomplete, and then
it may turn into (3).

(3) is up to the authors of each package, there's nothing we can do in
numpy to make `pip install scikit-image==0.22` work if that version of
scikit-image depends on an unconstrained numpy version.

(1) is what this thread, and the rollout plans for NumPy 2.0, are concerned
about. If we drop Python 3.9, we make that work more reliably. If we don't,
then on Python 3.9 a plain `pip install scikit-image` will be broken.


>
> At any rate, it's not like NumPy (or SciPy, or scikit-image) don't change
> APIs over several minor versions.
>

We have extensive docs on this, and a documented backwards compatibility
policy that we try hard to adhere to:
https://numpy.org/neps/nep-0023-backwards-compatibility.html
https://numpy.org/devdocs/dev/depending_on_numpy.html#runtime-dependency-version-ranges

I'm honestly a little tired of maintainers just ignoring that and
misunderstanding the problem.

In short, and independent of the Py3.9 issue, I don't think we should
> advocate for upper caps in general
>

No one is doing that "in general". It is specific to NumPy's situation, and
cannot be avoided. It's a price to pay of dealing 

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Mon, May 6, 2024 at 11:43 PM Aaron Meurer  wrote:

> On Mon, May 6, 2024 at 3:05 PM Ralf Gommers 
> wrote:
> >
>
> > So, I think I'm in favor of dropping Python 3.9 support after all to
> prevent problems. It is late in the game, but I do see that we're going to
> cause problems for packages that have already dropped 3.9, and I don't yet
> see an issue with dropping 3.9 for numpy 2.0 now. Does anyone see a
> potential failure mode if we go that way?
>
> Presumably dropping 3.9 support at this point would not mean removing
> anything that would actually break NumPy in Python 3.9. It would just
> mean adding the python_requires metadata and not building a 3.9 wheel.
> So if someone really needs a 3.9-compatible NumPy 2.0 they could build
> a wheel manually.
>
> I'm assuming the python_requires metadata is required though since
> otherwise pip would try to build a wheel from source.
>

Yes indeed, correct on all points.

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Juan Nunez-Iglesias]

On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote:
> This problem could have been avoided by proper use of upper bounds. 
> Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in 
> scikit-image (definitely for ABI reasons, and arguably also for API reasons). 
> It would really be useful if downstream packages started to take adding upper 
> bounds correctly more seriously. E.g., SciPy has always done it right, so the 
> failure mode that this thread is about doesn't exist for SciPy. That said, 
> this ship has sailed for 2.0 - most packages don't have upper bounds in some 
> or all of their recent releases.

I don't think this is a downstream problem, I think this is a "PyPI has 
immutable metadata" problem. I'm a big fan of Henry Schreiner's "Should You Use 
Upper Bound Version Constraints" 
, where he argues 
convincingly that the answer is almost always no. This highlighted bit contains 
the gist:

> A library that requires a manual version intervention is not broken, it’s 
> just irritating. A library that can’t be installed due to a version conflict 
> is broken. If that version conflict is fake, then you’ve created an 
> unsolvable problem where one didn’t exist.

Dropping Py 3.9 will fix the issue for a subset of users, but certainly not all 
users. Someone who pip installs scikit-image==0.22 on Py 3.10 will have a 
broken install. But importantly, they will be able to fix it in user space.

At any rate, it's not like NumPy (or SciPy, or scikit-image) don't change APIs 
over several minor versions. Quoting Henry again:

> Quite ironically, the better a package follows SemVer, the smaller the change 
> will trigger a major version, and therefore the less likely a major version 
> will break a particular downstream code.

In short, and independent of the Py3.9 issue, I don't think we should advocate 
for upper caps in general, because in general it is impossible to know whether 
an update is going to break your library, regardless of their SemVer practices, 
and a fake upper pin is worse than no upper pin.

Juan.___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Aaron Meurer]

On Mon, May 6, 2024 at 3:05 PM Ralf Gommers  wrote:
>
>
>
> On Mon, May 6, 2024 at 10:42 PM Oscar Benjamin  
> wrote:
>>
>> On Mon, 6 May 2024 at 19:59, Aaron Meurer  wrote:
>> >
>> > On Mon, May 6, 2024 at 6:34 AM Mark Harfouche  
>> > wrote:
>> > >
>> > > I'm asking that you let Python 3.9 support disappear with 1.26, and not 
>> > > "drop a final version" before you decide to move on with 3.10+ only.
>> >
>> > I don't understand NumPy supporting Python 3.9 means you have to also.
>> > A downstream dependent only has to support at most the versions you
>> > do. If NumPy dropped Python 3.9 but you wanted to keep it that would
>> > be a problem, but the reverse isn't because scikit-image depends on
>> > NumPy, not the other way around.
>>
>> A downstream package needs to provide updates for all the same Python
>> versions as its dependencies because otherwise e.g. a NumPy 2.0
>> release for CPython 3.9 breaks dependent packages that no longer
>> support 3.9. Those projects then need to add back support for older
>> Python versions at the same time as putting out an urgent
>> compatibility release. Perhaps usually this is not such an issue but
>> particularly for an intentionally compatibility breaking release
>> sending it out to more than the usual range of Python versions is not
>> helpful for downstream.
>
>
> I think this is an important argument indeed. The initial "follow SPEC 0" is 
> not really relevant, but let's focus on the practical issue at hand. When a 
> user types `pip install numpy scikit-image` in a Python 3.9 environment, 
> where numpy 2.0 would support 3.9 and scikit-image 0.23 does not, then the 
> end result will be that numpy 2.0 and scikit-image 0.22 (incompatible!) will 
> be installed. This is undesirable.
>
> This problem could have been avoided by proper use of upper bounds. 
> Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in 
> scikit-image (definitely for ABI reasons, and arguably also for API reasons). 
> It would really be useful if downstream packages started to take adding upper 
> bounds correctly more seriously. E.g., SciPy has always done it right, so the 
> failure mode that this thread is about doesn't exist for SciPy. That said, 
> this ship has sailed for 2.0 - most packages don't have upper bounds in some 
> or all of their recent releases.
>
> So, I think I'm in favor of dropping Python 3.9 support after all to prevent 
> problems. It is late in the game, but I do see that we're going to cause 
> problems for packages that have already dropped 3.9, and I don't yet see an 
> issue with dropping 3.9 for numpy 2.0 now. Does anyone see a potential 
> failure mode if we go that way?

Presumably dropping 3.9 support at this point would not mean removing
anything that would actually break NumPy in Python 3.9. It would just
mean adding the python_requires metadata and not building a 3.9 wheel.
So if someone really needs a 3.9-compatible NumPy 2.0 they could build
a wheel manually.

I'm assuming the python_requires metadata is required though since
otherwise pip would try to build a wheel from source.

Aaron Meurer

>
> Cheers,
> Ralf
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Ralf Gommers]

On Mon, May 6, 2024 at 10:42 PM Oscar Benjamin 
wrote:

> On Mon, 6 May 2024 at 19:59, Aaron Meurer  wrote:
> >
> > On Mon, May 6, 2024 at 6:34 AM Mark Harfouche 
> wrote:
> > >
> > > I'm asking that you let Python 3.9 support disappear with 1.26, and
> not "drop a final version" before you decide to move on with 3.10+ only.
> >
> > I don't understand NumPy supporting Python 3.9 means you have to also.
> > A downstream dependent only has to support at most the versions you
> > do. If NumPy dropped Python 3.9 but you wanted to keep it that would
> > be a problem, but the reverse isn't because scikit-image depends on
> > NumPy, not the other way around.
>
> A downstream package needs to provide updates for all the same Python
> versions as its dependencies because otherwise e.g. a NumPy 2.0
> release for CPython 3.9 breaks dependent packages that no longer
> support 3.9. Those projects then need to add back support for older
> Python versions at the same time as putting out an urgent
> compatibility release. Perhaps usually this is not such an issue but
> particularly for an intentionally compatibility breaking release
> sending it out to more than the usual range of Python versions is not
> helpful for downstream.
>

I think this is an important argument indeed. The initial "follow SPEC 0"
is not really relevant, but let's focus on the practical issue at hand.
When a user types `pip install numpy scikit-image` in a Python 3.9
environment, where numpy 2.0 would support 3.9 and scikit-image 0.23 does
not, then the end result will be that numpy 2.0 and scikit-image 0.22
(incompatible!) will be installed. This is undesirable.

This problem could have been avoided by proper use of upper bounds.
Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in
scikit-image (definitely for ABI reasons, and arguably also for API
reasons). It would really be useful if downstream packages started to take
adding upper bounds correctly more seriously. E.g., SciPy has always done
it right, so the failure mode that this thread is about doesn't exist for
SciPy. That said, this ship has sailed for 2.0 - most packages don't have
upper bounds in some or all of their recent releases.

So, I think I'm in favor of dropping Python 3.9 support after all to
prevent problems. It is late in the game, but I do see that we're going to
cause problems for packages that have already dropped 3.9, and I don't yet
see an issue with dropping 3.9 for numpy 2.0 now. Does anyone see a
potential failure mode if we go that way?

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Oscar Benjamin]

On Mon, 6 May 2024 at 19:59, Aaron Meurer  wrote:
>
> On Mon, May 6, 2024 at 6:34 AM Mark Harfouche  
> wrote:
> >
> > I'm asking that you let Python 3.9 support disappear with 1.26, and not 
> > "drop a final version" before you decide to move on with 3.10+ only.
>
> I don't understand NumPy supporting Python 3.9 means you have to also.
> A downstream dependent only has to support at most the versions you
> do. If NumPy dropped Python 3.9 but you wanted to keep it that would
> be a problem, but the reverse isn't because scikit-image depends on
> NumPy, not the other way around.

A downstream package needs to provide updates for all the same Python
versions as its dependencies because otherwise e.g. a NumPy 2.0
release for CPython 3.9 breaks dependent packages that no longer
support 3.9. Those projects then need to add back support for older
Python versions at the same time as putting out an urgent
compatibility release. Perhaps usually this is not such an issue but
particularly for an intentionally compatibility breaking release
sending it out to more than the usual range of Python versions is not
helpful for downstream.

--
Oscar
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Aaron Meurer]

On Mon, May 6, 2024 at 6:34 AM Mark Harfouche  wrote:
>
>
> On Mon, May 6, 2024 at 6:46 AM Sebastian Berg  
> wrote:
>>
>> On Mon, 2024-05-06 at 09:17 +1000, Matti Picus wrote:
>> > On 05/05/2024 11:32, Mark Harfouche wrote:
>> >
>> > but to me it makes sense to support it for the
>> > large 2.0 release.
>>
> The release will not be without problems rippling throughout the community. 
> SPEC0 effectively asks downstream packages to:
> "Focus on Python 3.10 in 2024". But the introduction of "numpy 2.0" without 
> updated downstream packages will surely break some.
> If these breakages occur, packages will have to think about whether or not to 
> backport a release for those stuck on Python 3.9.
> This is something that was identified as a potential risk  
> https://github.com/numpy/numpy/issues/26191 and you have all been very 
> proactive in helping resolve the issues.
> I'm asking that you let Python 3.9 support disappear with 1.26, and not "drop 
> a final version" before you decide to move on with 3.10+ only.

I don't understand NumPy supporting Python 3.9 means you have to also.
A downstream dependent only has to support at most the versions you
do. If NumPy dropped Python 3.9 but you wanted to keep it that would
be a problem, but the reverse isn't because scikit-image depends on
NumPy, not the other way around.

>
> On the list above is, Scikit-image, which I am a core developer of, and you 
> have all identified as being compatible with Numpy 2.0 starting 0.23.1 does 
> not have a CPython 3.9 build for 0.23.1 on pypi.
> https://pypi.org/project/scikit-image/0.23.1/#files
> It also lists as being compatible with numpy >=1.22 in its dependencies as 
> this was understood to be the best practice at the time
> https://github.com/scikit-image/scikit-image/blob/v0.22.x/pyproject.toml#L29
>
> Thus a project that uses Python 3.9 + pip + scikit-image will type
>
> pip install scikit-image
>
> and this can pull in numpy 2.0
> I sadly haven't followed too closely with the full list of fixes for numpy 
> 2.0 that were introduced in the 0.23.1 release, but I'm not looking forward 
> to explaining to users why "Python" broke "again" on python 3.9.

I rather expect the packaging issues that will arise when NumPy 2.0 is
released will be more general than this. The central issue being that
no one pinned NumPy < 2 until recently (and I'm sure many aren't even
doing it at all yet), and on PyPI, there's no way to update pinning
metadata except by doing a new patch release. Maybe someone could do
an analysis of the metadata on PyPI to see whether this specific issue
is likely to occur broadly or is just specific to scikit-image.

Aaron Meurer

>
> Scikit-image is an active project, with many eyes on it, and it seems many 
> fixes for Numpy2 were already in the 0.22.0 release, but smaller projects 
> will not have been so proactive.
>
>> I think it is late anyway and NumPy always had a slightly longer
>> support period and that seemed fine especially since NumPy is low in
>> the stack.
>>
>> The SPEC was written to give the community that precedence and show
>> that many agree with you (and numpy endorses it).
>> Maybe the "endorsed by" list should rather be grown to strengthen the
>> argument instead?
>
>
> I am asking that Numpy be included in the list of packages that adhere to it 
> making the whole SPEC stronger.
> "Endorsed" but not adhered to is a much weaker argument than: "this is what 
> numpy follows", "if the policy is good enough for them is good enough for us".
> It's very difficult to convince others when they see numpy as not following 
> things, so they say "well its working now, lets just keep the CIs the same".
> It only grows the support burden for the strained open source we have (our 
> time).
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Mark Harfouche]

On Mon, May 6, 2024 at 6:46 AM Sebastian Berg 
wrote:

> On Mon, 2024-05-06 at 09:17 +1000, Matti Picus wrote:
> > On 05/05/2024 11:32, Mark Harfouche wrote:
> >
> > but to me it makes sense to support it for the
> > large 2.0 release.
>
> The release will not be without problems rippling throughout the
community. SPEC0 effectively asks downstream packages to:
"Focus on Python 3.10 in 2024". But the introduction of "numpy 2.0" without
updated downstream packages will surely break some.
If these breakages occur, packages will have to think about whether or not
to backport a release for those stuck on Python 3.9.
This is something that was identified as a potential risk
https://github.com/numpy/numpy/issues/26191 and you have all been very
proactive in helping resolve the issues.
I'm asking that you let Python 3.9 support disappear with 1.26, and not
"drop a final version" before you decide to move on with 3.10+ only.

On the list above is, Scikit-image, which I am a core developer of, and you
have all identified as being compatible with Numpy 2.0 starting 0.23.1 does
not have a CPython 3.9 build for 0.23.1 on pypi.
https://pypi.org/project/scikit-image/0.23.1/#files
It also lists as being compatible with numpy >=1.22 in its dependencies as
this was understood to be the best practice at the time
https://github.com/scikit-image/scikit-image/blob/v0.22.x/pyproject.toml#L29

Thus a project that uses Python 3.9 + pip + scikit-image will type

pip install scikit-image

and this can pull in numpy 2.0
I sadly haven't followed too closely with the full list of fixes for numpy
2.0 that were introduced in the 0.23.1 release, but I'm not looking forward
to explaining to users why "Python" broke "again" on python 3.9.

Scikit-image is an active project, with many eyes on it, and it seems many
fixes for Numpy2 were already in the 0.22.0 release, but smaller projects
will not have been so proactive.

I think it is late anyway and NumPy always had a slightly longer
> support period and that seemed fine especially since NumPy is low in
> the stack.
>
> The SPEC was written to give the community that precedence and show
> that many agree with you (and numpy endorses it).
> Maybe the "endorsed by" list should rather be grown to strengthen the
> argument instead?
>

I am asking that Numpy be included in the list of packages that adhere to
it making the whole SPEC stronger.
"Endorsed" but not adhered to is a much weaker argument than: "this is what
numpy follows", "if the policy is good enough for them is good enough for
us".
It's very difficult to convince others when they see numpy as not following
things, so they say "well its working now, lets just keep the CIs the same".
It only grows the support burden for the strained open source we have (our
time).
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Sebastian Berg]

On Mon, 2024-05-06 at 09:17 +1000, Matti Picus wrote:
> On 05/05/2024 11:32, Mark Harfouche wrote:
> 

> > 
> > Thank you for considering this last minute request. I know it adds 
> > work at this stage.
> > 
> > Mark
> 
> 
> I think NumPy should not be the leader in dropping versions, rather 
> should be one of the more conservative packages since other packages 
> depend on it. We have indeed dropped 3.9 on HEAD, and will not be 
> supporting it for 2.1, but to me it makes sense to support it for the
> large 2.0 release.


I think it is late anyway and NumPy always had a slightly longer
support period and that seemed fine especially since NumPy is low in
the stack.

The SPEC was written to give the community that precedence and show
that many agree with you (and numpy endorses it).
Maybe the "endorsed by" list should rather be grown to strengthen the
argument instead?

(Of course there are true exceptions, IIRC scikit-learn chooses to have
much longer support windows.)

- Sebastian


> 
> 
> Matti
> 
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net
> 


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Matti Picus]

On 05/05/2024 11:32, Mark Harfouche wrote:

I know this is somewhat the 11th hour for the numpy 2.0 release but as 
downstream packager and user of numpy, I would really like to ask that 
Numpy strives to adhere to NEP29, and SPEC0


SEPC0 makes it pretty clear that Python 3.9 should not be included in 
any "future" versions.


Leading by example in this case will help downstream make the hard 
decision to adhere to the compatibility guidelines from SPEC0.


Thank you for considering this last minute request. I know it adds 
work at this stage.


Mark



I think NumPy should not be the leader in dropping versions, rather 
should be one of the more conservative packages since other packages 
depend on it. We have indeed dropped 3.9 on HEAD, and will not be 
supporting it for 2.1, but to me it makes sense to support it for the 
large 2.0 release.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com