[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Abdur-Rahmaan Janhangeer]

Thanks for info, that's becoming a black spot in the tool it seems.

It's more for production codes, now I can give it a try!

Kind Regards,

Abdur-Rahmaan Janhangeer
about  | blog

github 
Mauritius
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/26XIXA6EQR3N22YERVBWRUSKHDMZLM46/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Thomas Grainger]

I think that's unfortunate, you can still use and format the subset of Python 
3.10 syntax with black.

You already do this on python 3.9 for example black doesn't support 
parenthesized with
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5LWYHTZYZ4WJB7PSN3HWHECITMNHBT6O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Abdur-Rahmaan Janhangeer]

> What I have heard repeatedly, from people who are paid to know, is that most 
> users don’t care about the latest features, and would rather stick to a 
> release until it becomes unsupported. (Extreme example: Python 2.)

Just a quick note: Until black supports py3.10 fully, I'm sticking with py3.9
for dear life.

Kind Regards,

Abdur-Rahmaan Janhangeer
about | blog
github
Mauritius
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IJU6XLQHNNT5PFYTNLT2IPRYPRRTZLWK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Phil Thompson via Python-Dev]

On 05/10/2021 07:59, Nick Coghlan wrote:

On Tue, 28 Sep 2021, 6:55 am Brett Cannon,  wrote:




On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:



However the stable ABI is still a second class citizen as it is still
not possible (AFAIK) to specify a wheel name that doesn't need to
explicitly include each supported Python version (rather than a 
minimum

stable ABI version).



Actually you can do this. The list of compatible wheels for a platform
starts at CPython 3.2 when the stable ABI was introduced and goes 
forward
to the version of Python you are running. So you can build a wheel 
file
that targets the oldest version of CPython that you are targeting and 
its

version of the stable ABI and it is considered forward compatible. See
`python -m pip debug --verbose` for the complete list of wheel tags 
that

are supported for an interpreter.



I think Phil's point is a build side one: as far as I know, the process 
for
getting one of those more generic file names is still to build a wheel 
with

an overly precise name for the stable ABI declarations used, and then
rename it.

The correspondence between "I used these stable ABI declarations in my
module build" and "I can use this more broadly accepted wheel name" is
currently obscure enough that I couldn't tell you off the top of my 
head
how to do it, and I contributed to the design of both sides of the 
equation.


Actually improving the build ergonomics would be hard (and outside
CPython's own scope), but offering a table in the stable ABI docs 
giving

suggested wheel tags for different stable ABI declarations should be
feasible, and would be useful to both folks renaming already built 
wheels

and anyone working on improving the build automation tools.


Actually I was able to do what I wanted without renaming wheels...

Specify 'py_limited_api=True' as an argument to Extension() (using 
setuptools v57.0.0).


Specify...

[bdist_wheel]
py_limited_api = cp36

...in setup.cfg (using wheel v0.34.2).

The resulting wheel has a Python tag of 'cp36' and an ABI tag of 'abi3' 
for all platforms, which is interpreted by the current version of pip 
exactly as I want.


I'm not sure if this is documented anywhere.

Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XVBN3OWN5TAYAKTUYI6MEXATX3I62ZEZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Petr Viktorin]

On 05. 10. 21 8:59, Nick Coghlan wrote:
On Tue, 28 Sep 2021, 6:55 am Brett Cannon, > wrote:




On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev
mailto:python-dev@python.org>> wrote:


However the stable ABI is still a second class citizen as it is
still
not possible (AFAIK) to specify a wheel name that doesn't need to
explicitly include each supported Python version (rather than a
minimum
stable ABI version).


Actually you can do this. The list of compatible wheels for a
platform starts at CPython 3.2 when the stable ABI was introduced
and goes forward to the version of Python you are running. So you
can build a wheel file that targets the oldest version of CPython
that you are targeting and its version of the stable ABI and it is
considered forward compatible. See `python -m pip debug --verbose`
for the complete list of wheel tags that are supported for an
interpreter.


I think Phil's point is a build side one: as far as I know, the process 
for getting one of those more generic file names is still to build a 
wheel with an overly precise name for the stable ABI declarations used, 
and then rename it.


The correspondence between "I used these stable ABI declarations in my 
module build" and "I can use this more broadly accepted wheel name" is 
currently obscure enough that I couldn't tell you off the top of my head 
how to do it, and I contributed to the design of both sides of the equation.


Actually improving the build ergonomics would be hard (and outside 
CPython's own scope), but offering a table in the stable ABI docs giving 
suggested wheel tags for different stable ABI declarations should be 
feasible, and would be useful to both folks renaming already built 
wheels and anyone working on improving the build automation tools.



Indeed, thinking about proper wheel tags, and adding support for them in 
both pip/installer and setuptools/build/poetry/etc., would be a very 
helpful way to contribute to the stable ABI.

I don't think I will be able to get to this any time soon.

The current `abi3` tag doesn't encode the minimum ABI version. AFAIK 
that info should go in the "Requires-Python" wheel metadata, but there's 
not automation or clear guides for that. Putting it in the wheel tag 
might be a good idea.


There are vague ideas floating around about removing old stable ABI 
features (hopefully after they're deprecated for 5-10 years); if there's 
a new wheel tag scheme it should be made with that possibility in mind.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WSAF6H4P2GDORL6KOU6ZIBVUITDYAIBA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nick Coghlan]

On Tue, 28 Sep 2021, 6:55 am Brett Cannon,  wrote:

>
>
> On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
> python-dev@python.org> wrote:
>
>>
>> However the stable ABI is still a second class citizen as it is still
>> not possible (AFAIK) to specify a wheel name that doesn't need to
>> explicitly include each supported Python version (rather than a minimum
>> stable ABI version).
>>
>
> Actually you can do this. The list of compatible wheels for a platform
> starts at CPython 3.2 when the stable ABI was introduced and goes forward
> to the version of Python you are running. So you can build a wheel file
> that targets the oldest version of CPython that you are targeting and its
> version of the stable ABI and it is considered forward compatible. See
> `python -m pip debug --verbose` for the complete list of wheel tags that
> are supported for an interpreter.
>

I think Phil's point is a build side one: as far as I know, the process for
getting one of those more generic file names is still to build a wheel with
an overly precise name for the stable ABI declarations used, and then
rename it.

The correspondence between "I used these stable ABI declarations in my
module build" and "I can use this more broadly accepted wheel name" is
currently obscure enough that I couldn't tell you off the top of my head
how to do it, and I contributed to the design of both sides of the equation.

Actually improving the build ergonomics would be hard (and outside
CPython's own scope), but offering a table in the stable ABI docs giving
suggested wheel tags for different stable ABI declarations should be
feasible, and would be useful to both folks renaming already built wheels
and anyone working on improving the build automation tools.

Cheers,
Nick.


___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/E4EA6GW6N5NREEBM4QOA7GTLBGTGD4AV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NZRLQUMOJB5VD3CW5OQF6AKSNBJ2DW5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Christian Heimes]

On 27/09/2021 16.32, Ronald Oussoren via Python-Dev wrote:



On 26 Sep 2021, at 19:03, Christian Heimes > wrote:


On 26/09/2021 13.07, jack.jan...@cwi.nl  wrote:
The problem with the stable ABI is that very few developers are 
targeting it. I’m not sure why not, whether it has to do with 
incompleteness of the ABI, or with issues targeting it easily and 
your builds and then having pip/PyPI do the right things with wheels 
and all that. I’ve been on the capi-sig mailing list since its 
inception in 2007, but the discussions are really going over my head. 
I don’t understand what the problems are that keep people from 
targeting the stable ABI (or the various other attempts at 
standardising extensions over Python versions).


It takes some effort to port old extensions to stable ABI. Several old 
APIs are not supported in stable ABI extensions. For example 
developers have to port static type definitions to heap types. It's 
not complicated, but it takes some effort.


The stable ABI is also not complete, although it should be complete 
enough for a lot of projects.  A, fairly esoteric, issue I ran into is 
that it is currently not possible to define a class with a non-default 
meta class using the type-spec API (AFAIK), see #15870.


And as you write “it takes some effort”, that alone likely reduces the 
amount of projects that migrate to the stable ABI esp. for projects that 
already have a CI/CD setup that creates binary wheels for you (for 
example using cibuildwheel).


Indeed, the stable ABI is not complete. I just figured out that limited 
API < 3.9 cannot define objects with weakref support or dict offset. The 
__weaklistoffset__ and  __dictoffset__ PyMemberDefs were added in Python 
3.9.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G7YTJ62XB4QJJY43RDZTUFIYBQ5P5W57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Phil Thompson via Python-Dev]

On 27/09/2021 21:53, Brett Cannon wrote:

On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:


On 26/09/2021 05:21, Steven D'Aprano wrote:

[snip]





> These are not rhetorical questions, I genuinely do not know. I *think*
> that there was an attempt to make a stable C API back in 3.2 days:
>
> https://www.python.org/dev/peps/pep-0384/
>
> but I don't know what difference it has made to extension writers in
> practice. From your description, it sounds like perhaps not as big a
> difference as we would have liked.
>
> Maybe extension writers are not using the stable C API? Is that even
> possible? Please excuse my ignorance.

PyQt has used the stable ABI for many years. The main reason for using
it is to reduce the number of wheels. The PyQt ecosystem currently
contains 15 PyPI projects across 4 platforms supporting 5 Python
versions (including v3.10). Without the stable ABI a new release would
require 300 wheels. With the stable ABI it is a more manageable 60
wheels.

However the stable ABI is still a second class citizen as it is still
not possible (AFAIK) to specify a wheel name that doesn't need to
explicitly include each supported Python version (rather than a 
minimum

stable ABI version).



Actually you can do this. The list of compatible wheels for a platform
starts at CPython 3.2 when the stable ABI was introduced and goes 
forward

to the version of Python you are running. So you can build a wheel file
that targets the oldest version of CPython that you are targeting and 
its

version of the stable ABI and it is considered forward compatible. See
`python -m pip debug --verbose` for the complete list of wheel tags 
that

are supported for an interpreter.


Logical and it works.

Many thanks,
Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/22MU4QKR46SMFQWQFPWUIWM76JYJMJ3L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Tue, Sep 28, 2021 at 12:40 AM Guido van Rossum  wrote:
>
> What I have heard repeatedly, from people who are paid to know, is that most 
> users don’t care about the latest features, and would rather stick to a 
> release until it becomes unsupported. (Extreme example: Python 2.)
>
> Numpy isn’t random, it’s at the bottom of the food chain for a large 
> ecosystem or two — if it doesn’t support a new Python release, none of its 
> dependent packages can even start porting. (I guess only Cython is even 
> lower, but it’s a build-time tool. And indeed it has supported 3.10 for a 
> long time.)

Well, no, it wasn't entirely random :-).

Being on the bottom of the food chain is important, but I don't think
it's the full story -- Tensorflow is also at the bottom of a huge
ecosystem. I think it's also related to NumPy being mostly
volunteer-run, which means they're sensitive to feedback from
individual enthusiasts, and enthusiasts are the most aggressive early
adopters. OTOH Tensorflow is a huge commercial collaboration, and
companies *hate* upgrading.

Either way though, it doesn't seem to be anything to do with CPython's
ABI stability or release cadence.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNE7TY7ZFGBOOOG3H3YNB5MFGX5IWZ76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Guido van Rossum]

What I have heard repeatedly, from people who are paid to know, is that
most users don’t care about the latest features, and would rather stick to
a release until it becomes unsupported. (Extreme example: Python 2.)

Numpy isn’t random, it’s at the bottom of the food chain for a large
ecosystem or two — if it doesn’t support a new Python release, none of its
dependent packages can even start porting. (I guess only Cython is even
lower, but it’s a build-time tool. And indeed it has supported 3.10 for a
long time.)

—Guido

On Mon, Sep 27, 2021 at 23:01 Nathaniel Smith  wrote:

> On Sun, Sep 26, 2021 at 3:38 AM  wrote:
> > Open3D is an example. Will finally move to Python 3.9 some time the
> coming month. Its dependency graph contains about 70 other packages.
> >
> > In this specific case, the underlying problem was that TensorFlow was
> stuck at 3.8. The TensorFlow codebase got ported in November 2020, then
> released early 2021. Then Open3D included the new Tensorflow (plus whatever
> else needed to be adapted) in their codebase in May. They’re now going
> through their release schedule, and their 0.14 release should be up on PyPI
> soon.
>
> I took a minute to look up the release dates to fill in this timeline:
>
> Python 3.9 released: October 2020
> Tensorflow adds 3.9 support: November 2020
> Tensorflow v2.5.0 released with the new 3.9 support: May 2021
> Open3d adds 3.9 support: May 2021
> First Open3d release to include the new 3.9 support: ~October 2021
>
> So it seems like in this case at least, the year long delay consists
> of ~1 month of porting work, and ~11 months of projects letting the
> finished code sit in their repos without shipping to users.
>
> It seems like the core problem here is that these projects don't
> consider it important to keep up with the latest Python release. I'm
> not sure what CPython upstream can do about that. Maybe you could
> lobby these projects to ship releases more promptly?
>
> By contrast, to pick a random library that uses the unstable C API
> extensively, NumPy is already shipping wheels for 3.10 -- and 3.10
> isn't even out yet. So it's certainly possible to do, even for
> projects with a tiny fraction of Tensorflow's engineering budget.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OGBANTGGJAI2ZM5SYSLCWFGRIML5VWUL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2ETRHAUVP53BSHAESBWZQRBF4KTWRJP3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Sun, Sep 26, 2021 at 3:38 AM  wrote:
> Open3D is an example. Will finally move to Python 3.9 some time the coming 
> month. Its dependency graph contains about 70 other packages.
>
> In this specific case, the underlying problem was that TensorFlow was stuck 
> at 3.8. The TensorFlow codebase got ported in November 2020, then released 
> early 2021. Then Open3D included the new Tensorflow (plus whatever else 
> needed to be adapted) in their codebase in May. They’re now going through 
> their release schedule, and their 0.14 release should be up on PyPI soon.

I took a minute to look up the release dates to fill in this timeline:

Python 3.9 released: October 2020
Tensorflow adds 3.9 support: November 2020
Tensorflow v2.5.0 released with the new 3.9 support: May 2021
Open3d adds 3.9 support: May 2021
First Open3d release to include the new 3.9 support: ~October 2021

So it seems like in this case at least, the year long delay consists
of ~1 month of porting work, and ~11 months of projects letting the
finished code sit in their repos without shipping to users.

It seems like the core problem here is that these projects don't
consider it important to keep up with the latest Python release. I'm
not sure what CPython upstream can do about that. Maybe you could
lobby these projects to ship releases more promptly?

By contrast, to pick a random library that uses the unstable C API
extensively, NumPy is already shipping wheels for 3.10 -- and 3.10
isn't even out yet. So it's certainly possible to do, even for
projects with a tiny fraction of Tensorflow's engineering budget.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGBANTGGJAI2ZM5SYSLCWFGRIML5VWUL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Brett Cannon]

On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:

> On 26/09/2021 05:21, Steven D'Aprano wrote:
>
> [snip]
>


> > These are not rhetorical questions, I genuinely do not know. I *think*
> > that there was an attempt to make a stable C API back in 3.2 days:
> >
> > https://www.python.org/dev/peps/pep-0384/
> >
> > but I don't know what difference it has made to extension writers in
> > practice. From your description, it sounds like perhaps not as big a
> > difference as we would have liked.
> >
> > Maybe extension writers are not using the stable C API? Is that even
> > possible? Please excuse my ignorance.
>
> PyQt has used the stable ABI for many years. The main reason for using
> it is to reduce the number of wheels. The PyQt ecosystem currently
> contains 15 PyPI projects across 4 platforms supporting 5 Python
> versions (including v3.10). Without the stable ABI a new release would
> require 300 wheels. With the stable ABI it is a more manageable 60
> wheels.
>
> However the stable ABI is still a second class citizen as it is still
> not possible (AFAIK) to specify a wheel name that doesn't need to
> explicitly include each supported Python version (rather than a minimum
> stable ABI version).
>

Actually you can do this. The list of compatible wheels for a platform
starts at CPython 3.2 when the stable ABI was introduced and goes forward
to the version of Python you are running. So you can build a wheel file
that targets the oldest version of CPython that you are targeting and its
version of the stable ABI and it is considered forward compatible. See
`python -m pip debug --verbose` for the complete list of wheel tags that
are supported for an interpreter.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E4EA6GW6N5NREEBM4QOA7GTLBGTGD4AV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Ronald Oussoren via Python-Dev]

> On 26 Sep 2021, at 19:03, Christian Heimes  wrote:
> 
> On 26/09/2021 13.07, jack.jan...@cwi.nl wrote:
>> The problem with the stable ABI is that very few developers are targeting 
>> it. I’m not sure why not, whether it has to do with incompleteness of the 
>> ABI, or with issues targeting it easily and your builds and then having 
>> pip/PyPI do the right things with wheels and all that. I’ve been on the 
>> capi-sig mailing list since its inception in 2007, but the discussions are 
>> really going over my head. I don’t understand what the problems are that 
>> keep people from targeting the stable ABI (or the various other attempts at 
>> standardising extensions over Python versions).
> 
> It takes some effort to port old extensions to stable ABI. Several old APIs 
> are not supported in stable ABI extensions. For example developers have to 
> port static type definitions to heap types. It's not complicated, but it 
> takes some effort.

The stable ABI is also not complete, although it should be complete enough for 
a lot of projects.  A, fairly esoteric, issue I ran into is that it is 
currently not possible to define a class with a non-default meta class using 
the type-spec API (AFAIK), see #15870.

And as you write “it takes some effort”, that alone likely reduces the amount 
of projects that migrate to the stable ABI esp. for projects that already have 
a CI/CD setup that creates binary wheels for you (for example using 
cibuildwheel). 

Ronald
—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFLOCVRO2UP2PKWTFUTAQ3EKO6NLMG4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Christian Heimes]

On 26/09/2021 13.07, jack.jan...@cwi.nl wrote:
The problem with the stable ABI is that very few developers are 
targeting it. I’m not sure why not, whether it has to do with 
incompleteness of the ABI, or with issues targeting it easily and your 
builds and then having pip/PyPI do the right things with wheels and all 
that. I’ve been on the capi-sig mailing list since its inception in 
2007, but the discussions are really going over my head. I don’t 
understand what the problems are that keep people from targeting the 
stable ABI (or the various other attempts at standardising extensions 
over Python versions).


It takes some effort to port old extensions to stable ABI. Several old 
APIs are not supported in stable ABI extensions. For example developers 
have to port static type definitions to heap types. It's not 
complicated, but it takes some effort.


The other issue is Cython. Stable releases of Cython do not support 
stable ABI yet. It's an experimental feature in Cython 3.0.0 alpha.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O33XTH37NGAFHHPNV5653JVBCC4NXRHE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[jack . jansen]

> On 26 Sep 2021, at 04:09, MRAB  wrote:
> 
> On 2021-09-26 00:14, jack.jan...@cwi.nl wrote:
>> I think we really need to come up with some scheme whereby extension 
>> packages become more long-lived than a single Python release...
> You mean, something like the Python ABI (PEP 384, Stable Application Binary 
> Interface)?

And Steven D”Aprano also mentioned the stable ABI.

The problem with the stable ABI is that very few developers are targeting it. 
I’m not sure why not, whether it has to do with incompleteness of the ABI, or 
with issues targeting it easily and your builds and then having pip/PyPI do the 
right things with wheels and all that. I’ve been on the capi-sig mailing list 
since its inception in 2007, but the discussions are really going over my head. 
I don’t understand what the problems are that keep people from targeting the 
stable ABI (or the various other attempts at standardising extensions over 
Python versions).


> On 26 Sep 2021, at 06:21, Steven D'Aprano  wrote:
> 
> Do you have a reason to think that it is in danger in some way? Some 
> factor that didn't apply equally in 2001 and 2011 as it does in 2021?

Yes, very much so. Wheels.

Before we had wheels there were very few packages that were distributed in 
binary form, the NumPy family and the various GUI toolkits are the only ones 
that come to mind, and they had very active developer communities that tracked 
Python releases.

Wheels are absolutely wonderful, but the downside is that everyone has come to 
depend on them. Before wheels, extension modules were often optional, in that 
many packages would provide their basic functionality in pure Python, and then 
have some performance-enhancing or functionality-extending optional extension 
modules. Wheels have obviated the need for that. So now everything depends on 
extension modules (and on external packages that depend on extension modules, 
and so on).
--
Jack Jansen, , http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O5WTFG5WER7PALCYFCDJLP4VWYUWMAPS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[jack . jansen]

> On 26 Sep 2021, at 05:49, Nathaniel Smith  wrote:
> 
> On Sat, Sep 25, 2021 at 5:40 PM  wrote:
>> PyPI packages and wheels are targeted to specific Python versions, which 
>> means that any project that depends on some of the larger extension packages 
>> (of which there are many, and many of which are must-have for many projects) 
>> now start lagging Python versions by years, because somewhere deep down in 
>> the dependency graph there is something that is still stuck at Python 3.8 
>> (for example).
> 
> Can you give some examples of the packages you're thinking of, that
> are prominent/must-have and stuck on years-old Pythons?


Open3D is an example. Will finally move to Python 3.9 some time the coming 
month. Its dependency graph contains about 70 other packages.

In this specific case, the underlying problem was that TensorFlow was stuck at 
3.8. The TensorFlow codebase got ported in November 2020, then released early 
2021. Then Open3D included the new Tensorflow (plus whatever else needed to be 
adapted) in their codebase in May. They’re now going through their release 
schedule, and their 0.14 release should be up on PyPI soon. 

--
Jack Jansen, , http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PJYUB2R4VB7H4UBFQ6464OAKFAXBSRJQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Phil Thompson via Python-Dev]

On 26/09/2021 05:21, Steven D'Aprano wrote:

[snip]


As for the C-API... Python is 30 years old. Has it ever had a stable
C-API before now? Hasn't it *always* been the case that C packages have
targetted a single version and need to be rebuilt from source on every
release?


No.


These are not rhetorical questions, I genuinely do not know. I *think*
that there was an attempt to make a stable C API back in 3.2 days:

https://www.python.org/dev/peps/pep-0384/

but I don't know what difference it has made to extension writers in
practice. From your description, it sounds like perhaps not as big a
difference as we would have liked.

Maybe extension writers are not using the stable C API? Is that even
possible? Please excuse my ignorance.


PyQt has used the stable ABI for many years. The main reason for using 
it is to reduce the number of wheels. The PyQt ecosystem currently 
contains 15 PyPI projects across 4 platforms supporting 5 Python 
versions (including v3.10). Without the stable ABI a new release would 
require 300 wheels. With the stable ABI it is a more manageable 60 
wheels.


However the stable ABI is still a second class citizen as it is still 
not possible (AFAIK) to specify a wheel name that doesn't need to 
explicitly include each supported Python version (rather than a minimum 
stable ABI version). In other words it doesn't solve the OP's concern 
about unmaintained older packages being able to be installed in newer 
versions of Python (even though those packages had been explicitly 
designed to do so).


Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RPDUNMG6RS4FBG6GODZDZ4DCB252N4VP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Steven D'Aprano]

On Sun, Sep 26, 2021 at 01:14:18AM +0200, jack.jan...@cwi.nl wrote:

> I’m getting increasingly worried about the future of Python,

That Python will become even more popular?

TIOBE: second place, 0.16% below C.

PYPL: first place, 12.3% above Java.

RedMonk: equal second with Java.

https://www.tiobe.com/tiobe-index/

https://pypl.github.io/PYPL.html

https://redmonk.com/sogrady/2021/08/05/language-rankings-6-21/

We must be doing something right and maybe the problems that we see are 
not as big as we fear.

Language popularity is not in and of itself important. It doesn't really 
matter whether Python is 1st or 10th in language popularity. But the 
CPython part (at least) of the Python language and ecosystem is clearly 
thriving.

Do you have a reason to think that it is in danger in some way? Some 
factor that didn't apply equally in 2001 and 2011 as it does in 2021?


> and those 
> worries have been exacerbated by the new yearly release rhythm, and 
> the lack of a stable C-API that is full enough to entice extension 
> writers to use it.

Admittedly, the yearly release schedule *is* a new policy. I don't 
remember the rationale for it or who it is supposed to benefit, and I 
don't know what we are doing to objectively decide whether the change 
made things better or worse.

As for the C-API... Python is 30 years old. Has it ever had a stable 
C-API before now? Hasn't it *always* been the case that C packages have 
targetted a single version and need to be rebuilt from source on every 
release?

These are not rhetorical questions, I genuinely do not know. I *think* 
that there was an attempt to make a stable C API back in 3.2 days:

https://www.python.org/dev/peps/pep-0384/

but I don't know what difference it has made to extension writers in 
practice. From your description, it sounds like perhaps not as big a 
difference as we would have liked.

Maybe extension writers are not using the stable C API? Is that even 
possible? Please excuse my ignorance.



-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GN5QGYZEIFFMSJ66EIYG7H4ZNBIZM7FU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Sat, Sep 25, 2021 at 5:40 PM  wrote:
> PyPI packages and wheels are targeted to specific Python versions, which 
> means that any project that depends on some of the larger extension packages 
> (of which there are many, and many of which are must-have for many projects) 
> now start lagging Python versions by years, because somewhere deep down in 
> the dependency graph there is something that is still stuck at Python 3.8 
> (for example).

Can you give some examples of the packages you're thinking of, that
are prominent/must-have and stuck on years-old Pythons?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF7Z4EDN3DHBKPMYCIJFTI2JDTIJYHXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[MRAB]

On 2021-09-26 00:14, jack.jan...@cwi.nl wrote:
I’m getting increasingly worried about the future of Python, and those 
worries have been exacerbated by the new yearly release rhythm, and the 
lack of a stable C-API that is full enough to entice extension writers 
to use it.


PyPI packages and wheels are targeted to specific Python versions, which 
means that any project that depends on some of the larger extension 
packages (of which there are many, and many of which are must-have for 
many projects) now start lagging Python versions by years, because 
somewhere deep down in the dependency graph there is something that is 
still stuck at Python 3.8 (for example). I fully understand that 3.8 is 
good enough for the developers of that package, and that they have more 
pressing things to do than porting to 3.9 or 3.10, but it now keeps any 
project or package that depends on their software on 3.8 as well.


And I also fully understand that some other developer who creates a new 
package that is essential to my application only targets the current 
Python release, or maybe one release back, but now if I need both the 
new package and and older one I’m up the well-known creek without a paddle.


Building packages from source has become pretty much impossible 
nowadays, especially if your project is multi-platform and needs to 
interface to specific hardware, and you want to do the right thing with 
CI/CD builds and all that. On Linux/MacOS you have a chance when you try 
to specify all the dependencies for third party libraries and what not, 
but on Windows you’re dead in the water. And that is assuming you have 
the time and are smart enough to back port the new package to the old 
Python release, or the old package to the new Python release (and for 
the latter there’s probably a good reason why the developers haven’t 
done so already). Before you know it you have to install a couple of 
graphics card APIs for some obscure AI feature used by something you’ve 
never heard of, Cython for something else, and obscure vendor libraries 
for something else again.


I think we really need to come up with some scheme whereby extension 
packages become more long-lived than a single Python release...


You mean, something like the Python ABI (PEP 384, Stable Application 
Binary Interface)?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WFGX4ZRIRXQK4NZ6JSRZRQT7463FGFVG/
Code of Conduct: http://python.org/psf/codeofconduct/