[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 4:58 PM Stefan Behnel wrote: > I agree. Shipping the generated C sources was a very good choice as long > as > CPython's C-API was very stable and getting a build time dependency safely > installed on user side was very difficult. > > These days, it's the opposite way. >

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
On Tue, Feb 1, 2022 at 5:52 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 02.02.22 um 01:43: > Yes. HPy is certainly far from ready for anything real, but even for the > Limited API, it's still unclear whether it's actually complete enough to > cover Cython's needs. Basically, the API

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Guido van Rossum schrieb am 02.02.22 um 01:43: It may be hard to imagine if you're working on Cython, which only exists because of performance needs, but there are other things that people want to test with the upcoming CPython release in addition to performance I know. Cython (and originally

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Thomas Caswell schrieb am 01.02.22 um 23:15: I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
On Tue, Feb 1, 2022 at 4:14 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 02.02.22 um 00:21: > > I wonder if a solution during CPython's rocky alpha release cycle could > be > > to default (either in Cython or in projects using it) to the "not quite > as > > fast but not relying on a lot

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 12:32 pm, Christopher Barker wrote: I can make a "thick" high performance wrapper, e.g. if I want to call an expensive C function on each item in a sequence, I can do that in Cython, removing a lot of the overhead of Python. "Not as fast as possible" doesn't necessarily mean

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Guido van Rossum schrieb am 02.02.22 um 00:21: On Tue, Feb 1, 2022 at 3:07 David wrote: Greg Ewing wrote: To address this there could be an option to choose between "compatible code" and "fast code", with the former restricting itself to the stable API. To some extent, that exists at the

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Hi Irit, Irit Katriel via Python-Dev schrieb am 01.02.22 um 23:04: There two separate issues here. One is the timing of committing changes into cython, and the other is the process by which the cython devs learn about cpython development. On the first issue, you wrote: I'm reluctant to

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Thomas Caswell
I disagree with point (3). I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle ground of not

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Greg Ewing schrieb am 01.02.22 um 23:33: On 2/02/22 8:48 am, Guido van Rossum wrote: It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. I think the reason for this is that Cython is trying to be two things at once: (1) an interface

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 3:22 PM Greg Ewing wrote: > On 2/02/22 11:53 am, Christopher Barker wrote: > > As a long time Cython user, but not a Cython developer, I think (2) is > > the primary purpose, with (1) as a handy side benefit (otherwise > > we'd just use ctypes, yes?) > > Personally, no, I

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 11:53 am, Christopher Barker wrote: As a long time Cython user, but not a Cython developer, I think (2) is the primary purpose, with (1) as a handy side benefit (otherwise we'd just use ctypes, yes?) Personally, no, I would not "just use ctypes". The main reason I created Pyrex was

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
Hm... So maybe the issue is either with Cython's default settings (perhaps traditionally it defaults to "as fast as possible but relies on internal APIs a lot"?) or with the Cython settings selected by default by projects *using* Cython? I wonder if a solution during CPython's rocky alpha release

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread dw-git
Greg Ewing wrote: > To address this there could be an option to choose between > "compatible code" and "fast code", with the former restricting > itself to the stable API. To some extent, that exists at the moment - many of the real abuses of the CPython internals can be controlled by setting C

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 2:36 PM Greg Ewing wrote: > I think the reason for this is that Cython is trying to be two > things at once: (1) an interface between Python and C, (2) a > compiler that turns Python code into fast C code. > As a long time Cython user, but not a Cython developer, I think

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 8:48 am, Guido van Rossum wrote: It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. I think the reason for this is that Cython is trying to be two things at once: (1) an interface between Python and C, (2) a compiler that

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 5:42 am, Victor Stinner wrote: There is an on-going effort adding getter and setter functions on two structures which are causing most troubles on Python updates: * PyThreadState: https://bugs.python.org/issue39947 * PyFrameObject: https://bugs.python.org/issue40421 In the case of

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
Stefan, There two separate issues here. One is the timing of committing changes into cython, and the other is the process by which the cython devs learn about cpython development. On the first issue, you wrote: I'm reluctant to working on adapting Cython during alphas, because it > happened

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Christian Heimes schrieb am 01.02.22 um 16:42: On 01/02/2022 16.08, Victor Stinner wrote: I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. Another part is that there doesn't seem to be any Cython maintainer interested in communicating with the core devs who are changing those CPython internals. We have to resort to creating

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Miro Hrončok
On 01. 02. 22 20:17, Irit Katriel wrote: Miro, I have offered before and my offer still stands to help fix this. Thank You! This was already fixed in the cython main branch by Stefan. The discussion now is about when to backport it to cython 0.29. I'm actually working on the backport now

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
Miro, I have offered before and my offer still stands to help fix this. This was already fixed in the cython main branch by Stefan. The discussion now is about when to backport it to cython 0.29. I'm actually working on the backport now (learning cython in the process). But we will need to come

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Miro Hrončok
On 01. 02. 22 17:42, Victor Stinner wrote: The problem right now is the pressure put on Cython maintainers to fix Cython as soon as possible. IMO core developers who introduce incompatible changes should be more involved in the Cython changes, since Cython is a **key component** of the Python

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Victor Stinner
On Tue, Feb 1, 2022 at 5:37 PM Irit Katriel wrote: > _PyErr_StackItem is not part of the C API, it's an internal struct that > cython accesses directly. numpy currently fails on building Cython __Pyx_PyErr_GetTopmostException() function which access tstate->exc_info->exc_type, so it's about the

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Victor Stinner
On Tue, Feb 1, 2022 at 4:42 PM Christian Heimes wrote: > I would prefer if we can get Cython and all the other code generator and > bindings library off the unstable C-API. They should use the limited API > instead. If they require any C-APIs outside the limited API, then we > should investigate

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
_PyErr_StackItem is not part of the C API, it's an internal struct that cython accesses directly. On Tue, Feb 1, 2022 at 3:42 PM Christian Heimes wrote: > On 01/02/2022 16.08, Victor Stinner wrote: > > -- > > > > I would prefer to introduce C API incompatible changes differently: > > first

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

2022-02-01 Thread Martin Dengler
Hi, On Tue, Feb 01, 2022 at 10:19:12AM -0500, Éric Araujo wrote: Hi, Le 01/02/2022 à 07:25, Martin Dengler a écrit : The fact people are assuming bad faith and spilling ink about those getting involved with that extremely-underserved part when contributors are sorely > needed is

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christian Heimes
On 01/02/2022 16.08, Victor Stinner wrote: -- I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is released - (3) If possible, wait until numpy

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

2022-02-01 Thread Éric Araujo
Hi, Le 01/02/2022 à 07:25, Martin Dengler a écrit : The fact people are assuming bad faith and spilling ink about those getting involved with that extremely-underserved part when contributors are sorely > needed is counter-productive. I disagree, the original message was a good-faith

[Python-Dev] Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Victor Stinner
Hi, It became more and more common that a C API incompatible change introduced in Python breaks Cython and all Python projects using Cython (ex: numpy). Hopefully, usually only some projects using Cython are broken, not all of them. Some of you may remind the PEP 590 (vectorcall) implementation

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

2022-02-01 Thread Alex Waygood
Nikita found a very real (and slightly embarrassing!) bug in a patch I wrote for the enum module a few months back, due to his efforts to improve test coverage. And there is an entire section of the DevGuide devoted to "Improving test coverage", stating that PRs such as the ones Nikita has been

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

2022-02-01 Thread Martin Dengler
On Tue, Feb 01, 2022 at 12:35:02AM -0500, Terry Reedy wrote: On 1/31/2022 7:31 PM, Nikita Sobolev wrote: Hi, my name is Nikita and I think that I am the person behind these spammy PRs. Link: https://github.com/python/cpython/pulls/sobolevn I also encouraged multiple easily reviewable PRs from

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

2022-02-01 Thread Sebastian Rittau
Am 01.02.22 um 01:31 schrieb Nikita Sobolev: Hi, my name is Nikita and I think that I am the person behind these spammy PRs. Link: https://github.com/python/cpython/pulls/sobolevn As a typeshed maintainer, Nikita also "spams" typeshed with PRs. I highly appreciate those PRs, which I am sure