[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Pablo Galindo Salgado
That is on pourpose and is the public API for Python. In Python it returns an iterable of tuples, which is processed from the actual internal form. On Wed, 9 Feb 2022 at 18:56, Victor Stinner wrote: > On Wed, Feb 9, 2022 at 5:48 PM Pablo Galindo Salgado > wrote: > > We consider the

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Victor Stinner
On Wed, Feb 9, 2022 at 5:48 PM Pablo Galindo Salgado wrote: > We consider the representation of co_postions private, so we don't want (for > now) to ad > getters/setters. If you want to get the position of a instruction, you can > use PyCode_Addr2Location The code.co_positions() method is

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Victor Stinner
Hi, It's already possible to call PyObject_CallMethod(code, "co_positions", NULL) and then use the iterator in C. Is there an use case where performance of reading co_positions is critical? If not, there is no need to add a specialized function. Victor On Wed, Feb 9, 2022 at 5:23 PM Petr

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Ethan Furman
On 2/9/22 8:40 AM, Pablo Galindo Salgado wrote: > Petr Viktorin wrote: >> Should there be a getter/setter for co_positions? > > We consider the representation of co_postions private, so we don't want (for now) to ad > getters/setters. Isn't the whole point of getters/setters is to allow public

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Pablo Galindo Salgado
> Should there be a getter/setter for co_positions? We consider the representation of co_postions private, so we don't want (for now) to ad getters/setters. If you want to get the position of a instruction, you can use PyCode_Addr2Location On Wed, 9 Feb 2022 at 16:22, Petr Viktorin wrote: > On

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-09 Thread Petr Viktorin
On 04. 02. 22 15:23, Stefan Behnel wrote: Petr Viktorin schrieb am 03.02.22 um 13:47: On 02. 02. 22 11:50, Stefan Behnel wrote: Maybe we should advertise the two modes more. And make sure that both work. There are certainly issues with the current state of the "limited API" implementation,

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-04 Thread Ethan Furman
On 2/4/22 6:23 AM, Stefan Behnel wrote: > One recent example is the new error locations in tracebacks, where PEP 657 explicitly lists > the new "co_positions" field in code objects as an implementation detail of CPython. If we > want to implement this in Cython, then there is no other way than

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-04 Thread Stefan Behnel
Petr Viktorin schrieb am 03.02.22 um 13:47: On 02. 02. 22 11:50, Stefan Behnel wrote: Maybe we should advertise the two modes more. And make sure that both work. There are certainly issues with the current state of the "limited API" implementation, but that just needs work and testing. I

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-03 Thread Petr Viktorin
On 02. 02. 22 11:50, Stefan Behnel wrote: Petr Viktorin schrieb am 02.02.22 um 10:22: Moving off the internal (unstable) API would be great, but I don't think Cython needs to move all the way to the limited API. There are three "levels" in the C API: - limited API, with long-term ABI

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Stefan Behnel
Victor Stinner schrieb am 02.02.22 um 23:23: On Wed, Feb 2, 2022 at 3:54 PM Stefan Behnel wrote: So people using stable Python versions like Python 3.10 would not need Cython, but people testing the "next Python" (Python 3.11) would not have to manually removed generated C code. That sounds

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Victor Stinner
On Wed, Feb 2, 2022 at 3:54 PM Stefan Behnel wrote: > > So people using stable Python versions like Python 3.10 would not need > > Cython, but people testing the "next Python" (Python 3.11) would not > > have to manually removed generated C code. > > That sounds like an environment variable might

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Stefan Behnel
Ronald Oussoren via Python-Dev schrieb am 02.02.22 um 16:44: On 2 Feb 2022, at 11:50, Stefan Behnel wrote: Petr Viktorin schrieb am 02.02.22 um 10:22: - "normal" public API, covered by the backwards compatibility policy (users need to recompile for every minor release, and watch for

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Christopher Barker
> Maybe we should advertise the two modes more. And make sure that both work. > > That would be great — as a long time Cython user, I didn’t know they existed. To be fair, long-time means I figured out something that works years ago, and have kept doing that ever since. It might also help to make

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Ronald Oussoren via Python-Dev
> On 2 Feb 2022, at 11:50, Stefan Behnel wrote: > > Petr Viktorin schrieb am 02.02.22 um 10:22: >> Moving off the internal (unstable) API would be great, but I don't think >> Cython needs to move all the way to the limited API. >> There are three "levels" in the C API: >> - limited API, with

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Stefan Behnel
Petr Viktorin schrieb am 02.02.22 um 10:22: Moving off the internal (unstable) API would be great, but I don't think Cython needs to move all the way to the limited API. There are three "levels" in the C API: - limited API, with long-term ABI compatibility guarantees That's what

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Stefan Behnel
Victor Stinner schrieb am 02.02.22 um 11:35: I wish that there would be a 3rd option: ship C code generated by Cython *but* run Cython if this C code "looks" outdated, for example if building the C code fails with a compiler error. So, one thing I did yesterday was to make sure that .c files

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Victor Stinner
On Wed, Feb 2, 2022 at 9:25 AM wrote: > Guido van Rossum wrote: > > My question for you is if you're willing to write up a list of things in > > CPython that you depend on. Or is this just something you're not willing to > > commit to? It would be nice to know which it is, just so the CPython

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Victor Stinner
I wish that there would be a 3rd option: ship C code generated by Cython *but* run Cython if this C code "looks" outdated, for example if building the C code fails with a compiler error. So people using stable Python versions like Python 3.10 would not need Cython, but people testing the "next

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Petr Viktorin
On 01. 02. 22 16:42, Christian Heimes wrote: On 01/02/2022 16.08, Victor Stinner wrote: -- I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread Christian Heimes
On 02/02/2022 09.19, dw-...@d-woods.co.uk wrote: Guido van Rossum wrote: My question for you is if you're willing to write up a list of things in CPython that you depend on. Or is this just something you're not willing to commit to? It would be nice to know which it is, just so the CPython team

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-02 Thread dw-git
Guido van Rossum wrote: > My question for you is if you're willing to write up a list of things in > CPython that you depend on. Or is this just something you're not willing to > commit to? It would be nice to know which it is, just so the CPython team > knows what we're up against. I'm happy to

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 4:58 PM Stefan Behnel wrote: > I agree. Shipping the generated C sources was a very good choice as long > as > CPython's C-API was very stable and getting a build time dependency safely > installed on user side was very difficult. > > These days, it's the opposite way. >

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
On Tue, Feb 1, 2022 at 5:52 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 02.02.22 um 01:43: > Yes. HPy is certainly far from ready for anything real, but even for the > Limited API, it's still unclear whether it's actually complete enough to > cover Cython's needs. Basically, the API

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Guido van Rossum schrieb am 02.02.22 um 01:43: It may be hard to imagine if you're working on Cython, which only exists because of performance needs, but there are other things that people want to test with the upcoming CPython release in addition to performance I know. Cython (and originally

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Thomas Caswell schrieb am 01.02.22 um 23:15: I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
On Tue, Feb 1, 2022 at 4:14 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 02.02.22 um 00:21: > > I wonder if a solution during CPython's rocky alpha release cycle could > be > > to default (either in Cython or in projects using it) to the "not quite > as > > fast but not relying on a lot

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 12:32 pm, Christopher Barker wrote: I can make a "thick" high performance wrapper, e.g. if I want to call an expensive C function on each item in a sequence, I can do that in Cython, removing a lot of the overhead of Python. "Not as fast as possible" doesn't necessarily mean

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Guido van Rossum schrieb am 02.02.22 um 00:21: On Tue, Feb 1, 2022 at 3:07 David wrote: Greg Ewing wrote: To address this there could be an option to choose between "compatible code" and "fast code", with the former restricting itself to the stable API. To some extent, that exists at the

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Hi Irit, Irit Katriel via Python-Dev schrieb am 01.02.22 um 23:04: There two separate issues here. One is the timing of committing changes into cython, and the other is the process by which the cython devs learn about cpython development. On the first issue, you wrote: I'm reluctant to

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Thomas Caswell
I disagree with point (3). I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle ground of not

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Greg Ewing schrieb am 01.02.22 um 23:33: On 2/02/22 8:48 am, Guido van Rossum wrote: It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. I think the reason for this is that Cython is trying to be two things at once: (1) an interface

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 3:22 PM Greg Ewing wrote: > On 2/02/22 11:53 am, Christopher Barker wrote: > > As a long time Cython user, but not a Cython developer, I think (2) is > > the primary purpose, with (1) as a handy side benefit (otherwise > > we'd just use ctypes, yes?) > > Personally, no, I

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 11:53 am, Christopher Barker wrote: As a long time Cython user, but not a Cython developer, I think (2) is the primary purpose, with (1) as a handy side benefit (otherwise we'd just use ctypes, yes?) Personally, no, I would not "just use ctypes". The main reason I created Pyrex was

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
Hm... So maybe the issue is either with Cython's default settings (perhaps traditionally it defaults to "as fast as possible but relies on internal APIs a lot"?) or with the Cython settings selected by default by projects *using* Cython? I wonder if a solution during CPython's rocky alpha release

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread dw-git
Greg Ewing wrote: > To address this there could be an option to choose between > "compatible code" and "fast code", with the former restricting > itself to the stable API. To some extent, that exists at the moment - many of the real abuses of the CPython internals can be controlled by setting C

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker
On Tue, Feb 1, 2022 at 2:36 PM Greg Ewing wrote: > I think the reason for this is that Cython is trying to be two > things at once: (1) an interface between Python and C, (2) a > compiler that turns Python code into fast C code. > As a long time Cython user, but not a Cython developer, I think

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 8:48 am, Guido van Rossum wrote: It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. I think the reason for this is that Cython is trying to be two things at once: (1) an interface between Python and C, (2) a compiler that

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Greg Ewing
On 2/02/22 5:42 am, Victor Stinner wrote: There is an on-going effort adding getter and setter functions on two structures which are causing most troubles on Python updates: * PyThreadState: https://bugs.python.org/issue39947 * PyFrameObject: https://bugs.python.org/issue40421 In the case of

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
Stefan, There two separate issues here. One is the timing of committing changes into cython, and the other is the process by which the cython devs learn about cpython development. On the first issue, you wrote: I'm reluctant to working on adapting Cython during alphas, because it > happened

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Stefan Behnel
Christian Heimes schrieb am 01.02.22 um 16:42: On 01/02/2022 16.08, Victor Stinner wrote: I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Guido van Rossum
It seems to me that a big part of the problem is that Cython feels entitled to use arbitrary CPython internals. Another part is that there doesn't seem to be any Cython maintainer interested in communicating with the core devs who are changing those CPython internals. We have to resort to creating

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Miro Hrončok
On 01. 02. 22 20:17, Irit Katriel wrote: Miro, I have offered before and my offer still stands to help fix this. Thank You! This was already fixed in the cython main branch by Stefan. The discussion now is about when to backport it to cython 0.29. I'm actually working on the backport now

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
Miro, I have offered before and my offer still stands to help fix this. This was already fixed in the cython main branch by Stefan. The discussion now is about when to backport it to cython 0.29. I'm actually working on the backport now (learning cython in the process). But we will need to come

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Miro Hrončok
On 01. 02. 22 17:42, Victor Stinner wrote: The problem right now is the pressure put on Cython maintainers to fix Cython as soon as possible. IMO core developers who introduce incompatible changes should be more involved in the Cython changes, since Cython is a **key component** of the Python

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Victor Stinner
On Tue, Feb 1, 2022 at 5:37 PM Irit Katriel wrote: > _PyErr_StackItem is not part of the C API, it's an internal struct that > cython accesses directly. numpy currently fails on building Cython __Pyx_PyErr_GetTopmostException() function which access tstate->exc_info->exc_type, so it's about the

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Victor Stinner
On Tue, Feb 1, 2022 at 4:42 PM Christian Heimes wrote: > I would prefer if we can get Cython and all the other code generator and > bindings library off the unstable C-API. They should use the limited API > instead. If they require any C-APIs outside the limited API, then we > should investigate

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev
_PyErr_StackItem is not part of the C API, it's an internal struct that cython accesses directly. On Tue, Feb 1, 2022 at 3:42 PM Christian Heimes wrote: > On 01/02/2022 16.08, Victor Stinner wrote: > > -- > > > > I would prefer to introduce C API incompatible changes differently: > > first

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

2022-02-01 Thread Christian Heimes
On 01/02/2022 16.08, Victor Stinner wrote: -- I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change. - (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is released - (3) If possible, wait until numpy