[Python-Dev] Re: Making code object APIs unstable

2021-09-07 Thread Guido van Rossum
On Tue, Sep 7, 2021 at 10:00 AM Stefan Behnel wrote: > Guido van Rossum schrieb am 07.09.21 um 00:44: > > In addition, I just heard from the SC that they've approved the > exception. > > So we will remove these two APIs from 3.11 without deprecation. > > Erm, hang on – when I wrote that I'm

[Python-Dev] Re: Making code object APIs unstable

2021-09-07 Thread Stefan Behnel
Guido van Rossum schrieb am 07.09.21 um 00:44: In addition, I just heard from the SC that they've approved the exception. So we will remove these two APIs from 3.11 without deprecation. Erm, hang on – when I wrote that I'm fine with *changing* them, I wasn't thinking of actually *removing*

[Python-Dev] Re: Making code object APIs unstable

2021-09-06 Thread Victor Stinner
Oh, I didn't know this *existing* C API function: PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) So Cython could be modified to use it, no? Victor On Tue, Sep 7, 2021 at 12:44 AM Guido van Rossum wrote: > > On Fri, Sep 3, 2021 at 4:12 PM Victor Stinner

[Python-Dev] Re: Making code object APIs unstable

2021-09-06 Thread Guido van Rossum
On Fri, Sep 3, 2021 at 4:12 PM Victor Stinner wrote: > On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum wrote: > > FWIW I've applied for an exception from the two-release deprecation > policy from the SC: > > https://github.com/python/steering-council/issues/75 > > On the PyPI top 5000

[Python-Dev] Re: Making code object APIs unstable

2021-09-03 Thread Victor Stinner
On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum wrote: > FWIW I've applied for an exception from the two-release deprecation policy > from the SC: > https://github.com/python/steering-council/issues/75 On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I didn't check how many

[Python-Dev] Re: Making code object APIs unstable

2021-09-02 Thread Guido van Rossum
FWIW I've applied for an exception from the two-release deprecation policy from the SC: https://github.com/python/steering-council/issues/75 On Thu, Sep 2, 2021 at 1:12 AM Greg Ewing wrote: > On 2/09/21 7:46 pm, Antoine Pitrou wrote: > > Tracebacks are linked in a single direction, to go the

[Python-Dev] Re: Making code object APIs unstable

2021-09-02 Thread Greg Ewing
On 2/09/21 7:46 pm, Antoine Pitrou wrote: Tracebacks are linked in a single direction, to go the other direction you need to walk the frames attached to the traceback. So a (fake or not) frame object is still desirable, IMHO. Could we at least remove the necessity for a fake code object?

[Python-Dev] Re: Making code object APIs unstable

2021-09-02 Thread Antoine Pitrou
On Thu, 2 Sep 2021 13:31:32 +1200 Greg Ewing wrote: > On 2/09/21 4:46 am, Victor Stinner wrote: > > If creating a fake frame is a common use case, we can maybe write a > > public C API for that. For example, I saw parser injecting frames to > > show the file name and line number of the parsed

[Python-Dev] Re: Making code object APIs unstable

2021-09-02 Thread Petr Viktorin
On 01. 09. 21 22:28, Guido van Rossum wrote: I apologize, I keep making the same mistake. The PyCode_New[WithPosArgs] functions are *not* in the stable ABI or in the limited API, so there's no need to petition the SC, nor do I need Petr's approval. We may be bound by backwards

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Greg Ewing
On 2/09/21 4:46 am, Victor Stinner wrote: If creating a fake frame is a common use case, we can maybe write a public C API for that. For example, I saw parser injecting frames to show the file name and line number of the parsed file in the traceback. The way I would like to see this addressed

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Nick Coghlan
On Thu, 2 Sep 2021, 7:08 am Victor Stinner, wrote: > I saw Python projects injecting fake frames for XML and JSON parsers, > maybe also configuration file (.ini?) parsers. So text files which > have line numbers ;-) > > On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado > wrote: > > I don't

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
I saw Python projects injecting fake frames for XML and JSON parsers, maybe also configuration file (.ini?) parsers. So text files which have line numbers ;-) On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado wrote: > I don't think we should think on those terms. We certainly don't want to be

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Guido van Rossum
I apologize, I keep making the same mistake. The PyCode_New[WithPosArgs] functions are *not* in the stable ABI or in the limited API, so there's no need to petition the SC, nor do I need Petr's approval. We may be bound by backwards compatibility for the *cpython* API, but I think that if Cython

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Guido van Rossum
(context) > Guido van Rossum schrieb am 13.08.21 um 19:24: > > In 3.11 we're changing a lot of details about code objects. Part of this > is > > the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- > Fine > > Grained Error Locations in Tracebacks). > > > > As a result, the set

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Pablo Galindo Salgado
> If creating a fake frame is a common use case, we can maybe write a public C API for that. I don't think we should think on those terms. We certainly don't want to be on a case where yet again we cannot change the internals because we have an official C-API exposed. > For example, I saw

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
If creating a fake frame is a common use case, we can maybe write a public C API for that. For example, I saw parser injecting frames to show the file name and line number of the parsed file in the traceback. Victor On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel wrote: > > Guido van Rossum

[Python-Dev] Re: Making code object APIs unstable

2021-08-31 Thread Stefan Behnel
Guido van Rossum schrieb am 13.08.21 um 19:24: In 3.11 we're changing a lot of details about code objects. Part of this is the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine Grained Error Locations in Tracebacks). As a result, the set of fields of the code object is

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Spencer Brown
Victor Stinner wrote: > Since Cython is a common consumer of this C API, can somone please dig > into Cython to see exactly what it needs in terms of API? How does > Cython create all arguments of the __Pyx_PyCode_New() macro? Does it > copy an existing function to only override some fields,

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Guido van Rossum
Both locations Greg found seem to be creating a completely *empty* code object except for the name, filename and lineno. So that would actually work fine with my proposal. On Tue, Aug 17, 2021 at 4:24 PM Gregory P. Smith wrote: > +cc: cython-devel > > background reading for those new to the

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Gregory P. Smith
+cc: cython-devel background reading for those new to the thread: https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/ On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner wrote: > Since Cython is a common consumer of this C API, can somone please dig

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Victor Stinner
Since Cython is a common consumer of this C API, can somone please dig into Cython to see exactly what it needs in terms of API? How does Cython create all arguments of the __Pyx_PyCode_New() macro? Does it copy an existing function to only override some fields, something like

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Petr Viktorin
I'm late to the thread, and as I read it I see everything I wanted to say was covered already :) So just a few clarifications. The stable ABI is not defined by PEP 384 or PEP 652 or by the header something is defined in, but by the docs: - https://docs.python.org/dev/c-api/stable.html and

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Gregory P. Smith
Doing a search of a huge codebase (work), the predominant user of PyCode_New* APIs appears to be checked in Cython generated code (in all sorts of third_party OSS projects). It's in the boilerplate that Cython extensions make use of via it's __Pyx_PyCode_New macro.

[Python-Dev] Re: Making code object APIs unstable

2021-08-16 Thread Guido van Rossum
On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan wrote: > [...] > A cloning-with-replacement API that accepted the base code object and the > "safe to modify" fields could be a good complement to the API deprecation > proposal. > Yes (I forgot to mention that). > Moving actual "from scratch" code

[Python-Dev] Re: Making code object APIs unstable

2021-08-16 Thread Nick Coghlan
On Tue, 17 Aug 2021, 4:30 am Guido van Rossum, wrote: > On Mon, Aug 16, 2021 at 9:30 AM Steve Dower > wrote: > >> On 8/16/2021 12:47 AM, Guido van Rossum wrote: >> > My current proposal is to issue a DeprecationWarning in PyCode_New() >> and >> > PyCode_NewWithPosArgs(), which can be turned

[Python-Dev] Re: Making code object APIs unstable

2021-08-16 Thread Guido van Rossum
On Mon, Aug 16, 2021 at 9:30 AM Steve Dower wrote: > On 8/16/2021 12:47 AM, Guido van Rossum wrote: > > My current proposal is to issue a DeprecationWarning in PyCode_New() and > > PyCode_NewWithPosArgs(), which can be turned into an error using a > > command-line flag. If it's made an error, we

[Python-Dev] Re: Making code object APIs unstable

2021-08-16 Thread Steve Dower
On 8/16/2021 12:47 AM, Guido van Rossum wrote: My current proposal is to issue a DeprecationWarning in PyCode_New() and PyCode_NewWithPosArgs(), which can be turned into an error using a command-line flag. If it's made an error, we effectively have (B); by default, we have (A). Then in 3.13

[Python-Dev] Re: Making code object APIs unstable

2021-08-15 Thread Guido van Rossum
On Sat, Aug 14, 2021 at 4:56 AM Serhiy Storchaka wrote: > 13.08.21 20:24, Guido van Rossum пише: > > If these weren't part of the stable ABI, I'd choose (E). But because > > they are, I think only (A) or (B) are our options. The problem with (C) > > is that if there's code that links to them but

[Python-Dev] Re: Making code object APIs unstable

2021-08-15 Thread Jim J. Jewett
Guido van Rossum wrote: > On Fri, Aug 13, 2021 at 11:17 AM Terry Reedy tjre...@udel.edu wrote: > > On 8/13/2021 1:24 PM, Guido van Rossum wrote: > I'm actually not sure what use case you're talking about. "source/code that hadn't actually changed and was just being re-compiled and run". I've

[Python-Dev] Re: Making code object APIs unstable

2021-08-14 Thread Serhiy Storchaka
13.08.21 20:24, Guido van Rossum пише: > If these weren't part of the stable ABI, I'd choose (E). But because > they are, I think only (A) or (B) are our options. The problem with (C) > is that if there's code that links to them but doesn't call them (except > in some corner case that the user can

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Terry Reedy
On 8/13/2021 8:45 PM, Guido van Rossum wrote: Now, backwards compatibility is still nothing to sneeze at, but at least we don't have to hem and haw about ABI compatibility. If back compatibility were our sacred prime directive, then we would not (or should not) have changed code objects in

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Guido van Rossum
On Fri, Aug 13, 2021 at 11:17 AM Terry Reedy wrote: > On 8/13/2021 1:24 PM, Guido van Rossum wrote: > [...] > > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the > > PEP 387 stable ABI. What should we do? > > PEP 387 is Backwards Compatibility Policy >

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Jim J. Jewett
How badly would the code objects be crippled? >(no exception table, no endline/column tables, > qualname defaults to name) That sounds like it would be a pain for debugging, but might still work for source/code that hadn't actually changed and was just being re-compiled and run (possibly with

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Eric Snow
On Fri, Aug 13, 2021 at 11:29 AM Guido van Rossum wrote: > If these weren't part of the stable ABI, I'd choose (E). They aren't in the stable ABI (or limited API). Instead, they are part of the broader public API (declared in Include/cpython/code.h, along with "struct PyCodeObject" and others).

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Patrick Reader
I'm a major consumer of these APIs as part of some commercial projects (unfortunately I can't discuss too much further) but we find it worth the effort of keeping up with CPython internal changes to continue using them. Option D seems like the best option from my point of view; any user would

[Python-Dev] Re: Making code object APIs unstable

2021-08-13 Thread Terry Reedy
On 8/13/2021 1:24 PM, Guido van Rossum wrote: In 3.11 we're changing a lot of details about code objects. Part of this is the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine Grained Error Locations in Tracebacks). As a result, the set of fields of the code object is