[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-02 Thread Eric V. Smith
On 7/2/2020 11:36 AM, Stefan Behnel wrote: Victor Stinner schrieb am 02.07.20 um 00:07: Le mer. 1 juil. 2020 à 23:43, Eric V. Smith a écrit : On 7/1/2020 3:43 PM, Stefan Behnel wrote: Petr Viktorin schrieb am 30.06.20 um 14:51: For example, could we only deprecate the bad parts, but not

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-02 Thread Stefan Behnel
Victor Stinner schrieb am 02.07.20 um 00:07: > Le mer. 1 juil. 2020 à 23:43, Eric V. Smith a écrit : >> >> On 7/1/2020 3:43 PM, Stefan Behnel wrote: >>> Petr Viktorin schrieb am 30.06.20 um 14:51: For example, could we only deprecate the bad parts, but not remove them until the

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Victor Stinner
There is the https://github.com/pyhandle/hpy project which is implemented on top of the existing C API. But this project doesn't solve problems listed in PEP 620, since CPython must continue to support existing C extensions. Victor Le mer. 1 juil. 2020 à 23:43, Eric V. Smith a écrit : > > On

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Eric V. Smith
On 7/1/2020 3:43 PM, Stefan Behnel wrote: Petr Viktorin schrieb am 30.06.20 um 14:51: For example, could we only deprecate the bad parts, but not remove them until the experiments actually show that they are preventing a beneficial change? Big nod on this one. At one of the core sprints

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Stefan Behnel
Petr Viktorin schrieb am 30.06.20 um 14:51: > For example, could we only deprecate the bad parts, but not remove them > until the experiments actually show that they are preventing a beneficial > change? Big nod on this one. Stefan ___ Python-Dev

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Emily Bowman
On Wed, Jul 1, 2020 at 4:09 AM Antoine Pitrou wrote: > How does this help third-party extensions? > If the cost is high enough, exposing the guts of a function to allow the compiler to inline it is not unreasonable; all of the major compilers have ways to inline things that are technically

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Thomas Wouters
On Tue, Jun 30, 2020 at 3:09 PM Petr Viktorin wrote: > On 2020-06-30 02:46, Victor Stinner wrote: > > You missed the point of the PEP: "It becomes possible to experiment > > with more advanced optimizations in CPython than just > > micro-optimizations, like tagged pointers." > > I don't think

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Antoine Pitrou
On Wed, 1 Jul 2020 12:50:01 +0200 Victor Stinner wrote: > Le mer. 1 juil. 2020 à 03:53, Inada Naoki a écrit : > > I confirmed the performance regression, although the difference is 12%. > > And I find the commit cause the regression. > > > >

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-07-01 Thread Victor Stinner
Le mer. 1 juil. 2020 à 03:53, Inada Naoki a écrit : > I confirmed the performance regression, although the difference is 12%. > And I find the commit cause the regression. > > https://github.com/python/cpython/commit/45ec5b99aefa54552947049086e87ec01bc2fc9a > https://bugs.python.org/issue40170 >

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-30 Thread Inada Naoki
On Tue, Jun 30, 2020 at 6:45 AM Raymond Hettinger wrote: > > > > Converting macros to static inline functions should only impact very few > > C extensions which use macros in unusual ways. > > These should be individually verified to make sure they actually get inlined > by the compiler. In

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-30 Thread Petr Viktorin
On 2020-06-30 02:46, Victor Stinner wrote: You missed the point of the PEP: "It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations, like tagged pointers." I don't think experiments are a good motivation. When the C API is broken, everyone

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-30 Thread Antoine Pitrou
On Mon, 29 Jun 2020 23:31:31 -0700 Emily Bowman wrote: > On Mon, Jun 29, 2020 at 7:41 PM Raymond Hettinger < > raymond.hettin...@gmail.com> wrote: > > > Perhaps you don't want to believe the results, but the timings are > > careful, stable, repeatable, and backed-up by a disassembly that shows

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-30 Thread Emily Bowman
On Mon, Jun 29, 2020 at 7:41 PM Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > Perhaps you don't want to believe the results, but the timings are > careful, stable, repeatable, and backed-up by a disassembly that shows the > exact cause. The builds used for the timings were the

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-29 Thread Raymond Hettinger
> On Jun 29, 2020, at 5:46 PM, Victor Stinner wrote: > > You missed the point of the PEP: "It becomes possible to experiment > with more advanced optimizations in CPython than just > micro-optimizations, like tagged pointers." > > IMHO it's time to stop wasting our limited developer resources

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-29 Thread Victor Stinner
You missed the point of the PEP: "It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations, like tagged pointers." IMHO it's time to stop wasting our limited developer resources on micro-optimizations and micro-benchmarks, but think about overall

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-29 Thread Raymond Hettinger
> On Jun 22, 2020, at 5:10 AM, Victor Stinner wrote: > > Introduce C API incompatible changes to hide implementation details. How much of the existing C extension ecosystem do you expect to break as a result of these incompatible changes? > It will be way easier to add new features. This

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-29 Thread Victor Stinner
Hi Stefan, I'm interested in experimenting with a moving GC in CPython, but also by modifying the C API to make sure that it is efficient on PyPy or another Python implementation which uses a moving GC. As Carl in the other thread, currently, other Python implementations have to emulate

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-26 Thread Stefan Behnel
Victor Stinner schrieb am 26.06.20 um 14:39: > Well, the general problem is to track when the caller ends using a resource. Although that is less of a problem if you only allow exposing the internal data representation and nothing else. In that case, you can tie the lifetime of the data access to

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-26 Thread Victor Stinner
,Hi Carl, Le ven. 26 juin 2020 à 07:36, Carl Shapiro a écrit : > It would be very helpful to broaden the objective of avoiding functions > returning PyObject** to other types of pointers. I have in mind several > functions in the C-API that return a char* pointer to the contents of an >

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-25 Thread Carl Shapiro
Thank you very much for putting this PEP together. It would be very helpful to broaden the objective of avoiding functions returning PyObject** to other types of pointers. I have in mind several functions in the C-API that return a char* pointer to the contents of an object. While these

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-25 Thread Victor Stinner
Le mer. 24 juin 2020 à 21:16, Stefan Behnel a écrit : > It couldn't because even today it is still fairly difficult to convert > existing code to the limited API. Some code cannot even be migrated at all, > e.g. because the entire buffer protocol is missing from it. Some bugs were > only fixed in

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-24 Thread Stefan Behnel
Gustavo Carneiro schrieb am 24.06.20 um 19:19: > On Wed, 24 Jun 2020 at 17:22, Victor Stinner wrote: >> The question becomes: how to promote the limited C API? Should it >> become the default, rather than an opt-in option? > > It would be interesting to find out what is the performance impact of

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-24 Thread Stefan Behnel
Victor Stinner schrieb am 24.06.20 um 17:40: > My practical problem is how to prevent C extensions accessing the > PyFloatObject.ob_fval member directly. Do extensions really do that in their code? I mean, there *is* a macro for doing exactly this thing, which suggests that users should exactly

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-24 Thread Gustavo Carneiro
On Wed, 24 Jun 2020 at 17:22, Victor Stinner wrote: [...] > The question becomes: how to promote the limited C API? Should it > become the default, rather than an opt-in option? > It would be interesting to find out what is the performance impact of using limited C API, vs normal API, on some

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-24 Thread Victor Stinner
Le mer. 24 juin 2020 à 16:20, Stefan Behnel a écrit : > Note, I understand the difference between ABI and API. Keeping > PyTuple_GET_ITEM() a macro or inline function can break the ABI at some > point once PyTupleObject changes in an incompatible way in Py3.14, and it > may do different things in

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-24 Thread Stefan Behnel
Victor Stinner schrieb am 24.06.20 um 02:14: > Le mar. 23 juin 2020 à 16:56, Stefan Behnel a écrit : >>> Members of ``PyObject`` and ``PyTupleObject`` structures have not >>> changed since the "Initial revision" commit (1990) >> >> While I see an advantage in hiding the details of PyObject

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Victor Stinner
Le mar. 23 juin 2020 à 16:56, Stefan Behnel a écrit : > > Adding a new member breaks the stable ABI (PEP 384), especially for > > types declared statically (e.g. ``static PyTypeObject MyType = > > {...};``). In Python 3.4, the PEP 442 "Safe object finalization" added > > the ``tp_finalize``

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Victor Stinner
Le mar. 23 juin 2020 à 20:37, Neil Schemenauer a écrit : > My thinking is that, eventually, we would like to allow CPython to > use something other than reference counting for internal PyObject > memory management. In other systems with garbage collection, the > memory allocator is typically

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Neil Schemenauer
On 2020-06-23, Thomas Wouters wrote: > I think the ability for per-type allocation/deallocation routines isn't > really about efficiency, but more about giving more control to embedding > systems (or libraries wrapped by extension modules) about how *their* > objects are allocated. It doesn't make

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Antoine Pitrou
On Tue, 23 Jun 2020 16:34:28 +0200 Victor Stinner wrote: > Le mar. 23 juin 2020 à 15:56, Petr Viktorin a écrit : > > > Adding or removing members of C structures is causing multiple backward > > > compatibility issues. > > > > > > Adding a new member breaks the stable ABI (PEP 384), especially

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Stefan Behnel
Hi Victor, thanks for your continued work on improving the C-API. I'll comment on the PEP inline. Victor Stinner schrieb am 22.06.20 um 14:10: > PEP available at: https://www.python.org/dev/peps/pep-0620/ > [...] > Motivation > == > > The C API blocks CPython evolutions >

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Victor Stinner
Le mar. 23 juin 2020 à 15:56, Petr Viktorin a écrit : > > Adding or removing members of C structures is causing multiple backward > > compatibility issues. > > > > Adding a new member breaks the stable ABI (PEP 384), especially for > > types declared statically (e.g. ``static PyTypeObject MyType

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Petr Viktorin
On 2020-06-22 14:10, Victor Stinner wrote: Hi, PEP available at: https://www.python.org/dev/peps/pep-0620/ This PEP is the result of 4 years of research work on the C API: https://pythoncapi.readthedocs.io/ It's the third version. The first version (2017) proposed to add a "new C API" and

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Thomas Wouters
On Tue, Jun 23, 2020 at 11:33 AM Victor Stinner wrote: > Hi Neil, > > Le mar. 23 juin 2020 à 03:47, Neil Schemenauer > > > One aspect of the API that could be improved is memory management > > for PyObjects. The current API is quite a mess and for no good > > reason except legacy, IMHO. The

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Victor Stinner
Hi Neil, Le mar. 23 juin 2020 à 03:47, Neil Schemenauer a écrit : > Thanks for putting work into this. You're welcome, I took some ideas from your tagged pointer proof of concept ;-) I recall that we met the same C API issues in our experiments ;-) > Changes must be made for > well founded

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-22 Thread Neil Schemenauer
Hi Victor, Thanks for putting work into this. I support the idea of slowly evolving the C API. It must be done carefully so as to not unnecessarily break 3rd party extensions. Changes must be made for well founded reasons and not just because we think it makes a "cleaner" API. I believe you

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-22 Thread Dong-hee Na
Thanks, Victor for awesome PEP I am a big +1 on this proposal since some of the core developers already need the evolution of C API. I believe this proposal is not only for alternative python compiler implementation but also gives a chance for enhancing CPython performance. And I love this