[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-15 Thread Brett Cannon
It seems a little odd to be dictating website updates about other VMs in this PEP. I'm not arguing that we shouldn't update the site, I just think requiring it as part of this PEP seems tangential to what the PEP is focusing on. ___ Python-Dev mailing

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-15 Thread Ronald Oussoren via Python-Dev
> On 15 Apr 2020, at 03:39, Victor Stinner wrote: > > Hi Ronald, > > Le mar. 14 avr. 2020 à 18:25, Ronald Oussoren a > écrit : >> Making “PyObject” opaque will also affect the stable ABI because even types >> defined using the PyTypeSpec API embed a “PyObject” value in the structure >>

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Victor Stinner
Hi Ronald, Le mar. 14 avr. 2020 à 18:25, Ronald Oussoren a écrit : > Making “PyObject” opaque will also affect the stable ABI because even types > defined using the PyTypeSpec API embed a “PyObject” value in the structure > defining the instance layout. It is easy enough to change this in a

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread André Malo
Steve Dower wrote: > On 14Apr2020 1557, André Malo wrote: > > > Stefan Behnel wrote: > > > >> André Malo schrieb am 14.04.20 um 13:39: > >> > >>> A good way to test that promise (or other implications like > >>> performance) > >>> might > >>> > > also be to rewrite the standard library

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Steve Dower
On 14Apr2020 1557, André Malo wrote: Stefan Behnel wrote: André Malo schrieb am 14.04.20 um 13:39: A good way to test that promise (or other implications like performance) might also be to rewrite the standard library extensions in Cython and see where it leads. Not sure I understand

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Matěj Cepl
On 2020-04-14, 12:35 GMT, Stefan Behnel wrote: >> A good way to test that promise (or other implications like performance) >> might >> also be to rewrite the standard library extensions in Cython and see where >> it >> leads. > > Not sure I understand what you're saying here. stdlib extension

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Matěj Cepl
On 2020-04-13, 17:39 GMT, Eric Fahlgren wrote: > Ok, so put that in a Pros/Cons list that provides guidance as to what > interface and tools to choose when writing a new extension module. > Personally, I'd put Cython (and other "big" packages, numpy, requests and > such) on par with CPython itself

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Ronald Oussoren via Python-Dev
> On 10 Apr 2020, at 19:20, Victor Stinner wrote: > […] > > > PEP xxx: Modify the C API to hide implementation details > > > Abstract > > > * Hide implementation

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread André Malo
Stefan Behnel wrote: > André Malo schrieb am 14.04.20 um 13:39: > > > I think, it does not serve well as a policy for CPython. Since we're > > talking hypotheticals right now, if Cython vanishes tomorrow, we're > > kind of left empty handed. Such kind of a runtime, if considered part of > > the

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Stefan Behnel
Paul Moore schrieb am 13.04.20 um 14:25: > On a related but different note, what is the recommended policy > (assuming it's not to use the C API) for embedding Python, and for > exposing the embedding app to Python as a C extension? My standard > example of this is the Vim interface to Python -

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Stefan Behnel
André Malo schrieb am 14.04.20 um 13:39: > I think, it does not serve well as a policy for CPython. Since we're talking > hypotheticals right now, if Cython vanishes tomorrow, we're kind of left > empty > handed. Such kind of a runtime, if considered part of the compatibility > "promise",

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread Stefan Behnel
Steve Dower schrieb am 14.04.20 um 00:27: > On 13Apr2020 2308, André Malo wrote: >> For one thing, if you open up APIs for Cython, they're open for everybody >> (Cython being "just" another C extension). >> More to the point: The ABIs have the same problem as they have now, >> regardless >> how

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-14 Thread André Malo
Steve Dower wrote: > On a policy level, we don't make changes that would break users of the C > API. Because we can't track everyone who's using it, we have to assume > that everything is used and any change will cause breakage. > > To make sure it's possible to keep developing CPython, we

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 13Apr2020 2308, André Malo wrote: For one thing, if you open up APIs for Cython, they're open for everybody (Cython being "just" another C extension). More to the point: The ABIs have the same problem as they have now, regardless how responsive the Cython developers are. Once you compiled the

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread André Malo
Steve Dower wrote: > On 11Apr2020 0025, Antoine Pitrou wrote: > > On Fri, 10 Apr 2020 23:33:28 +0100 > > > > Steve Dower wrote: > >> On 10Apr2020 2055, Antoine Pitrou wrote: > >>> On Fri, 10 Apr 2020 19:20:00 +0200 > >>> > >>> Victor Stinner wrote: > Note: Cython and cffi should be

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 13Apr2020 2105, Chris Meyer wrote: How would I call a Python function from the C++ application that returns a Python object to C++ and then call a method on that Python object from C++? My specific example is that I create Python handlers for Qt windows and then from the Qt/C++ I call

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Daniel Holth
Sorry that this is a bit off-topic. cffi would be a user of any new C API. I've tried to make sure ABI3 is supported in setuptools and wheel, with varying success. Apparently virtualenvs and Windows have problems. I'm excited about the possibility of a better C API and possibly ABI.

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Chris Meyer
> On Apr 13, 2020, at 11:26 AM, Daniel Holth wrote: > > Was it regular cffi or cffi's embedding API, which is used a bit differently > than regular cffi, that "seems to only solve a fraction of the problem"? Was > just playing around with the embedding API and was impressed. > > In Python: >

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Daniel Holth
It can be done exactly as passing a void* when registering a C callback, and getting it passed back to your callback function. https://cffi.readthedocs.io/en/latest/ref.html#ffi-new-handle-ffi-from-handle

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Daniel Holth
Was it regular cffi or cffi's embedding API, which is used a bit differently than regular cffi, that "seems to only solve a fraction of the problem"? Was just playing around with the embedding API and was impressed. In Python: @ffi.def_extern() def uwsgi_pyexample_init(): print("init called")

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Eric Fahlgren
On Mon, Apr 13, 2020 at 9:00 AM Steve Dower wrote: > On 13Apr2020 1325, Paul Moore wrote: > > Personally, I'd say that "recommended 3rd party tools" reads as saying > > "if you want a 3rd party tool to build extensions, these are good (and > > are a lot easier than using the raw C API)". That's

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Chris Meyer
> On Apr 13, 2020, at 5:25 AM, Paul Moore wrote: > > On a related but different note, what is the recommended policy > (assuming it's not to use the C API) for embedding Python, and for > exposing the embedding app to Python as a C extension? My standard > example of this is the Vim interface to

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 13Apr2020 1325, Paul Moore wrote: Personally, I'd say that "recommended 3rd party tools" reads as saying "if you want a 3rd party tool to build extensions, these are good (and are a lot easier than using the raw C API)". That's a lot different than saying "we recommend that people writing C

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Fabio Zadrozny
> * Hide implementation details from the C API to be able to `optimize > CPython`_ and make PyPy more efficient. > * The expectation is that `most C extensions don't rely directly on > CPython internals`_ and so will remain compatible. > * Continue to support old unmodified C extensions by

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Rhodri James
On 13/04/2020 11:17, Steve Dower wrote: On 11Apr2020 1156, Rhodri James wrote: On 10/04/2020 18:20, Victor Stinner wrote: Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. If this is true, the

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Paul Moore
On Mon, 13 Apr 2020 at 11:20, Steve Dower wrote: > > On 11Apr2020 1156, Rhodri James wrote: > > On 10/04/2020 18:20, Victor Stinner wrote: > >> Note: Cython and cffi should be preferred to write new C extensions. > >> This PEP is about existing C extensions which cannot be rewritten with > >>

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 13Apr2020 1122, Steve Dower wrote: On 11Apr2020 0111, Victor Stinner wrote: Steve: the use case is to debug very rare Python crashes (ex: once every two months) of customers who fail to provide a reproducer. My *expectation* is that a debug build should help to reproduce the bug and/or

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 13Apr2020 1157, Antoine Pitrou wrote: On Mon, 13 Apr 2020 11:35:34 +0100 Steve Dower wrote: and this code that they're using doesn't have any system dependencies that differ in debug builds (spoiler: they do). Are you talking about Windows? On non-Windows systems, I don't think there are

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Antoine Pitrou
On Mon, 13 Apr 2020 11:35:34 +0100 Steve Dower wrote: > > Neither Windows not macOS support fork (macOS only recently). Victor's argument: "fork() is not terrific with inline reference counts". My argument: people shouldn't generally use fork() anyway, because it has other issues. My

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 11Apr2020 0025, Antoine Pitrou wrote: On Fri, 10 Apr 2020 23:33:28 +0100 Steve Dower wrote: On 10Apr2020 2055, Antoine Pitrou wrote: On Fri, 10 Apr 2020 19:20:00 +0200 Victor Stinner wrote: Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 11Apr2020 1156, Rhodri James wrote: On 10/04/2020 18:20, Victor Stinner wrote: Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. If this is true, the documentation on python.org needs a

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-13 Thread Steve Dower
On 11Apr2020 0111, Victor Stinner wrote: Steve: the use case is to debug very rare Python crashes (ex: once every two months) of customers who fail to provide a reproducer. My *expectation* is that a debug build should help to reproduce the bug and/or provide more information when the bug

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-11 Thread Phil Thompson
On 11/04/2020 13:08, Ivan Pozdeev via Python-Dev wrote: On 10.04.2020 20:20, Victor Stinner wrote: Stable ABI -- The idea is to build a C extension only once: the built binary will be usable on multiple Python runtimes and different versions of the same runtime (stable ABI). The

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-11 Thread Ivan Pozdeev via Python-Dev
On 10.04.2020 20:20, Victor Stinner wrote: Hi, Here is a first draft a PEP which summarize the research work I'm doing on CPython C API since 2017 and the changes that me and others already made since Python 3.7 towards an "opaque" C API. The PEP is also a collaboration with developers of

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-11 Thread Rhodri James
On 10/04/2020 18:20, Victor Stinner wrote: Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. If this is true, the documentation on python.org needs a serious rewrite. I am in the throes of

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-11 Thread Antoine Pitrou
On Sat, 11 Apr 2020 01:52:13 +0200 Victor Stinner wrote: > > By the way, CPython currently uses statically allocated types for > builtin types like str or list. This may have to change to run > efficiently multiple subinterepters in parallel: each subinterpeter > should have its own

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-11 Thread Antoine Pitrou
On Sat, 11 Apr 2020 02:11:41 +0200 Victor Stinner wrote: > Le ven. 10 avr. 2020 à 22:00, Antoine Pitrou a écrit : > > > Debug runtime and remove debug checks in release mode > > > . > > > > > > If the C extensions are no longer tied to CPython

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Victor Stinner
Le ven. 10 avr. 2020 à 22:00, Antoine Pitrou a écrit : > > Debug runtime and remove debug checks in release mode > > . > > > > If the C extensions are no longer tied to CPython internals, it becomes > > possible to switch to a Python runtime

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Victor Stinner
Le ven. 10 avr. 2020 à 22:00, Antoine Pitrou a écrit : > How do you keep fast type checking such as PyTuple_Check() if extension > code doesn't have access e.g. to tp_flags? > > I notice you did: > """ > Add fast inlined version _PyType_HasFeature() and _PyType_IS_GC() > for object.c and

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Victor Stinner
Le ven. 10 avr. 2020 à 22:00, Antoine Pitrou a écrit : > > Examples of issues to make structures opaque: > > > > * ``PyGC_Head``: https://bugs.python.org/issue40241 > > * ``PyObject``: https://bugs.python.org/issue39573 > > * ``PyTypeObject``: https://bugs.python.org/issue40170 > > How do you

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Antoine Pitrou
On Fri, 10 Apr 2020 23:33:28 +0100 Steve Dower wrote: > On 10Apr2020 2055, Antoine Pitrou wrote: > > On Fri, 10 Apr 2020 19:20:00 +0200 > > Victor Stinner wrote: > >> > >> Note: Cython and cffi should be preferred to write new C extensions. > >> This PEP is about existing C extensions which

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Steve Dower
On 10Apr2020 2055, Antoine Pitrou wrote: On Fri, 10 Apr 2020 19:20:00 +0200 Victor Stinner wrote: Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. Using Cython does not make the C API

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

2020-04-10 Thread Antoine Pitrou
On Fri, 10 Apr 2020 19:20:00 +0200 Victor Stinner wrote: > > Note: Cython and cffi should be preferred to write new C extensions. > This PEP is about existing C extensions which cannot be rewritten with > Cython. Using Cython does not make the C API irrelevant. In some applications, the C API