Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-29 Thread Nathaniel Smith
On Mon, Apr 29, 2019 at 5:01 PM Neil Schemenauer wrote: > As far as I understand, we have a similar problem already for > gc.get_objects() because those static type objects don't have a > PyGC_Head. My 2-cent proposal for fixing things in the long term > would be to introduce a function like

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-29 Thread Victor Stinner
You have my support is you work on removing static types :-) Here are my notes on the current C APIs to define a type: https://pythoncapi.readthedocs.io/type_object.html IMHO static types should go away in the long term. They are causing too many practical issues. Victor Le mar. 30 avr. 2019

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-29 Thread Neil Schemenauer
On 2019-04-27, Nathaniel Smith wrote: > For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce > a list of all live objects on demand. If that's the goal, then static type > objects aren't a huge deal. You can't add extra data into the type objects > themselves, but since

[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-28 Thread Victor Stinner
FYI I pushed my 3 changes to implement my idea. It is now possible to install some extensions in release mode and some others in debug mode. Python in debug mode prefers debug extensions. I documented changes here:

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-27 Thread Nathaniel Smith
On Sat, Apr 27, 2019, 04:27 Armin Rigo wrote: > Hi Neil, > > On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer > wrote: > > Regarding the Py_TRACE_REFS fields, I think we can't do them without > > breaking the ABI because of the following. For GC objects, they are > > always allocated by

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-27 Thread Stefan Behnel
Matthias Klose schrieb am 25.04.19 um 13:48: > Are there use cases where you only want to load *some* > debug extensions, even if more are installed? Not sure if there are _important_ use cases (that could justify certain design decisions), but I can certainly imagine using a non-debug (and

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-27 Thread Armin Rigo
Hi Neil, On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer wrote: > Regarding the Py_TRACE_REFS fields, I think we can't do them without > breaking the ABI because of the following. For GC objects, they are > always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we > can allocate the

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Matthias Klose
On 25.04.19 13:14, Victor Stinner wrote: > Le jeu. 25 avr. 2019 à 09:34, Matthias Klose a écrit : >> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So >> depending on the package maintainer, you already have that available, but it >> is >> extra maintenance cost.

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Matthias Klose
On 25.04.19 13:26, Victor Stinner wrote: > I looked how fonforge gets compiler and linker flags to embed Python: > it seems like to "pkg-config --libs python-2.7" which returns > "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify > Misc/python.pc as well... or not? :-) I'm not

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Victor Stinner
I looked how fonforge gets compiler and linker flags to embed Python: it seems like to "pkg-config --libs python-2.7" which returns "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I don't know if it's common that

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Victor Stinner
Le jeu. 25 avr. 2019 à 09:34, Matthias Klose a écrit : > there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So > depending on the package maintainer, you already have that available, but it > is > extra maintenance cost. Simplifying that would be a good idea. Fedora

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Victor Stinner
Le jeu. 25 avr. 2019 à 09:30, Matthias Klose a écrit : > the purpose of python-config here is not clear. Whether it's intended to be > used > for linking extensions, or embedded interpreters. Currently you are using the > same for both use cases. My PR 12946 removes libpython from distutils,

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Victor Stinner
Hi, I'm now convinced that C extensions must *not* be linked to libpython on Unix. I wrote PR 12946: https://github.com/python/cpython/pull/12946 bpo-21536: C extensions are no longer linked to libpython On Unix, C extensions are no longer linked to libpython. It is now possible

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Matthias Klose
On 25.04.19 08:31, Nathaniel Smith wrote: > You don't necessarily need rpath actually. The Linux loader has a > bug/feature where once it has successfully loaded a library with a given > soname, then any future requests for that soname within the same process > will automatically return that same

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Matthias Klose
On 24.04.19 01:44, Victor Stinner wrote: > Hi, > > Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by > default in debug build", but I lost myself in details, I forgot the > main purpose of my proposal... > > Let me retry from scratch with a more explicit title: I would like to

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Matthias Klose
On 24.04.19 18:02, Victor Stinner wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-25 Thread Nathaniel Smith
You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Neil Schemenauer
On 2019-04-24, Victor Stinner wrote: > The current blocker issue is that the Py_DEBUG define imply the > Py_TRACE_REFS define I think your change to make Py_TRACE_REFS as separate configure flag is fine. I've used the trace fields to debug occasionally but I don't use it often enough to need it

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Ivan Pozdeev via Python-Dev
On 24.04.2019 17:03, Antoine Pitrou wrote: On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner wrote: The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define:

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Stefan Behnel
Jeroen Demeyer schrieb am 24.04.19 um 09:24: > On 2019-04-24 01:44, Victor Stinner wrote: >> I would like to >> be able to run C extensions compiled in release mode on a Python >> compiled in debug mode > > That seems like a very good idea. I would certainly use the debug mode > while developing

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Antoine Pitrou
On Wed, 24 Apr 2019 18:02:18 +0200 Victor Stinner wrote: > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Victor Stinner
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded. When Python is built with --enable-shared, the python3.7

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Antoine Pitrou
On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner wrote: > > The first requirement for the use case is that a Python debug build > supports the ABI of a release build. The current blocker issue is that > the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 > extra fields (_ob_prev

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Ivan Pozdeev via Python-Dev
On 24.04.2019 3:50, Ivan Pozdeev via Python-Dev wrote: On 24.04.2019 2:44, Victor Stinner wrote: Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-24 Thread Jeroen Demeyer
On 2019-04-24 01:44, Victor Stinner wrote: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode That seems like a very good idea. I would certainly use the debug mode while developing CPython or C extensions.

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit : > The current blocker issue is that > the Py_DEBUG define imply the Py_TRACE_REFS define (...): > > https://bugs.python.org/issue36465 > https://github.com/python/cpython/pull/12615 I updated my PR: """ Release build and debug build are now

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Ivan Pozdeev via Python-Dev
On 24.04.2019 2:44, Victor Stinner wrote: Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit : > The first requirement for the use case is that a Python debug build > supports the ABI of a release build. (...) I > propose to no longer imply Py_TRACE_REFS (...) > > Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in >

[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode