[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[Michael Haubenwallner]

Change by Michael Haubenwallner :


--
pull_requests: +13592
pull_request: https://github.com/python/cpython/pull/737

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:

The initial issue has been fixed. As far as I know, all known issues have been 
fixed, so I close the issue. Thanks Miro for the help in reviews and tests!

FYI waf is already fixed for the future Python 3.8 beta1! It now attempts to 
use --embed for its "pyembed" config ;-) Miro wrote a fix:

https://gitlab.com/ita1024/waf/merge_requests/2236
https://gitlab.com/ita1024/waf/issues/2239

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:


New changeset bc66faccb8a6140e7e07b5e67843b7f21152c144 by Victor Stinner in 
branch 'master':
bpo-36721: Fix pkg-config symbolic links on "make install" (GH-13551)
https://github.com/python/cpython/commit/bc66faccb8a6140e7e07b5e67843b7f21152c144


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:

By the way, I'm not sure that the current layout of .pc files. The name 
"module" give a different configuration. Is that correct?

$ grep ^Libs: /opt/py38/lib/pkgconfig/python-3.8-embed.pc
Libs: -L${libdir} -lpython3.8

$ grep ^Libs: /opt/py38dbg/lib/pkgconfig/python-3.8-embed.pc
Libs: -L${libdir} -lpython3.8d

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13462

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:

Miro commented my PR:
https://github.com/python/cpython/pull/13500#issuecomment-495510268

"""
This is what it gave me in Fedora, feels a bit inconsistent:

   /usr/lib64/pkgconfig/python-3.8-embed.pc
   /usr/lib64/pkgconfig/python-embed-3.8d.pc
   /usr/lib64/pkgconfig/python3-embed.pc
"""

I installed Python in release mode:

$ ./configure --enable-shared --prefix=/opt/py38 CFLAGS="-O0" && make && make 
install

It gives me:

vstinner@apu$ ls -l /opt/py38/lib/pkgconfig/
-rw-r--r--. 1 vstinner vstinner 312 24 mai   17:39 python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 286 24 mai   17:39 python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner  19 24 mai   17:39 python3-embed.pc -> 
python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner  13 24 mai   17:39 python3.pc -> python-3.8.pc

Debug build:

$ ./configure --enable-shared --prefix=/opt/py38dbg --with-pydebug CFLAGS="-O0" 
&& make && make install

It gives me:

vstinner@apu$ ls -l /opt/py38dbg/lib/pkgconfig/
lrwxrwxrwx. 1 vstinner vstinner  13 24 mai   17:43 python-3.8d.pc -> 
python-3.8.pc
-rw-r--r--. 1 vstinner vstinner 317 24 mai   17:43 python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 290 24 mai   17:43 python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner  19 24 mai   17:43 python3-embed.pc -> 
python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner  13 24 mai   17:43 python3.pc -> python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner  19 24 mai   17:43 python-embed-3.8d.pc -> 
python-embed-3.8.pc

Oh! "python-embed-3.8d.pc" is a broken symbolic link to "python-embed-3.8.pc" 
(which doesn't exist).


Ok, now I get it: I messed up in names of symbolic links :-) I wrote PR 13551 
which gives me the following files for a debug build:

vstinner@apu$ ls -l /opt/py38dbg/lib/pkgconfig/*.pc
lrwxrwxrwx. 1 vstinner vstinner  19 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python-3.8d-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner  13 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python-3.8d.pc -> python-3.8.pc
-rw-r--r--. 1 vstinner vstinner 317 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 290 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner  19 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python3-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner  13 24 mai   17:51 
/opt/py38dbg/lib/pkgconfig/python3.pc -> python-3.8.pc


The format is now: "python" "version" "embed suffix".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:

Even if I'm not confident in my change (add --embed option), I chose to merge 
it anyway since at least "waf" build system is broken by my other changes (no 
longer link C extensions to libpython). I would like to get this change into 
Python 3.8 beta1 to attempt to fix most applications embedding Python.

Anyway, if something goes wrong, we still have plenty of time to decide what to 
do before 3.8.0 final, scheduled for 2019-10-21: 
https://www.python.org/dev/peps/pep-0569/

--

Since I merged my change, I reset the priority from Release Blocker to normal.

--
priority: release blocker -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0a8e57248b913851640c64375600f05157c997df by Victor Stinner in 
branch 'master':
bpo-36721: Add --embed option to python-config (GH-13500)
https://github.com/python/cpython/commit/0a8e57248b913851640c64375600f05157c997df


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +13415
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed and --embed to python3.8-config

[STINNER Victor]

Change by STINNER Victor :


--
title: Add pkg-config python-3.8-embed -> Add pkg-config python-3.8-embed and 
--embed to python3.8-config

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[Matthias Klose]

Matthias Klose  added the comment:

"AFAICT, the purpose of python-config is to provide configuration info for 
embedding Python"

If that's the intention, then at least it's not used as such.  It's also used 
to build/configure extensions using automake/cmake based build systems.  There 
is one tool, and two different use cases.  I think Victor's suggestion is 
appropriate

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[STINNER Victor]

STINNER Victor  added the comment:

I mark this issue as a release blocker: bpo-21536 basically broke the 
compilation of all applications which embed Python. IMHO this issue is the best 
solution to fix it.


> I don't understand the need for this.

Oh. Let me explain this issue differently. There are two use cases for 
libpython:

* Build a C extension and load it in Python: use case called "pyext" by waf
* Embed Python into an application: use case called "pyembed" by waf

My bpo-21536 broke waf "pyembed" which fails to detect Python. waf "pyembed" 
creates a C program which calls Py_Initialize(), but the linker fails to locate 
Py_Initialize() symbol. To embed Python into an application, we really need to 
link the application to libpython: we need -lpython3.8.

In Python 3.7, "pyext" and "pyembed" use cases were covered both by default 
"python3.7-config --libs" and "pkg-config python3.7 --libs" configuration.

In Python 3.8, we now have to distinguish both use cases and provide 
*different* configuration depending if binary must be linked to libpython or 
not.

More info about waf breakage in my waf bug report:
https://gitlab.com/ita1024/waf/issues/2239

FYI the waf breakage was discovered by trying to build libtdb on Fedora Rawhide 
with Python 3.8a4, it's a dependency of Samba which embeds Python:
https://bugzilla.redhat.com/show_bug.cgi?id=1711638

--
nosy: +lukasz.langa
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[Miro Hrončok]

Miro Hrončok  added the comment:

As a note, waf seems to use python3-config for both embedded and extension 
modules. Currently, embedded is broken.

See for example https://bugzilla.redhat.com/show_bug.cgi?id=1711638

--
nosy: +hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[Ned Deily]

Ned Deily  added the comment:

I don't understand the need for this.  AFAICT, the purpose of python-config is 
to provide configuration info for embedding Python (Issue1161914 and 
https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems).
  At some point, a pkg-config template was added (in Issue3585) but it seems to 
duplicate the purpose of python-config and thus is also intended for use in 
embedding Python (although we don't seem to document it, it is familiar to 
users of automake). I don't know what other purposes either python-config or 
the pkg-config template serve other than for embedding. AFAIK, they should not 
be used for building C extension modules. But admittedly this area is not 
well-documented.  In any case, I think pkg-config and python-config should 
return the same values for similar parameters.  Anyone else have an opinion?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[Scott Talbert]

Change by Scott Talbert :


--
nosy: +swt2c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36721] Add pkg-config python-3.8-embed

[STINNER Victor]

New submission from STINNER Victor :

The bpo-21536 modified how C extensions are built: they are no longer linked to 
libpython. The problem is that when an application embeds Python: the 
application wants to be linked to libpython.

commit 8c3ecc6bacc8d0cd534f2b5b53ed962dd1368c7b (HEAD -> master, 
upstream/master)
Author: Victor Stinner 
Date:   Thu Apr 25 20:13:10 2019 +0200

bpo-21536: C extensions are no longer linked to libpython (GH-12946)

On Unix, C extensions are no longer linked to libpython.

It is now possible to load a C extension built using a shared library
Python with a statically linked Python.

When Python is embedded, libpython must not be loaded with
RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it
was already not possible to load C extensions which were not linked
to libpython, like C extensions of the standard library built by the
"*shared*" section of Modules/Setup.

distutils, python-config and python-config.py have been modified.

I chose to modify distutils, python-config (shell) and python-config.py 
(Python), but *not* Misc/python.pc (pkg-config).

Previously, we had:

$ pkg-config python-3.7 --libs
-lpython3.7m 
$ python3.7-config --libs
-lpython3.7m -lcrypt -lpthread -ldl -lutil -lm 
$ python3.7-config.py --libs
-lpython3.7m -lcrypt -lpthread -ldl -lutil -lm

Now, we get:

$ pkg-config python-3.8 --libs
-lpython3.8
$ python3.8-config --libs
-lcrypt -lpthread -ldl -lutil -lm -lm 
$ python-config.py --libs
-lcrypt -lpthread -ldl -lutil -lm -lm

python-config and python-config.py can now be used to build C extensions, but 
not to build an application embedding Python.

pkg-config should not be used to build a C extenstion since it links to 
libpython, but we don't want to do that (see bpo-21536).

I'm not sure that different tools should return different results.

I propose:

* Add a new command "pkg-config python-3.8-embed"
* Add a new "--embed" option to python3.8-config and python3.8-config.py 
commands
* Remove "-lpython@VERSION@@ABIFLAGS@" from "Libs: -L${libdir} 
-lpython@VERSION@@ABIFLAGS@" of Misc/python.pc.in

On Windows, we already provide different binaries for embedded Python with 
"-embed" suffix:

* Download Windows x86-64 embeddable zip file: python-3.7.3-embed-amd64.zip
* Download Windows x86-64 executable installer: python-3.7.3-amd64.exe

https://www.python.org/downloads/windows/

--
components: Build
messages: 340853
nosy: doko, vstinner
priority: normal
severity: normal
status: open
title: Add pkg-config python-3.8-embed
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com