Re: sysconfig default scheme change in Python 3.10

2022-03-28 Thread Stefano Rivera
Hi Simon (2022.03.28_21:44:38_+)

> On Mon, 28 Mar 2022 at 21:17:40 +, Stefano Rivera wrote:
> > We've fixed this issue in pybind11 and automake-1.16, so far.
> 
> Does this mean that packages that build using Automake, but use the
> pregenerated configure/Makefile.in provided by the upstream maintainer
> (often on an older or non-Debian distro) instead of regenerating the
> build system with dh_autoreconf or equivalent, are going to be broken now?

The change is to python.m4, so it gets included in aclocal.m4 and
configure. If you rebuild those, you'll get the correct behaviour.

I don't think the generated makefiles are affected, just the configured
paths.

We have upstreamed the patch, https://debbugs.gnu.org/54412 so
eventually upstreams will ship something that works correctly, again.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272



Re: sysconfig default scheme change in Python 3.10

2022-03-28 Thread Simon McVittie
On Mon, 28 Mar 2022 at 21:17:40 +, Stefano Rivera wrote:
> We've fixed this issue in pybind11 and automake-1.16, so far.

Does this mean that packages that build using Automake, but use the
pregenerated configure/Makefile.in provided by the upstream maintainer
(often on an older or non-Debian distro) instead of regenerating the
build system with dh_autoreconf or equivalent, are going to be broken now?

If true, this will hopefully not affect too many packages, because using
dh_autoreconf is the debhelper default and has been considered a Debian
best-practice for a while; but it seems worth being aware of, because I'm
reasonably sure there are still some packages that deliberately do not
regenerate the build system, and some of those probably build CPython
modules.

smcv



sysconfig default scheme change in Python 3.10

2022-03-28 Thread Stefano Rivera
I realise we made a consequential changes in Python 3.10, that was never
announced to the team. At the time, I didn't realise how consequential
it would be, but it has forced a few packages to change in response.

The change was in python3.10 3.10.2-3 (25 Feb 2022).

The default sysconfig change has changed from posix_prefix to
posix_local. This matches the long-existing behaviour in Debian's
distutils.sysconfig. distutils is being removed from Python's stdlib in
3.12 [PEP-632], so setuptools etc. are being migrated to be configured
by sysconfig instead.

The practical reason for this is to make "python3 setup.py install"
install to /usr/local/lib/... rather than into the generally
package-managed /usr/lib/python3/.

$ python3.10 -m sysconfig
...
Current installation scheme: "posix_local"

Paths:
data = "/usr/local"
include = "/usr/include/python3.10"
platinclude = "/usr/include/python3.10"
platlib = "/usr/local/lib/python3.10/dist-packages"
platstdlib = "/usr/lib/python3.10"
purelib = "/usr/local/lib/python3.10/dist-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/lib/python3.10"
...

This has caused some changes for some Debian packages. Generally it can
be fixed at a build-tool level, and shouldn't break other Python
packages, but for non-standard tools that haven't been fixed yet, you
may notice that modules install into /usr/local instead of /usr.
dh_python3 will move modules from /usr/local back to /usr, but it's
called after dh_install, so dh_install configuration can get broken by
this change.

The solution in build tools is to explicitly select the posix_prefix
scheme, and specify the appropriate prefix (/usr or /usr/local). For
build tools that default to /usr/local, this should be all you need to
do.

We've fixed this issue in pybind11 and automake-1.16, so far.
If you find any more tools that need work, feel free to ask me to help.

There have been some tweaks to this, since 3.10.2-3, to reduce breakage.
One of the significant wants was to not prefix the include paths with
/local/, as they are defined as referencing the location of cPython's
headers, not the install location for headers [bpo-5]. But some
build tools may have their own interpretations of this...

Related to this change is something else worth noting:

Because distutils is being removed, modern setuptools now carries its
own distutils fork, internally. This doesn't have Debian's patches
implementing these scheme differences. Instead, it's configured by the
presence of a _distutils_system_mod module, which monkey-patches
distutils.

[PEP-632]: https://peps.python.org/pep-0632/
[bpo-5]: https://bugs.python.org/issue5

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272