Re: [gentoo-dev] [PATCH 04/12] python-utils-r1.eclass: Use wrapper scripts to fix cross-compiling
On Fri, 04 Jan 2019 16:53:46 +0100
Michał Górny wrote:
> On Thu, 2019-01-03 at 21:39 +, James Le Cuirot wrote:
> > Python has little concept of cross-compiling but it turns out that
> > making it work isn't so hard after all.
> >
> > Platform-specific details are held within _sysconfigdata.py,
> > sysconfig.py, and various distutils files. If we simply symlink these
> > from SYSROOT into an empty directory and add that directory to
> > PYTHONPATH then we can utilise the build host's Python with the target
> > host's settings.
> >
> > The pkg-config files were already being symlinked in a similar manner
> > but now we source them from within SYSROOT.
> >
> > In order for PYTHONPATH to be respected, Python must be executed via
> > the wrapper, which was not the case before. We previously relied
> > solely on the PATH but now PYTHON must point to the wrapper rather
> > than the usual location under /usr/bin. However, we only do this when
> > SYSROOT or EPREFIX are effectively set to avoid unnecessary
> > complexity. This has required some rejigging in the way that PYTHON is
> > set but it should remain compatible with existing packages.
> >
> > The python_wrapper_setup function that handles all this has had its
> > arguments reordered because no one ever uses the path/workdir
> > argument, which makes specifying other arguments tedious.
>
> This really belongs in a separate patch.
Fair enough.
> > Some packages rely on python-config but luckily this is just a shell
> > script so it can be executed from within SYSROOT. This is bending the
> > rules of PMS slightly but Python leaves us with little choice. I also
> > wrote those rules so I'm allowed to bend them. ;)
> >
> > PYTHON_INCLUDEDIR, PYTHON_LIBPATH, and their associated functions are
> > generally used during src_configure or src_compile and, as such, they
> > now need to have SYSROOT prepended.
> >
> > python_doheader uses PYTHON_INCLUDEDIR to install new headers and
> > therefore needs the value without SYSROOT. It was already stripping
> > EPREFIX before so now it simply strips SYSROOT as well. Similar
> > instances of this can do likewise or call the functions with SYSROOT
> > unset to achieve the same effect.
> >
> > To make all this work, we are assuming that CPython is located at
> > /usr/$(get_libdir)/${EPYTHON}, which is admittedly a little circular
> > when we are querying Python for the right paths. I feel the reason
> > that python_export was rewritten to query these dynamically was less
> > because someone might install Python to some weird location and more
> > to avoid special handling for each of the different
> > implementations. And what of those other implementations?
>
> This is a wrong assumption. CPython 3.7 is in /usr/lib/python3.7.
It was true at the time I wrote it. I addressed 3.7 later in the patch
series. I figured this diff was long enough already so I kept it
separate.
> > Being Java-based, Jython is installed under the platform-neutral
> > /usr/share and presumably has few other platform-specific
> > aspects. Enabling native extensions appears non-trivial and none of
> > our module packages have enabled support for it anyway.
> >
> > I think PyPy could potentially support cross-compiling but it
> > hardcodes the native extension filename suffix within its own binaries
> > with no way to override it. Perhaps we could patch this in somehow but
> > I'm afraid I don't care enough.
> >
> > Together with the following changes to distutils-r1.eclass, I have now
> > been able to cross-compile a large number of packages with native
> > Python extensions, most with no changes at all, and the rest with only
> > minor fixes.
> >
> > Closes: https://bugs.gentoo.org/503874
> > Bug: https://bugs.gentoo.org/648652
> > Signed-off-by: James Le Cuirot
> > ---
> > eclass/python-utils-r1.eclass | 120 ++
> > 1 file changed, 92 insertions(+), 28 deletions(-)
> >
> > diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> > index 1a549f49f3f2..607af1b52f75 100644
> > --- a/eclass/python-utils-r1.eclass
> > +++ b/eclass/python-utils-r1.eclass
> > @@ -211,9 +211,15 @@ _python_impl_matches() {
> > #
> > # distutils-r1: Set within any of the python sub-phase functions.
> > #
> > -# Example value:
> > +# If SYSROOT or EPREFIX are effectively set then this will point to an
> > +# automatically generated wrapper rather than the usual path under
> > +# /usr/bin in order to accommodate cross-compiling. We could do this all
> > +# the time but it would add unnecessary complexity.
> > +#
> > +# Example values:
> > # @CODE
> > # /usr/bin/python2.7
> > +# /var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7
> > # @CODE
> >
> > # @ECLASS-VARIABLE: EPYTHON
> > @@ -256,6 +262,10 @@ _python_impl_matches() {
> > # Set and exported on request using python_export().
> > # Requires a proper build-time dependency on the Python implementation.
> > #
> >
Re: [gentoo-dev] [PATCH 04/12] python-utils-r1.eclass: Use wrapper scripts to fix cross-compiling
On Thu, 2019-01-03 at 21:39 +, James Le Cuirot wrote:
> Python has little concept of cross-compiling but it turns out that
> making it work isn't so hard after all.
>
> Platform-specific details are held within _sysconfigdata.py,
> sysconfig.py, and various distutils files. If we simply symlink these
> from SYSROOT into an empty directory and add that directory to
> PYTHONPATH then we can utilise the build host's Python with the target
> host's settings.
>
> The pkg-config files were already being symlinked in a similar manner
> but now we source them from within SYSROOT.
>
> In order for PYTHONPATH to be respected, Python must be executed via
> the wrapper, which was not the case before. We previously relied
> solely on the PATH but now PYTHON must point to the wrapper rather
> than the usual location under /usr/bin. However, we only do this when
> SYSROOT or EPREFIX are effectively set to avoid unnecessary
> complexity. This has required some rejigging in the way that PYTHON is
> set but it should remain compatible with existing packages.
>
> The python_wrapper_setup function that handles all this has had its
> arguments reordered because no one ever uses the path/workdir
> argument, which makes specifying other arguments tedious.
This really belongs in a separate patch.
>
> Some packages rely on python-config but luckily this is just a shell
> script so it can be executed from within SYSROOT. This is bending the
> rules of PMS slightly but Python leaves us with little choice. I also
> wrote those rules so I'm allowed to bend them. ;)
>
> PYTHON_INCLUDEDIR, PYTHON_LIBPATH, and their associated functions are
> generally used during src_configure or src_compile and, as such, they
> now need to have SYSROOT prepended.
>
> python_doheader uses PYTHON_INCLUDEDIR to install new headers and
> therefore needs the value without SYSROOT. It was already stripping
> EPREFIX before so now it simply strips SYSROOT as well. Similar
> instances of this can do likewise or call the functions with SYSROOT
> unset to achieve the same effect.
>
> To make all this work, we are assuming that CPython is located at
> /usr/$(get_libdir)/${EPYTHON}, which is admittedly a little circular
> when we are querying Python for the right paths. I feel the reason
> that python_export was rewritten to query these dynamically was less
> because someone might install Python to some weird location and more
> to avoid special handling for each of the different
> implementations. And what of those other implementations?
This is a wrong assumption. CPython 3.7 is in /usr/lib/python3.7.
>
> Being Java-based, Jython is installed under the platform-neutral
> /usr/share and presumably has few other platform-specific
> aspects. Enabling native extensions appears non-trivial and none of
> our module packages have enabled support for it anyway.
>
> I think PyPy could potentially support cross-compiling but it
> hardcodes the native extension filename suffix within its own binaries
> with no way to override it. Perhaps we could patch this in somehow but
> I'm afraid I don't care enough.
>
> Together with the following changes to distutils-r1.eclass, I have now
> been able to cross-compile a large number of packages with native
> Python extensions, most with no changes at all, and the rest with only
> minor fixes.
>
> Closes: https://bugs.gentoo.org/503874
> Bug: https://bugs.gentoo.org/648652
> Signed-off-by: James Le Cuirot
> ---
> eclass/python-utils-r1.eclass | 120 ++
> 1 file changed, 92 insertions(+), 28 deletions(-)
>
> diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> index 1a549f49f3f2..607af1b52f75 100644
> --- a/eclass/python-utils-r1.eclass
> +++ b/eclass/python-utils-r1.eclass
> @@ -211,9 +211,15 @@ _python_impl_matches() {
> #
> # distutils-r1: Set within any of the python sub-phase functions.
> #
> -# Example value:
> +# If SYSROOT or EPREFIX are effectively set then this will point to an
> +# automatically generated wrapper rather than the usual path under
> +# /usr/bin in order to accommodate cross-compiling. We could do this all
> +# the time but it would add unnecessary complexity.
> +#
> +# Example values:
> # @CODE
> # /usr/bin/python2.7
> +# /var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7
> # @CODE
>
> # @ECLASS-VARIABLE: EPYTHON
> @@ -256,6 +262,10 @@ _python_impl_matches() {
> # Set and exported on request using python_export().
> # Requires a proper build-time dependency on the Python implementation.
> #
> +# This is prepended with SYSROOT in order to accommodate
> +# cross-compiling. You may need to strip SYSROOT and EPREFIX if using it
> +# to install new headers.
> +#
> # Example value:
> # @CODE
> # /usr/include/python2.7
> @@ -270,6 +280,9 @@ _python_impl_matches() {
> # Valid only for CPython. Requires a proper build-time dependency
> # on the Python implementation.
> #
> +# This is prepended with
