Re: [gentoo-dev] [PATCH 02/12] python-utils-r1.eclass: New python_fix_shebang approach

[Michał Górny]

On Thu, 2019-01-03 at 21:39 +, James Le Cuirot wrote:
> The previous approach would erroneously match foopython. The new
> approach requires the match to start the string or be preceeded by a
> slash, the only two cases we actually want. It does this with slightly
> less code and allows the replacement of whole path strings that would
> be problematic when passed to sed. This will be needed when
> cross-compiling is addressed.
> 
> Signed-off-by: James Le Cuirot 
> ---
>  eclass/python-utils-r1.eclass | 78 ++-
>  1 file changed, 31 insertions(+), 47 deletions(-)
> 
> diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> index da76a755fb34..1eca0764a202 100644
> --- a/eclass/python-utils-r1.eclass
> +++ b/eclass/python-utils-r1.eclass
> @@ -1165,8 +1165,7 @@ python_fix_shebang() {
>   [[ -d ${path} ]] && is_recursive=1
>  
>   while IFS= read -r -d '' f; do
> - local shebang i
> - local error= from=
> + local shebang i= error= fix=
>  
>   # note: we can't ||die here since read will fail if file
>   # has no newline characters
> @@ -1175,65 +1174,56 @@ python_fix_shebang() {
>   # First, check if it's shebang at all...
>   if [[ ${shebang} == '#!'* ]]; then
>   local split_shebang=()
> - read -r -a split_shebang <<<${shebang} || die
> + read -r -a split_shebang <<<${shebang#\#\!} || 
> die

Does '!' really need to be escaped?

>  
>   # Match left-to-right in a loop, to avoid 
> matching random
>   # repetitions like 'python2.7 python2'.
> - for i in "${split_shebang[@]}"; do
> - case "${i}" in
> - *"${EPYTHON}")
> + for i in $(seq 0 $((${#split_shebang[@]} - 
> 1))); do

for i in "${!split_shebang[@]}"; do

> + case "/${split_shebang[${i}]}" in

case "/${split_shebang[i]}" in

Also below.

> + */${EPYTHON})
>   debug-print 
> "${FUNCNAME}: in file ${f#${D%/}}"
>   debug-print 
> "${FUNCNAME}: shebang matches EPYTHON: ${shebang}"
>  
>   # Nothing to do, move 
> along.
>   any_correct=1
> - from=${EPYTHON}
> + continue 2
> + ;;
> + */python)
> + fix=1
>   break
>   ;;
> - *python|*python[23])
> - debug-print 
> "${FUNCNAME}: in file ${f#${D%/}}"
> - debug-print 
> "${FUNCNAME}: rewriting shebang: ${shebang}"
> -
> - if [[ ${i} == *python2 
> ]]; then
> - from=python2
> - if [[ ! 
> ${force} ]]; then
> - 
> python_is_python3 "${EPYTHON}" && error=1
> - fi
> - elif [[ ${i} == 
> *python3 ]]; then
> - from=python3
> - if [[ ! 
> ${force} ]]; then
> - 
> python_is_python3 "${EPYTHON}" || error=1
> - fi
> - else
> - from=python
> + */python2)
> + if [[ ! ${force} ]]; 
> then
> + 
> python_is_python3 "${EPYTHON}" && error=1
>   fi
> + fix=1
>   break
>   ;;
> -

Re: [gentoo-dev] [PATCH] Eclass changes for cross-compiling Python modules

[Benda Xu]

Hi James,

James Le Cuirot  writes:

> Once this is in place, I can finish my long-awaited revamp of my
> cross-boss project that will allow you to cross-compile @system from
> scratch with very little effort.

I haven't gone through the patches yet.  But I want to say thank you!
The cross-boss project has been on my list for ages and you've made it
real.  Your efforts on EAPI-7 and cross python modules are much
appreciated.

Yours,
Benda

[gentoo-portage-dev] Re: [PATCH] file_copy: handle EUCLEAN from copy_file_range (bug 674332)

[Zac Medico]

On 1/2/19 2:29 PM, Zac Medico wrote:
> EXT4 can set the errno to EUCLEAN for copy_file_range.
> 
> Bug: https://bugs.gentoo.org/674332
> Signed-off-by: Zac Medico 
> ---
>  src/portage_util_file_copy_reflink_linux.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/portage_util_file_copy_reflink_linux.c 
> b/src/portage_util_file_copy_reflink_linux.c
> index 352342c06..1422232a5 100644
> --- a/src/portage_util_file_copy_reflink_linux.c
> +++ b/src/portage_util_file_copy_reflink_linux.c
> @@ -271,7 +271,7 @@ _reflink_linux_file_copy(PyObject *self, PyObject *args)
>  
>  if (copyfunc_ret < 0) {
>  error = errno;
> -if ((errno == EXDEV || errno == ENOSYS || errno == 
> EOPNOTSUPP) &&
> +if ((errno == EXDEV || errno == ENOSYS || errno == 
> EOPNOTSUPP || errno == EUCLEAN) &&
>  copyfunc == cfr_wrapper) {
>  /* Use sendfile instead of copy_file_range for
>   * cross-device copies, or when the copy_file_range
> 

The EUCLEAN might be triggered by a kernel bug, so I don't want to merge
a patch until we have more info.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 11/12] distutils-r1.eclass: Fix cross-compiling

[James Le Cuirot]

Following the changes in python-utils-r1.eclass, Python is now needed
in both BDEPEND and DEPEND.

As distutils does not call our eclass helpers, we need to pass the
correct include and library directories in setup.cfg. Unfortunately it
is still hardcoded to add -I/usr/include/pythonX.X and -L/usr/lib but
these appear after the SYSROOT paths so the build succeeds anyway. If
a missing header or library were to cause it to fall back to the build
host paths then it would most likely fail but it does not matter as it
would have failed either way.

Closes: https://bugs.gentoo.org/648652
Signed-off-by: James Le Cuirot 
---
 eclass/distutils-r1.eclass | 22 --
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index fa7a3ab5c12b..ff7792f11a87 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2019 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: distutils-r1.eclass
@@ -99,11 +99,8 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
 
 if [[ ! ${DISTUTILS_OPTIONAL} ]]; then
RDEPEND=${PYTHON_DEPS}
-   if [[ ${EAPI} != [56] ]]; then
-   BDEPEND=${PYTHON_DEPS}
-   else
-   DEPEND=${PYTHON_DEPS}
-   fi
+   DEPEND=${PYTHON_DEPS}
+   [[ ${EAPI} != [56] ]] && BDEPEND=${PYTHON_DEPS}
REQUIRED_USE=${PYTHON_REQUIRED_USE}
 fi
 
@@ -407,8 +404,21 @@ _distutils-r1_create_setup_cfg() {
# setuptools like to create .egg files for install --home.
[bdist_egg]
dist-dir = ${BUILD_DIR}/dist
+
+   # this is needed when cross-compiling
+   [build_ext]
_EOF_
 
+   if [[ ${EPYTHON} != jython* ]]; then
+   echo "include-dirs = $(python_get_includedir)" \
+   >> "${HOME}"/.pydistutils.cfg || die
+   fi
+
+   if [[ ${EPYTHON} == python* ]]; then
+   echo "library-dirs = $(dirname $(python_get_library_path))" \
+   >> "${HOME}"/.pydistutils.cfg || die
+   fi
+
# we can't refer to ${D} before src_install()
if [[ ${EBUILD_PHASE} == install ]]; then
cat >> "${HOME}"/.pydistutils.cfg <<-_EOF_ || die
-- 
2.19.2

[gentoo-dev] [PATCH 12/12] distutils-r1.eclass: Make distutils-r1_create_setup_cfg external

[James Le Cuirot]

It is useful on its own when the build system calls setup.py for us,
from a Makefile, for example. ${BUILD_DIR} is unlikely to be set in
this instance so it falls back to ${PWD}.

Signed-off-by: James Le Cuirot 
---
 eclass/distutils-r1.eclass | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index ff7792f11a87..2e969e205b16 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -250,7 +250,7 @@ esetup.py() {
local die_args=()
[[ ${EAPI} != [45] ]] && die_args+=( -n )
 
-   [[ ${BUILD_DIR} ]] && _distutils-r1_create_setup_cfg
+   [[ ${BUILD_DIR} ]] && distutils-r1_create_setup_cfg
 
set -- "${EPYTHON:-python}" setup.py "${mydistutilsargs[@]}" "${@}"
 
@@ -288,7 +288,7 @@ distutils_install_for_testing() {
#so we need to set it properly and mkdir them,
# 4) it runs a bunch of commands which write random files to cwd,
#in order to avoid that, we add the necessary path overrides
-   #in _distutils-r1_create_setup_cfg.
+   #in distutils-r1_create_setup_cfg.
 
TEST_DIR=${BUILD_DIR}/test
local bindir=${TEST_DIR}/scripts
@@ -377,15 +377,16 @@ distutils-r1_python_configure() {
[[ ${EAPI} == [45] ]] || die "${FUNCNAME} is banned in EAPI 6 (it was a 
no-op)"
 }
 
-# @FUNCTION: _distutils-r1_create_setup_cfg
-# @INTERNAL
+# @FUNCTION: distutils-r1_create_setup_cfg
 # @DESCRIPTION:
 # Create implementation-specific configuration file for distutils,
 # setting proper build-dir (and install-dir) paths.
-_distutils-r1_create_setup_cfg() {
+distutils-r1_create_setup_cfg() {
+   local build_base=${BUILD_DIR:-${PWD}}
+
cat > "${HOME}"/.pydistutils.cfg <<-_EOF_ || die
[build]
-   build-base = ${BUILD_DIR}
+   build-base = ${build_base}
 
# using a single directory for them helps us export
# ${PYTHONPATH} and ebuilds find the sources independently
@@ -403,7 +404,7 @@ _distutils-r1_create_setup_cfg() {
# this is needed by distutils_install_for_testing since
# setuptools like to create .egg files for install --home.
[bdist_egg]
-   dist-dir = ${BUILD_DIR}/dist
+   dist-dir = ${build_base}/dist
 
# this is needed when cross-compiling
[build_ext]
-- 
2.19.2

[gentoo-dev] [PATCH 09/12] python-any-r1.eclass: Export PYTHON after checking whether installed

[James Le Cuirot]

This previously happened before checking but now exporting PYTHON
involves creating the wrappers, which requires the requested Python to
be installed.

We could also drop all the calls to python_wrapper_setup as this is
now already done when exporting PYTHON but it doesn't hurt to be
explicit.

Signed-off-by: James Le Cuirot 
---
 eclass/python-any-r1.eclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/python-any-r1.eclass b/eclass/python-any-r1.eclass
index 7a91507a600f..7c5ae87872eb 100644
--- a/eclass/python-any-r1.eclass
+++ b/eclass/python-any-r1.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2019 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: python-any-r1.eclass
@@ -327,8 +327,8 @@ python_setup() {
# fallback to best installed impl.
# (reverse iteration over _PYTHON_SUPPORTED_IMPLS)
for (( i = ${#_PYTHON_SUPPORTED_IMPLS[@]} - 1; i >= 0; i-- )); do
-   python_export "${_PYTHON_SUPPORTED_IMPLS[i]}" EPYTHON PYTHON
-   if _python_EPYTHON_supported "${EPYTHON}"; then
+   if _python_EPYTHON_supported "${_PYTHON_SUPPORTED_IMPLS[i]}"; 
then
+   python_export "${_PYTHON_SUPPORTED_IMPLS[i]}" EPYTHON 
PYTHON
python_wrapper_setup
return
fi
-- 
2.19.2

[gentoo-dev] [PATCH 10/12] python-any-r1.eclass: Create wrappers against build system's config

[James Le Cuirot]

python-any-r1 is not to be used in packages requiring Python at
runtime. It may not even be installed.

Signed-off-by: James Le Cuirot 
---
 eclass/python-any-r1.eclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/eclass/python-any-r1.eclass b/eclass/python-any-r1.eclass
index 7c5ae87872eb..c46ca0e2444b 100644
--- a/eclass/python-any-r1.eclass
+++ b/eclass/python-any-r1.eclass
@@ -283,6 +283,11 @@ _python_EPYTHON_supported() {
 python_setup() {
debug-print-function ${FUNCNAME} "${@}"
 
+   # Ensure Python wrappers are created against the build system's
+   # configuration as python-any-r1 is not to be used in packages
+   # requiring Python at runtime. It may not even be installed.
+   local SYSROOT ESYSROOT=${BROOT-${EPREFIX}}
+
# support developer override
if [[ ${PYTHON_COMPAT_OVERRIDE} ]]; then
local impls=( ${PYTHON_COMPAT_OVERRIDE} )
-- 
2.19.2

[gentoo-dev] [PATCH 08/12] python-utils-r1.eclass: Adjust wrappers for Python 3.7 libdir change

[James Le Cuirot]

We assumed Python to be located at /usr/$(get_libdir) in recent
changes to this eclass but this has changed from Python 3.7. We now
follow upstream, installing most files to /usr/lib, with only certain
files, such as the pkg-config files, going into /usr/$(get_libdir).

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 17de9ca9c44f..5e54cc618212 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1063,7 +1063,14 @@ python_wrapper_setup() {
 
# CPython-specific
if [[ ${EPYTHON} == python* ]]; then
-   local pysysrootlib=${pysysroot}/usr/$(get_libdir)
+   local pysysrootlib=${pysysroot}/usr/
+
+   case "${EPYTHON}" in
+   python2.*|python3.[0-6])
+   pysysrootlib+=$(get_libdir) ;;
+   *)
+   pysysrootlib+=lib ;;
+   esac
 
cat > "${workdir}/bin/python-config" <<-_EOF_ || die
#!/bin/sh
@@ -1093,7 +1100,7 @@ python_wrapper_setup() {
ln -s "${EPYTHON/python/2to3-}" "${workdir}"/bin/2to3 
|| die
 
# Python 2.7+.
-   ln -s "${pysysrootlib}"/pkgconfig/${EPYTHON/n/n-}.pc \
+   ln -s 
"${pysysroot}"/usr/$(get_libdir)/pkgconfig/${EPYTHON/n/n-}.pc \
"${workdir}"/pkgconfig/python.pc || die
ln -s python.pc 
"${workdir}"/pkgconfig/python${pyver}.pc || die
else
-- 
2.19.2

[gentoo-dev] [PATCH 07/12] python-utils-r1.eclass: Add special wrapper handling for Python itself

[James Le Cuirot]

dev-lang/python sometimes used to install the epython module into the
wrong libdir (e.g. lib vs lib64). The earlier eclass changes actually
break the build entirely as sysconfigdata is now sourced from within
SYSROOT, where it may not even be installed yet. This therefore needs
to be handled as a special case where sysconfigdata is sourced from
within ED instead.

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 91e457f3cf14..17de9ca9c44f 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1028,7 +1028,11 @@ python_wrapper_setup() {
# overridden but hopefully it will be somewhere under ${T}.
local ${lpyvar}=$(PATH=${PATH//${T}//dev/null} type -P "${EPYTHON}" || 
die "${FUNCNAME}: can't find ${EPYTHON} in PATH")
 
-   local pysysroot=${ESYSROOT-${SYSROOT%/}${EPREFIX}}
+   if [[ ${CATEGORY}/${PN} == dev-lang/python ]]; then
+   local pysysroot=${ED%/}
+   else
+   local pysysroot=${ESYSROOT-${SYSROOT%/}${EPREFIX}}
+   fi
 
if [[ ! -x ${workdir}/bin/python ]]; then
_python_check_dead_variables
-- 
2.19.2

[gentoo-dev] [PATCH 06/12] python-utils-r1.eclass: Don't die in python_fix_shebang if all fixed

[James Le Cuirot]

Shebangs may need fixing on prefix systems or when cross-building but
not at other times.

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 19cfaf2798ab..91e457f3cf14 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1328,16 +1328,12 @@ python_fix_shebang() {
fi
done < <(find -H "${path}" -type f -print0 || die)
 
-   if [[ ! ${any_fixed} ]]; then
+   if [[ ! ${any_fixed} && ! ${any_correct} ]]; then
local cmd=eerror
[[ ${EAPI:-0} == [012345] ]] && cmd=eqawarn
 
"${cmd}" "QA warning: ${FUNCNAME}, ${path#${D%/}} did 
not match any fixable files."
-   if [[ ${any_correct} ]]; then
-   "${cmd}" "All files have ${EPYTHON} shebang 
already."
-   else
-   "${cmd}" "There are no Python files in 
specified directory."
-   fi
+   "${cmd}" "There are no Python files in specified 
directory."
 
[[ ${cmd} == eerror ]] && die "${FUNCNAME} did not 
match any fixable files (QA warning fatal in EAPI ${EAPI})"
fi
-- 
2.19.2

[gentoo-dev] [PATCH 05/12] python-utils-r1.eclass: Replace temporary paths in python_fix_shebang

[James Le Cuirot]

This will deal with cases where the shebang is defined using ${PYTHON}
or the location of python in the PATH.

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 22 --
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 607af1b52f75..19cfaf2798ab 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -989,7 +989,8 @@ python_doheader() {
 # @DESCRIPTION:
 # Create proper 'python' executable and pkg-config wrappers
 # (if available) in the directory named by . Set up PATH
-# and PKG_CONFIG_PATH appropriately.  defaults to ${T}/${EPYTHON}.
+# and PKG_CONFIG_PATH appropriately.  defaults to
+# ${T}/python-wrappers/${EPYTHON}${SYSROOT:+-sysroot}.
 #
 # The wrappers will be created for implementation named by ,
 # or for one named by ${EPYTHON} if no  passed.
@@ -1010,7 +1011,7 @@ python_wrapper_setup() {
 
# Use separate directories for SYSROOT in case we need to execute
# Python in the context of the build host by unsetting SYSROOT.
-   local workdir=${3:-${T}/${impl}${SYSROOT:+-sysroot}}
+   local workdir=${3:-${T}/python-wrappers/${impl}${SYSROOT:+-sysroot}}
 
[[ ${workdir} ]] || die "${FUNCNAME}: no workdir specified."
[[ ${impl} ]] || die "${FUNCNAME}: no impl nor EPYTHON specified."
@@ -1243,8 +1244,25 @@ python_fix_shebang() {
# Match left-to-right in a loop, to avoid 
matching random
# repetitions like 'python2.7 python2'.
for i in $(seq 0 $((${#split_shebang[@]} - 
1))); do
+   # Fix the leading path if it points to 
a wrapper
+   # script created by this eclass or 
points to
+   # ${BROOT} rather than ${EPREFIX}. We 
generally only
+   # touch the tail end of the string as 
this is safer
+   # but if a path starts with ${T} or 
${BROOT}, where
+   # the latter differs from ${EPREFIX}, 
then it
+   # definitely needs fixing. We preserve 
the tail for
+   # further processing in the case block 
below.
+   if [[ ${split_shebang[${i}]} == 
${T}/python-wrappers/*/bin/* || ( ${split_shebang[${i}]} == ${BROOT}/usr/bin/* 
&& ${BROOT} != ${EPREFIX} ) ]]; then
+   
split_shebang[${i}]=${EPREFIX}/usr/bin/${split_shebang[${i}]##*/}
+   fix=1
+   fi
+
case "/${split_shebang[${i}]}" in
*/${EPYTHON})
+   # EPYTHON matched but 
the leading path may
+   # still need fixing.
+   [[ ${fix} ]] && break
+
debug-print 
"${FUNCNAME}: in file ${f#${D%/}}"
debug-print 
"${FUNCNAME}: shebang matches EPYTHON: ${shebang}"
 
-- 
2.19.2

[gentoo-dev] [PATCH 04/12] python-utils-r1.eclass: Use wrapper scripts to fix cross-compiling

[James Le Cuirot]

Python has little concept of cross-compiling but it turns out that
making it work isn't so hard after all.

Platform-specific details are held within _sysconfigdata.py,
sysconfig.py, and various distutils files. If we simply symlink these
from SYSROOT into an empty directory and add that directory to
PYTHONPATH then we can utilise the build host's Python with the target
host's settings.

The pkg-config files were already being symlinked in a similar manner
but now we source them from within SYSROOT.

In order for PYTHONPATH to be respected, Python must be executed via
the wrapper, which was not the case before. We previously relied
solely on the PATH but now PYTHON must point to the wrapper rather
than the usual location under /usr/bin. However, we only do this when
SYSROOT or EPREFIX are effectively set to avoid unnecessary
complexity. This has required some rejigging in the way that PYTHON is
set but it should remain compatible with existing packages.

The python_wrapper_setup function that handles all this has had its
arguments reordered because no one ever uses the path/workdir
argument, which makes specifying other arguments tedious.

Some packages rely on python-config but luckily this is just a shell
script so it can be executed from within SYSROOT. This is bending the
rules of PMS slightly but Python leaves us with little choice. I also
wrote those rules so I'm allowed to bend them. ;)

PYTHON_INCLUDEDIR, PYTHON_LIBPATH, and their associated functions are
generally used during src_configure or src_compile and, as such, they
now need to have SYSROOT prepended.

python_doheader uses PYTHON_INCLUDEDIR to install new headers and
therefore needs the value without SYSROOT. It was already stripping
EPREFIX before so now it simply strips SYSROOT as well. Similar
instances of this can do likewise or call the functions with SYSROOT
unset to achieve the same effect.

To make all this work, we are assuming that CPython is located at
/usr/$(get_libdir)/${EPYTHON}, which is admittedly a little circular
when we are querying Python for the right paths. I feel the reason
that python_export was rewritten to query these dynamically was less
because someone might install Python to some weird location and more
to avoid special handling for each of the different
implementations. And what of those other implementations?

Being Java-based, Jython is installed under the platform-neutral
/usr/share and presumably has few other platform-specific
aspects. Enabling native extensions appears non-trivial and none of
our module packages have enabled support for it anyway.

I think PyPy could potentially support cross-compiling but it
hardcodes the native extension filename suffix within its own binaries
with no way to override it. Perhaps we could patch this in somehow but
I'm afraid I don't care enough.

Together with the following changes to distutils-r1.eclass, I have now
been able to cross-compile a large number of packages with native
Python extensions, most with no changes at all, and the rest with only
minor fixes.

Closes: https://bugs.gentoo.org/503874
Bug: https://bugs.gentoo.org/648652
Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 120 ++
 1 file changed, 92 insertions(+), 28 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 1a549f49f3f2..607af1b52f75 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -211,9 +211,15 @@ _python_impl_matches() {
 #
 # distutils-r1: Set within any of the python sub-phase functions.
 #
-# Example value:
+# If SYSROOT or EPREFIX are effectively set then this will point to an
+# automatically generated wrapper rather than the usual path under
+# /usr/bin in order to accommodate cross-compiling. We could do this all
+# the time but it would add unnecessary complexity.
+#
+# Example values:
 # @CODE
 # /usr/bin/python2.7
+# /var/tmp/portage/dev-python/pyblake2-1.2.3/temp/python2.7/bin/python2.7
 # @CODE
 
 # @ECLASS-VARIABLE: EPYTHON
@@ -256,6 +262,10 @@ _python_impl_matches() {
 # Set and exported on request using python_export().
 # Requires a proper build-time dependency on the Python implementation.
 #
+# This is prepended with SYSROOT in order to accommodate
+# cross-compiling. You may need to strip SYSROOT and EPREFIX if using it
+# to install new headers.
+#
 # Example value:
 # @CODE
 # /usr/include/python2.7
@@ -270,6 +280,9 @@ _python_impl_matches() {
 # Valid only for CPython. Requires a proper build-time dependency
 # on the Python implementation.
 #
+# This is prepended with SYSROOT in order to accommodate
+# cross-compiling.
+#
 # Example value:
 # @CODE
 # /usr/lib64/libpython2.7.so
@@ -314,6 +327,10 @@ _python_impl_matches() {
 # Valid only for CPython. Requires a proper build-time dependency
 # on the Python implementation and on pkg-config.
 #
+# This is prepended with SYSROOT in order to accommodate
+# cross-compiling. You generally should not 

[gentoo-dev] [PATCH 03/12] python-utils-r1.eclass: Have wrapper workdir default to "impl" arg

[James Le Cuirot]

It currently defaults to EPYTHON, which might not even be defined
yet. The "impl" arg defaults to EPYTHON anyway.

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 1eca0764a202..1a549f49f3f2 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -980,8 +980,8 @@ python_doheader() {
 python_wrapper_setup() {
debug-print-function ${FUNCNAME} "${@}"
 
-   local workdir=${1:-${T}/${EPYTHON}}
local impl=${2:-${EPYTHON}}
+   local workdir=${1:-${T}/${impl}}
 
[[ ${workdir} ]] || die "${FUNCNAME}: no workdir specified."
[[ ${impl} ]] || die "${FUNCNAME}: no impl nor EPYTHON specified."
-- 
2.19.2

[gentoo-dev] [PATCH 02/12] python-utils-r1.eclass: New python_fix_shebang approach

[James Le Cuirot]

The previous approach would erroneously match foopython. The new
approach requires the match to start the string or be preceeded by a
slash, the only two cases we actually want. It does this with slightly
less code and allows the replacement of whole path strings that would
be problematic when passed to sed. This will be needed when
cross-compiling is addressed.

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 78 ++-
 1 file changed, 31 insertions(+), 47 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index da76a755fb34..1eca0764a202 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1165,8 +1165,7 @@ python_fix_shebang() {
[[ -d ${path} ]] && is_recursive=1
 
while IFS= read -r -d '' f; do
-   local shebang i
-   local error= from=
+   local shebang i= error= fix=
 
# note: we can't ||die here since read will fail if file
# has no newline characters
@@ -1175,65 +1174,56 @@ python_fix_shebang() {
# First, check if it's shebang at all...
if [[ ${shebang} == '#!'* ]]; then
local split_shebang=()
-   read -r -a split_shebang <<<${shebang} || die
+   read -r -a split_shebang <<<${shebang#\#\!} || 
die
 
# Match left-to-right in a loop, to avoid 
matching random
# repetitions like 'python2.7 python2'.
-   for i in "${split_shebang[@]}"; do
-   case "${i}" in
-   *"${EPYTHON}")
+   for i in $(seq 0 $((${#split_shebang[@]} - 
1))); do
+   case "/${split_shebang[${i}]}" in
+   */${EPYTHON})
debug-print 
"${FUNCNAME}: in file ${f#${D%/}}"
debug-print 
"${FUNCNAME}: shebang matches EPYTHON: ${shebang}"
 
# Nothing to do, move 
along.
any_correct=1
-   from=${EPYTHON}
+   continue 2
+   ;;
+   */python)
+   fix=1
break
;;
-   *python|*python[23])
-   debug-print 
"${FUNCNAME}: in file ${f#${D%/}}"
-   debug-print 
"${FUNCNAME}: rewriting shebang: ${shebang}"
-
-   if [[ ${i} == *python2 
]]; then
-   from=python2
-   if [[ ! 
${force} ]]; then
-   
python_is_python3 "${EPYTHON}" && error=1
-   fi
-   elif [[ ${i} == 
*python3 ]]; then
-   from=python3
-   if [[ ! 
${force} ]]; then
-   
python_is_python3 "${EPYTHON}" || error=1
-   fi
-   else
-   from=python
+   */python2)
+   if [[ ! ${force} ]]; 
then
+   
python_is_python3 "${EPYTHON}" && error=1
fi
+   fix=1
break
;;
-   
*python[23].[0123456789]|*pypy|*pypy3|*jython[23].[0123456789])
-   # Explicit mismatch.
+   */python3)

[gentoo-dev] [PATCH 01/12] python-utils-r1.eclass: Fix some double slash paths

[James Le Cuirot]

If Python returns relative site or script directories then something
is very wrong!

Signed-off-by: James Le Cuirot 
---
 eclass/python-utils-r1.eclass | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index e3cf82b4b58f..da76a755fb34 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2019 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: python-utils-r1.eclass
@@ -797,11 +797,11 @@ python_newexe() {
 
# install the wrapper
_python_ln_rel "${ED%/}"/usr/lib/python-exec/python-exec2 \
-   "${ED%/}/${wrapd}/${newfn}" || die
+   "${ED%/}${wrapd}/${newfn}" || die
 
# don't use this at home, just call python_doscript() instead
if [[ ${_PYTHON_REWRITE_SHEBANG} ]]; then
-   python_fix_shebang -q "${ED%/}/${d}/${newfn}"
+   python_fix_shebang -q "${ED%/}${d}/${newfn}"
fi
 }
 
@@ -927,7 +927,7 @@ python_domodule() {
doins -r "${@}" || return ${?}
)
 
-   python_optimize "${ED%/}/${d}"
+   python_optimize "${ED%/}${d}"
 }
 
 # @FUNCTION: python_doheader
-- 
2.19.2

[gentoo-dev] [PATCH] Eclass changes for cross-compiling Python modules

[James Le Cuirot]

Here's a series of eclass changes I've been working on since August to
allow Python modules to be cross-compiled. We previously believed this
to be practically impossible without significant changes upstream and
in the wider Python ecosystem so getting here feels like quite an
achievement. While the approach is a little unconventional, it doesn't
feel overly hacky and the results are quite consistently good.

These changes should also benefit prefix and I did a full prefix
bootstrap for the first time this week just to make sure I didn't
break it.

A handful of other Python packages that don't quite follow the usual
mantra have needed fixing up. I haven't included those changes here
but you can view them on this GitHub pull request.

https://github.com/gentoo/gentoo/pull/9822

Once this is in place, I can finish my long-awaited revamp of my
cross-boss project that will allow you to cross-compile @system from
scratch with very little effort.

Re: [gentoo-dev] AUTHORS file for portage repository

[Kent Fredric]

On Thu, 3 Jan 2019 00:25:29 +0100
Jonas Stein  wrote:

> git log | grep "Author: "| sort | uniq | sed "s/Author: //g" | wc -c

That's a rather round about way of doing :
  git shortlog -e -s | cut -f 2 

 git shortlog -e -s  | cut -f 2 | wc -c
37471

git shortlog -e -s  | wc -l
998


pgpFfkBYRuopC.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] cargo.eclass: allow passing additional arguments to cargo

[Sergei Trofimovich]

On Wed,  2 Jan 2019 13:58:26 -0800
Georgy Yakovlev  wrote:

> +# @CODE
> +# ECARGO_BUILD_FLAGS="$(usex pcre "--features pcre2" "")"
> +# @CODE

You may also want to provide user's hooks as well.

Similar to EXTRA_ECONF and EXTRA_EMAKE those might be
ECARGO_EXTRA_BUILD_FLAGS
ECARGO_EXTRA_INSTALL_FLAGS

That way user would be able to tweak cargo behaviour without
breaking ebuild behaviour too much via make.conf/package.env, like
ECARGO_EXTRA_BUILD_FLAGS=--color=never

-- 

  Sergei

Re: [gentoo-dev] [PATCH] cargo.eclass: allow passing additional arguments to cargo

[Michał Górny]

Dnia January 3, 2019 5:46:35 AM UTC, Georgy Yakovlev  
napisał(a):
>On Wednesday, January 2, 2019 8:52:45 PM PST Michał Górny wrote:
>> On Wed, 2019-01-02 at 13:58 -0800, Georgy Yakovlev wrote:
>> > This adds 2 eclass variables
>> > 
>> > ECARGO_BUILD_FLAGS
>> > ECARGO_INSTALL_FLAGS
>> > 
>> > contents will be passed to "cargo build" and "cargo install" calls
>in
>> > cargo_src_compile() and cargo_src_install() respectively.
>> > 
>> > Closes: https://github.com/gentoo/gentoo/pull/10725
>> > Signed-off-by: Georgy Yakovlev 
>> > ---
>> >  eclass/cargo.eclass | 25 +++--
>> >  1 file changed, 23 insertions(+), 2 deletions(-)
>> > 
>> > diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
>> > index 50f7830c51b..ea58c63b456 100644
>> > --- a/eclass/cargo.eclass
>> > +++ b/eclass/cargo.eclass
>> > @@ -1,142 +1,163 @@
>> >  # Copyright 1999-2018 Gentoo Authors
>> >  # Distributed under the terms of the GNU General Public License v2
>> >  
>> >  # @ECLASS: cargo.eclass
>> >  # @MAINTAINER:
>> >  # r...@gentoo.org
>> >  # @AUTHOR:
>> >  # Doug Goldstein 
>> >  # @SUPPORTED_EAPIS: 6 7
>> >  # @BLURB: common functions and variables for cargo builds
>> >  
>> >  if [[ -z ${_CARGO_ECLASS} ]]; then
>> >  _CARGO_ECLASS=1
>> >  
>> >  CARGO_DEPEND=""
>> >  [[ ${CATEGORY}/${PN} != dev-util/cargo ]] &&
>CARGO_DEPEND="virtual/cargo"
>> >  
>> >  case ${EAPI} in
>> >6) DEPEND="${CARGO_DEPEND}";;
>> >7) BDEPEND="${CARGO_DEPEND}";;
>> >*) die "EAPI=${EAPI:-0} is not supported" ;;
>> >  esac
>> >  
>> >  inherit multiprocessing
>> >  
>> >  EXPORT_FUNCTIONS src_unpack src_compile src_install
>> >  
>> >  IUSE="${IUSE} debug"
>> >  
>> >  ECARGO_HOME="${WORKDIR}/cargo_home"
>> >  ECARGO_VENDOR="${ECARGO_HOME}/gentoo"
>> >  
>> > +# @ECLASS-VARIABLE: ECARGO_BUILD_FLAGS
>> > +# @DEFAULT_UNSET
>> > +# @DESCRIPTION:
>> > +# This allows to pass additional build flags to cargo in
>cargo_src_compile()
>> > +#
>> > +# Example:
>> > +# @CODE
>> > +# ECARGO_BUILD_FLAGS="$(usex pcre "--features pcre2" "")"
>> > +# @CODE
>> > +
>> > +# @ECLASS-VARIABLE: ECARGO_INSTALL_FLAGS
>> > +# @DEFAULT_UNSET
>> > +# @DESCRIPTION:
>> > +# This allows to pass additional install flags to cargo in
>cargo_src_install()
>> > +#
>> > +# Example:
>> > +# @CODE
>> > +# ECARGO_INSTALL_FLAGS="--path=."
>> > +# @CODE
>> > +
>> > +
>> >  # @FUNCTION: cargo_crate_uris
>> >  # @DESCRIPTION:
>> >  # Generates the URIs to put in SRC_URI to help fetch dependencies.
>> >  cargo_crate_uris() {
>> >local crate
>> >for crate in "$@"; do
>> >local name version url pretag
>> >name="${crate%-*}"
>> >version="${crate##*-}"
>> >pretag="^[a-zA-Z]+"
>> >if [[ $version =~ $pretag ]]; then
>> >version="${name##*-}-${version}"
>> >name="${name%-*}"
>> >fi
>> >url="https://crates.io/api/v1/crates/${name}/${version}/download
>-> ${crate}.crate"
>> >echo "${url}"
>> >done
>> >  }
>> >  
>> >  # @FUNCTION: cargo_src_unpack
>> >  # @DESCRIPTION:
>> >  # Unpacks the package and the cargo registry
>> >  cargo_src_unpack() {
>> >debug-print-function ${FUNCNAME} "$@"
>> >  
>> >mkdir -p "${ECARGO_VENDOR}" || die
>> >mkdir -p "${S}" || die
>> >  
>> >local archive shasum pkg
>> >for archive in ${A}; do
>> >case "${archive}" in
>> >*.crate)
>> >ebegin "Loading ${archive} into Cargo registry"
>> >tar -xf "${DISTDIR}"/${archive} -C 
>> > "${ECARGO_VENDOR}/" || die
>> ># generate sha256sum of the crate itself as 
>> > cargo needs this
>> >shasum=$(sha256sum "${DISTDIR}"/${archive} | 
>> > cut -d ' ' -f 1)
>> >pkg=$(basename ${archive} .crate)
>> >cat <<- EOF > 
>> > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json
>> >{
>> >"package": "${shasum}",
>> >"files": {}
>> >}
>> >EOF
>> ># if this is our target package we need it in 
>> > ${WORKDIR} too
>> ># to make ${S} (and handle any revisions too)
>> >if [[ ${P} == ${pkg}* ]]; then
>> >tar -xf "${DISTDIR}"/${archive} -C 
>> > "${WORKDIR}" || die
>> >fi
>> >eend $?
>> >;;
>> >cargo-snapshot*)
>> >ebegin "Unpacking ${archive}"
>> >mkdir -p "${S}"/target/snapshot
>> >tar -xzf "${DISTDIR}"/${archive} -C 
>> > "${S}"/target/snapshot
>--strip-components 2 || die
>> ># cargo's