[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye  added the comment:

PYTHON_FOR_BUILD does not use PYTHONPATH in the implementation of (closed) PR 
17420 and should fix the current issue as a side effect.

FWIW, PR 17420 fixes cross-compilation of third-party extension modules by 
replacing the complex PYTHON_FOR_BUILD command line set in configure.ac by the 
more simple command:

"PYTHON_FOR_BUILD='PYTHON_PROJECT_BASE=$(abs_builddir) '$interp"

In this PR finding the path name of the target sysconfigdata module when needed 
is done by the sysconfig module and by distutils using the 
Misc/python-config.sh.in script instead of using PYTHONPATH.

Since PYTHONPATH is not used in PR 17420, the current issue should also be 
fixed by the PR. See also Benedikt OP that provided a hint for the fix.

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Andrew Aladjev]

Andrew Aladjev  added the comment:

I've made a quick test of all patches provided. All patches have failed to fix 
it. The problem is more heavy than it looks like.

Please look at the following code:

PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f 
pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) 
'${EPYTHON_FOR_BUILD:-${interp}}

It means that we will use "interp" to load cross compiled modules, env, etc 
from "_PYTHON_PROJECT_BASE". This code is broken by design. Loading of cross 
compiled modules, env, etc is not possible in general. For example you have 
different libc: glibc, musl, uclibc.

glibc -> cross compile -> glibc = loads fine.
musl -> cross compile -> musl = loads fine.
uclibc -> cross compile -> uclibc = loads fine.

glibc -> cross compile -> musl = failed to load.
glibc -> cross compile -> uclibc = failed to load.
musl -> cross compile -> glibc = failed to load.
musl -> cross compile -> uclibc = failed to load.
uclibc -> cross compile -> glibc = failed to load.
uclibc -> cross compile -> musl = failed to load.

Cross compilation doesn't mean only another arch, it means different libc, 
kernel headers, binutils and compiler. You will receive same issue while using 
clang toolchains.

Please assign this issue to python core developers. This issue can't be fixed 
by regular users without experience in python development. It requires complete 
change of cross compilation design.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Andrew Aladjev]

Andrew Aladjev  added the comment:

Hello. I've provided a double recompilation workaround in the following gentoo 
bug https://bugs.gentoo.org/705970. You can use it for now if you don't want to 
wait until this issue will be fixed.

I will try patches provided by people in this issue today's evening. Thank you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Andrew Aladjev]

Change by Andrew Aladjev :


--
nosy: +puchenyaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye added the comment:

Cross-compilation of Android x86_64 on a linux x86_64 host followed by 'make 
install' does not fail anymore after the changes made in issue 27917. As 
expected since the triplets are different now when the target is an Android 
platform.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye added the comment:

Patch rebased to current tip.

To summarize:
* The root cause of the problem reported by Benedikt still exists.
* The problem is hidden now that the shared libraries names include the 
PLATFORM_TRIPLET.
* But byte-compilation still fails for cross-builds when the 'host' and 'build' 
PLATFORM_TRIPLET are the same and this occurs for the x86_64 Android build on a 
x86_64 'build' system.

Matthias, do you have any comment on this patch?

--
Added file: http://bugs.python.org/file43953/py_build_environ_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Changes by Xavier de Gaye :


--
nosy: +thomas-petazzoni

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye added the comment:

Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install because in my Android 
cross-build setup, the native interpreter is built from the same source tree as 
the cross-build. However, currently the interpreter used for the cross-build 
may be any interpreter found in PATH and can import an incorrect 
importlib.util.MAGIC_NUMBER if PYTHONPATH is not set properly. So this new 
patch does not use -E and splits PYTHON_FOR_BUILD into two variables, 
PY_BUILD_ENVIRON that sets the environment variables and PYTHON_FOR_BUILD, the 
interpreter. PY_BUILD_ENVIRON is not used when running compileall.

Tested by running the test suite on the Android i686 and x86_64 emulators 
(excluding test_ctypes that segfaults python on x86_64). The tests cannot be 
run with a cross-build where PYTHON_FOR_BUILD is python 3.5.1 because of issue 
27434.

--
Added file: http://bugs.python.org/file43603/py_build_environ.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye added the comment:

> On Android cross-builds, it should be possible to reproduce the failure when 
> the build system and the host system have the same PLATFORM_TRIPLET

I confirm that this is the case. The following backtrace is produced when 
cross-building and installing Android x86_64 on a linux x86_64:

PYTHONPATH=/sdcard/org.bitbucket.pyona/lib/python3.6   \

_PYTHON_PROJECT_BASE=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64
 _PYTHON_HOST_PLATFORM=linux-x86_64 
PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu
 python3.6 -Wi /sdcard/org.bitbucket.pyona/lib/python3.6/compileall.py \
-d /sdcard/org.bitbucket.pyona/lib/python3.6 -f \
-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
/sdcard/org.bitbucket.pyona/lib/python3.6
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/compileall.py", line 17, in 

import struct
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/struct.py", line 13, in 

from _struct import *
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file 
or directory
make[3]: [Makefile:1270: libinstall] Error 1 (ignored)


'python3.6' is the native interpreter built from the same source tree as the 
source tree used for the cross-build. PATH has been set so that this native 
interpreter is run. Running this same command interactively with the native 
interpreter gives the same result:

[xavier@bilboquet python-native]$   
PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu
 ./python3.6
Python 3.6.0a2+ (default:64ed56fbc5e7+, Jun 27 2016, 10:36:45) 
[GCC 6.1.1 20160602] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/home/xavier/config/.pystartup", line 14, in 
import readline
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file 
or directory
>>> 


Cross-building and installing Android i686 on a linux x86_64 with the same 
Makefile does not cause the problem.

Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Xavier de Gaye]

Xavier de Gaye added the comment:

The byte-compiling for cross-builds fails because PYTHON_FOR_BUILD defined in 
configure, sets PYTHONPATH with the directory of the newly built extension 
modules shared libraries and because PYTHON_FOR_BUILD is used to run 
compileall.py in the Makefile. PYTHON_FOR_BUILD, i.e. the native python running 
on the build system, attempts to load the newly compiled libraries for the 
foreign host system and fails.

The problem is more difficult to reproduce after changeset 11c4f3f936e7 in 
issue 22980 since the shared libraries names include now a PLATFORM_TRIPLET 
that prevents the wrong library to be loaded in most cases. On Android 
cross-builds, it should be possible to reproduce the failure when the build 
system and the host system have the same PLATFORM_TRIPLET. In all cases, the 
interpreter used to run compileall.py imports the wrong _sysconfigdata module.

To fix this problem, this patch adds the -E option to PYTHON_FOR_BUILD when 
running compileall in the Makefile and 'cross_compiling' is set. The patch does 
not fix the fact that, when 'cross_compiling' is not set, PYTHONPATH is set to 
'$(DESTDIR)$(LIBDEST)' in the Makefile at the beginning of the statement that 
runs compileall.py while PYTHON_FOR_BUILD is set to './$(BUILDPYTHON) -E', 
something is obviously wrong here. So the patch would be simpler if '-E' was 
used in both cases, but I don't know what would be lost by removing this 
setting of PYTHONPATH at the beginning of those compileall statements, IOW what 
was their initial purpose.

--
keywords: +patch
nosy: +xdegaye
stage:  -> patch review
versions:  -Python 3.3, Python 3.4
Added file: http://bugs.python.org/file43559/python_for_build_flag.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Nick Coghlan]

Nick Coghlan added the comment:

Issue 23968 moves _sysconfigdata.py to an arch dependendent directory

--
dependencies: +rename the platform directory from plat-$(MACHDEP) to 
plat-$(PLATFORM_TRIPLET)
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Roumen Petrov]

Changes by Roumen Petrov bugtr...@roumenpetrov.info:


--
nosy: +rpetrov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22724
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Stefan Krah]

Changes by Stefan Krah stefan-use...@bytereef.org:


--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22724
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Benedikt Morbach]

Changes by Benedikt Morbach benedikt.morb...@googlemail.com:


--
versions: +Python 3.5, Python 3.6 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22724
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Benedikt Morbach]

New submission from Benedikt Morbach:

For cross-builds the byte-compiling and 2to3 Grammar generation fail.

This is because PYTHONPATH/sys.path includes 'build/lib.$PLATFORM', in which 
the compiled .so modules reside.
The host python obviously barfs on those, as they are compiled for a different 
architecture.

However, this directory also contains _sysconfigdata.py, which is needed at all 
times.

A possible fix would be to separate those two.

--
components: Cross-Build
messages: 229969
nosy: Benedikt.Morbach
priority: normal
severity: normal
status: open
title: byte-compile fails for cross-builds
type: compile error
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22724
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22724] byte-compile fails for cross-builds

[Benedikt Morbach]

Benedikt Morbach added the comment:

alternatively, introduce a $(PYTHON_FOR_INSTALL), which doesn't set PYTHONPATH.

Then run those steps using that, with the Makefile setting 
PYTHONPATH=$(DESTDIR)$(LIBDEST), which doesn't contain the compiled modules at 
that point.

The first solution looks cleaner to me, but might be more tricky to implement, 
as e.g. Modules/getpath.c makes some assumptions about the location of the 
build dir.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22724
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com