[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye  added the comment:

PR 17420 fixes cross-compilation of third-party extension modules.

The PYTHON_PROJECT_BASE environment variable is the path to the directory where 
Python has been cross-compiled. It is used by the native python interpreter to 
find the target sysconfigdata module.

For example the following command builds a wheel file to be transfered and 
installed with pip on the target platform, provided the native python 
interpreter and the cross-compiled one both have the wheel package installed:

  $ PYTHON_PROJECT_BASE=/path/to/builddir python setup.py bdist_wheel

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Change by Xavier de Gaye :


--
pull_requests: +16901
pull_request: https://github.com/python/cpython/pull/17420

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[pmp-p]

Change by pmp-p :


--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Changes by Xavier de Gaye :


--
assignee: xdegaye -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Matthias Klose]

Matthias Klose added the comment:

> Hum, you still need to provide the native python interpreter
> with the _path_ to the -python-config executable that
> can be anywhere on the file system

No, it's found in the same path. No contradiction.

> So I suggest we start with this patch as it works for:

Please do not.

> Then we can later extend the semantics of XBUILD_PYTHON_DIR

Again, I think that's the wrong way to go rely on any path as the primary id 
for the target, then "extending" on it makes it worse.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Matthias Klose]

Matthias Klose added the comment:

again, I don't think relying on a specific target path for a cross target is a 
good idea. and now deciding that the last possibility to use a target id to 
identify is better is internal doesn't make it better.  I'd appreciate if we 
could sit together where your work will lead to, and see how we accomplish 
everybody's goals.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

I know. I guess it can be determined without manually specifying but haven't 
investigated into details. Maybe some patches are necessary.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye added the comment:

$ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android 
PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr 
python3.7 setup.py build_ext
>

Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are 
private, please do not use them, they may be changed or removed at any time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

Well, cross compiling extension modules already works fine...

$ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android 
PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr 
python3.7 setup.py build_ext
running build_ext
building 'xx' extension
creating build
creating build/temp.linux-x86_64-3.7
/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang 
-Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -target aarch64-none-linux-android -gcc-toolchain 
/opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 
--sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE 
-fno-integrated-as -target aarch64-none-linux-android -gcc-toolchain 
/opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 
--sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE 
-fno-integrated-as -fPIC 
-I/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/include/python3.7m
 -c xxmodule.c -o build/temp.linux-x86_64-3.7/xxmodule.o
creating build/lib.linux-x86_64-3.7
/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -shared 
-target aarch64-none-linux-android -gcc-toolchain 
/opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 
--sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie 
-L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib 
-target aarch64-none-linux-android -gcc-toolchain 
/opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 
--sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie 
-L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib 
build/temp.linux-x86_64-3.7/xxmodule.o -o 
build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so
clang: warning: argument unused during compilation: '-pie'
clang: warning: argument unused during compilation: '-pie'

And it's running fine, too:

$ adb push build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so 
/data/local/tmp
[100%] /data/local/tmp/xx.cpython-37m-aarch64-linux-android.so

$ adb shell
shell@ASUS_Z00E_2:/ $ cd /data/local/tmp
shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh
shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m
Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14) 
[GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xx
>>> xx

>>>

Here's my xxmodule.c:

#include "Python.h"

PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");

static struct PyModuleDef xxmodule = {
PyModuleDef_HEAD_INIT,
"xx",
module_doc,
0,
NULL,
NULL,
NULL,
NULL,
NULL
};

PyMODINIT_FUNC
PyInit_xx(void)
{
return PyModuleDef_Init();
}

And setup.py:

from distutils.core import setup, Extension

module1 = Extension('xx',
sources=['xxmodule.c'])

setup(name='PackageName',
  version='1.0',
  description='This is a demo package',
  ext_modules=[module1])

Both the host and target Python are built from a6e59a2e880e

There's a little bug in the built filename: 
build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which 
should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.

Also, I guess there may be quirks if NDK or the cross-built CPython is at 
different locations than built-time configurations. Those issues can be left to 
the future.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye added the comment:

So I suggest we start with this patch as it works for:
* The standard library extension modules (for debian as well).
* The third-party extension modules on platforms that have multiarch defined 
and on platforms that do not have multiarch defined (on debian systems one can 
only use the build directory for XBUILD_PYTHON_DIR, the other systems can use 
the three options described in msg281993).

Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the 
multiarch triplet if this is necessary for debian systems, unless the debian 
developers want to continue using the system they have developed and are 
currently using to cross compile third-party extension modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye added the comment:

> I don't think that identifying the target by some path is the right way to 
> go, and you should be able to identify the target by giving the target 
> triplet as used by the configure parameters and then deduce the location from 
> the target (or have an explicit location given).
> 
> The idea here is to use the platform identifier which we already had in the 
> trunk before the PLATDIR was removed (the multiarch id or if not defined the 
> platform). So by having a  specifier, you could deduce a path, or a 
> -python-config executable and you're done. No need to know about a 
> path directly.  Of course python cross builds would have to install the 
> -python-config executable or symlink.

Hum, you still need to provide the native python interpreter with the _path_ to 
the -python-config executable that can be anywhere on the file system 
(its location is relative to the cross compiled build, not the native 
interpreter), so it seems that this contradicts your previous sentence saying 
"I don't think that identifying the target by some path is the right way to go".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Changes by Xavier de Gaye :


--
nosy: +Chi Hsuan Yen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye added the comment:

> This approach will not work with a "multiarch" enabled environment, and break 
> cross builds on Debian and Ubuntu.

No, the patch does not break cross builds on Debian and Ubuntu, unless you can 
demonstrate it does.

> Afaics, the proposal assumes that the python executable for the target 
> architecture is installed (which it is not for the multiarch cross-build 
> layout), and that each target architecture has to be identified by it's own 
> directory tree (again, not in the multiarch environment, you can install 
> multiple targets into the same path).

No, you are mistaken, the path name of the build tree may be used to cross 
build third-party extension modules as stated in case a) of msg281993, and this 
should also work with debian. BTW the same code is run to cross build a 
standard library extension module and a third-party extension module, and 
obviously python is not yet installed by the time the standard library 
extension modules are built ;-)

> The idea here is to use the platform identifier which we already had in the 
> trunk before the PLATDIR was removed (the multiarch id or if not defined the 
> platform). So by having a  specifier, you could deduce a path, or a 
> -python-config executable and you're done. No need to know about a 
> path directly.  Of course python cross builds would have to install the 
> -python-config executable or symlink.

It is not clear why, because debian has this multiarch design, this should 
constrain our build system to follow their paradigm. After all, debian has 
already found some ways to cross-build the extension modules for their 
supported multiarch platforms since it is not possible, as of today, with 
upstream CPython.

> Minor issue: please s/xbuild/cross_build/.

Agreed.

> > * The sysconfigdata file name was terminated with a dangling
> >   underscore when 'multiarch' is not defined.
> 
> That only solves part of the problem in that the kernel/os version gets 
> encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the 
> version of your runtime and build time kernel differ.

No idea what is the problem you are refering to. It cannot be the "dangling 
underscore" problem since this is a cosmetic issue. Please explain.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Matthias Klose]

Matthias Klose added the comment:

> * The sysconfigdata file name was terminated with a dangling
>   underscore when 'multiarch' is not defined.

That only solves part of the problem in that the kernel/os version gets encoded 
as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of 
your runtime and build time kernel differ.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Matthias Klose]

Matthias Klose added the comment:

This approach will not work with a "multiarch" enabled environment, and break 
cross builds on Debian and Ubuntu.

Afaics, the proposal assumes that the python executable for the target 
architecture is installed (which it is not for the multiarch cross-build 
layout), and that each target architecture has to be identified by it's own 
directory tree (again, not in the multiarch environment, you can install 
multiple targets into the same path).

I don't think that identifying the target by some path is the right way to go, 
and you should be able to identify the target by giving the target triplet as 
used by the configure parameters and then deduce the location from the target 
(or have an explicit location given).

The idea here is to use the platform identifier which we already had in the 
trunk before the PLATDIR was removed (the multiarch id or if not defined the 
platform). So by having a  specifier, you could deduce a path, or a 
-python-config executable and you're done. No need to know about a path 
directly.  Of course python cross builds would have to install the 
-python-config executable or symlink.

Minor issue: please s/xbuild/cross_build/.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

New submission from Xavier de Gaye:

With this patch, cross compiling a third-party extension module is done with 
the command:

  XBUILD_PYTHON_DIR=/path/to/python/dir python setup.py build

where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled 
python executable.
It may be:
a) The build tree, which is the source tree when the cross compilation is not 
out of the source tree.
b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed 
with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' 
may be different.
c) When the result of the cross compilation has been manually copied (for 
example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is 
/some/path/bin.
In case b), one can use the 'install' setup.py command instead of the 'build' 
command, to build and install the extension module to 
'$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 
'egg-info' file.

The patch uses the 'python-config' shell script (created at issue 16235 [1]) 
that is initialized with the proper variables at the configure stage to provide 
the minimum information required by the sysconfig module to create (with the 
option --generate-posix-vars) the sysconfigdata file or to import the 
sysconfigdata module.
The patch also fixes issue 22724 [2] as sys.path is only modified during the 
time needed to import the sysconfigdata module.

The patch fixes two minor problems:
* '_PYTHON_HOST_PLATFORM' in Makefile.pre.in was not added to the sysconfig 
variables as intended, since those variables may not be prefixed with an 
underscore.
* The sysconfigdata file name was terminated with a dangling underscore when 
'multiarch' is not defined.
Patch also tested with pyephem on an Android emulator.
The patch misses the documentation.
Please run autoconf after installing the patch.

[1] issue 16235: add python-config.sh for use during cross compilation
http://bugs.python.org/issue16235
[2] issue 22724: byte-compile fails for cross-builds
http://bugs.python.org/issue22724

--
assignee: xdegaye
components: Cross-Build
files: cross-build-extension.patch
keywords: patch
messages: 281993
nosy: Alex.Willmer, doko, dstufft, eric.araujo, haypo, martin.panter, xdegaye, 
zach.ware
priority: normal
severity: normal
stage: patch review
status: open
title: cross compilation of third-party extension modules
type: behavior
versions: Python 3.6, Python 3.7
Added file: http://bugs.python.org/file45688/cross-build-extension.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com