Gerlando Falauto created ARROW-2616:
---------------------------------------
Summary: Cross-compiling Pyarrow
Key: ARROW-2616
URL: https://issues.apache.org/jira/browse/ARROW-2616
Project: Apache Arrow
Issue Type: Bug
Components: Packaging, Python
Affects Versions: 0.9.0
Reporter: Gerlando Falauto
Hi there,
not quite sure this could be filed in as a bug.
I'm trying to cross-compile Pyarrow on a regular linux x86_64 host for an arm32
target, using LEDE/OpenWRT as the build/packaging system.
The process miserably fails as soon as cmake gets involved:
{noformat}
-- Runnning cmake for pyarrow
cmake
-DPYTHON_EXECUTABLE=/workdir/rpi/LEDE/staging_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/host/bin/python3.6
-DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUNDLE_BOOST=OFF
-DCMAKE_BUILD_TYPE=debug
/workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python
INFOCompiler command:
/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
INFOCompiler version: Reading specs from
/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../lib/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/specs
COLLECT_GCC=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
COLLECT_LTO_WRAPPER=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../libexec/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/lto-wrapper
Target: arm-openwrt-linux-muslgnueabi
Configured with:
/workdir/LEDE/build_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/gcc-5.4.0/configure
--with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC
5.4.0 r3533-d0bf257c46'
--prefix=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi
--build=x86_64-linux-gnu --host=x86_64-linux-gnu
--target=arm-openwrt-linux-muslgnueabi --with-gnu-ld --enable-target-optspace
--disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx
--disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++
--with-gmp=/workdir/LEDE/staging_dir/host
--with-mpfr=/workdir/LEDE/staging_dir/host
--with-mpc=/workdir/LEDE/staging_dir/host --disable-decimal-float
--with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit
--with-float=hard
--with-headers=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/include
--disable-libsanitizer --enable-languages=c,c++ --enable-shared
--enable-threads
--with-slibdir=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/lib
--enable-lto --with-libelf=/workdir/LEDE/staging_dir/host
Thread model: posix
gcc version 5.4.0 (LEDE GCC 5.4.0 r3533-d0bf257c46)
INFOCompiler id: GNU
Selected compiler gcc 5.4.0
Configured for DEBUG build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: DEBUG
-- Build output directory:
/workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/debug/
CMake Error at cmake_modules/FindPythonLibsNew.cmake:124 (message):
Python config failure: Python is 64-bit, chosen compiler is 32-bit
Call Stack (most recent call first):
CMakeLists.txt:183 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/CMakeFiles/CMakeOutput.log".
error: command 'cmake' failed with exit status 1
{noformat}
I had a look at the affected area (FindPythonLibsNew.cmake) and I'm under the
impression the whole CMake-based build system for pyarrow *was never designed
for cross-compilation in the first place*. It just calls the configured
interpreter ({{PYTHON_EXECUTABLE}}) and tries to locate libraries by running
sample code.
That's just great for native builds (where you have e.g. multiple installations
on the same filesystem), but it's never gonna work on a cross-build environment.
I believe I could patch the cmake files so to be able to pass those paths (e.g.
PYTHON_INCLUDE_DIRS) from the command line (i.e. the way it's done by
{{CMAKE_BUILD_TYPE}}), and skip the auto-detect portions. Before I go down that
path though, I'd like to know whether I'm bound to eventual failure due to
inherent design decisions.
[~pitrou] could perhaps back me up here?
Thank you!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
