[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Martin Panter]

Changes by Martin Panter :


--
resolution:  -> out of date
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Xavier de Gaye]

Xavier de Gaye added the comment:

I confirm that, on 3.6 and after the changesets that fixed issue 22359, pgen is 
always cross-compiled whatever the timestamps and that pgen is not run on a 
cross-build for Android. So I think it should work as well now for Trevor with 
v2.7.12.

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Martin Panter]

Martin Panter added the comment:

Since 2.7.12 and 3.5.2, pgen should not be executed when in cross-compilation 
mode, thanks to Issue 22359 and Issue 22625. So I suspect the main problem is 
basically solved now.

Even though it should no longer be used, pgen is still cross-compiled depending 
on the timestamps and “make touch”. (Evidence: 
Issue 27490.)

My guess of why “make touch” did not work for Trevor was that he was compiling 
Python 2. In this case, “make touch” was added in revision da3f4774b939, and 
still only affects two files, neither of which are generated by pgen. Maybe 
Python 2’s “make touch” rule could be expanded for other generated files 
including Include/graminit.h etc.

--
nosy: +martin.panter
superseder:  -> When cross-compiling, don’t try to execute binaries

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Marc Duponcheel]

Marc Duponcheel added the comment:

one workaround is to build native in separate directory and to use the native 
build Parser/pgen and Programs/_freeze_importlib for cross compilation host 
build.

--
nosy: +Marc Duponcheel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Gregory P. Smith]

Changes by Gregory P. Smith g...@krypto.org:


--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Antoine Pitrou]

Antoine Pitrou added the comment:

Have you tried make touch to avoid rebuilding pgen and stuff?

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

I thought make touch was only for those trying to build from the Mecurial 
source, as opposed to building from the released tar-ball source.  I thought my 
efforts laid on the other side of the need for that command.  If I understood 
wrong, when would I use it and why? ... Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[R. David Murray]

R. David Murray added the comment:

make touch avoids rebuilding pgen and stuff, and just uses what was checked 
out or provided in the tarball.  The release tarballs are supposed to have the 
time stamps in the correct order so that the compiletime/boostrapping utilities 
don't get built/rebuilt.

So for cross compilation, you should be able to use the existing files to build 
the target python.  If you then also want pgen *on the target*, you will need 
to build it for the target.  Like I said, I don't have much experience with 
this stuff, but if I understand correctly, you only really the utilities if you 
want to be able to do development *of python* on the target.

So, if running make in cross-compile mode (whatever that looks like, I have 
no clue) tries to rebuild pgen in your scenario, then *that* is what needs to 
be fixed, I think.

Of course, I could be completely wrong, which is why I've been pretty tentative 
about handing out advice :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

Thanks, David!  I have no interest in running pgen on the target/host.  My only 
interest is building python and its various modules to run on my embedded host. 
 I do not want to develop Python on the embedded host.  Unfortunately, the 
build process requires Parser/pgen to build the grammar files, which are needed 
for several object files.  Here's the relevant snippet from the Makefile.pre.in:

$(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGENSRCS)
@$(MKDIR_P) Include
$(MAKE) $(PGEN)
$(PGEN) $(GRAMMAR_INPUT) $(GRAMMAR_H) $(GRAMMAR_C)
$(GRAMMAR_C): $(GRAMMAR_H) $(GRAMMAR_INPUT) $(PGENSRCS)
$(MAKE) $(GRAMMAR_H)
touch $(GRAMMAR_C)

$(PGEN):$(PGENOBJS)
$(CC) $(OPT) $(LDFLAGS) $(PGENOBJS) $(LIBS) -o $(PGEN)

Parser/grammar.o:   $(srcdir)/Parser/grammar.c \
$(srcdir)/Include/token.h \
$(srcdir)/Include/grammar.h
...
Python/compile.o Python/symtable.o Python/ast.o: $(GRAMMAR_H) $(AST_H)

If there is a way to eliminate the need for Parser/pgen to run on the build 
system to cross-compile the default all target, that would be great. ... I'll 
experiment with make touch. ... Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

I executed make touch between configure and make, but the build process 
still created Parser/pgen and then tried to use it, which of course crashed the 
build, since pgen was compiled for the embedded host not the build system.  :(  
Was that the wrong usage?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[R. David Murray]

R. David Murray added the comment:

I believe that's the correct usage, in which case there must be a bug in the 
process somewhere.  My guess would be that it is looking for a file in the 
wrong place when doing a cross compile, but that's just a guess.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

In the vein of:

http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html

I have created a patch and top-level build script, which builds the requisite 
python interpreter and Parser/pgen binary to run on the build system, which are 
both used during the cross-compile process.

In some ways, this patch is simpler, because of the new cross-compile 
capabilities in 2.7.4 and 2.7.5.  However, previous work is still insufficient, 
because the Makefile assumes the cross-compiled Parser/pgen can be executed on 
the build system during build.  This patch primarily compensates for that, but 
it also works around some misplaced installation file issues.

My top-level cross-compile script, which applies the patch *during* the build 
process is also attached.  It depends on a VCS to restore the original version, 
but I could have just as easily made 2 copies of the patched files (before and 
after patching) and copied the original versions as needed.

I hope this helps others.  Hopefully, bits of this can be integrated to help 
further simplify future cross-compile efforts.

--
keywords: +patch
Added file: http://bugs.python.org/file31991/Python-2.7.5-xcompile.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

Short version of cross-compile script without error checking:

#!/bin/bash
export RFS=/local/my_root_file_system
make distclean
rm -rf python_for_build Parser/pgen_for_build
git checkout -- Makefile.pre.in Modules/Setup.dist configure setup.py
./configure
make python Parser/pgen
mv python python_for_build
mv Parser/pgen Parser/pgen_for_build
patch -p3  Python-2.7.5-xcompile.patch
export 
PATH=/opt/my_cross_compile_toolchain/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin:${PATH}
make distclean
./configure --host=powerpc-none-linux-gnuspe --build=i586-linux-gnu --prefix=/  
\
--disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no 
ac_cv_have_long_long_format=yes
make --jobs=4
sudo make install DESTDIR=${RFS} PATH=${PATH}

--
Added file: http://bugs.python.org/file31992/cross-compile.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

FWIW, I also explored my original proposal, which essentially moved the above 
script and modifications into the Python configure.ac and Makefile.pre.in 
files, so that Python's internal build process would create the native build 
system versions of Python and Parser/pgen, if necessary.  I made some progress 
in creating additional Make targets just for the native build system, which 
redefined the toolchain after completion; however, I encountered unexepected, 
overwhelming configure contamination in the auto-generated header files.

Maybe someone can push this farther than me?  I'm attaching my patch, in case 
it helps someone else and just as reference.

As for me, I think it is much easier to use the above solution, which would 
require very little patching to the sources to make a 2-step compilation to be 
trivial.

--
Added file: 
http://bugs.python.org/file31993/Python-2.7.5-xcompile_improvements.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[R. David Murray]

R. David Murray added the comment:

I have no idea, frankly :)

I guess my point is that the tool chain was not designed with cross compilation 
in mind, so sorting out how to make it work and writing a howto is something 
that needs to be done.  The patches that have already been applied address a 
narrow need felt and fixed by the Debian folk, dealing with cross compiling 64 
bit vs 32 bit, if I understand correctly.  I don't know if it is a general 
cross compile fix, but in theory it should at least have moved us closer.

Now, that said, as far as build order goes, what we do in general is to check 
in the sources that are built by the tools.  If you get your sources out of a 
tarball, make shouldn't try to rebuild those.  If you are working from a 
checkout, try 'make touch' before the make, to update the timestamps the way 
they would be in a release tarball.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

Ok, thanks for the tips.  I'm new to developing on Python itself.  I'll start 
simple by trying to develop a set of patches for the 2.7.5 source tar-ball, 
which I usually use to build Python. If that succeeds, I'll look into pushing 
it into the related source files.  So far, I'm anticipating modifying these 
files: configure, Makefile.pre.in, and setup.py. ... Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

New submission from Trevor Bowen:

FWIW, I'm using a Freescale cross-compile tool-chain on a Linux x86-64 build 
host, although I have duplicated the cross-compile error on an x86 Ubunutu 
10.04 build host.

Steps to reproduce:

$ wget http://www.python.org/ftp/python/2.7.5/Python-2.7.5.tar.bz2
$ tar -jxf Python-2.7.5.tar.bz2
$ cd Python-2.7.5
$ export 
PATH=/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin:${PATH}
# For info only:
$ ls -l 
/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin
 
total 10472
-rwxrwxrwx 1 root root  590485 Mar  5  2012 powerpc-none-linux-gnuspe-addr2line
-rwxrwxrwx 1 root root  614647 Mar  5  2012 powerpc-none-linux-gnuspe-ar
-rwxrwxrwx 1 root root  897161 Mar  5  2012 powerpc-none-linux-gnuspe-as
-rwxrwxrwx 1 root root  235382 Mar  5  2012 powerpc-none-linux-gnuspe-c++
-rwxrwxrwx 1 root root  589227 Mar  5  2012 powerpc-none-linux-gnuspe-c++filt
-rwxrwxrwx 1 root root  234277 Mar  5  2012 powerpc-none-linux-gnuspe-cpp
-rwxrwxrwx 1 root root8503 Mar  5  2012 powerpc-none-linux-gnuspe-embedspu
-rwxrwxrwx 1 root root  235382 Mar  5  2012 powerpc-none-linux-gnuspe-g++
-rwxrwxrwx 1 root root  233126 Mar  5  2012 powerpc-none-linux-gnuspe-gcc
-rwxrwxrwx 1 root root  233126 Mar  5  2012 powerpc-none-linux-gnuspe-gcc-4.3.2
-rwxrwxrwx 1 root root   16512 Mar  5  2012 powerpc-none-linux-gnuspe-gccbug
-rwxrwxrwx 1 root root   28017 Mar  5  2012 powerpc-none-linux-gnuspe-gcov
-rwxrwxrwx 1 root root  655127 Mar  5  2012 powerpc-none-linux-gnuspe-gprof
-rwxrwxrwx 1 root root 1036372 Mar  5  2012 powerpc-none-linux-gnuspe-ld
-rwxrwxrwx 1 root root  603678 Mar  5  2012 powerpc-none-linux-gnuspe-nm
-rwxrwxrwx 1 root root  750617 Mar  5  2012 powerpc-none-linux-gnuspe-objcopy
-rwxrwxrwx 1 root root  895336 Mar  5  2012 powerpc-none-linux-gnuspe-objdump
-rwxrwxrwx 1 root root  614647 Mar  5  2012 powerpc-none-linux-gnuspe-ranlib
-rwxrwxrwx 1 root root  264063 Mar  5  2012 powerpc-none-linux-gnuspe-readelf
-rwxrwxrwx 1 root root  593901 Mar  5  2012 powerpc-none-linux-gnuspe-size
-rwxrwxrwx 1 root root  591853 Mar  5  2012 powerpc-none-linux-gnuspe-strings
-rwxrwxrwx 1 root root  750617 Mar  5  2012 powerpc-none-linux-gnuspe-strip
$ ./configure --host=powerpc-none-linux-gnuspe \
--build=i586-linux-gnu --prefix=/  \
--disable-ipv6 ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no
$ make

Make fails because the foreign (powerpc, in this case) pgen binary is used to 
build the $(GRAMMAR_H).  Instead, a native version of the python interpreter 
(preferrable) and pgen (essential) need to be built first using the host's 
toolchain.  Then, these tools can be used to build the full foreign suite.

FWIW, This 2 step process is well documented here:

http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html

However, the cross-compile fixes from Issue17086 may have helped this process.  
(It's hard for me to determine how to take advantage of these updates.)  
Certainly, they changed the above process dramatically.

Incidentally, Python 3.3.2 exhibits almost the exact same build error, if not 
the same.

Build log on an x86-64 host for Python-2.7.5 for a PowerPC-linux target is 
attached.

Tail of log:

make Parser/pgen
make[1]: Entering directory `/home/user/Python-2.7.5'
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   
-DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   
-DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   
-DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   
-DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   
-DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
powerpc-none-linux-gnuspe-gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes 
 Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o 
Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o 
Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o 
Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o 
Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl  -lpthread -lutil -o 
Parser/pgen
make[1]: Leaving directory `/home/user/Python-2.7.5'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
Parser/pgen: Parser/pgen: cannot execute binary file
make: *** [Include/graminit.h] 

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[R. David Murray]

R. David Murray added the comment:

It is a known issue that the Python build infrastructure does not currently 
support cross compiling.  There are a number of issues and patches in this 
tracker that address pieces of this puzzle.  Help sorting it all out will be 
welcome.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19142] Cross-compile fails trying to execute foreign pgen on build host

[Trevor Bowen]

Trevor Bowen added the comment:

Sorry, I do not mean to compound an already open and complex problem.  I 
thought that the fixes in 2.7.4 were meant in part to help alleviate this 
problem.  I had not found any feedback or tutorials, so I wanted to provide a 
status update of sorts.  I'm sure there is some.  I just have not been able to 
find it myself.

If the configure script passed in the cross-compile tool-chain *and* the 
build-host tool-chain, could the Makefile be enhanced to build pgen and python 
first with the build-host and then use them to build the full cross-compiled 
versions?  Or, would there be a better general approach?

I am not familiar with the MINGW requirements.  Would this satisfy their needs?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19142
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com