[issue45886] Fix Program/_freeze_module for cross compiling Python

[Brett Cannon]

Change by Brett Cannon :


--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Christian Heimes  added the comment:

The issue is related to bpo-45873. The Debian buildbot has an ancient Python 
version that does not support f-strings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Irit Katriel]

Irit Katriel  added the comment:

Is this related to this issue or something else?

https://buildbot.python.org/all/#/builders/310/builds/344

gcc -pthread -c -Wsign-compare -g -Og -Wall-std=c99 -Wextra 
-Wno-unused-parameter -Wno-missing-field-initializers 
-Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal 
 -I. -I./Include-DPy_BUILD_CORE -o Programs/_freeze_module.o 
Programs/_freeze_module.c
gcc -pthread -o Programs/_freeze_module Programs/_freeze_module.o 
Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o 
Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o 
Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o 
Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o 
Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o 
Objects/classobject.o Objects/codeobject.o Objects/complexobject.o 
Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o 
Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o 
Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o 
Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o 
Objects/longobject.o Objects/dictobject.o Objects/odictobject.o 
Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o 
Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o 
Objects/picklebufob
 ject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o 
Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o 
Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o 
Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o 
Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o 
Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o 
Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o 
Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o 
Python/getcompiler.o Python/getcopyright.o Python/getplatform.o 
Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o 
Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o 
Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o 
Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o 
Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o 
Python/pytime.o Py
 thon/bootstrap_hash.o Python/specialize.o Python/structmember.o 
Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o 
Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o 
Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o 
Python/suggestions.o Python/dynload_shlib.oModules/config.o 
Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  
Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  
Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  
Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o 
Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o 
Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  
Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  
Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  
Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  
Modules/symtablemodule.o  Modules/pwdmodule
 .o  Modules/xxsubtype.o -lpthread -ldl  -lutil-lm 
Programs/_freeze_module importlib._bootstrap ./Lib/importlib/_bootstrap.py 
Python/frozen_modules/importlib._bootstrap.h
python3 ./Tools/scripts/deepfreeze.py 
Python/frozen_modules/importlib._bootstrap.h -m importlib._bootstrap -o 
Python/deepfreeze/importlib._bootstrap.c
  File "./Tools/scripts/deepfreeze.py", line 29
res.append(f"\\x{i:02x}")
   ^
SyntaxError: invalid syntax
Makefile:1069: recipe for target 'Python/deepfreeze/importlib._bootstrap.c' 
failed
make: *** [Python/deepfreeze/importlib._bootstrap.c] Error 1
program finished with exit code 2
elapsedTime=45.181814

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 765b2a3ad2e8abf9a06d5e9b3802b575ec115d76 by Christian Heimes in 
branch 'main':
bpo-45886: Fix OOT build when srcdir has frozen module headers (GH-29793)
https://github.com/python/cpython/commit/765b2a3ad2e8abf9a06d5e9b3802b575ec115d76


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +28028
pull_request: https://github.com/python/cpython/pull/29793

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Guido van Rossum]

Guido van Rossum  added the comment:

Imthink I prefer the ‘make clean’ extension.--
--Guido (mobile)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Christian Heimes  added the comment:

The FREEZE_MODULE trick fails with out-of-tree builds when SRCDIR contains the 
frozen_modules header files from a previous build. When doing OOT building, 
make includes the SRCDIR in the search path for dependencies (VPATH). It 
considers $(srcdir)/Python/frozen_modules/importlib._bootstrap_external.h as 
recent enough and therefore does not generate 
$(builddir)/Python/frozen_modules/importlib._bootstrap_external.h. The 
freeze_module program does not know about VPATH and looks up the file in the 
$(builddir) tree:

$ make 
FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module
...
python3.9 ../../Tools/scripts/deepfreeze.py 
Python/frozen_modules/importlib._bootstrap_external.h -m 
importlib._bootstrap_external -o 
Python/deepfreeze/importlib._bootstrap_external.c
Traceback (most recent call last):
  File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 463, in 

main()
  File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 451, in 
main
with open(args.file, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 
'Python/frozen_modules/importlib._bootstrap_external.h'
make: *** [Makefile:1055: Python/deepfreeze/importlib._bootstrap_external.c] 
Error 1

$ make 
FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module 
Python/frozen_modules/importlib._bootstrap_external.h  
make: '../../Python/frozen_modules/importlib._bootstrap_external.h' is up to 
date.


I see two possible solutions for the problem:

* extend ``make clean`` to also remove $(FROZEN_FILES_OUT). The make clean 
command is suppose to remove all files that interferes with OOT builds (see 
check-clean-src in Makefile.pre.in).
* prefix $(FROZEN_FILES_OUT) and their targets with $(abs_builddir) to avoid 
VPATH lookups. abs_builddir is the absolute path to the build directory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Christian Heimes  added the comment:


New changeset dd8ce9ea8d7e45224a348ea7962b5f50cbf86dc5 by Christian Heimes in 
branch 'main':
bpo-45886: Allow overriding freeze command for cross compiling (GH-29735)
https://github.com/python/cpython/commit/dd8ce9ea8d7e45224a348ea7962b5f50cbf86dc5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Guido van Rossum]

Guido van Rossum  added the comment:

To the question "why does _freeze_module need to be a (C) binary" (IOW, why 
can't it be replaced by a Python script that is run with PYTHON_FOR_REGEN, 
which always runs on the build host), the reason is that it uses the bytecode 
compiler to generate correct output. The bytecode compiler is written in C and 
depends on many other parts of the runtime. Bytecode is not compatible between 
versions (and in 3.11 it is an especially moving target) so we absolutely must 
use the bytecode compiler from the current source tree.

Fortunately bytecode *is* portable across OS and CPU versions, so there is no 
problem with taking a _freeze_module binary compiled for the build host and 
running it there (on the build host). It *is* complicated to build a binary for 
the build host in a tree configured for cross compilation though -- you'd have 
to do an out-of-tree build configured for the build host platform. Rather than 
solving *that* problem, Christian proposes to let the user solve that, and 
allowing the user to pass in the path to the host platform _freeze_module 
binary.

--
nosy: +eric.snow, gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

Change by Christian Heimes :


--
keywords: +patch
pull_requests: +27973
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/29735

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45886] Fix Program/_freeze_module for cross compiling Python

[Christian Heimes]

New submission from Christian Heimes :

The build process creates the binary Program/_freeze_module, which is used to 
create frozen modules. The program takes a Python file, compiles it, and stores 
its marshalled output in a header file.

The approach does not work when cross compiling Python. In cross building case 
Program/_freeze_module cannot be executed on the build host. For example a 
cross build with build host "x86_64" and target host "aarch64" would create a 
aarch64 Program/_freeze_module. The current x86_64 host cannot executed binary 
(unless you use qemu and binfmt, which I'm deliberately ignoring here).

To unblock cross building and until we find a better solution, I propose that 
we allow developers to override the freeze module command on the command line. 
This allows developers to use a freeze_module program from a non-cross build:

   ../../configure -C --host=aarch64-linux-gnu-gcc --build=x86_64-pc-linux-gnu
   make FREEZE_MODULE=../x86_64/Program/_freeze_module

--
assignee: christian.heimes
components: Build
messages: 406893
nosy: christian.heimes
priority: normal
severity: normal
status: open
title: Fix Program/_freeze_module for cross compiling Python
type: enhancement
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com