[issue39221] Cross compiled python installed wrong version of lib2to3/Grammar pickle
Andrew Aladjev added the comment:
Hello. I've received more problems with pickle generation, reviewed source
code, invented better bike for solving it. So I can provide full description of
this issue and reduce other developers time on debugging.
Problem:
1. Martin v. Löwis introduced pickle generation Wed Mar 19
04:43:46 2008 commit
https://github.com/python/cpython/commit/5e37baea8007cb64b65a180e4d6c80de292a8a4a#diff-c9bd4064884726c21716df7118dcaeb72a63cc8ddd49484c7f4676934b37d9bb
2. Martin provided code for generating of pickle file: head + tail +
".".join(map(str, sys.version_info)) + ".pickle". This code has never been
changed and comes as is in 2021 year.
Code means that current python interpreter uses its own version for creating a
new pickle file for new cpython interpreter. It will work only when current
python interpreter equals to new cpython interpreter. This code is broken by
design.
3. We can see file "Lib/lib2to3/patcomp.py" in the commit above, today (2021)
it is a part of "Lib/lib2to3/pygram.py". It comes with the following code:
_GRAMMAR_FILE = ... "Grammar.txt"
_PATTERN_GRAMMAR_FILE = ... "PatternGrammar.txt"
4. We can review step-by-step "load_grammar" real world usage from
"Lib/lib2to3/pgen2/driver.py":
driver.load_grammar(_GRAMMAR_FILE)
driver.load_grammar(_PATTERN_GRAMMAR_FILE)
def load_grammar(gt="Grammar.txt", gp=None
gp = head + tail + ".".join(map(str, sys.version_info)) + ".pickle"
if force or not _newer(gp, gt):
logger.info("Generating grammar tables from %s", gt)
def _newer(a, b):
if not os.path.exists(a):
return False
This "code" comes as is today as a part of latest stable python 3.9.5. This
workaround means that when pickle files doesn't exist - python will recreate it
in runtime inside "lib/lib2to3" folder.
This workaround will ruin your production if "lib/lib2to3" is readonly and
pickle files are not inside. Everybody will try to use this workaround as a
security hole. But it looks like nobody from cpython cares.
Solution:
1. Apply provided remove_python_version_from_pickle.patch.
2. Cross compile new cpython.
3. Cross compile portage.
4. Chroot.
5. Run python -c "import lib2to3.pygram", it will create pickles without
version postfix, stop python from creating junk in runtime, fix sandbox
permission issues temporaly.
6. Reinstall portage.
7. Reinstall cpython without patch, it will recreate pickles during
installation.
8. Reinstall portage.
This is just bike on the top of Martin's workaround. The best variant is to
ignore cpython (and maybe python) completely, ignore software written in python
where possible.
--
Added file:
https://bugs.python.org/file50153/remove_python_version_from_pickle.patch
___
Python tracker
<https://bugs.python.org/issue39221>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22724] byte-compile fails for cross-builds
Andrew Aladjev added the comment:
I've made a quick test of all patches provided. All patches have failed to fix
it. The problem is more heavy than it looks like.
Please look at the following code:
PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir)
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f
pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)
'${EPYTHON_FOR_BUILD:-${interp}}
It means that we will use "interp" to load cross compiled modules, env, etc
from "_PYTHON_PROJECT_BASE". This code is broken by design. Loading of cross
compiled modules, env, etc is not possible in general. For example you have
different libc: glibc, musl, uclibc.
glibc -> cross compile -> glibc = loads fine.
musl -> cross compile -> musl = loads fine.
uclibc -> cross compile -> uclibc = loads fine.
glibc -> cross compile -> musl = failed to load.
glibc -> cross compile -> uclibc = failed to load.
musl -> cross compile -> glibc = failed to load.
musl -> cross compile -> uclibc = failed to load.
uclibc -> cross compile -> glibc = failed to load.
uclibc -> cross compile -> musl = failed to load.
Cross compilation doesn't mean only another arch, it means different libc,
kernel headers, binutils and compiler. You will receive same issue while using
clang toolchains.
Please assign this issue to python core developers. This issue can't be fixed
by regular users without experience in python development. It requires complete
change of cross compilation design.
--
___
Python tracker
<https://bugs.python.org/issue22724>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22724] byte-compile fails for cross-builds
Andrew Aladjev added the comment: Hello. I've provided a double recompilation workaround in the following gentoo bug https://bugs.gentoo.org/705970. You can use it for now if you don't want to wait until this issue will be fixed. I will try patches provided by people in this issue today's evening. Thank you. -- ___ Python tracker <https://bugs.python.org/issue22724> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22724] byte-compile fails for cross-builds
Change by Andrew Aladjev : -- nosy: +puchenyaka ___ Python tracker <https://bugs.python.org/issue22724> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39399] Cross compilation using different libc is broken
Andrew Aladjev added the comment: Hello, it seems like issue 22724 is completely the same. Let's discuss solutions there. Thank you. -- ___ Python tracker <https://bugs.python.org/issue39399> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39399] Cross compilation using different libc is broken
New submission from Andrew Aladjev : Hello. I am implementing python cross compilation using "x86_64-pc-linux-musl" toolchain: "x86_64-pc-linux-musl-emerge -v1 python:3.6". Please see the following build log https://gist.github.com/andrew-aladev/e10fa5a8151ffb3c5782edd64ae08b28. We can see the following part: Traceback (most recent call last): File "/usr/x86_64-pc-linux-musl/tmp/portage/dev-lang/python-3.6.9/image//usr/lib/python3.6/compileall.py", line 17, in import struct File "/usr/x86_64-pc-linux-musl/tmp/portage/dev-lang/python-3.6.9/image/usr/lib/python3.6/struct.py", line 13, in from _struct import * ImportError: libc.so: cannot open shared object file: No such file or directory It means that cross compilation of python is not reliable today by design. Python is trying to use PYTHON_FOR_BUILD for loading cross compiled modules. It is not possible in general case. PYTHON_FOR_BUILD should not try to load cross compiled modules. Please see the following gentoo issue https://bugs.gentoo.org/705970. I've attached a gentoo specific workaround there. -- components: Build messages: 360330 nosy: puchenyaka priority: normal severity: normal status: open title: Cross compilation using different libc is broken type: behavior versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue39399> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39221] Cross compiled python installed wrong version of lib2to3/Grammar pickle
Andrew Aladjev added the comment: For now I've created a workaround - just removed python version from pickle generator. For my current container it works perfect. -- keywords: +patch Added file: https://bugs.python.org/file48828/remove_python_version_from_pickle.patch ___ Python tracker <https://bugs.python.org/issue39221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39221] Cross compiled python installed wrong version of lib2to3/Grammar pickle
Andrew Aladjev added the comment: This is build.log: https://gist.githubusercontent.com/andrew-aladev/f36e8e754278e4fad50ff13238326f7a/raw/245a72e8bd0fb3dc6e4185ec757b2d3ab59d5861/gistfile1.txt We can see that python installed "Grammar3.6.9.final.0.pickle" instead of "Grammar3.6.10.final.0.pickle" -- ___ Python tracker <https://bugs.python.org/issue39221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39221] Cross compiled python installed wrong version of lib2to3/Grammar pickle
New submission from Andrew Aladjev : Please see the following gentoo bug https://bugs.gentoo.org/704816 https://github.com/python/cpython/blob/master/Lib/lib2to3/pgen2/driver.py#L110 > head + tail + ".".join(map(str, sys.version_info)) + ".pickle" I've tried "print(sys.version_info)" during compilation and received: > sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0) "sys.version_info" is not the target python version, this is the version of python that is running compilation. This variable needs to be replace with something like "sys.target_python_version". This issue looks simple but I can't fix it by myself. Please assign this issue to core developer. We need to find all places where "sys.version_info" is used as target python version during compilation and replace it. -- components: Library (Lib) messages: 359343 nosy: puchenyaka priority: normal severity: normal status: open title: Cross compiled python installed wrong version of lib2to3/Grammar pickle type: enhancement versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue39221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
