some updates on the issue.
I did some further investigation and it turns out that only python cache
files are affected - not only when the manifest is created but also when
files are being staged.
Here is a dump from one of my runs
2020-03-29T03:09:17.3893569Z ERROR: simple-c-1.0-r0
do_prepare_recipe_sysroot: Error executing a python function in
exec_python_func() autogenerated:
2020-03-29T03:09:17.3893931Z
2020-03-29T03:09:17.3894095Z The stack trace of python calls that
resulted in this exception/failure was:
2020-03-29T03:09:17.3894483Z File: 'exec_python_func() autogenerated',
lineno: 2, function: <module>
2020-03-29T03:09:17.3894608Z 0001:
2020-03-29T03:09:17.3894727Z *** 0002:extend_recipe_sysroot(d)
2020-03-29T03:09:17.3894837Z 0003:
2020-03-29T03:09:17.3895143Z File:
'/opt/build/poky/meta/classes/staging.bbclass', lineno: 570, function:
extend_recipe_sysroot
2020-03-29T03:09:17.3895282Z 0566: if "/bin/" in
l or "/sbin/" in l:
2020-03-29T03:09:17.3895415Z 0567: # defer
/*bin/* files until last in case they need libs
2020-03-29T03:09:17.3895535Z 0568: binfiles[l] = (targetdir, dest)
2020-03-29T03:09:17.3895652Z 0569: else:
2020-03-29T03:09:17.3895780Z *** 0570: staging_copyfile(l, targetdir,
dest, postinsts, seendirs)
2020-03-29T03:09:17.3895901Z 0571:
2020-03-29T03:09:17.3896183Z 0572: # Handle deferred binfiles
2020-03-29T03:09:17.3896313Z 0573: for l in binfiles:
2020-03-29T03:09:17.3896429Z 0574: (targetdir, dest) =
binfiles[l]
2020-03-29T03:09:17.3896763Z File:
'/opt/build/poky/meta/classes/staging.bbclass', lineno: 152, function:
staging_copyfile
2020-03-29T03:09:17.3896904Z 0148: os.symlink(linkto, dest)
2020-03-29T03:09:17.3897018Z 0149: #bb.warn(c)
2020-03-29T03:09:17.3897129Z 0150: else:
2020-03-29T03:09:17.3897233Z 0151: try:
2020-03-29T03:09:17.3897341Z *** 0152: os.link(c, dest)
2020-03-29T03:09:17.3897455Z 0153: except OSError as err:
2020-03-29T03:09:17.3897571Z 0154: if err.errno ==
errno.EXDEV:
2020-03-29T03:09:17.3897691Z 0155: bb.utils.copyfile(c, dest)
2020-03-29T03:09:17.3897807Z 0156: else:
2020-03-29T03:09:17.3898436Z Exception: FileNotFoundError: [Errno 2] No
such file or directory:
'/opt/build/build/tmp/sysroots-components/x86_64/python3-vulture-native/usr/lib/python3.8/site-packages/vulture/whitelists/__pycache__/unittest_whitelist.cpython-38.pyc'
->
'/opt/build/build/tmp/work/core2-32-scatest-linux/simple-c/1.0-r0/recipe-sysroot-native/usr/lib/python3.8/site-packages/vulture/whitelists/__pycache__/unittest_whitelist.cpython-38.pyc'
2020-03-29T03:09:17.3898588Z
2020-03-29T03:09:17.3898936Z ERROR: Logfile of failure stored in:
/opt/build/build/tmp/work/core2-32-scatest-linux/simple-c/1.0-r0/temp/log.do_prepare_recipe_sysroot.122175
2020-03-29T03:09:17.3899233Z NOTE: recipe simple-c-1.0-r0: task
do_prepare_recipe_sysroot: Failed
2020-03-29T03:09:17.3936565Z ERROR: Task
(/opt/build/meta-sca/recipes-sca-test/code-from-elsewhere/simple-c.bb:do_prepare_recipe_sysroot)
failed with exit code '1'
2020-03-29T03:09:17.3956520Z NOTE: Running task 3920 of 5451
(/opt/build/poky/meta/recipes-devtools/python/python3-git_3.0.5.bb:do_fetch)
After reading a bit about
pyc-files(https://docs.python.org/3/library/py_compile.html#py_compile.PycInvalidationMode),
I suspect some of the files being invalidated, although I still haven't
figured out how that is done.
After forcefully removing all cache files from all native packages, it
seems to work - but still I think it just hides the general issue
causing this.
Current summary:
- a lot of native python packages are required by multiple recipes (via
DEPENDS)
- this effect is only visible under heavy load (on a 16+ core machine:
never happened so far, 8 core: happens occasionally, 2 core: happens
nearly every time)
- only python cache files (.pyc) are affected
- error occurs at random packages in do_prepare_recipe_sysroot
- some pyc files set in the package manifest are not present in the
filesystem
On 28.03.20 13:26, Konrad Weihmann wrote:
Hi,
I'm facing the following error message sporadically on all branches I
tried so far (master, zeus, warrior and thud)
The stack trace of python calls that resulted in this
exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:extend_recipe_sysroot(d)
0003:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 551,
function: extend_recipe_sysroot
0547: dest = newmanifest[l]
0548: if l.endswith("/"):
0549: staging_copydir(l, targetdir, dest,
seendirs)
0550: continue
*** 0551: staging_copyfile(l, targetdir, dest,
postinsts, seendirs)
0552:
0553: bb.note("Installed into sysroot: %s" % str(msg_adding))
0554: bb.note("Skipping as already exists in sysroot: %s" %
str(msg_exists))
0555:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 152,
function: staging_copyfile
0148: os.symlink(linkto, dest)
0149: #bb.warn(c)
0150: else:
0151: try:
*** 0152: os.link(c, dest)
0153: except OSError as err:
0154: if err.errno == errno.EXDEV:
0155: bb.utils.copyfile(c, dest)
0156: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'
->
'/build/poky/build/tmp/work/qemux86_64-mine-linux/core-image-minimal-mine/1.0-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'
I already had a look at the manifest
cat manifest-x86_64-python3-msgcheck-native.populate_sysroot
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/python3-msgcheck-native
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
which states the file should be there, but when doing
find
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share
the file isn't there.
This happens to random python packages compiled as native (sometimes
even for python-native itself), but (afaik) not for cross or target
packages (I'm pretty sure because of the different packaging applied).
So I digged a little into the code
classes/sstate.bbclass:sstate_install, which seems to create the
sysroot-component dir and the manifest.
There is a gap between the manifest creation (line 285) and the
hardlinking (line till 311).
Now my question is there any place where a file potentially could get
lost? - at first glance there shouldn't be one - I have to admit that
I don't fully understand all this subprocess magic in
lib/oe/path.py:copyhardlinktree.
What I do to fix the issue is reopening the manifest and double check
for missing files and remove them from the manifest, but this
feels wrong - so any advise is welcome...
Hope that someone more familiar with the topic could have a look.
Thanks
Konrad
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#48978): https://lists.yoctoproject.org/g/yocto/message/48978
Mute This Topic: https://lists.yoctoproject.org/mt/72608082/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
