HI,
sorry I didn't specify it further.
To me it seems to be a corner case scenario.
This happens when doing a from scratch build with a lot on native python
libraries (as it is setup in a test distribution I'm working on).
Roughly it pulls in ~100 native python site packages.
The setup where I almost every time can reproduce the issue
- Ubuntu 16.04 or 18.04 container
- 2 VCPUs (can't say what it actually is)
- 14GB of hard disk (I guess it's some virtual storage)
- 12GB of RAM
-> as offered by GitHub's Actions Azure interface
The setup where it occasionally happens
- Ubuntu 19.04/19.10
- 8 cores i7-6th gen
- 16GB Ram
- some Samsung-SSD with more than enough space
The setup where it never happened so far
- Ubuntu 18.04
- 48 core threadripper
- lots of RAM and SSD
and
- Debian 9
- 24 core XEN guest
- >80GB of RAM
- HDD doesn't matter as it was a ramdisk only build
This doesn't happen (afaik) on differential builds when the initial one
is good (same for sstate cache backed builds).
I just wanted to know if somebody ever encountered the same issue.
As the nature of this issue is hard to catch, I'm open to ideas...
What currently works for me is to remove all python cache files before
they are packaged - at least none of my recent build failed so far.
Any ideas how to tackle this problem, any suggestions?
Best
Konrad
On 02.04.20 05:42, Randy MacLeod wrote:
On 2020-03-28 8:26 a.m., Konrad Weihmann wrote:
Hi,
I'm facing the following error message sporadically on all branches I
tried so far (master, zeus, warrior and thud)
The stack trace of python calls that resulted in this
exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:extend_recipe_sysroot(d)
0003:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 551,
function: extend_recipe_sysroot
0547: dest = newmanifest[l]
0548: if l.endswith("/"):
0549: staging_copydir(l, targetdir,
dest, seendirs)
0550: continue
*** 0551: staging_copyfile(l, targetdir, dest,
postinsts, seendirs)
0552:
0553: bb.note("Installed into sysroot: %s" % str(msg_adding))
0554: bb.note("Skipping as already exists in sysroot: %s" %
str(msg_exists))
0555:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 152,
function: staging_copyfile
0148: os.symlink(linkto, dest)
0149: #bb.warn(c)
0150: else:
0151: try:
*** 0152: os.link(c, dest)
0153: except OSError as err:
0154: if err.errno == errno.EXDEV:
0155: bb.utils.copyfile(c, dest)
0156: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'
->
'/build/poky/build/tmp/work/qemux86_64-mine-linux/core-image-minimal-mine/1.0-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'
I already had a look at the manifest
cat manifest-x86_64-python3-msgcheck-native.populate_sysroot
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/python3-msgcheck-native
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
which states the file should be there, but when doing
find
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share
the file isn't there.
This happens to random python packages compiled as native (sometimes
even for python-native itself), but (afaik) not for cross or target
packages (I'm pretty sure because of the different packaging applied).
So I digged a little into the code
classes/sstate.bbclass:sstate_install, which seems to create the
sysroot-component dir and the manifest.
There is a gap between the manifest creation (line 285) and the
hardlinking (line till 311).
Now my question is there any place where a file potentially could get
lost? - at first glance there shouldn't be one - I have to admit that
I don't fully understand all this subprocess magic in
lib/oe/path.py:copyhardlinktree.
What I do to fix the issue is reopening the manifest and double check
for missing files and remove them from the manifest, but this
feels wrong - so any advise is welcome...
Hope that someone more familiar with the topic could have a look.
Hi Konrad,
I'm not really familiar with that code but it's being run buy 1000s of
builder around the world so let's try to eliminate a few possibilities.
When did you start having this problem?
How often do you think it's happening: 1 in 3 builds, 1 in 10?
Tell us about your machine: OS,version, disk, CPUs, ram
Do you do anything special in your conf dir? Send local.conf perhaps.
Do you have any local bbappends or commits on top of poky or
in other layers?
Have you tried to simplify the build to eliminate problems
potentially caused by other layers?
Are you able to reproduce the problem on more than one build machine?
Are you able to reproduce the problem on a different Linux distro?
Are there other builds or users on the machine that may be causing
extra load?
../Randy
Thanks
Konrad
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#49026): https://lists.yoctoproject.org/g/yocto/message/49026
Mute This Topic: https://lists.yoctoproject.org/mt/72608082/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
