Re: [OE-core] [PATCH] git: set --with-gitconfig=/etc/gitconfig for -native builds
Polite ping. On 23/04/2024 11.57, Rasmus Villemoes via lists.openembedded.org wrote: > From: Rasmus Villemoes > > Commit 6c2ae2346db0 (kern-tools: depend on git-replacement-native) > broke our kernel builds. For saving space and time, we have a DL_DIR > shared between multiple users/buildbots, not all of which run with the > same uid (and with appropriate sticky bits set so that files > downloaded by one user become owned by a common group and are readable > by others). This works fine also for git sources because the docker > images we use all have a /etc/gitconfig with > > [safe] > directory = * > > But with the mentioned commit, the host's git is no longer used for > do_unpack (nor for do_fetch if re-building and sysroot has already > been populated by a previous build), causing spurious "fatal: detected > dubious ownership..." failures. > > Currently, the path where the git-native binary searches for system > gitconfig is the sysroot from it was built, which obviously doesn't > contain a /etc/gitconfig. As for the nativesdk variant, respect the > host's /etc/gitconfig if present. > > Signed-off-by: Rasmus Villemoes > --- > meta/recipes-devtools/git/git_2.44.0.bb | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-devtools/git/git_2.44.0.bb > b/meta/recipes-devtools/git/git_2.44.0.bb > index 90e555eba7..78b00dd19f 100644 > --- a/meta/recipes-devtools/git/git_2.44.0.bb > +++ b/meta/recipes-devtools/git/git_2.44.0.bb > @@ -40,6 +40,7 @@ EXTRA_OECONF = > "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \ > --without-iconv \ > " > EXTRA_OECONF:append:class-nativesdk = " --with-gitconfig=/etc/gitconfig " > +EXTRA_OECONF:append:class-native = " --with-gitconfig=/etc/gitconfig " > > # Needs brokensep as this doesn't use automake > inherit autotools-brokensep perlnative bash-completion manpages > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199445): https://lists.openembedded.org/g/openembedded-core/message/199445 Mute This Topic: https://lists.openembedded.org/mt/105686820/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] systemd: sed ROOT_HOME only if sysusers PACKAGECONFIG is set
On 24/04/2024 10.36, Alexander Kanavin via lists.openembedded.org wrote: > On Wed, 24 Apr 2024 at 10:19, Christian B. Sørensen via > lists.openembedded.org > wrote: > >> Any input on the patchset ? Would appreciate if it could be included in the >> scarthgap release. > > A little patience would help. The patch is fine (and so there's not > input), it's in a staging branch, How would one know? It's not in master-next, and I don't see any other branches containing it. Is there some git repo other than https://git.openembedded.org/openembedded-core/ one could follow to see that a patch has actually been picked up? The complete radio silence when submitting to oe-core makes it extremely hard to know if anybody has even seen a given patch. Ideally, one would get an email notification (not sent to the list), but just being able to peek at some staging repo would help avoid "impatient" pings. > but integration to master would happen after scarthgap is out. So the plan is to release scarthgap with a known bug and a known fix for it? Or are you saying this will go into the scarthgap branch before release, and after that be applied to master? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198685): https://lists.openembedded.org/g/openembedded-core/message/198685 Mute This Topic: https://lists.openembedded.org/mt/105593219/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] git: set --with-gitconfig=/etc/gitconfig for -native builds
From: Rasmus Villemoes Commit 6c2ae2346db0 (kern-tools: depend on git-replacement-native) broke our kernel builds. For saving space and time, we have a DL_DIR shared between multiple users/buildbots, not all of which run with the same uid (and with appropriate sticky bits set so that files downloaded by one user become owned by a common group and are readable by others). This works fine also for git sources because the docker images we use all have a /etc/gitconfig with [safe] directory = * But with the mentioned commit, the host's git is no longer used for do_unpack (nor for do_fetch if re-building and sysroot has already been populated by a previous build), causing spurious "fatal: detected dubious ownership..." failures. Currently, the path where the git-native binary searches for system gitconfig is the sysroot from it was built, which obviously doesn't contain a /etc/gitconfig. As for the nativesdk variant, respect the host's /etc/gitconfig if present. Signed-off-by: Rasmus Villemoes --- meta/recipes-devtools/git/git_2.44.0.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-devtools/git/git_2.44.0.bb b/meta/recipes-devtools/git/git_2.44.0.bb index 90e555eba7..78b00dd19f 100644 --- a/meta/recipes-devtools/git/git_2.44.0.bb +++ b/meta/recipes-devtools/git/git_2.44.0.bb @@ -40,6 +40,7 @@ EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \ --without-iconv \ " EXTRA_OECONF:append:class-nativesdk = " --with-gitconfig=/etc/gitconfig " +EXTRA_OECONF:append:class-native = " --with-gitconfig=/etc/gitconfig " # Needs brokensep as this doesn't use automake inherit autotools-brokensep perlnative bash-completion manpages -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198611): https://lists.openembedded.org/g/openembedded-core/message/198611 Mute This Topic: https://lists.openembedded.org/mt/105686820/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bmaptool: add PROVIDES bmap-tools for compatibility
On 19/04/2024 23.00, Alexander Kanavin wrote: > This was already proposed, and rejected. > https://lists.openembedded.org/g/openembedded-core/topic/104753355 > > You need to fix the metadata that refers to the old name. So, how can a layer then be compatible with both nanbield and scarthgap? Could this perhaps be applied just to the scarthgap branch, but not master? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198565): https://lists.openembedded.org/g/openembedded-core/message/198565 Mute This Topic: https://lists.openembedded.org/mt/105625834/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] bmaptool: add PROVIDES bmap-tools for compatibility
From: Rasmus Villemoes This is very often a build dependency, such as in my case using a class from meta-ptx, which fails with ERROR: Nothing PROVIDES 'bmap-tools-native'. Close matches: bmaptool-native bpftool-native mtools-native due to the renaming. Signed-off-by: Rasmus Villemoes --- meta/recipes-support/bmaptool/bmaptool_git.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-support/bmaptool/bmaptool_git.bb b/meta/recipes-support/bmaptool/bmaptool_git.bb index fd53c21c06..b4b3fa1fbd 100644 --- a/meta/recipes-support/bmaptool/bmaptool_git.bb +++ b/meta/recipes-support/bmaptool/bmaptool_git.bb @@ -23,6 +23,7 @@ RDEPENDS:${PN} = "python3-core python3-compression python3-misc python3-mmap pyt inherit setuptools3 # For compatibility with layers before scarthgap +PROVIDES = "bmap-tools" RREPLACES:${PN} = "bmap-tools" RCONFLICTS:${PN} = "bmap-tools" -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198562): https://lists.openembedded.org/g/openembedded-core/message/198562 Mute This Topic: https://lists.openembedded.org/mt/105625834/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] openssh: add After dependencies on nss-user-lookup.target
From: Rasmus Villemoes Quoting 'man systemd.special': nss-user-lookup.target A target that should be used as synchronization point for all regular UNIX user/group name service lookups. [...] All services for which the availability of the full user/group database is essential should be ordered after this target, but not pull it in. All services which provide parts of the user/group database should be ordered before this target, and pull it in. When no service providing parts of the user/group database exists and thus pulls in the nss-user-lookup.target, this added dependency is a no-op. However, when such a service does exist, and e.g. modifies /etc/shadow to change password or enable/disable certain accounts, it is essential that no ssh connections are accepted until those changes are made. Signed-off-by: Rasmus Villemoes --- meta/recipes-connectivity/openssh/openssh/sshd.service | 1 + meta/recipes-connectivity/openssh/openssh/sshd.socket | 1 + 2 files changed, 2 insertions(+) diff --git a/meta/recipes-connectivity/openssh/openssh/sshd.service b/meta/recipes-connectivity/openssh/openssh/sshd.service index 2a997b656a..3e570ab1e5 100644 --- a/meta/recipes-connectivity/openssh/openssh/sshd.service +++ b/meta/recipes-connectivity/openssh/openssh/sshd.service @@ -2,6 +2,7 @@ Description=OpenSSH server daemon Wants=sshdgenkeys.service After=sshdgenkeys.service +After=nss-user-lookup.target [Service] Environment="SSHD_OPTS=" diff --git a/meta/recipes-connectivity/openssh/openssh/sshd.socket b/meta/recipes-connectivity/openssh/openssh/sshd.socket index 8d76d62309..7dd2ed0626 100644 --- a/meta/recipes-connectivity/openssh/openssh/sshd.socket +++ b/meta/recipes-connectivity/openssh/openssh/sshd.socket @@ -1,6 +1,7 @@ [Unit] Conflicts=sshd.service Wants=sshdgenkeys.service +After=nss-user-lookup.target [Socket] ExecStartPre=@BASE_BINDIR@/mkdir -p /var/run/sshd -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198463): https://lists.openembedded.org/g/openembedded-core/message/198463 Mute This Topic: https://lists.openembedded.org/mt/105575252/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] perf: add jevents PACKAGECONFIG item
From: Rasmus Villemoes Building for an arm64 target, e.g. qemuarm64 or a raspberrypi3, without "python" in PACKAGECONFIG, results in | Makefile.config:892: *** ERROR: No python interpreter needed for jevents generation. Install python or build with NO_JEVENTS=1.. Stop. Signed-off-by: Rasmus Villemoes --- v2: add comment on (non-)relationship to 'python' PC item. meta/recipes-kernel/perf/perf.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb index a392166e73..5672f7d338 100644 --- a/meta/recipes-kernel/perf/perf.bb +++ b/meta/recipes-kernel/perf/perf.bb @@ -28,6 +28,9 @@ PACKAGECONFIG[audit] = ",NO_LIBAUDIT=1,audit" PACKAGECONFIG[manpages] = ",,xmlto-native asciidoc-native" PACKAGECONFIG[cap] = ",,libcap" PACKAGECONFIG[libtraceevent] = ",NO_LIBTRACEEVENT=1,libtraceevent" +# jevents requires host python for generating a .c file, but is +# unrelated to the python item. +PACKAGECONFIG[jevents] = ",NO_JEVENTS=1,python3-native" # Arm CoreSight PACKAGECONFIG[coresight] = "CORESIGHT=1,,opencsd" PACKAGECONFIG[pfm4] = ",NO_LIBPFM4=1,libpfm4" -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190260): https://lists.openembedded.org/g/openembedded-core/message/190260 Mute This Topic: https://lists.openembedded.org/mt/102439731/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: add jevents PACKAGECONFIG item
On 06/11/2023 16.07, Richard Purdie wrote: > On Mon, 2023-11-06 at 16:00 +0100, Rasmus Villemoes wrote: >> >> When I set PACKAGECONFIG for perf in a perf.bbappend file, without >> python being in there, the build breaks: >> [...] >> This difference is, to put it mildly, quite confusing and surprising >> (and took quite some time to track down). I don't know if there's >> anything to do about it. Is setting PACKAGECONFIG via :pn-foo overrides >> the only supported method, or is this just a quirk of the perf recipe >> that one has to live with? > > That is a pretty horrible interaction of things. As you say, that > inherit is effectively an immediate expansion of the data and wouldn't > "see" changes in any bbappend. > > Could you file a bug please? Done. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15276 I now see that the same problem probably exists for the perl item (just the other way around because perl is not initially in PACKAGECONFIG); the line include ${@bb.utils.contains('PACKAGECONFIG', 'perl', 'perf-perl.inc', '', d)} won't have effect if perl is only set in PACKAGECONFIG in a .bbappend. > I'm wondering if we should have some other form of inherit which > applies at the end of parsing which would avoid that issue. Sometimes > order is important, sometimes it is less important with the inherits. Dunno. Maybe a new bitbake keyword "defer" (or "late" or whatever): defer inherit ${@something(PACKAGECONFIG)} that would parse the rest of the line as usual, but instead of adding directly to the statements list, instead to some new deferred_statements (which StatementGroup could track itself, it would probably just need to be taught .append_deferred and .eval_deferred methods and have a .deferred member). I don't really know much about bitbake internals, it doesn't sound too hard to do, but I'm probably missing something. Also, perhaps it's too subtle. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190258): https://lists.openembedded.org/g/openembedded-core/message/190258 Mute This Topic: https://lists.openembedded.org/mt/102420698/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: add jevents PACKAGECONFIG item
On 06/11/2023 15.49, Bruce Ashfield wrote: > On Mon, Nov 6, 2023 at 9:22 AM Rasmus Villemoes > wrote: >> >> From: Rasmus Villemoes >> >> Building for an arm64 target, e.g. qemuarm64 or a raspberrypi3, >> without "python" in PACKAGECONFIG, results in >> >> | Makefile.config:892: *** ERROR: No python interpreter needed for jevents >> generation. Install python or build with NO_JEVENTS=1.. Stop. >> >> Signed-off-by: Rasmus Villemoes >> --- >> meta/recipes-kernel/perf/perf.bb | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/meta/recipes-kernel/perf/perf.bb >> b/meta/recipes-kernel/perf/perf.bb >> index a392166e73..37b53ee7df 100644 >> --- a/meta/recipes-kernel/perf/perf.bb >> +++ b/meta/recipes-kernel/perf/perf.bb >> @@ -28,6 +28,7 @@ PACKAGECONFIG[audit] = ",NO_LIBAUDIT=1,audit" >> PACKAGECONFIG[manpages] = ",,xmlto-native asciidoc-native" >> PACKAGECONFIG[cap] = ",,libcap" >> PACKAGECONFIG[libtraceevent] = ",NO_LIBTRACEEVENT=1,libtraceevent" >> +PACKAGECONFIG[jevents] = ",NO_JEVENTS=1,python3-native" > > Since we already have a python packageconfig and it takes care of > the dependencies that we need, we shouldn't be repeating the python3-native > in this one. We'll end up having to maintain that dependency in > two locations. > > i.e. we don't have python3-native as a dependency of our main python > packageconfig .. is that an error (the python packageconfig) or should > this be depending on python3 python3-setuptools-native, just like the > python one ? The two are as I understand it entirely unrelated. The existing python knob controls whether some stuff that requires python at runtime is built/installed (and that build may of course itself require some native python, in this case in the guise of setuptools-native). The jevents stuff is purely about generating some tables [as a .c file] at build time, and that requires native python, but does not impose a dependency on python on target. > Since there's no way to make packageconfig 'a' depend on > packageconfig 'b' that I know of (RP will correct me if I'm missing > or forgetting something here), we could always add a test on the > final PACKAGECONFIG and error/warn if jevents was enabled and > python wasn't. I don't think it's correct to force jevents pc to include python pc, as the latter will make the perf binary pull in the whole python interpreter etc. If feature A depends on foo-native and foo and bar, and feature B requires foo-native, I don't see why one should be forced to enable feature A in order to get B. [Conversely, even if the dependencies of feature A today happen to be a superset of those of feature B, I don't think selecting feature A should automatically imply feature B.] > It is always a bit annoying that perf turns these on, and we have > to explicitly opt-out .. but that is out of our control :) Indeed. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190227): https://lists.openembedded.org/g/openembedded-core/message/190227 Mute This Topic: https://lists.openembedded.org/mt/102420698/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: add jevents PACKAGECONFIG item
Somewhat related, but maybe not really: When I set PACKAGECONFIG for perf in a perf.bbappend file, without python being in there, the build breaks: | Makefile.config:277: *** .../tmp/work/qemuarm64-poky-linux/perf/1.0/recipe-sysroot-native/usr/bin/python3-native/python3-config not found. Stop. | make[1]: *** [Makefile.perf:242: sub-make] Error 2 If I instead set the same PACKAGECONFIG value, but by doing PACKAGECONFIG:pn-perf = "tui libunwind libtraceevent" in some global configuration file, it works as expected. I think the difference is due to the inherit ${@bb.utils.contains('PACKAGECONFIG', 'python', 'python3targetconfig', '', d)} line in the perf recipe. At the time where this line is evaluated, if PACKAGECONFIG is set via a .bbappend, this ends up evaluated using the ??= value which does include python, and hence we end up inheriting python3targetconfig, which in turn inherits python3native, which does export PYTHON And since PYTHON is now exported, perf's logic ends up in this fragment in tools/perf/Makefile.config: # If PYTHON is defined but PYTHON_CONFIG isn't, then take $(PYTHON)-config as if it was the user # supplied value for PYTHON_CONFIG. Because it's "user supplied", error out if it doesn't exist. ifdef PYTHON ifndef PYTHON_CONFIG PYTHON_CONFIG_AUTO := $(call get-executable,$(PYTHON)-config) PYTHON_CONFIG := $(if $(PYTHON_CONFIG_AUTO),$(PYTHON_CONFIG_AUTO),\ $(call $(error $(PYTHON)-config not found))) endif endif which breaks because the value of PACKAGECONFIG that ended up determining buildtime dependencies did not pull in that python3-config. This difference is, to put it mildly, quite confusing and surprising (and took quite some time to track down). I don't know if there's anything to do about it. Is setting PACKAGECONFIG via :pn-foo overrides the only supported method, or is this just a quirk of the perf recipe that one has to live with? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190222): https://lists.openembedded.org/g/openembedded-core/message/190222 Mute This Topic: https://lists.openembedded.org/mt/102420698/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] perf: add jevents PACKAGECONFIG item
From: Rasmus Villemoes Building for an arm64 target, e.g. qemuarm64 or a raspberrypi3, without "python" in PACKAGECONFIG, results in | Makefile.config:892: *** ERROR: No python interpreter needed for jevents generation. Install python or build with NO_JEVENTS=1.. Stop. Signed-off-by: Rasmus Villemoes --- meta/recipes-kernel/perf/perf.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb index a392166e73..37b53ee7df 100644 --- a/meta/recipes-kernel/perf/perf.bb +++ b/meta/recipes-kernel/perf/perf.bb @@ -28,6 +28,7 @@ PACKAGECONFIG[audit] = ",NO_LIBAUDIT=1,audit" PACKAGECONFIG[manpages] = ",,xmlto-native asciidoc-native" PACKAGECONFIG[cap] = ",,libcap" PACKAGECONFIG[libtraceevent] = ",NO_LIBTRACEEVENT=1,libtraceevent" +PACKAGECONFIG[jevents] = ",NO_JEVENTS=1,python3-native" # Arm CoreSight PACKAGECONFIG[coresight] = "CORESIGHT=1,,opencsd" PACKAGECONFIG[pfm4] = ",NO_LIBPFM4=1,libpfm4" -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190219): https://lists.openembedded.org/g/openembedded-core/message/190219 Mute This Topic: https://lists.openembedded.org/mt/102420698/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] valgrind: split helper scripts to separate packages, update dependencies
From: Rasmus Villemoes The cachegrind scripts have been rewritten in python3, so the RDEPENDS on perl is no longer sufficient. This is unfortunately not caught by QA checks since the scripts use #! /usr/bin/env python3 as shebang line. Since the valgrind binary by itself can be quite useful on target, allow including that in the rootfs without also having to pull in the whole perl and python interpreters and stdlibs. Signed-off-by: Rasmus Villemoes --- meta/recipes-devtools/valgrind/valgrind_3.21.0.bb | 14 -- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/meta/recipes-devtools/valgrind/valgrind_3.21.0.bb b/meta/recipes-devtools/valgrind/valgrind_3.21.0.bb index 717d82830e..68ab319a1b 100644 --- a/meta/recipes-devtools/valgrind/valgrind_3.21.0.bb +++ b/meta/recipes-devtools/valgrind/valgrind_3.21.0.bb @@ -92,6 +92,18 @@ CACHED_CONFIGUREVARS += "ac_cv_path_PERL='/usr/bin/env perl'" # which fixes build path issue in DWARF. SELECTED_OPTIMIZATION = "${DEBUG_FLAGS}" +# Split out various helper scripts to separate packages to avoid the +# main package depending on perl and python. +PACKAGES =+ "${PN}-cachegrind ${PN}-massif ${PN}-callgrind" + +FILES:${PN}-cachegrind = "${bindir}/cg_*" +FILES:${PN}-massif = "${bindir}/ms_*" +FILES:${PN}-callgrind = "${bindir}/callgrind_*" + +RDEPENDS:${PN}-cachegrind = "${PN} python3-core" +RDEPENDS:${PN}-massif = "${PN} perl" +RDEPENDS:${PN}-callgrind = "${PN} perl" + do_configure:prepend () { rm -rf ${S}/config.h sed -i -e 's:$(abs_top_builddir):$(pkglibdir)/ptest:g' ${S}/none/tests/Makefile.am @@ -116,8 +128,6 @@ VALGRINDARCH:powerpc64le = "ppc64le" INHIBIT_PACKAGE_STRIP_FILES = "${PKGD}${libexecdir}/valgrind/vgpreload_memcheck-${VALGRINDARCH}-linux.so" -RDEPENDS:${PN} += "perl" - # valgrind needs debug information for ld.so at runtime in order to # redirect functions like strlen. RRECOMMENDS:${PN} += "${TCLIBC}-dbg" -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190163): https://lists.openembedded.org/g/openembedded-core/message/190163 Mute This Topic: https://lists.openembedded.org/mt/102365327/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] perf: lift TARGET_CC_ARCH modification out of security_flags.inc
From: Rasmus Villemoes Building perf without security_flags.inc being included in one's distro results in the buildpaths warning WARNING: perf-1.0-r9 do_package_qa: QA Issue: File /usr/bin/trace in package perf contains reference to TMPDIR because the ${DEBUG_PREFIX_MAP} does not get used. Most recipes get that from CFLAGS, but the perf recipe explicitly unsets that. Now ${SELECTED_OPTIMIZATION} of course contains more than just ${DEBUG_FLAGS}/${DEBUG_PREFIX_MAP}. For most TUs, perf's build system adds its own optimization flags (-O6 for odd reasons), so for those including the -O2 or -Og doesn't change anything. But looking at the .o.cmd files show that there are some TUs which currently get built without any -O flag. So for those adding the distro's SELECTED_OPTIMIZATION seem to be the right thing to do. Signed-off-by: Rasmus Villemoes --- v2: add comment meta/conf/distro/include/security_flags.inc | 1 - meta/recipes-kernel/perf/perf.bb| 9 + 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/meta/conf/distro/include/security_flags.inc b/meta/conf/distro/include/security_flags.inc index 2972f05b4e..d97a6edb0f 100644 --- a/meta/conf/distro/include/security_flags.inc +++ b/meta/conf/distro/include/security_flags.inc @@ -69,4 +69,3 @@ SECURITY_LDFLAGS:pn-xserver-xorg = "${SECURITY_X_LDFLAGS}" TARGET_CC_ARCH:append:pn-binutils = " ${SELECTED_OPTIMIZATION}" TARGET_CC_ARCH:append:pn-gcc = " ${SELECTED_OPTIMIZATION}" TARGET_CC_ARCH:append:pn-gdb = " ${SELECTED_OPTIMIZATION}" -TARGET_CC_ARCH:append:pn-perf = " ${SELECTED_OPTIMIZATION}" diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb index 675acfaf26..a392166e73 100644 --- a/meta/recipes-kernel/perf/perf.bb +++ b/meta/recipes-kernel/perf/perf.bb @@ -73,6 +73,15 @@ SPDX_S = "${S}/tools/perf" # supported kernel. LDFLAGS="-ldl -lutil" +# Perf's build system adds its own optimization flags for most TUs, +# overriding the flags included here. But for some, perf does not add +# any -O option, so ensure the distro's chosen optimization gets used +# for those. Since ${SELECTED_OPTIMIZATION} always includes +# ${DEBUG_FLAGS} which in turn includes ${DEBUG_PREFIX_MAP}, this also +# ensures perf is built with appropriate -f*-prefix-map options, +# avoiding the 'buildpaths' QA warning. +TARGET_CC_ARCH += "${SELECTED_OPTIMIZATION}" + EXTRA_OEMAKE = '\ V=1 \ VF=1 \ -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189615): https://lists.openembedded.org/g/openembedded-core/message/189615 Mute This Topic: https://lists.openembedded.org/mt/102130591/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: lift TARGET_CC_ARCH modification out of security_flags.inc
On 20/10/2023 12.13, Richard Purdie wrote: > On Fri, 2023-10-20 at 12:03 +0200, Rasmus Villemoes wrote: >> On 20/10/2023 11.38, Richard Purdie wrote: >>> On Fri, 2023-10-20 at 10:10 +0200, Rasmus Villemoes wrote: >>>> On 19/10/2023 14.48, Richard Purdie wrote: >> >>>>> The fact this works suggests perf is ignoring TARGET_CFLAGS. Is there >>>>> anything in the perf build system where we should be passing in cflags >>>>> we really want used? >>>> >>>> Well, IIUC, the normal way TARGET_CFLAGS would take effect is via the >>>> 'export CFLAGS = "${TARGET_CFLAGS}'. But in the perf recipe, both >>>> do_compile and do_install start with >>>> >>>> # Linux kernel build system is expected to do the right thing >>>> unset CFLAGS >>>> >>>> And perf's build system does indeed add lots of flags of its own, at >>>> least for most TUs, but nowhere is any -f(macro/debug/file)-prefix being >>>> set. >>>> >>>> So I do think that TARGET_CC_ARCH is the best place for flags that we >>>> really want used. At least as an initial step, because this is known to >>>> work when using the security_flags.inc file. Maybe there's some cleaner >>>> way where we don't unset CFLAGS, but that could be a giant can of worms. >>>> >>>> And yes, a comment should be added. Is something like >>>> >>>> # Perf's build system adds its own optimization flags for most TUs, >>>> # overriding the flags included here. But for some, perf does not add >>>> # any -O option, so ensure the distro's chosen optimization gets used >>>> # for those. Since ${SELECTED_OPTIMIZATION} always includes >>>> # ${DEBUG_FLAGS} which in turn includes ${DEBUG_PREFIX_MAP}, this also >>>> # ensures perf is built with appropriate -f*-prefix-map options, >>>> # avoiding the 'buildpaths' QA warning. >>>> >>>> too verbose? >>> >>> If it were me I'd probably write: >>> >>> # The perf makefile has "unset CFLAGS" as "Linux kernel build system is >>> # expected to do the right thing". >> >> It's not the perf makefile, it's right there in our recipe itself. I >> don't think upstream perf is to blame. > > Sorry, I misunderstood. I thought this was the perf environment that > was doing this so in that case your comment does make much more sense > than mine. OK. Bruce, should I resend? > Out of interest did you see what happens if you allow our CFLAGS in and > remove that unset? I don't think I've tried that, but I've tried a few other things that I thought would be cleaner but which then exploded, so in the end I decided that just lifting exactly what the poky distro does should be safest and have the highest likelyhood of being accepted. > I also wondered if EXTRA_CFLAGS in EXTRA_OEMAKE > might be an option? Based on quick reading of perf build documentation, yes, EXTRA_CFLAGS seems like a reasonable approach. What I'm a little worried about is the TUs where perf apparently does not by itself add all its cflags (-O6 and whatnot), so I'm not sure if they would be built with what is passed via EXTRA_CFLAGS. I will experiment sometime next week if I remember and nothing else preempts that plan. > I'm a bit worried that the current situation all feels a little > fragile. Agree. But the QA checks do help quite a lot here. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189518): https://lists.openembedded.org/g/openembedded-core/message/189518 Mute This Topic: https://lists.openembedded.org/mt/102058904/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: lift TARGET_CC_ARCH modification out of security_flags.inc
On 20/10/2023 11.38, Richard Purdie wrote: > On Fri, 2023-10-20 at 10:10 +0200, Rasmus Villemoes wrote: >> On 19/10/2023 14.48, Richard Purdie wrote: >>> The fact this works suggests perf is ignoring TARGET_CFLAGS. Is there >>> anything in the perf build system where we should be passing in cflags >>> we really want used? >> >> Well, IIUC, the normal way TARGET_CFLAGS would take effect is via the >> 'export CFLAGS = "${TARGET_CFLAGS}'. But in the perf recipe, both >> do_compile and do_install start with >> >> # Linux kernel build system is expected to do the right thing >> unset CFLAGS >> >> And perf's build system does indeed add lots of flags of its own, at >> least for most TUs, but nowhere is any -f(macro/debug/file)-prefix being >> set. >> >> So I do think that TARGET_CC_ARCH is the best place for flags that we >> really want used. At least as an initial step, because this is known to >> work when using the security_flags.inc file. Maybe there's some cleaner >> way where we don't unset CFLAGS, but that could be a giant can of worms. >> >> And yes, a comment should be added. Is something like >> >> # Perf's build system adds its own optimization flags for most TUs, >> # overriding the flags included here. But for some, perf does not add >> # any -O option, so ensure the distro's chosen optimization gets used >> # for those. Since ${SELECTED_OPTIMIZATION} always includes >> # ${DEBUG_FLAGS} which in turn includes ${DEBUG_PREFIX_MAP}, this also >> # ensures perf is built with appropriate -f*-prefix-map options, >> # avoiding the 'buildpaths' QA warning. >> >> too verbose? > > If it were me I'd probably write: > > # The perf makefile has "unset CFLAGS" as "Linux kernel build system is > # expected to do the right thing". It's not the perf makefile, it's right there in our recipe itself. I don't think upstream perf is to blame. > It doesn't and we need our prefix > # map options Yes. > and security flags amongst other things so force our > # cflags in. This has nothing to do with the security flags. That file still contains TARGET_CC_ARCH:append:class-target = " ${SECURITY_CFLAGS}" and TARGET_CC_ARCH is included in the CC value we pass to perf's make, with or without this patch. So if the distro includes security_flags.inc, perf gets built with those flags as expected. But when that file is _not_ included, we do miss the prefix-map flags, which should be passed regardless of any security distro settings. I think the line I'm moving out of security_flags.inc have never really had any business being there in the first place, regardless of whether adding ${SELECTED_OPTIMIZATION} to TARGET_CC_ARCH is the most appropriate method to get those prefix-map options passed. But since that's how the poky distro has achieved that until now, it seems to be the least invasive and least risky fix for distros that don't use security_flags.inc. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189513): https://lists.openembedded.org/g/openembedded-core/message/189513 Mute This Topic: https://lists.openembedded.org/mt/102058904/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] perf: lift TARGET_CC_ARCH modification out of security_flags.inc
On 19/10/2023 14.48, Richard Purdie wrote: > On Thu, 2023-10-19 at 14:32 +0200, Rasmus Villemoes via > lists.openembedded.org wrote: >> From: Rasmus Villemoes >> >> Building perf without security_flags.inc being included in one's >> distro results in the buildpaths warning >> >> WARNING: perf-1.0-r9 do_package_qa: QA Issue: File /usr/bin/trace in >> package perf contains reference to TMPDIR >> >> because the ${DEBUG_PREFIX_MAP} does not get used. Most recipes get >> that from CFLAGS, but the perf recipe explicitly unsets that. >> >> Now ${SELECTED_OPTIMIZATION} of course contains more than just >> ${DEBUG_FLAGS}/${DEBUG_PREFIX_MAP}. For most TUs, perf's build system >> adds its own optimization flags (-O6 for odd reasons), so for those >> including the -O2 or -Og doesn't change anything. But looking at the >> .o.cmd files show that there are some TUs which currently get built >> without any -O flag. So for those adding the distro's >> SELECTED_OPTIMIZATION seem to be the right thing to do. >> >> Signed-off-by: Rasmus Villemoes >> --- >> meta/conf/distro/include/security_flags.inc | 1 - >> meta/recipes-kernel/perf/perf.bb| 1 + >> 2 files changed, 1 insertion(+), 1 deletion(-) > > The fact this works suggests perf is ignoring TARGET_CFLAGS. Is there > anything in the perf build system where we should be passing in cflags > we really want used? Well, IIUC, the normal way TARGET_CFLAGS would take effect is via the 'export CFLAGS = "${TARGET_CFLAGS}'. But in the perf recipe, both do_compile and do_install start with # Linux kernel build system is expected to do the right thing unset CFLAGS And perf's build system does indeed add lots of flags of its own, at least for most TUs, but nowhere is any -f(macro/debug/file)-prefix being set. So I do think that TARGET_CC_ARCH is the best place for flags that we really want used. At least as an initial step, because this is known to work when using the security_flags.inc file. Maybe there's some cleaner way where we don't unset CFLAGS, but that could be a giant can of worms. And yes, a comment should be added. Is something like # Perf's build system adds its own optimization flags for most TUs, # overriding the flags included here. But for some, perf does not add # any -O option, so ensure the distro's chosen optimization gets used # for those. Since ${SELECTED_OPTIMIZATION} always includes # ${DEBUG_FLAGS} which in turn includes ${DEBUG_PREFIX_MAP}, this also # ensures perf is built with appropriate -f*-prefix-map options, # avoiding the 'buildpaths' QA warning. too verbose? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189508): https://lists.openembedded.org/g/openembedded-core/message/189508 Mute This Topic: https://lists.openembedded.org/mt/102058904/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] perf: lift TARGET_CC_ARCH modification out of security_flags.inc
From: Rasmus Villemoes Building perf without security_flags.inc being included in one's distro results in the buildpaths warning WARNING: perf-1.0-r9 do_package_qa: QA Issue: File /usr/bin/trace in package perf contains reference to TMPDIR because the ${DEBUG_PREFIX_MAP} does not get used. Most recipes get that from CFLAGS, but the perf recipe explicitly unsets that. Now ${SELECTED_OPTIMIZATION} of course contains more than just ${DEBUG_FLAGS}/${DEBUG_PREFIX_MAP}. For most TUs, perf's build system adds its own optimization flags (-O6 for odd reasons), so for those including the -O2 or -Og doesn't change anything. But looking at the .o.cmd files show that there are some TUs which currently get built without any -O flag. So for those adding the distro's SELECTED_OPTIMIZATION seem to be the right thing to do. Signed-off-by: Rasmus Villemoes --- meta/conf/distro/include/security_flags.inc | 1 - meta/recipes-kernel/perf/perf.bb| 1 + 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/conf/distro/include/security_flags.inc b/meta/conf/distro/include/security_flags.inc index 2972f05b4e..d97a6edb0f 100644 --- a/meta/conf/distro/include/security_flags.inc +++ b/meta/conf/distro/include/security_flags.inc @@ -69,4 +69,3 @@ SECURITY_LDFLAGS:pn-xserver-xorg = "${SECURITY_X_LDFLAGS}" TARGET_CC_ARCH:append:pn-binutils = " ${SELECTED_OPTIMIZATION}" TARGET_CC_ARCH:append:pn-gcc = " ${SELECTED_OPTIMIZATION}" TARGET_CC_ARCH:append:pn-gdb = " ${SELECTED_OPTIMIZATION}" -TARGET_CC_ARCH:append:pn-perf = " ${SELECTED_OPTIMIZATION}" diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb index 675acfaf26..a6110dedc9 100644 --- a/meta/recipes-kernel/perf/perf.bb +++ b/meta/recipes-kernel/perf/perf.bb @@ -72,6 +72,7 @@ SPDX_S = "${S}/tools/perf" # symbols and this is preferred than requiring patches to every old # supported kernel. LDFLAGS="-ldl -lutil" +TARGET_CC_ARCH += "${SELECTED_OPTIMIZATION}" EXTRA_OEMAKE = '\ V=1 \ -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189451): https://lists.openembedded.org/g/openembedded-core/message/189451 Mute This Topic: https://lists.openembedded.org/mt/102058904/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 04/15] perf: fix buildpaths QA warning in 6.4+
On 10/10/2023 14.31, Bruce Ashfield wrote: > > > On Tue, Oct 10, 2023 at 7:44 AM Rasmus Villemoes > It seems we can fix it with > > $ git diff > diff --git a/meta/recipes-kernel/perf/perf.bb <http://perf.bb> > b/meta/recipes-kernel/perf/perf.bb <http://perf.bb> > index 420286e..59c0c10 100644 > --- a/meta/recipes-kernel/perf/perf.bb <http://perf.bb> > +++ b/meta/recipes-kernel/perf/perf.bb <http://perf.bb> > @@ -81,7 +81,7 @@ EXTRA_OEMAKE = '\ > LDSHARED="${CC} -shared" \ > AR="${AR}" \ > LD="${LD}" \ > - EXTRA_CFLAGS="-ldw -I${S}" \ > + EXTRA_CFLAGS="-ldw -I${S} ${DEBUG_PREFIX_MAP}" \ > YFLAGS='-y > --file-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}' > \ > EXTRA_LDFLAGS="${PERF_EXTRA_LDFLAGS}" \ > perfexecdir=${libexecdir} \ > > but I'm quite curious what the difference between our setups is, since > apparently the only problem oe-core has/had was that new file introduced > in v6.4. > > > You can see any patches that we have against perf in the kernel-cache > repository (https://git.yoctoproject.org/yocto-kernel-cache/ > <https://git.yoctoproject.org/yocto-kernel-cache/>). That being > said, we unfortunately cannot really patch perf, since the recipe is > supported against more than our reference kernels. I didn't suggest to patch perf, I suggested to patch the perf recipe. > > Other than that, there's no special setup (in particular no setup that > I have) as it is the autobuilder infrastructure that typically finds and > reports these issues. So I found the difference, and those autobuilder builds do in fact make the compiler use all the right -f*-prefix-map options. It's because meta/conf/distro/include/security_flags.inc contains TARGET_CC_ARCH:append:pn-perf = " ${SELECTED_OPTIMIZATION}" and that includes the DEBUG_PREFIX_MAP setting, and in turn gets included in the CC setting via HOST_CC_ARCH. And of course the poky distro includes that security_flags.inc file, while our custom distro does not (up until five minutes ago, I never knew that file existed). Of course one wonders why those :append:pn-* settings that don't actually append any security related flags, but do affect stuff like the recipes passing QA, are hidden away in a file called security_flags.inc instead of those settings just being directly in the recipes themselves. There's no explanation in oe-core e93765ffb57 ; it removed a whole bunch of append_pn-* stuff which had apparently become redundant (the commit log hints as much), but doesn't offer any explanation as to why pn-perf should grow that setting. And AFAICT that long predates the invention of the -f*-prefix-map flags, so it seems to be mostly by accident that perf, with the poky distro, actually does build with those flags. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188910): https://lists.openembedded.org/g/openembedded-core/message/188910 Mute This Topic: https://lists.openembedded.org/mt/100051819/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 04/15] perf: fix buildpaths QA warning in 6.4+
On 10/07/2023 05.20, Bruce Ashfield via lists.openembedded.org wrote: > From: Bruce Ashfield > > kernel version 6.4 introduces a new file that need to have > absolute paths removed, so we can avoid the buildpaths QA > warning and have relocatable packages. > > We add pmu-flex.h to the processing, and the issue is resolved. Hi Bruce We've seen those buildpaths qa warning when building perf for a long time, and they happen whether we build against a 5.15, 6.2 or 6.5 kernel. More specifically, what we see is WARNING: perf-1.0-r9 do_package_qa: QA Issue: File /usr/bin/.debug/perf in package perf-dbg contains reference to TMPDIR [buildpaths] WARNING: perf-1.0-r9 do_package_qa: QA Issue: File /usr/bin/trace in package perf contains reference to TMPDIR File /usr/bin/perf in package perf contains reference to TMPDIR [buildpaths] Looking at the .o.cmd files, there is indeed nowhere a -fmacro-prefix-map or -fdebug-prefix-map, and apart from the bison/yacc stuff, I also don't see where the perf recipe attempts to pass such flags. It seems we can fix it with $ git diff diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb index 420286e..59c0c10 100644 --- a/meta/recipes-kernel/perf/perf.bb +++ b/meta/recipes-kernel/perf/perf.bb @@ -81,7 +81,7 @@ EXTRA_OEMAKE = '\ LDSHARED="${CC} -shared" \ AR="${AR}" \ LD="${LD}" \ -EXTRA_CFLAGS="-ldw -I${S}" \ +EXTRA_CFLAGS="-ldw -I${S} ${DEBUG_PREFIX_MAP}" \ YFLAGS='-y --file-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}' \ EXTRA_LDFLAGS="${PERF_EXTRA_LDFLAGS}" \ perfexecdir=${libexecdir} \ but I'm quite curious what the difference between our setups is, since apparently the only problem oe-core has/had was that new file introduced in v6.4. FWIW, what we build are vanilla -stable kernels, i.e. "6.2" means v6.2.16 as tagged by Greg. The only modification of the perf recipe we do is removing "scripting" from PACKAGECONFIG. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11): https://lists.openembedded.org/g/openembedded-core/message/11 Mute This Topic: https://lists.openembedded.org/mt/100051819/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] openssh: update sshd_check_keys script to make use of 'sshd -G'
From: Rasmus Villemoes Parsing sshd's config file with 'sed' does not work in for example the case where somebody has made use of the new ability to add a config fragment in /etc/ssh/sshd_config.d/ with one or more HostKey stanzas. Also, sshd_config keywords are case-insensitive, but the current sed pattern only matches the CamelCase spelling of HostKey. In openssh 9.3, sshd learnt a new command line flag '-G', which causes sshd to parse the given configuration file and print the resulting effective configuration on stdout. So use that instead. Furthermore, since that "effective configuration" includes the default set of host keys if the configuration file has no HostKey stanzas, we also avoid the script needing to know what sshd's default is - that could plausibly change with some future release. Signed-off-by: Rasmus Villemoes --- meta/recipes-connectivity/openssh/openssh/sshd_check_keys | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_check_keys b/meta/recipes-connectivity/openssh/openssh/sshd_check_keys index ef117de897..606d1894b5 100644 --- a/meta/recipes-connectivity/openssh/openssh/sshd_check_keys +++ b/meta/recipes-connectivity/openssh/openssh/sshd_check_keys @@ -57,8 +57,7 @@ while true ; do esac done -HOST_KEYS=$(sed -n 's/^[ \t]*HostKey[ \t]\+\(.*\)/\1/p' "${sshd_config}") -[ -z "${HOST_KEYS}" ] && HOST_KEYS="$SYSCONFDIR/ssh_host_rsa_key $SYSCONFDIR/ssh_host_ecdsa_key $SYSCONFDIR/ssh_host_ed25519_key" +HOST_KEYS=$(sshd -G -f "${sshd_config}" | grep -i '^hostkey ' | cut -f2 -d' ') for key in ${HOST_KEYS} ; do [ -f $key ] && continue -- 2.40.1.1.g1c60b9335d -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188396): https://lists.openembedded.org/g/openembedded-core/message/188396 Mute This Topic: https://lists.openembedded.org/mt/101653899/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Dilemma on changes - merge or not to merge (e.g. 6.4)
On 15/08/2023 15.08, Paul Gortmaker via lists.openembedded.org wrote: > [Dilemma on changes - merge or not to merge (e.g. 6.4)] On 14/08/2023 (Mon > 10:54) Richard Purdie wrote: > >> Remaining are: >> * an error upon boot on preempt-rt on qemux86-64 >> (e.g. >> https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/7616/steps/36/logs/stdio) >> We'll probably just have to ignore it in parselogs as it has been?? >> around for a while and nobody seems interested in fixing it upstream. > > Just back from vacation and I see an internal report of 10-ish at boot it seems to be rate-limited to 10 per boot, so it should never appear more than those 10ish times: static bool report_idle_softirq(void) { ... if (ratelimit >= 10) return false; ... ratelimit++; ... } (it's all non-atomic/lockfree, so ofc it could just happen to get emitted 11 or 12 times if the stars align just right...) > > NOHZ tick-stop error: local softirq work is pending, handler #80!!! > > ..on the 6.1.43-rt10-yocto-preempt-rt kernel, on real hardware. So it > seems we can't blame that one entirely on v6.4 kernel (or qemu). > > We used to get (late 3.x and 4.x era) pretty common "NOHZ: local softirq > pending" messages even on common/popular distro kernels. But I haven't > seen those for a long time and they didn't scream "error" or have the > alarmist three exclamation marks either. FWIW, we're also seeing exactly that "NOHZ tick-stop error" message on 6.4.6-rt8 running on a couple of different imx8mp based boards. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186117): https://lists.openembedded.org/g/openembedded-core/message/186117 Mute This Topic: https://lists.openembedded.org/mt/100733646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC] [PATCH] Allow fitimage + initramfs rebuild to be accelerated
On 10/11/2022 21.32, Paul Eggleton via lists.openembedded.org wrote: > Hi Richard > > On Thursday, 10 November 2022 11:14:24 NZDT Richard Purdie wrote: >> Adding in yet further if/else paths with magic variables to control >> them isn't filling me with confidence. > > I understand that. I was hoping to figure out a way to avoid adding > significant > extra complexity. > FWIW, we've long since given up on using kernel-fitimage.bbclass, it is way to inflexible (e.g. there's no way to build two FIT images with the same kernel but different initramfs). It is much simpler to just have a separate small kernel-fit.bb recipe which pulls vmlinux or Image or whatnot, plus the .dtbs, from DEPLOY_DIR_IMAGE along with the initramfs images, provide our own .its file(s), and build the FIT image(s) there. That automatically avoids rebuilding the kernel when only something in initramfs has changed. Building the kernel and packaging the binary into some container format really doesn't need to be done in the same recipe. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#173128): https://lists.openembedded.org/g/openembedded-core/message/173128 Mute This Topic: https://lists.openembedded.org/mt/94747626/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] bitbake.conf: set BB_DEFAULT_UMASK using ??=
Currently, there's no way for the user's site.conf, local.conf or similar to set BB_DEFAULT_UMASK, because those files are included by bitbake.conf prior to the unconditional assignment of BB_DEFAULT_UMASK. To make that possible, use a weak default assignment instead. This is also consistent with most other variable assignments in the lower half of bitbake.conf. I believe the risk of a regression is very small; it would require something like somebody having a definition of BB_DEFAULT_UMASK in a local configuration file, and having been relying on that _not_ taking effect. Signed-off-by: Rasmus Villemoes --- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index dd2df8a552..52a36d788b 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -928,7 +928,7 @@ SHELL[unexport] = "1" TRANSLATED_TARGET_ARCH ??= "${@d.getVar('TARGET_ARCH').replace("_", "-")}" # Set a default umask to use for tasks for determinism -BB_DEFAULT_UMASK = "022" +BB_DEFAULT_UMASK ??= "022" # Complete output from bitbake BB_CONSOLELOG ?= "${LOG_DIR}/cooker/${MACHINE}/${DATETIME}.log" -- 2.37.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169915): https://lists.openembedded.org/g/openembedded-core/message/169915 Mute This Topic: https://lists.openembedded.org/mt/93268429/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Should BB_DEFAULT_UMASK be set with ?=
On 30/05/2022 11.19, Rasmus Villemoes via lists.openembedded.org wrote: > Hi, > > Suppose I want to define BB_DEFAULT_UMASK in site.conf or local.conf (I > do have a use case for setting it to 002). But bitbake.conf includes > those files before it then goes on to use an unconditional assignment > > BB_DEFAULT_UMASK = "022" > > So how/where is one supposed to be able to set BB_DEFAULT_UMASK? Should > the assignment in bitbake.conf be ?= or perhaps ??= instead? Or should > the assigment instead be moved above the inclusion of the user's > configuration files? ping (and cc += RP) -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169815): https://lists.openembedded.org/g/openembedded-core/message/169815 Mute This Topic: https://lists.openembedded.org/mt/93226320/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] e2fsprogs: add alternatives handling of lsattr as well
On 09/06/2022 09.20, Luca Ceresoli wrote: > Hi Rasmus, > > On Wed, 8 Jun 2022 15:12:05 +0200 > "Rasmus Villemoes via lists.openembedded.org" > wrote: > ^^^ > > As you can see above, your sender address is getting mangled. This is > not your fault, but it makes applying your patches annoying. > > Can you please try to work around that by setting the sendemail.from > parameter in your git config? Sorry about that, didn't know it. It's not completely clear if I should put just the email address or the full "Name " in that configuration item, i.e. whether it's [sendemail] from = rasmus.villem...@prevas.dk or [sendemail] from = Rasmus Villemoes For now I've gone with the latter, please let me know if that's correct. In any case we'll see if it works the next time I submit a patch. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166769): https://lists.openembedded.org/g/openembedded-core/message/166769 Mute This Topic: https://lists.openembedded.org/mt/91622646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] e2fsprogs: add alternatives handling of lsattr as well
Building busybox with CONFIG_LSATTR=y and installing that in the same filesystem as e2fsprogs breaks: ERROR: ... do_rootfs: Postinstall scriptlets of ['busybox'] have failed. If the intention is to defer them to first boot, then please place them into pkg_postinst_ontarget:${PN} (). Deferring to first boot via 'exit 1' is no longer supported. Fix that by also alternatifying lsattr just as chattr already is. Signed-off-by: Rasmus Villemoes --- meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb index ec48f419c7a..5b2d1921f08 100644 --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb @@ -51,6 +51,7 @@ do_install () { oe_multilib_header ext2fs/ext2_types.h install -d ${D}${base_bindir} mv ${D}${bindir}/chattr ${D}${base_bindir}/chattr.e2fsprogs + mv ${D}${bindir}/lsattr ${D}${base_bindir}/lsattr.e2fsprogs install -v -m 755 ${S}/contrib/populate-extfs.sh ${D}${base_sbindir}/ @@ -99,10 +100,12 @@ FILES:libe2p = "${base_libdir}/libe2p.so.*" FILES:libext2fs = "${libdir}/e2initrd_helper ${base_libdir}/libext2fs.so.*" FILES:${PN}-dev += "${datadir}/*/*.awk ${datadir}/*/*.sed ${base_libdir}/*.so ${bindir}/compile_et ${bindir}/mk_cmds" -ALTERNATIVE:${PN} = "chattr" +ALTERNATIVE:${PN} = "chattr lsattr" ALTERNATIVE_PRIORITY = "100" ALTERNATIVE_LINK_NAME[chattr] = "${base_bindir}/chattr" ALTERNATIVE_TARGET[chattr] = "${base_bindir}/chattr.e2fsprogs" +ALTERNATIVE_LINK_NAME[lsattr] = "${base_bindir}/lsattr" +ALTERNATIVE_TARGET[lsattr] = "${base_bindir}/lsattr.e2fsprogs" ALTERNATIVE:${PN}-doc = "fsck.8" ALTERNATIVE_LINK_NAME[fsck.8] = "${mandir}/man8/fsck.8" -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166707): https://lists.openembedded.org/g/openembedded-core/message/166707 Mute This Topic: https://lists.openembedded.org/mt/91622646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] vim: put xxd in its own package
On 08/06/2022 14.30, Paulo Neves wrote: > On 6/8/22 14:13, Rasmus Villemoes wrote: > >> On 08/06/2022 13.58, Paulo Neves via lists.openembedded.org wrote: > I mean that your patch allows for a new standalone package ${PN}-xxd. > Were somebody to use this package standalone, meaning without installing vim, > the ncurses-terminfo-base would not be installed, possibly breaking xxd. > > Before your patch this risk did not occur due to xxd being include in the > ${PN} > which had an RDEPENDS to ncurses-terminfo-base. Thanks, now I understand what you were saying. And no, xxd does not in any way depend on those terminfo files; it's a simple utility for converting binary files to hex (or the reverse, which is what I happen to need, taking input in the form of ascii hex chars and producing binary output). Yes, I've tested it on target as well. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166705): https://lists.openembedded.org/g/openembedded-core/message/166705 Mute This Topic: https://lists.openembedded.org/mt/91618936/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] vim: put xxd in its own package
On 08/06/2022 13.58, Paulo Neves via lists.openembedded.org wrote: > Forgive me if it is a stupid question, but does xxd not rdepend on > ncurses-terminfo-base as well? No, it does not, it's a trivial standalone utility whose only dynamic dependency is libc. > I ask because before your patch it could > was implied it that it did, s part of PN. I don't understand what you're trying to say. I just put the extra RDEPENDS from vim to vim-xxd into the existing RDEPENDS:vim line. That doesn't imply any dependency either way between the two items listed on the RHS. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166702): https://lists.openembedded.org/g/openembedded-core/message/166702 Mute This Topic: https://lists.openembedded.org/mt/91618936/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] vim: put xxd in its own package
The xxd tool can be quite handy by itself, and doesn't have anything to do with vim per se. Make it possible to include the rather tiny xxd in a rootfs without pulling in the several MB vim binary and associated data. For backwards compatibility, add an RDEPENDS from the main package to the new vim-xxd package. Signed-off-by: Rasmus Villemoes --- meta/recipes-support/vim/vim_8.2.bb | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/meta/recipes-support/vim/vim_8.2.bb b/meta/recipes-support/vim/vim_8.2.bb index f358e61132b..fee9f055e9a 100644 --- a/meta/recipes-support/vim/vim_8.2.bb +++ b/meta/recipes-support/vim/vim_8.2.bb @@ -2,14 +2,18 @@ require vim.inc PROVIDES = "xxd" -RDEPENDS:${PN} = "ncurses-terminfo-base" +RDEPENDS:${PN} = "ncurses-terminfo-base ${PN}-xxd" # Recommend that runtime data is installed along with vim RRECOMMENDS:${PN} = "${PN}-syntax ${PN}-help ${PN}-tutor ${PN}-vimrc ${PN}-common" PACKAGECONFIG:class-native = "" BBCLASSEXTEND = "native nativesdk" -ALTERNATIVE:${PN}:append = " xxd" +PACKAGES =+ "${PN}-xxd" +FILES:${PN}-xxd = "${bindir}/xxd" +RPROVIDES:${PN}-xxd = "xxd" + +ALTERNATIVE:${PN}-xxd = "xxd" ALTERNATIVE_TARGET[xxd] = "${bindir}/xxd" ALTERNATIVE_LINK_NAME[xxd] = "${bindir}/xxd" -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166693): https://lists.openembedded.org/g/openembedded-core/message/166693 Mute This Topic: https://lists.openembedded.org/mt/91618936/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Should BB_DEFAULT_UMASK be set with ?=
Hi, Suppose I want to define BB_DEFAULT_UMASK in site.conf or local.conf (I do have a use case for setting it to 002). But bitbake.conf includes those files before it then goes on to use an unconditional assignment BB_DEFAULT_UMASK = "022" So how/where is one supposed to be able to set BB_DEFAULT_UMASK? Should the assignment in bitbake.conf be ?= or perhaps ??= instead? Or should the assigment instead be moved above the inclusion of the user's configuration files? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#166274): https://lists.openembedded.org/g/openembedded-core/message/166274 Mute This Topic: https://lists.openembedded.org/mt/91427323/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] git: make expat and curl into PACKAGECONFIG items
It can be useful to use git on target (e.g. with some wrapper like etckeeper for keeping track of changes to /etc), and for such cases, it is likely one has no need for pulling from/pushing to http[s] repositories. From the INSTALL file: - "libcurl" library ... If you do not use http:// or https:// repositories, and do not want to put patches into an IMAP mailbox, you do not have to have them (use NO_CURL). - "expat" library; git-http-push uses it for remote lock management over DAV. Similar to "curl" above, this is optional (with NO_EXPAT). Setting --without-expat and --without-curl reduces the size of the installed "git" package from 18M to 12M, in addition to avoiding pulling those libraries into the rootfs. Signed-off-by: Rasmus Villemoes --- meta/recipes-devtools/git/git_2.35.1.bb | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/meta/recipes-devtools/git/git_2.35.1.bb b/meta/recipes-devtools/git/git_2.35.1.bb index 0cff3ce1001..47c22118640 100644 --- a/meta/recipes-devtools/git/git_2.35.1.bb +++ b/meta/recipes-devtools/git/git_2.35.1.bb @@ -3,7 +3,7 @@ HOMEPAGE = "http://git-scm.com; DESCRIPTION = "Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency." SECTION = "console/utils" LICENSE = "GPL-2.0-only" -DEPENDS = "openssl curl zlib expat" +DEPENDS = "openssl zlib" PROVIDES:append:class-native = " git-replacement-native" @@ -18,10 +18,12 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=7c0d7ef03a7eb04ce795b0f60e68e7e1" CVE_PRODUCT = "git-scm:git" -PACKAGECONFIG ??= "" +PACKAGECONFIG ??= "expat curl" PACKAGECONFIG[cvsserver] = "" PACKAGECONFIG[svn] = "" PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native" +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl" +PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat" EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \ --without-tcltk \ -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#163781): https://lists.openembedded.org/g/openembedded-core/message/163781 Mute This Topic: https://lists.openembedded.org/mt/90128519/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bitbake.conf: add BB_NUMBER_THREADS to BB_HASHEXCLUDE_COMMON
On 28/02/2022 14.41, Richard Purdie wrote: > On Mon, 2022-02-28 at 09:42 +0100, Rasmus Villemoes wrote: >> The imx-gpu-sdk recipe in the meta-imx layer references >> ${BB_NUMBER_THREADS} in its do_compile function. Changing >> BB_NUMBER_THREADS between bitbake invocations leads to the well-known >> >> When reparsing >> ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, >> the basehash value changed from >> 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to >> 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The >> metadata is not deterministic and this needs to be fixed. >> >> And I'm not the first to hit this problem with that recipe: >> https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 >> >> This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, >> so changing it does not cause the recipe to be reparsed, but it is not >> included in BB_HASHEXCLUDE_COMMON and thus >> BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to >> both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has >> ${BB_NUMBER_THREADS} as default value. > > Technically imx-gpu-sdk is incorrect. BB_NUMBER_THREADS is the number of tasks > bitbake should run. PARALLEL_MAKE is what is used for parallelism in > do_compile. > > I appreciate that has -j in but you can use: ${@oe.utils.parallel_make(d)} to > obtain the value that recipe needs. Oh, I'm not saying that that recipe isn't buggy, but is still seems awfully user-unfriendly and inconsistent to treat BB_NUMBER_THREADS different from PARALLEL_MAKE wrt. hash computations. But now I've raised the issue, and I don't care much what happens next; I now know the root cause of the error I saw and rarely build using meta-imx in the first place. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162498): https://lists.openembedded.org/g/openembedded-core/message/162498 Mute This Topic: https://lists.openembedded.org/mt/89446852/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] bitbake.conf: add BB_NUMBER_THREADS to BB_HASHEXCLUDE_COMMON
The imx-gpu-sdk recipe in the meta-imx layer references ${BB_NUMBER_THREADS} in its do_compile function. Changing BB_NUMBER_THREADS between bitbake invocations leads to the well-known When reparsing ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, the basehash value changed from 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The metadata is not deterministic and this needs to be fixed. And I'm not the first to hit this problem with that recipe: https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, so changing it does not cause the recipe to be reparsed, but it is not included in BB_HASHEXCLUDE_COMMON and thus BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has ${BB_NUMBER_THREADS} as default value. Signed-off-by: Rasmus Villemoes --- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6fb7bfeb23c..6bc18016056 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -942,7 +942,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ -OMP_NUM_THREADS BB_CURRENTTASK" +OMP_NUM_THREADS BB_CURRENTTASK BB_NUMBER_THREADS" BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \ -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162452): https://lists.openembedded.org/g/openembedded-core/message/162452 Mute This Topic: https://lists.openembedded.org/mt/89446852/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] kernel.bbclass: remove unnecessary dead code
The grep pattern seems to have been wrong ever since we stopped adding the -ffile-prefix-map via a patch in commit 20aea61385e, because the actual upstream gcc produces -ffile-prefix-map== and not -ffile-prefix-map= Besides, these *-prefix-map options are already used when building the kernel due to the setting of KERNEL_CC, and the kernel build system itself also probes for and uses at least -fmacro-prefix-map (hence taking care of __FILE__ etc., but not necessarily things that go in .debug_info sections). Signed-off-by: Rasmus Villemoes --- meta/classes/kernel.bbclass | 15 ++- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index d13c38fb02..4acec1877e 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -322,15 +322,6 @@ python do_devshell:prepend () { addtask bundle_initramfs after do_install before do_deploy -get_cc_option () { - # Check if KERNEL_CC supports the option "file-prefix-map". - # This option allows us to build images with __FILE__ values that do not - # contain the host build path. - if ${KERNEL_CC} -Q --help=joined | grep -q "\-ffile-prefix-map="; then - echo "-ffile-prefix-map=${S}=/kernel-source/" - fi -} - kernel_do_compile() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then @@ -361,9 +352,8 @@ kernel_do_compile() { copy_initramfs use_alternate_initrd=CONFIG_INITRAMFS_SOURCE=${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio fi - cc_extra=$(get_cc_option) for typeformake in ${KERNEL_IMAGETYPE_FOR_MAKE} ; do - oe_runmake ${typeformake} CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd + oe_runmake ${typeformake} CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd done # vmlinux.gz is not built by kernel if (echo "${KERNEL_IMAGETYPES}" | grep -wq "vmlinux\.gz"); then @@ -390,8 +380,7 @@ do_compile_kernelmodules() { bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then - cc_extra=$(get_cc_option) - oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} + oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} # Module.symvers gets updated during the # building of the kernel modules. We need to -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#156260): https://lists.openembedded.org/g/openembedded-core/message/156260 Mute This Topic: https://lists.openembedded.org/mt/85810078/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] RRECOMMENDS "masking" unsatisfiable IMAGE_INSTALL
On 06/07/2021 21.39, Alex Stewart wrote: > Hey Rasmus, > > Sorry for the delay; I was OOO for the holidays and I'm just now working > through my inbox. No need to apologize; as it happens I've just now returned from vacation. > On 6/28/21 4:17 AM, Rasmus Villemoes wrote: >> On 25/06/2021 18.13, Richard Purdie wrote: >>> That is probably a bug that needs opening against opkg in bugzilla then. >>> Added Alex to Cc: (opkg maintainer). >> Indeed, thanks. Alex, do you have enough context from the above, or do I >> need to do anything explicitly to file a bug? >> >> Rasmus > > Could you expand a bit on how OE is supposed to fail when the kernel > module package is unsatisfiable? Does it propagate a opkg satisfaction > error, or is the misconfiguration caught in the OE python modules? > > I ask because it seems most likely to me that the inconsistency here is > in the OE opkg interface modules, rather than opkg itself - which > obviously has no concept of IMAGE_INSTALL or the kernel configuration. > > Can you identify the set of packages that OE is requesting opkg to > install, with and without the RRECOMMENDS? Looking at log.do_rootfs, it seems that opkg gets invoked in exactly the same way with and without the RRECOMMENDS in play: With RRECOMMENDS, where the build succeeds: NOTE: Installing the following packages: [...] glib-2.0 iproute2 kernel-image kernel-module-abcd less libgettextlib [...] NOTE: [...]/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/opkg.conf -t [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/temp/ipktemp/ -o [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version --add-ignore-recommends busybox-syslog --add-ignore-recommends busybox-udhcpc --add-ignore-recommends kernel-vmlinux --add-ignore-recommends udev-cache install [...] glib-2.0 iproute2 kernel-image kernel-module-abcd less libgettextlib [...] And no further mention of kernel-module-abcd. Without the RRECOMMENDS: NOTE: Installing the following packages: [...] glib-2.0 iproute2 kernel-image kernel-module-abcd less libgettextlib [...] NOTE: [...]/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/opkg.conf -t [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/temp/ipktemp/ -o [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version --add-ignore-recommends busybox-syslog --add-ignore-recommends busybox-udhcpc --add-ignore-recommends kernel-vmlinux --add-ignore-recommends udev-cache install [...] glib-2.0 iproute2 kernel-image kernel-module-abcd less libgettextlib [...] ERROR: Unable to install packages. Command '[...]/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/opkg.conf -t [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/temp/ipktemp/ -o [...]/work/rpi3-oe-linux/pil-rootfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version --add-ignore-recommends busybox-syslog --add-ignore-recommends busybox-udhcpc --add-ignore-recommends kernel-vmlinux --add-ignore-recommends udev-cache install [...] glib-2.0 iproute2 kernel-image kernel-module-abcd less libgettextlib [...]' returned 255: * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'kernel-module-abcd'. So I assume the difference must be in some of the metadata files (opkg.conf or the ipktemp/ dir argument to the -t flag), but I know nothing about how opkg/ipk works internally or what to look for. Thanks, Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#154143): https://lists.openembedded.org/g/openembedded-core/message/154143 Mute This Topic: https://lists.openembedded.org/mt/83779506/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] recipes-kernel: vdsotest: new recipe
On 30/06/2021 23.19, Alexander Kanavin wrote: > Should this be in meta-oe rather than oe-core? Perhaps. Could you share some of your thought process that led you to ask that question? I honestly didn't think much about it, but it did seem to fit well with the other "userspace tools closely tied to how the linux kernel works". Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#153450): https://lists.openembedded.org/g/openembedded-core/message/153450 Mute This Topic: https://lists.openembedded.org/mt/83902323/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] recipes-kernel: vdsotest: new recipe
vdsotest is a handy tool for testing and benchmarking the vDSO, e.g. to measure the overhead of clock_gettime() done via the vDSO compared to an actual system call. Signed-off-by: Rasmus Villemoes --- Resending because the list rejected the first attempt, sent from a wrong email address. Apologies to anyone receiving this again. meta/recipes-kernel/vdsotest/vdsotest_0.3.bb | 11 +++ 1 file changed, 11 insertions(+) create mode 100644 meta/recipes-kernel/vdsotest/vdsotest_0.3.bb diff --git a/meta/recipes-kernel/vdsotest/vdsotest_0.3.bb b/meta/recipes-kernel/vdsotest/vdsotest_0.3.bb new file mode 100644 index 00..5f94909102 --- /dev/null +++ b/meta/recipes-kernel/vdsotest/vdsotest_0.3.bb @@ -0,0 +1,11 @@ +SUMMARY = "vdsotest is a utility for testing and benchmarking a Linux VDSO" +HOMEPAGE = "https://github.com/nlynch-mentor/vdsotest; +LICENSE = "GPL-2.0" +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263" + +SRC_URI = "git://github.com/nlynch-mentor/vdsotest.git" +SRCREV = "4ad733fb25364e4afaf4060e57c2429a93b686c5" + +S = "${WORKDIR}/git" + +inherit autotools -- 2.31.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#153446): https://lists.openembedded.org/g/openembedded-core/message/153446 Mute This Topic: https://lists.openembedded.org/mt/83902323/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] RRECOMMENDS "masking" unsatisfiable IMAGE_INSTALL
On 25/06/2021 18.13, Richard Purdie wrote: > On Fri, 2021-06-25 at 17:45 +0200, Rasmus Villemoes wrote: >> On 25/06/2021 14.16, Richard Purdie wrote: >>> On Fri, 2021-06-25 at 09:37 +0200, Rasmus Villemoes via >>> lists.openembedded.org wrote: >>>> I noticed that if I have an image recipe that says >>>> >>>> IMAGE_INSTALL += "kernel-module-foo" >>>> >>>> it fails as expected when the kernel hasn't been built with CONFIG_FOO=m. >>>> >>>> However, if at the same time some other package which happens to get >>>> installed in the same image says >>>> >>>> RRECOMMENDS_${PN} += "kernel-module-foo" >>>> >>>> the image build silently succeeds, obviously without any foo.ko >>>> included. This doesn't seem right. [I'm well aware that the RRECOMMENDS >>>> line alone works as expected, this is about the case where both the >>>> above lines are in play.] >>> >>> Which package backend are you using and does this happen if you >>> use a different one, particularly swapping ipk and rpm? >>> >>> That might tell us if this is a package manager issue or a bitbake >>> metadata one. >> >> I was using ipk. It seems to work as expected (i.e. fail the build) when >> using rpm. > > That is probably a bug that needs opening against opkg in bugzilla then. > Added Alex to Cc: (opkg maintainer). Indeed, thanks. Alex, do you have enough context from the above, or do I need to do anything explicitly to file a bug? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#153352): https://lists.openembedded.org/g/openembedded-core/message/153352 Mute This Topic: https://lists.openembedded.org/mt/83779506/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] RRECOMMENDS "masking" unsatisfiable IMAGE_INSTALL
On 25/06/2021 14.16, Richard Purdie wrote: > On Fri, 2021-06-25 at 09:37 +0200, Rasmus Villemoes via > lists.openembedded.org wrote: >> I noticed that if I have an image recipe that says >> >> IMAGE_INSTALL += "kernel-module-foo" >> >> it fails as expected when the kernel hasn't been built with CONFIG_FOO=m. >> >> However, if at the same time some other package which happens to get >> installed in the same image says >> >> RRECOMMENDS_${PN} += "kernel-module-foo" >> >> the image build silently succeeds, obviously without any foo.ko >> included. This doesn't seem right. [I'm well aware that the RRECOMMENDS >> line alone works as expected, this is about the case where both the >> above lines are in play.] > > Which package backend are you using and does this happen if you > use a different one, particularly swapping ipk and rpm? > > That might tell us if this is a package manager issue or a bitbake > metadata one. I was using ipk. It seems to work as expected (i.e. fail the build) when using rpm. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#153280): https://lists.openembedded.org/g/openembedded-core/message/153280 Mute This Topic: https://lists.openembedded.org/mt/83779506/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] RRECOMMENDS "masking" unsatisfiable IMAGE_INSTALL
I noticed that if I have an image recipe that says IMAGE_INSTALL += "kernel-module-foo" it fails as expected when the kernel hasn't been built with CONFIG_FOO=m. However, if at the same time some other package which happens to get installed in the same image says RRECOMMENDS_${PN} += "kernel-module-foo" the image build silently succeeds, obviously without any foo.ko included. This doesn't seem right. [I'm well aware that the RRECOMMENDS line alone works as expected, this is about the case where both the above lines are in play.] = So I discovered this while trying to figure out how to be able to say IMAGE_INSTALL += "kernel-module-ext4" , and have that succeed whenever ext4 is builtin or modular, and have it be a build time error if the kernel was built with EXT4=n - so the recommends approach is out. It obviously doesn't work out-of-the-box for the EXT4=y case. So now I'm contemplating doing python kernel_base_provide_builtin () { kname = d.getVar("KERNEL_PACKAGE_NAME") or "kernel" builtin = os.path.join(d.getVar("PKGD"), "lib/modules", d.getVar("KERNEL_VERSION"), "modules.builtin") if not os.path.exists(builtin): return builtin_mods = [] with open(builtin) as f: for mod in f: mod = mod.strip() mod = os.path.basename(mod) if not mod.endswith(".ko"): continue mod = mod[:-3] mod = mod.lower().replace("_", "-") builtin_mods.append("%s-module-%s" % (kname, mod)) basepkg = kname + "-base" rprovides = (d.getVar("RPROVIDES_" + basepkg) or "").split() rprovides += builtin_mods d.setVar("RPROVIDES_" + basepkg, " ".join(rprovides)) } PACKAGESPLITFUNCS += "kernel_base_provide_builtin" i.e., make sure that the kernel-base package is an rprovider of kernel-module-ext4 for the EXT4=y case [*]. That seems to work as expected in the cases I've tried (provided nobody rrecommends ext4, per above). Does anybody see anything wrong with that approach? EXT4 is just an example, I'm gonna use it for other modules/subsystems with more complicated dependencies that might not be satisfied in a base .config, so a fragment saying CONFIG_FOO=m might have been ignored due to that, and I'd prefer catching that no later than at image build time. I'm not necessarily suggesting the above for oe-core, I can live with it in my own kernel recipes. [*] kernel-base and not the package containing the actual kernel binary, because kernel-base contains the modules.builtin file which is what modprobe consults first to see if it actually has to do anything, so it makes sense to have kernel-module-ext4 pull in that file and the other /lib/modules/ metadata when EXT4=y. The kernel binary is as often as not not actually in the rootfs image, and wouldn't be useful to modprobe anyway. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#153266): https://lists.openembedded.org/g/openembedded-core/message/153266 Mute This Topic: https://lists.openembedded.org/mt/83779506/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 0/4] fix externalsrc task dependency issues
On 25/09/2020 15.14, Rasmus Villemoes via lists.openembedded.org wrote: > You've convinced me that changing deltask to noexec is too > wide-reaching, which means that I think just patch 1/4 should be > applied. Please consider doing that. Or rather, please apply the v2 I just sent to update the commit log to reflect my new understanding. Thanks, Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142789): https://lists.openembedded.org/g/openembedded-core/message/142789 Mute This Topic: https://lists.openembedded.org/mt/77015064/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] kernel.bbclass: ensure symlink_kernsrc task gets run even with externalsrc
Commit c5dfc2586b41 (kernel.bbclass: run do_symlink_kernsrc before do_patch) fixed a race between do_symlink_kernsrc and do_populate_lic. However, I missed the fact that when externalsrc.bbclass is in use, the do_patch task doesn't exist, meaning that do_symlink_kernsrc now doesn't get run at all, breaking the build. We need this task to run regardless of whether do_patch exists or not, so reinstate the configure->symlink_kernsrc dependency explicitly. Fixes: c5dfc2586b41 (kernel.bbclass: run do_symlink_kernsrc before do_patch) Reported-by: Chanho Park Signed-off-by: Rasmus Villemoes --- v2: update commit log to not say that externalsrc.bbclass is at fault for using deltask over [noexec]. Rest of the series is dropped. meta/classes/kernel.bbclass | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 48135b3d41..78def5bbc1 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -172,7 +172,10 @@ python do_symlink_kernsrc () { shutil.move(s, kernsrc) os.symlink(kernsrc, s) } -addtask symlink_kernsrc before do_patch after do_unpack +# do_patch is normally ordered before do_configure, but +# externalsrc.bbclass deletes do_patch, breaking the dependency of +# do_configure on do_symlink_kernsrc. +addtask symlink_kernsrc before do_patch do_configure after do_unpack inherit kernel-arch deploy -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142788): https://lists.openembedded.org/g/openembedded-core/message/142788 Mute This Topic: https://lists.openembedded.org/mt/77080010/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 0/4] fix externalsrc task dependency issues
On 25/09/2020 13.43, Richard Purdie wrote: > On Fri, 2020-09-25 at 12:31 +0200, Rasmus Villemoes wrote: >> Well, perhaps listtasks could be taught to say "[noexec flag set]" in >> the description - currently it seems to be somewhat random whether a >> class uses noexec or deltask. > > No. That would just confuse things even more. That would hide do_build > for example so the default target would be missing? I did not say "don't display noexec tasks". I said 'say "[noexec flag set]" in the description' - i.e., have the output include an indication that that task will not be actually be executed. Perhaps a slightly longer prose would be helpful, say something like "[disabled for this recipe due to noexec flag being set]" appended after the current description. There are already lots of noexec tasks, e.g. all image recipes have a lot of ordinary tasks disabled via noexec, so that should help address your concern that > ... If > someone can "see" the task in listtasks and trys to run it but nothing > happens, its very confusing for the user. >> Oh, and for debugging task ordering issues, it would also be nice if >> listtasks showed tasks in topological rather than alphabetical order - >> that won't necessarily always show an implicit dependency that went >> missing due to a deltask, but it would at least reveal some of them. > > I think showing half of some information would be much more confusing > that clearly not showing it. We'd just get bug reports that the order > was "wrong" and it can't be fixed in all cases. Can we then (perhaps via some listtasksfull or whatnot) get bitbake to display the tasks along with their dependencies. Or how would I, today, go about figuring out what (intra-recipe) dependencies some-recipe:do_configure actually has? >>> I think there were technical issues as well, such as recipes which >>> inject other tasks between unpack and patch. We needed to stop those >>> other tasks running too which deltasks does effectively, noexec would >>> not. >> >> Eh, but the other side of that coin is that there are obviously tasks >> injected between unpack and patch, or some other combination, which must >> _not_ be prevented from running > > Some need to be prevented, for example where they do extra unpacking > type operations on files. Those would actively break externalsrc. > >> - cf. at least the two or three workarounds that seem to become unnecessary. > > Some would be unnecessary, it would break other use cases. > >> So can you give examples of >> such tasks that _should_ implicitly not run when externalsrc disables >> (one way or another) the SRCTREECOVEREDTASKS tasks? That's actually the >> RFC part of this, I hope somebody with far more knowledge of various >> corner cases and odd recipes can demonstrate why deltask is more >> appropriate - and then, as I said, some comments to that effect in >> externalsrc.bbclass would be much appreciated. > > I will try my best to find them. I don't have a list of these things I > can just recite from memory. One that did come up in a quick search is: > > classes/dos2unix.bbclass:addtask convert_crlf_to_lf after do_unpack before > do_patch > > I would note that the class doesn't appear used in OE-Core any longer. OK, that counts, even if that class appears unused - I can see how such things can exist. > Proving OE-Core is "safe" for such a change is only part of the > challenge too, there is the wider ecosystem to consider. Absolutely. >> I think it would be weird for a recipe to inherit externalsrc and at the >> same time define a task ordered before do_patch, then not expect that >> task to run at all. So I'm genuinely curious how such examples would look. > > devtool takes recipes and adds externalsrc behind the scenes to them to > enable it to do things. Its expected these scenarios work, its used for > example by the automated upgrade helper. We're not risking breaking all > these tools just because you think the code might be cleaner. See, that's the kind of explanation I'm looking for - and again, I'd like to see in externalsrc.bbclass itself. The thing is, I never knew about the existence of the externalsrc class until I noticed the bug reported on c5dfc2586b. So when I tried to figure out how to fix that bug, I had to understand why externalsrc removed the do_patch task, breaking the configure->patch->symlink_kernsrc chain. > Your approach also concerns me as you are basically asking me to > "prove" your code has issues and it seems aren't interested in testing > even the patch you did send. That is not how I intended this to be interpreted. But I was r
Re: [OE-core] [RFC PATCH 0/4] fix externalsrc task dependency issues
On 25/09/2020 11.22, Richard Purdie wrote: > On Fri, 2020-09-25 at 11:06 +0200, Rasmus Villemoes wrote: >> On 22/09/2020 17.15, Rasmus Villemoes wrote: >>> This is all entirely untested, hence RFC. But if this doesn't work >>> for >>> some reason, I think externalsrc.bbclass should grow some comments >>> as >>> to why deltask is used instead of the noexec flag. >>> >>> Patch 4 is a revert of patch 1, but I did it that way to facilitate >>> backporting, in case patch 2 is too dangerous for backporting. If >>> that >>> is considered safe enough, it should be enough to apply patch 2 to >>> avoid the churn. In any case, patch 3 is just a little >>> simplification >>> made possible by patch 2. >>> >>> >> >> Ping. Chanho, can you tell if this would fix the externalsrc problem? >> Richard, any comments on the deltask vs noexec usage? > > You started this patchset with "This is all entirely untested" which > I'm afraid made me rather nervous. Much as I'd love to, I don't have > time to debug every patch series people send. > > I'm fairly sure the deltask vs. noexec was deliberate and Paul and I > went around in circles on this several times. > > I have some memory that at least part of the problem was usability. If > someone can "see" the task in listtasks and trys to run it but nothing > happens, its very confusing for the user. By deleting it, we make it > very clear its not there. Well, perhaps listtasks could be taught to say "[noexec flag set]" in the description - currently it seems to be somewhat random whether a class uses noexec or deltask. Oh, and for debugging task ordering issues, it would also be nice if listtasks showed tasks in topological rather than alphabetical order - that won't necessarily always show an implicit dependency that went missing due to a deltask, but it would at least reveal some of them. > I think there were technical issues as well, such as recipes which > inject other tasks between unpack and patch. We needed to stop those > other tasks running too which deltasks does effectively, noexec would > not. Eh, but the other side of that coin is that there are obviously tasks injected between unpack and patch, or some other combination, which must _not_ be prevented from running - cf. at least the two or three workarounds that seem to become unnecessary. So can you give examples of such tasks that _should_ implicitly not run when externalsrc disables (one way or another) the SRCTREECOVEREDTASKS tasks? That's actually the RFC part of this, I hope somebody with far more knowledge of various corner cases and odd recipes can demonstrate why deltask is more appropriate - and then, as I said, some comments to that effect in externalsrc.bbclass would be much appreciated. I think it would be weird for a recipe to inherit externalsrc and at the same time define a task ordered before do_patch, then not expect that task to run at all. So I'm genuinely curious how such examples would look. Thanks, Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142779): https://lists.openembedded.org/g/openembedded-core/message/142779 Mute This Topic: https://lists.openembedded.org/mt/77015064/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 0/4] fix externalsrc task dependency issues
On 22/09/2020 17.15, Rasmus Villemoes wrote: > This is all entirely untested, hence RFC. But if this doesn't work for > some reason, I think externalsrc.bbclass should grow some comments as > to why deltask is used instead of the noexec flag. > > Patch 4 is a revert of patch 1, but I did it that way to facilitate > backporting, in case patch 2 is too dangerous for backporting. If that > is considered safe enough, it should be enough to apply patch 2 to > avoid the churn. In any case, patch 3 is just a little simplification > made possible by patch 2. > > Ping. Chanho, can you tell if this would fix the externalsrc problem? Richard, any comments on the deltask vs noexec usage? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142773): https://lists.openembedded.org/g/openembedded-core/message/142773 Mute This Topic: https://lists.openembedded.org/mt/77015064/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 0/4] fix externalsrc task dependency issues
This is all entirely untested, hence RFC. But if this doesn't work for some reason, I think externalsrc.bbclass should grow some comments as to why deltask is used instead of the noexec flag. Patch 4 is a revert of patch 1, but I did it that way to facilitate backporting, in case patch 2 is too dangerous for backporting. If that is considered safe enough, it should be enough to apply patch 2 to avoid the churn. In any case, patch 3 is just a little simplification made possible by patch 2. Rasmus Villemoes (4): kernel.bbclass: ensure symlink_kernsrc task gets run even with externalsrc externalsrc.bbclass: make SRCTREECOVEREDTASKS noexec instead of deleting them kernel-yocto.bbclass: remove now unnecessary externalsrc workaround Revert "kernel.bbclass: ensure symlink_kernsrc task gets run even with externalsrc" meta/classes/externalsrc.bbclass | 5 + meta/classes/kernel-yocto.bbclass | 6 -- 2 files changed, 1 insertion(+), 10 deletions(-) -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142720): https://lists.openembedded.org/g/openembedded-core/message/142720 Mute This Topic: https://lists.openembedded.org/mt/77015064/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 4/4] Revert "kernel.bbclass: ensure symlink_kernsrc task gets run even with externalsrc"
With the changes to externalsrc.bbclass to make tasks noexec instead of deleting them, we no longer need to explicitly make do_symlink_kernsrc a dependency of do_configure. Signed-off-by: Rasmus Villemoes --- meta/classes/kernel.bbclass | 5 + 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 78def5bbc1..48135b3d41 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -172,10 +172,7 @@ python do_symlink_kernsrc () { shutil.move(s, kernsrc) os.symlink(kernsrc, s) } -# do_patch is normally ordered before do_configure, but -# externalsrc.bbclass deletes do_patch, breaking the dependency of -# do_configure on do_symlink_kernsrc. -addtask symlink_kernsrc before do_patch do_configure after do_unpack +addtask symlink_kernsrc before do_patch after do_unpack inherit kernel-arch deploy -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142724): https://lists.openembedded.org/g/openembedded-core/message/142724 Mute This Topic: https://lists.openembedded.org/mt/77015068/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 2/4] externalsrc.bbclass: make SRCTREECOVEREDTASKS noexec instead of deleting them
The preferred way to prevent a task from running is not to deltask it, but setting the noexec flag. That way various implicit dependencies through the task in question are preserved, avoiding ad hoc workarounds, such as the one right here. Signed-off-by: Rasmus Villemoes --- meta/classes/externalsrc.bbclass | 5 + 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index d200129987..7b4d8b369f 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -100,14 +100,11 @@ python () { d.setVarFlag(task, 'cleandirs', ' '.join(cleandirs)) fetch_tasks = ['do_fetch', 'do_unpack'] -# If we deltask do_patch, there's no dependency to ensure do_unpack gets run, so add one -# Note that we cannot use d.appendVarFlag() here because deps is expected to be a list object, not a string -d.setVarFlag('do_configure', 'deps', (d.getVarFlag('do_configure', 'deps', False) or []) + ['do_unpack']) for task in d.getVar("SRCTREECOVEREDTASKS").split(): if local_srcuri and task in fetch_tasks: continue -bb.build.deltask(task, d) +d.setVarFlag(task, 'noexec', '1') d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ") d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ") -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142722): https://lists.openembedded.org/g/openembedded-core/message/142722 Mute This Topic: https://lists.openembedded.org/mt/77015066/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 3/4] kernel-yocto.bbclass: remove now unnecessary externalsrc workaround
Now that SRCTREECOVEREDTASKS gets marked as noexec rather than being deltask'ed, do_kernel_configme is still ordered after do_patch (and recursively, after do_unpack), so this workaround is no longer needed. Signed-off-by: Rasmus Villemoes --- meta/classes/kernel-yocto.bbclass | 6 -- 1 file changed, 6 deletions(-) diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass index a35c5923df..741b0b961c 100644 --- a/meta/classes/kernel-yocto.bbclass +++ b/meta/classes/kernel-yocto.bbclass @@ -656,12 +656,6 @@ python () { # If diffconfig is available, ensure it runs after kernel_configme if 'do_diffconfig' in d: bb.build.addtask('do_diffconfig', None, 'do_kernel_configme', d) - -externalsrc = d.getVar('EXTERNALSRC') -if externalsrc: -# If we deltask do_patch, do_kernel_configme is left without -# dependencies and runs too early -d.setVarFlag('do_kernel_configme', 'deps', (d.getVarFlag('do_kernel_configme', 'deps', False) or []) + ['do_unpack']) } # extra tasks -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142723): https://lists.openembedded.org/g/openembedded-core/message/142723 Mute This Topic: https://lists.openembedded.org/mt/77015067/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 1/4] kernel.bbclass: ensure symlink_kernsrc task gets run even with externalsrc
Commit c5dfc2586b41 (kernel.bbclass: run do_symlink_kernsrc before do_patch) fixed a race between do_symlink_kernsrc and do_populate_lic. However, I missed the fact that when externalsrc.bbclass is in use, the do_patch task doesn't exist, meaning that do_symlink_kernsrc now doesn't get run at all, breaking the build. While I think the proper fix is to stop deltask'ing do_patch and instead make it noexec in externalsrc.bbclass, a more targeted fix suitable for backporting is to simply make this a dependency of both do_patch and do_configure. Fixes: c5dfc2586b41 (kernel.bbclass: run do_symlink_kernsrc before do_patch) Reported-by: Chanho Park Signed-off-by: Rasmus Villemoes --- meta/classes/kernel.bbclass | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 48135b3d41..78def5bbc1 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -172,7 +172,10 @@ python do_symlink_kernsrc () { shutil.move(s, kernsrc) os.symlink(kernsrc, s) } -addtask symlink_kernsrc before do_patch after do_unpack +# do_patch is normally ordered before do_configure, but +# externalsrc.bbclass deletes do_patch, breaking the dependency of +# do_configure on do_symlink_kernsrc. +addtask symlink_kernsrc before do_patch do_configure after do_unpack inherit kernel-arch deploy -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142721): https://lists.openembedded.org/g/openembedded-core/message/142721 Mute This Topic: https://lists.openembedded.org/mt/77015065/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][dunfell 23/25] kernel.bbclass: run do_symlink_kernsrc before do_patch
On 16/09/2020 17.18, Steve Sakoman wrote: > Since there is an upcoming dunfell release and we don't have a fix for > this issue I am going to revert this patch. > > When it is fixed in master I will reconsider taking this patch and the > fix for dunfell. Sorry for not seeing this sooner, and for the bug in the first place. However, looking into externalsrc.bbclass, isn't the bug there, considering the "if we delete do_patch, do_unpack doesn't run" fixup done - kernel-yocto.bbclass has a similar fixup for do_kernel_configme. Why doesn't that class just set [noexec]=1 instead of deleting the tasks outright, killing all those random fixups? But if that's too invasive, I suppose just ordering the symlink task before both patch and configure should work, i.e. --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -172,7 +172,7 @@ python do_symlink_kernsrc () { shutil.move(s, kernsrc) os.symlink(kernsrc, s) } -addtask symlink_kernsrc before do_patch after do_unpack +addtask symlink_kernsrc before do_patch do_configure after do_unpack inherit kernel-arch deploy Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142672): https://lists.openembedded.org/g/openembedded-core/message/142672 Mute This Topic: https://lists.openembedded.org/mt/76518932/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] rootfs-postcommands.bbclass: make /etc/timestamp path configurable
On 04/09/2020 13.51, Richard Purdie wrote: > On Fri, 2020-09-04 at 09:08 +0200, Rasmus Villemoes wrote: >> Not every image necessarily wants a timestamp file, > > Wouldn't you just not have rootfs_update_timestamp in > ROOTFS_POSTPROCESS_COMMAND in that case? Perhaps, but removing stuff from ROOTFS_POSTPROCESS_COMMAND is a little tedious and error-prone, 8c835f63 isn't all that old. And with the other use case in mind, it really isn't a lot of extra logic to assign a meaning to ROOTFS_TIMESTAMP_FILE being empty. > > I have a feeling there are other consumers of this file which would > also need to be changed to reference ROOTFS_TIMESTAMP_FILE? Well, there are two files associated to the initscripts recipe mentioning it, and a mention in devtool. But I don't imagine ROOTFS_TIMESTAMP_FILE being set globally, it's really something one can set in individual image recipes for images that don't create an ordinary linux rootfs (e.g. bootfiles for RPi). Dunno, if somebody sets ROOTFS_TIMESTAMP_FILE for their proper rootfs and also include the initscripts package in that rootfs, I suppose they should match, but I don't know where one should set the variable. Btw., is there some good reason that by default, with reprocible_builds turned on, the rootfs gets two files with the exact same content (/etc/timestamp and /etc/version)? Because we'll need to fix (or avoid) the creation of /etc/version in the same way for some of our images. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142250): https://lists.openembedded.org/g/openembedded-core/message/142250 Mute This Topic: https://lists.openembedded.org/mt/76624850/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] rootfs-postcommands.bbclass: make /etc/timestamp path configurable
Not every image necessarily wants a timestamp file, and even if it does, it's possible to create filesystem images that do not have a /etc directory or where some other location for the file is more appropriate. So add a ROOTFS_TIMESTAMP_FILE variable weakly defaulting to /etc/timestamp - setting that to the empty string opts out of creating the timestamp file. I wasn't sure whether to spell /etc as ${sysconfdir}, but chose /etc to stay completely backwards compatible. Signed-off-by: Rasmus Villemoes --- meta/classes/rootfs-postcommands.bbclass | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass index 984730ebe8..cf54f22566 100644 --- a/meta/classes/rootfs-postcommands.bbclass +++ b/meta/classes/rootfs-postcommands.bbclass @@ -12,6 +12,7 @@ ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [ 'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}' # Create /etc/timestamp during image construction to give a reasonably sane default time setting +ROOTFS_TIMESTAMP_FILE ??= "/etc/timestamp" ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp; " # Tweak the mount options for rootfs in /etc/fstab if read-only-rootfs is enabled @@ -275,14 +276,18 @@ python write_image_manifest () { # Can be used to create /etc/timestamp during image construction to give a reasonably # sane default time setting rootfs_update_timestamp () { + if [ -z "${ROOTFS_TIMESTAMP_FILE}" ]; then + return + fi if [ "${REPRODUCIBLE_TIMESTAMP_ROOTFS}" != "" ]; then # Convert UTC into %4Y%2m%2d%2H%2M%2S sformatted=`date -u -d @${REPRODUCIBLE_TIMESTAMP_ROOTFS} +%4Y%2m%2d%2H%2M%2S` else sformatted=`date -u +%4Y%2m%2d%2H%2M%2S` fi - echo $sformatted > ${IMAGE_ROOTFS}/etc/timestamp - bbnote "rootfs_update_timestamp: set /etc/timestamp to $sformatted" + mkdir -p $(dirname ${IMAGE_ROOTFS}${ROOTFS_TIMESTAMP_FILE}) + echo $sformatted > ${IMAGE_ROOTFS}${ROOTFS_TIMESTAMP_FILE} + bbnote "rootfs_update_timestamp: set ${ROOTFS_TIMESTAMP_FILE} to $sformatted" } # Prevent X from being started -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#142182): https://lists.openembedded.org/g/openembedded-core/message/142182 Mute This Topic: https://lists.openembedded.org/mt/76624850/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] kernel.bbclass: run do_symlink_kernsrc before do_patch
There's a race between do_symlink_kernsrc and do_populate_lic, since the latter is ordered "after do_patch"; so the two may run in parallel. In some cases, that actually causes do_populate_lic to fail if it happens to look for a license file somewhere under ${S} in the short window after shutil.move and before the symlink has been created. Fix that by simply ordering symlink_kernsrc before do_patch. Any task that pokes around in ${S} looking for files should be ordered after do_patch, so this should also fix similar latent races with other ad hoc tasks. Signed-off-by: Rasmus Villemoes --- meta/classes/kernel.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index e2ceb6a333..cba77daa7a 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -172,7 +172,7 @@ python do_symlink_kernsrc () { shutil.move(s, kernsrc) os.symlink(kernsrc, s) } -addtask symlink_kernsrc before do_configure after do_unpack +addtask symlink_kernsrc before do_patch after do_unpack inherit kernel-arch deploy -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141825): https://lists.openembedded.org/g/openembedded-core/message/141825 Mute This Topic: https://lists.openembedded.org/mt/76407816/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] do_populate_lic problems
There are a couple of problems with do_populate_lic, especially around the kernel and u-boot recipes. First, we have a race condition in kernel recipes: addtask populate_lic after do_patch before do_build addtask symlink_kernsrc before do_configure after do_unpack so there's no ordering between those two tasks. I have in fact seen do_populate_lic fail because (part of) it runs just between the two lines shutil.move(s, kernsrc) os.symlink(kernsrc, s) WARNING: Could not copy license file /mnt/... ... ERROR: QA Issue: ...: LIC_FILES_CHKSUM points to an invalid file: and it's fairly easy to reproduce with a monkey-patch: @@ -148,7 +148,9 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc) else: import shutil +import time shutil.move(s, kernsrc) +time.sleep(15) os.symlink(kernsrc, s) } I think this could be fixed by making symlink_kernsrc run before do_patch rather than do_configure - in fact, AFAICT, there's the exact same lack of ordering with do_patch, it's just that nobody usually has any .patch files for the kernel when it's maintained in git. Second, as I've reported previously [1], creating hard-links to files in kernel or u-boot source repositories, as do_populate_lic does, leads to scripts/setlocalversion falsely appending a "-dirty" string. It doesn't happen if one is building a kernel containing ff64dd485730 or a backport of it (e.g. v5.0+, v4.19.82+) _and_ one has git >= 2.14. U-Boot hasn't copied ff64dd485730 yet, so it can always happen for U-Boot. Since there's no ordering between do_populate_lic and do_compile, it's not even deterministic whether that -dirty string is appended. So while appropriate tools and patched upstream projects will make the problem go away, I'm wondering the hardlink optimization is really worth the trouble it causes. [1] https://lists.openembedded.org/g/openembedded-core/message/137702 Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141806): https://lists.openembedded.org/g/openembedded-core/message/141806 Mute This Topic: https://lists.openembedded.org/mt/76403349/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] cleansstate
On 24/08/2020 23.08, Richard Purdie wrote: > On Mon, 2020-08-24 at 23:05 +0200, Rasmus Villemoes wrote: >> On 22/08/2020 09.27, Richard Purdie via lists.openembedded.org wrote: >>> On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote: >>>> What would be a use case for cleanstate? >>> >>> I understand why people want it although it should never really >>> need to >>> be used. >> >> Well, except when you discover that your sstate is effectively >> corrupted/polluted by objects that happen to have the same hashes as >> those computed from current recipes, but the build artifacts >> would/should be different. Such as when using a file://foo/* glob >> pattern in SRC_URI. > > Where such issues are reported, we do try and fix them. I hadn't quite > connected that the globbing issue was actually breaking sstate hashes, > that is more worrying :( It does, and you can reproduce yourself using upstream core recipes. Try building connman-gnome. Then alter one of the png files under images/, and build again. The recipe does not get rebuilt. Then cleansstate, build, and yes, then the modified png does get picked up. Of course, those pngs don't change often, if ever. But in our case the recipe was for including some firmware blobs for some I/O modules. We found out when the CI builds that used sstate cache failed to produce BSPs containing the updated firmware, while the nightly builds that build from scratch did. >> >> Richard, I'm still hoping you'd respond in the >> https://lists.openembedded.org/g/openembedded-core/topic/75235594 >> thread. As I see it, half-way supporting glob patterns in file:// >> URIs is a giant footgun. > > My leaning is to remove them. You can just do file://foo and I think > that actually works? In some of the cases where we had recipes using globs, yes, that was the solution. In a few other cases we had more complicated patterns (e.g. *.sh or something else that wasn't just including a whole dir). Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141805): https://lists.openembedded.org/g/openembedded-core/message/141805 Mute This Topic: https://lists.openembedded.org/mt/76298145/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] cleansstate
On 22/08/2020 09.27, Richard Purdie via lists.openembedded.org wrote: > On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote: >> What would be a use case for cleanstate? > > I never really wanted to add it at all. There is/was some case for > wanting to remove sstate objects and ensure something really does > rebuild from source. When sstate was new and had more issues, people > really wanted it and I was pushed into having it exist. > > I understand why people want it although it should never really need to > be used. Well, except when you discover that your sstate is effectively corrupted/polluted by objects that happen to have the same hashes as those computed from current recipes, but the build artifacts would/should be different. Such as when using a file://foo/* glob pattern in SRC_URI. Richard, I'm still hoping you'd respond in the https://lists.openembedded.org/g/openembedded-core/topic/75235594 thread. As I see it, half-way supporting glob patterns in file:// URIs is a giant footgun. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141789): https://lists.openembedded.org/g/openembedded-core/message/141789 Mute This Topic: https://lists.openembedded.org/mt/76298145/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] file checksums versus SRC_URI globs
On 09/07/2020 21.31, Rasmus Villemoes via lists.openembedded.org wrote: > On 02/07/2020 08.42, Rasmus Villemoes via lists.openembedded.org wrote: >> On 01/07/2020 16.03, Quentin Schulz wrote: >>> Hi Rasmus, >>> >>> On Wed, Jul 01, 2020 at 03:51:19PM +0200, Rasmus Villemoes wrote: >>>> Hi, >>>> >>>> We have a recipe that uses >>>> >>>> SRC_URI += "file://somedir/*" >>>> >>> >>> Glob aren't supported. Use "file://somedir/" instead. >> >> Thanks, that actually works for one of the cases we have (there are >> others that use globs which cannot be solved quite that simply, but for >> now I'm just listing files explicitly instead). >> >> However, I'm not sure that "globs aren't supported". The commit I >> referenced clearly tried to make that work (better), it also "works" in >> the sense of unpacking the expected things when building from scratch - >> there's even >> >> def test_local_wildcard(self): >> tree = self.fetchUnpack(['file://a', 'file://dir/*']) >> self.assertEqual(tree, ['a', 'dir/c', 'dir/d', 'dir/subdir/e']) >> >> in bitbake/lib/bb/tests/fetch.py. And the two upstream recipes >> connman-gnome_0.7.bb and matchbox-desktop_2.2.bb both use that exact >> pattern. >> >> So either >> >> - this is a plain bug in the signature computation, > > I'm guessing the culprit is > > commit 6c0706a28d72c591f1b75b6e3f3b645859387c7e > Author: Richard Purdie > Date: Mon Dec 8 21:25:23 2014 + > > cache/fetch2/siggen: Ensure we track include history for file checksums > > which was the one that introduced the ":True: or ":False" suffixing via > > +filelist.append(f + ":" + str(os.path.exists(f))) > > and then also did > > @@ -981,6 +980,10 @@ def get_file_checksums(filelist, pn): > > checksums = [] > for pth in filelist.split(): > +exist = pth.split(":")[1] > +if exist == "False": > +continue > +pth = pth.split(":")[0] > if '*' in pth: > # Handle globs > for f in glob.glob(pth): > > which practically guaranteed that the "if '*' in pth" would be dead code. > > Richard, do you agree that this is a bug in the signature computation? Richard, ping? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#141199): https://lists.openembedded.org/g/openembedded-core/message/141199 Mute This Topic: https://lists.openembedded.org/mt/75235594/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] move find_cfgs() helper to cml1.bbclass
u-boot.inc and busybox.inc contain identical copies of this little helper. They both inherit the cml1 class and use cml1_do_configure right after having used this helper, and other recipes that want to write similar logic for doing Kconfig via fragments will also need it or something equivalent. Signed-off-by: Rasmus Villemoes --- Those other recipes can certainly also just copy-paste find_cfgs(), but I think it's nice to avoid even small pieces of code duplication (it all adds up). What does cml1 stand for? I can't find anything in git history or comments, and "basic support for the Linux kernel style build configuration system" doesn't provide much of a clue either. meta/classes/cml1.bbclass | 10 ++ meta/recipes-bsp/u-boot/u-boot.inc| 10 -- meta/recipes-core/busybox/busybox.inc | 10 -- 3 files changed, 10 insertions(+), 20 deletions(-) diff --git a/meta/classes/cml1.bbclass b/meta/classes/cml1.bbclass index c7f6723cb3..8ab240589a 100644 --- a/meta/classes/cml1.bbclass +++ b/meta/classes/cml1.bbclass @@ -1,3 +1,13 @@ +# returns all the elements from the src uri that are .cfg files +def find_cfgs(d): +sources=src_patches(d, True) +sources_list=[] +for s in sources: +if s.endswith('.cfg'): +sources_list.append(s) + +return sources_list + cml1_do_configure() { set -e unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS diff --git a/meta/recipes-bsp/u-boot/u-boot.inc b/meta/recipes-bsp/u-boot/u-boot.inc index c91da02959..5ec3d90764 100644 --- a/meta/recipes-bsp/u-boot/u-boot.inc +++ b/meta/recipes-bsp/u-boot/u-boot.inc @@ -70,16 +70,6 @@ UBOOT_EXTLINUX_INSTALL_DIR ?= "/boot/extlinux" UBOOT_EXTLINUX_CONF_NAME ?= "extlinux.conf" UBOOT_EXTLINUX_SYMLINK ?= "${UBOOT_EXTLINUX_CONF_NAME}-${MACHINE}-${PR}" -# returns all the elements from the src uri that are .cfg files -def find_cfgs(d): -sources=src_patches(d, True) -sources_list=[] -for s in sources: -if s.endswith('.cfg'): -sources_list.append(s) - -return sources_list - do_configure () { if [ -n "${UBOOT_CONFIG}" ]; then unset i j diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc index 6cfdcd7344..45aaa2b41c 100644 --- a/meta/recipes-core/busybox/busybox.inc +++ b/meta/recipes-core/busybox/busybox.inc @@ -133,16 +133,6 @@ do_prepare_config () { fi } -# returns all the elements from the src uri that are .cfg files -def find_cfgs(d): -sources=src_patches(d, True) -sources_list=[] -for s in sources: -if s.endswith('.cfg'): -sources_list.append(s) - -return sources_list - do_configure () { set -x do_prepare_config -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140493): https://lists.openembedded.org/g/openembedded-core/message/140493 Mute This Topic: https://lists.openembedded.org/mt/75405917/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] file checksums versus SRC_URI globs
On 02/07/2020 08.42, Rasmus Villemoes via lists.openembedded.org wrote: > On 01/07/2020 16.03, Quentin Schulz wrote: >> Hi Rasmus, >> >> On Wed, Jul 01, 2020 at 03:51:19PM +0200, Rasmus Villemoes wrote: >>> Hi, >>> >>> We have a recipe that uses >>> >>> SRC_URI += "file://somedir/*" >>> >> >> Glob aren't supported. Use "file://somedir/" instead. > > Thanks, that actually works for one of the cases we have (there are > others that use globs which cannot be solved quite that simply, but for > now I'm just listing files explicitly instead). > > However, I'm not sure that "globs aren't supported". The commit I > referenced clearly tried to make that work (better), it also "works" in > the sense of unpacking the expected things when building from scratch - > there's even > > def test_local_wildcard(self): > tree = self.fetchUnpack(['file://a', 'file://dir/*']) > self.assertEqual(tree, ['a', 'dir/c', 'dir/d', 'dir/subdir/e']) > > in bitbake/lib/bb/tests/fetch.py. And the two upstream recipes > connman-gnome_0.7.bb and matchbox-desktop_2.2.bb both use that exact > pattern. > > So either > > - this is a plain bug in the signature computation, I'm guessing the culprit is commit 6c0706a28d72c591f1b75b6e3f3b645859387c7e Author: Richard Purdie Date: Mon Dec 8 21:25:23 2014 + cache/fetch2/siggen: Ensure we track include history for file checksums which was the one that introduced the ":True: or ":False" suffixing via +filelist.append(f + ":" + str(os.path.exists(f))) and then also did @@ -981,6 +980,10 @@ def get_file_checksums(filelist, pn): checksums = [] for pth in filelist.split(): +exist = pth.split(":")[1] +if exist == "False": +continue +pth = pth.split(":")[0] if '*' in pth: # Handle globs for f in glob.glob(pth): which practically guaranteed that the "if '*' in pth" would be dead code. Richard, do you agree that this is a bug in the signature computation? As I wrote previously, there's no warning anywhere that using globs in SRC_URI will fail to take the contents of the referenced file into account in hashes, but there's obviously still a some code that makes file://*.c work wrt. unpacking (and a test case for that). Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140490): https://lists.openembedded.org/g/openembedded-core/message/140490 Mute This Topic: https://lists.openembedded.org/mt/75235594/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] [dunfell] coreutils: don't split stdbuf to own package with single-binary
Commit 992cec44 (coreutils: Move stdbuf into an own package coreutils-stdbuf) breaks package-qa when the single-binary PACKAGECONFIG is used: ERROR: coreutils-8.32-r0 do_package_qa: QA Issue: /usr/bin/stdbuf contained in package coreutils-stdbuf requires /usr/bin/coreutils, but no providers found in RDEPENDS_coreutils-stdbuf? [file-rdeps] ERROR: coreutils-8.32-r0 do_package_qa: QA run found fatal errors. Please consider fixing them. With that PACKAGECONFIG, /usr/bin/stdbuf is just a simple "script" containing the single line #!/usr/bin/coreutils --coreutils-prog-shebang=stdbuf Since there's no point splitting stdbuf to its own package when all the functionality is in the single big coreutils binary anyway, fix this by not creating the separate stdbuf package for the single-binary case. But also make sure that the coreutils-stdbuf item always exists so recipes can always RDEPEND on coreutils-stdbuf. Signed-off-by: Rasmus Villemoes Signed-off-by: Richard Purdie (cherry picked from commit 74d24b5b895198898944260136d05e991a203c11) --- meta/recipes-core/coreutils/coreutils_8.31.bb | 15 +-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/meta/recipes-core/coreutils/coreutils_8.31.bb b/meta/recipes-core/coreutils/coreutils_8.31.bb index 2b196b7512..0c8452da98 100644 --- a/meta/recipes-core/coreutils/coreutils_8.31.bb +++ b/meta/recipes-core/coreutils/coreutils_8.31.bb @@ -59,9 +59,20 @@ base_bindir_progs = "cat chgrp chmod chown cp date dd echo false hostname kill l sbindir_progs= "chroot" -PACKAGE_BEFORE_PN_class-target += "coreutils-stdbuf" +# Split stdbuf into its own package, so one can include +# coreutils-stdbuf without getting the rest of coreutils, but make +# coreutils itself pull in stdbuf, so IMAGE_INSTALL += "coreutils" +# always provides all coreutils +PACKAGE_BEFORE_PN_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" FILES_coreutils-stdbuf = "${bindir}/stdbuf ${libdir}/coreutils/libstdbuf.so" -RDEPENDS_coreutils_class-target += "coreutils-stdbuf" +RDEPENDS_coreutils_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" + +# However, when the single-binary PACKAGECONFIG is used, stdbuf +# functionality is built into the single coreutils binary, so there's +# no point splitting /usr/bin/stdbuf to its own package. Instead, add +# an RPROVIDE so that rdepending on coreutils-stdbuf will work +# regardless of whether single-binary is in effect. +RPROVIDES_coreutils += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', 'coreutils-stdbuf', '', d)}" # Let aclocal use the relative path for the m4 file rather than the # absolute since coreutils has a lot of m4 files, otherwise there might -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140465): https://lists.openembedded.org/g/openembedded-core/message/140465 Mute This Topic: https://lists.openembedded.org/mt/75392946/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] coreutils: don't split stdbuf to own package with single-binary
Commit 992cec44 (coreutils: Move stdbuf into an own package coreutils-stdbuf) breaks package-qa when the single-binary PACKAGECONFIG is used: ERROR: coreutils-8.32-r0 do_package_qa: QA Issue: /usr/bin/stdbuf contained in package coreutils-stdbuf requires /usr/bin/coreutils, but no providers found in RDEPENDS_coreutils-stdbuf? [file-rdeps] ERROR: coreutils-8.32-r0 do_package_qa: QA run found fatal errors. Please consider fixing them. With that PACKAGECONFIG, /usr/bin/stdbuf is just a simple "script" containing the single line #!/usr/bin/coreutils --coreutils-prog-shebang=stdbuf Since there's no point splitting stdbuf to its own package when all the functionality is in the single big coreutils binary anyway, fix this by not creating the separate stdbuf package for the single-binary case. But also make sure that the coreutils-stdbuf item always exists so recipes can always RDEPEND on coreutils-stdbuf. Signed-off-by: Rasmus Villemoes --- meta/recipes-core/coreutils/coreutils_8.32.bb | 15 +-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/meta/recipes-core/coreutils/coreutils_8.32.bb b/meta/recipes-core/coreutils/coreutils_8.32.bb index 2422f8634e..f8ae3fd94c 100644 --- a/meta/recipes-core/coreutils/coreutils_8.32.bb +++ b/meta/recipes-core/coreutils/coreutils_8.32.bb @@ -59,9 +59,20 @@ base_bindir_progs = "cat chgrp chmod chown cp date dd echo false hostname kill l sbindir_progs= "chroot" -PACKAGE_BEFORE_PN_class-target += "coreutils-stdbuf" +# Split stdbuf into its own package, so one can include +# coreutils-stdbuf without getting the rest of coreutils, but make +# coreutils itself pull in stdbuf, so IMAGE_INSTALL += "coreutils" +# always provides all coreutils +PACKAGE_BEFORE_PN_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" FILES_coreutils-stdbuf = "${bindir}/stdbuf ${libdir}/coreutils/libstdbuf.so" -RDEPENDS_coreutils_class-target += "coreutils-stdbuf" +RDEPENDS_coreutils_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" + +# However, when the single-binary PACKAGECONFIG is used, stdbuf +# functionality is built into the single coreutils binary, so there's +# no point splitting /usr/bin/stdbuf to its own package. Instead, add +# an RPROVIDE so that rdepending on coreutils-stdbuf will work +# regardless of whether single-binary is in effect. +RPROVIDES_coreutils += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', 'coreutils-stdbuf', '', d)}" # Let aclocal use the relative path for the m4 file rather than the # absolute since coreutils has a lot of m4 files, otherwise there might -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140322): https://lists.openembedded.org/g/openembedded-core/message/140322 Mute This Topic: https://lists.openembedded.org/mt/75328937/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH][dunfell] coreutils: don't split stdbuf to own package with single-binary
On 04/07/2020 23.10, Adrian Bunk wrote: > On Fri, Jul 03, 2020 at 11:48:36AM +0100, Richard Purdie wrote: >> ... >> I hadn't realised there was already the reverse dependency. I think we >> need to let the package exist in both cases so the two dependencies >> need to reverse direction depending on whether single-binary is set. >> ... > > What about no separate coreutils-stdbuf package with single-binary, > instead an RPROVIDES of coreutils-stdbuf? That would probably work. That's essentially the first patch, plus a RPROVIDES_coreutils += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', 'coreutils-stdbuf', '', d)}" line, right? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140320): https://lists.openembedded.org/g/openembedded-core/message/140320 Mute This Topic: https://lists.openembedded.org/mt/75275273/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH][dunfell] coreutils: don't split stdbuf to own package with single-binary
On 03/07/2020 12.48, Richard Purdie wrote: >>> Whilst I realise there is a problem here is the correct fix not: >>> >>> RDEPENDS_coreutils-stdbuf_class-target += >>> "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils', >>> d)}" >>> >>> ? >> >> [Well, the coreutils should be in the true branch of that.] I dunno, it >> creates a cyclic rdepends between coreutils and coreutils-stdbuf. Is >> that ok? Seems a bit ugly to me, even if it would work. > > I hadn't realised there was already the reverse dependency. I think we > need to let the package exist in both cases so the two dependencies > need to reverse direction depending on whether single-binary is set. > > So something like: > > RDEPENDS_coreutils_class-target += "${@bb.utils.contains('PACKAGECONFIG', > 'single-binary', '', 'coreutils-stdbuf', d)}" > RDEPENDS_coreutils-stdbuf_class-target += > "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', 'coreutils', '', d)}" > > ? The argument for creating the coreutils->coreutils-stdbuf RDEPENDS was that people would get the same stuff in their rootfs as they used to (but they could also just get stdbuf if they needed that but use busybox for all the rest). The above would mean people who set single-binary and include coreutils in their rootfs no longer get the stdbuf utility (the functionality is in the binary, but they no longer get the /usr/bin/stdbuf wrapper script). But I suppose few enough people use single-binary (or the qa issue would surely have surfaced sometime in the past half year) that we can live with having to explicitly add coreutils-stdbuf. But then perhaps we can just unconditionally drop the coreutils->coreutils-stdbuf dependency, and have anybody who need stdbuf include it explicitly? In general, I do support splitting things to their own packages without automatically pulling them back in by the main package; it can be a little annoying for downstreams, but surely everybody have tests that ensure the utilities they need are in their images. [I'm not being sarcastic.] Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140280): https://lists.openembedded.org/g/openembedded-core/message/140280 Mute This Topic: https://lists.openembedded.org/mt/75275273/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH][dunfell] coreutils: don't split stdbuf to own package with single-binary
On 03/07/2020 11.19, Richard Purdie wrote: > On Fri, 2020-07-03 at 09:36 +0200, Rasmus Villemoes wrote: >> Commit 992cec44 (coreutils: Move stdbuf into an own package >> coreutils-stdbuf) breaks package-qa when the single-binary >> PACKAGECONFIG is used: >> >> ERROR: coreutils-8.31-r0 do_package_qa: QA Issue: /usr/bin/stdbuf >> contained in package coreutils-stdbuf requires /usr/bin/coreutils, >> but no providers found in RDEPENDS_coreutils-stdbuf? [file-rdeps] >> ERROR: coreutils-8.31-r0 do_package_qa: QA run found fatal errors. >> Please consider fixing them. > > Does this problem exist for other packages in coreutils? What other packages? stdbuf is the only one being split to its own package: coreutils/8.31-r0$ ls -l packages-split/ total 32 drwxr-xr-x 4 ravi abcdef 4096 Jul 3 09:49 coreutils drwxr-xr-x 3 ravi abcdef 4096 Jul 3 09:50 coreutils-dbg drwxr-xr-x 2 ravi abcdef 4096 Jul 3 09:49 coreutils-dev drwxr-xr-x 3 ravi abcdef 4096 Jul 3 09:49 coreutils-doc drwxr-xr-x 2 ravi abcdef 4096 Jul 3 09:49 coreutils-locale -rw-r--r-- 1 ravi abcdef 48 Jul 3 09:49 coreutils.shlibdeps drwxr-xr-x 3 ravi abcdef 4096 Jul 3 09:50 coreutils-src drwxr-xr-x 2 ravi abcdef 4096 Jul 3 09:49 coreutils-staticdev > I'd suspect it > does in which case why is stdbuf special? Because it's the only util that gets special packaging treatment. Unless there's some magic 'please auto-split all utils to their own packages' that I don't know about and which we don't happen to set. > Whilst I realise there is a problem here is the correct fix not: > > RDEPENDS_coreutils-stdbuf_class-target += > "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils', d)}" > > ? [Well, the coreutils should be in the true branch of that.] I dunno, it creates a cyclic rdepends between coreutils and coreutils-stdbuf. Is that ok? Seems a bit ugly to me, even if it would work. > As Alex says, this would need to be merged in master before we can even > consider it for dunfell. I'll do it for master once I know which way you want to fix it. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140256): https://lists.openembedded.org/g/openembedded-core/message/140256 Mute This Topic: https://lists.openembedded.org/mt/75275273/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH][dunfell] coreutils: don't split stdbuf to own package with single-binary
Commit 992cec44 (coreutils: Move stdbuf into an own package coreutils-stdbuf) breaks package-qa when the single-binary PACKAGECONFIG is used: ERROR: coreutils-8.31-r0 do_package_qa: QA Issue: /usr/bin/stdbuf contained in package coreutils-stdbuf requires /usr/bin/coreutils, but no providers found in RDEPENDS_coreutils-stdbuf? [file-rdeps] ERROR: coreutils-8.31-r0 do_package_qa: QA run found fatal errors. Please consider fixing them. ERROR: Logfile of failure stored in: /mnt/ext4/projects/deif/pdsystems/yocto/tmp-glibc/work/ppce300c3-oe-linux/coreutils/8.31-r0/temp/log.do_package_qa.5594 ERROR: Task (/mnt/ext4/projects/deif/pdsystems/yocto/meta-poky/meta/recipes-core/coreutils/coreutils_8.31.bb:do_package_qa) failed with exit code '1' With that PACKAGECONFIG, /usr/bin/stdbuf is just a simple "script" containing the single line #!/usr/bin/coreutils --coreutils-prog-shebang=stdbuf Since there's no point splitting stdbuf to its own package when all the functionality is in the single big coreutils binary anyway, fix this by not creating the separate stdbuf package for the single-binary case. Signed-off-by: Rasmus Villemoes --- The same issue exists in master, which has been bumped to 8.32, but other than that the patch should apply cleanly there as well. meta/recipes-core/coreutils/coreutils_8.31.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/recipes-core/coreutils/coreutils_8.31.bb b/meta/recipes-core/coreutils/coreutils_8.31.bb index 57b2c1bdba..80e8161367 100644 --- a/meta/recipes-core/coreutils/coreutils_8.31.bb +++ b/meta/recipes-core/coreutils/coreutils_8.31.bb @@ -58,9 +58,9 @@ base_bindir_progs = "cat chgrp chmod chown cp date dd echo false hostname kill l sbindir_progs= "chroot" -PACKAGE_BEFORE_PN_class-target += "coreutils-stdbuf" +PACKAGE_BEFORE_PN_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" FILES_coreutils-stdbuf = "${bindir}/stdbuf ${libdir}/coreutils/libstdbuf.so" -RDEPENDS_coreutils_class-target += "coreutils-stdbuf" +RDEPENDS_coreutils_class-target += "${@bb.utils.contains('PACKAGECONFIG', 'single-binary', '', 'coreutils-stdbuf', d)}" # Let aclocal use the relative path for the m4 file rather than the # absolute since coreutils has a lot of m4 files, otherwise there might -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140251): https://lists.openembedded.org/g/openembedded-core/message/140251 Mute This Topic: https://lists.openembedded.org/mt/75275273/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] file checksums versus SRC_URI globs
On 01/07/2020 16.03, Quentin Schulz wrote: > Hi Rasmus, > > On Wed, Jul 01, 2020 at 03:51:19PM +0200, Rasmus Villemoes wrote: >> Hi, >> >> We have a recipe that uses >> >> SRC_URI += "file://somedir/*" >> > > Glob aren't supported. Use "file://somedir/" instead. Thanks, that actually works for one of the cases we have (there are others that use globs which cannot be solved quite that simply, but for now I'm just listing files explicitly instead). However, I'm not sure that "globs aren't supported". The commit I referenced clearly tried to make that work (better), it also "works" in the sense of unpacking the expected things when building from scratch - there's even def test_local_wildcard(self): tree = self.fetchUnpack(['file://a', 'file://dir/*']) self.assertEqual(tree, ['a', 'dir/c', 'dir/d', 'dir/subdir/e']) in bitbake/lib/bb/tests/fetch.py. And the two upstream recipes connman-gnome_0.7.bb and matchbox-desktop_2.2.bb both use that exact pattern. So either - this is a plain bug in the signature computation, - documentation lacks a big fat warning that using globs in file:// should be avoided unless one has a fetish for tracking down bugs due to confused sstate, or - all code that makes glob patterns work should be ripped out so that it is indeed not supported, thus eliminating this footgun. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140215): https://lists.openembedded.org/g/openembedded-core/message/140215 Mute This Topic: https://lists.openembedded.org/mt/75235594/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] file checksums versus SRC_URI globs
Hi, We have a recipe that uses SRC_URI += "file://somedir/*" and we have noticed that changing one of the files inside somedir does not cause the recipe to be rebuilt - in fact, no task hashes are affected; we discovered this because a clean build worked correctly, while one that uses an sstate mirror would end up with an old version of those files on the target. Digging around, it _seems_ that this should be supported, e.g. commit bb66113bde5361b869dce2bdaece5b938f077ea8 Author: Richard Purdie Date: Sun Apr 6 11:08:22 2014 +0100 bitbake: fetch2: Fix bug in file checksum generation However, it also seems that something since then has broken the checksumming of files specified via a glob pattern. At least if I take the recipe mentioned in that commit and run "bitbake-dumpsig -t connman-gnome do_fetch", I get Variable SRC_URI value is git://github.com/connectivity/connman-gnome.git file://0001-Removed-icon-from-connman-gnome-about-applet.patch file://null_check_for_ipv4_config.patchfile://images/* file://connman-gnome-fix-dbus-interface-name.patch file://0001-Port-to-Gtk3.patch ... This task depends on the checksums of files: [('0001-Removed-icon-from-connman-gnome-about-applet.patch', '4c1b4d89f92949f34d5c232d85398bb4'), ('connman-gnome-fix-dbus-interface-name.patch', '51f2ebbc91ec35184876dfdf8aec35fd'), ('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'), ('0001-Port-to-Gtk3.patch', 'fa8ffcd77c330fd5063c7a7c62dd7c69')] i.e., none of the files in images/ are mentioned, contrary to what was apparently the case back in 2014. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140182): https://lists.openembedded.org/g/openembedded-core/message/140182 Mute This Topic: https://lists.openembedded.org/mt/75235594/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] curl: add debug info
Currently, curl (and libcurl) is built without debug info, making the curl-dbg package rather useless. Since debug symbols are automatically stripped and put in that package by the build system, making sure that curl is built with -g shouldn't hurt anything, but will help those that try to debug a libcurl-using application and hence explicitly include curl-dbg in their rootfs. Unfortunately, setting --enable-debug then changes the default value of the optimize option from (assume yes) to (assume no), while also changing the default value of the curldebug option [which is a separate thing that actually changes generated code to add some memory tracking] from (assume no) to (assume yes). So explicitly pass the appropriate options that make those two have the same value as they used to have by default. Signed-off-by: Rasmus Villemoes --- meta/recipes-support/curl/curl_7.70.0.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-support/curl/curl_7.70.0.bb b/meta/recipes-support/curl/curl_7.70.0.bb index baf72f8e75..bb25677ef7 100644 --- a/meta/recipes-support/curl/curl_7.70.0.bb +++ b/meta/recipes-support/curl/curl_7.70.0.bb @@ -57,6 +57,9 @@ EXTRA_OECONF = " \ --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ --without-libmetalink \ --without-libpsl \ +--enable-debug \ +--enable-optimize \ +--disable-curldebug \ " do_install_append_class-target() { -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139587): https://lists.openembedded.org/g/openembedded-core/message/139587 Mute This Topic: https://lists.openembedded.org/mt/74933778/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
On 04/06/2020 22.05, Andre McCurdy wrote: > but I can try to summarise the details related to the actual patch... [snip] > There's a genuine bug and I don't see an obviously better fix, so I > think the patch should be merged. Thank you for that summary, that's much better than what I could have written, and also thanks for the conclusion. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139258): https://lists.openembedded.org/g/openembedded-core/message/139258 Mute This Topic: https://lists.openembedded.org/mt/74625854/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
On 03/06/2020 22.44, Andre McCurdy wrote: > On Wed, Jun 3, 2020 at 12:38 AM Rasmus Villemoes > wrote: >> On 02/06/2020 23.46, Andre McCurdy wrote: >>> On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via >>> lists.openembedded.org wrote: >>>> >>> >>> It seems like a weird corner case to me. Where do these libraries come >>> from and why can't they be moved or symlinked into a standard path? >> >> Pre-compiled proprietary binaries/libraries that go in >> /opt//..., just as /opt is meant for (and no, we can't just >> install them to /usr/lib, /usr/bin etc - partly because they contain >> hard-coded absolute paths, but also due to some not-entirely-techical >> reasons). So there's an /etc/ld.so.conf.d/.conf that needs to be >> picked up. >> >>> If they are somehow special and only used by special applications then >>> setting rpath or using LD_LIBRARY_PATH just for those applications >>> would seem to be a better solution than enabling them globally. >> >> We don't control the compilation, so rpath is out. We did consider >> LD_LIBRARY_PATH, but figured that the ld.conf solution is more robust >> (no problems with setuid binaries or an application that misguidedly >> sanitizes the environment for subprocesses). > > Unless the supplier of your binaries is completely unresponsive (if > so, bad luck) then educating them on the usage of rpath and asking for > a new release is probably the cleanest solution. For various reasons that is not a viable path. > If you start your special application via a wrapper script which sets > LD_LIBRARY_PATH then it won't be seen by anything else, so the > concerns about other apps messing with it etc should not apply? "Other apps" could be the apps in /opt themselves, that start other binaries from under /opt, perhaps via some maze of shell scripts or whatnot (the usual mess one sees with proprietary stuff) - which is also the reason I'm worried they might have done some "value-add" sanitization somewhere. > Relying on /etc/ld.so.conf.d/*.conf will certainly work too, but has > two disadvantages: it will add the custom library path globally > instead of just for the special apps and it's not portable (it's not > supported by musl etc - although if you only care about your > proprietary binaries pre-compiled for glibc then that won't matter to > you). One could also ask the question: If one is not supposed to use the drop-in dir ld.so.conf.d/ for exactly what I'm using it for, then why does Yocto provide a ld.so.conf that mentions it? Sure, for glibc-based distros (of course, it's from the glibc recipe), but that's fine. Please remember that this is all just about ensuring that the ldconfig run at image build time produces the same result, regardless of whether one has opted to have ldconfig-the-binary in the rootfs or not (and everybody seems to agree there are valid reasons not to have that). > Anyway, the original patch has been merged, so this now just a side > discussion. Indeed. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139206): https://lists.openembedded.org/g/openembedded-core/message/139206 Mute This Topic: https://lists.openembedded.org/mt/74625854/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
On 02/06/2020 23.46, Andre McCurdy wrote: > On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via > lists.openembedded.org wrote: >> >> On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote: >>> I understand the concern, I am a little torn on this as adding in too >>> many different controls and options complicates the test matrix and >>> makes things harder for users. >>> >>> You're effectively suggesting a new DISTRO_FEATURE to control this? or >>> maybe better, perhaps a glibc PACKAGECONFIG? >> >> Yes, right. I don't think it need be a DISTRO_FEATURE because nothing >> outside glibc needs to care. >> >> Here are the scenarios that I think matter, ordered from simplest to >> most complex: >> >> 1. Immutable filesystem, every library installed in the place where >> ld.so would first look for it anyway (i.e. everything in {/usr}/lib). >> In this case, we don't want ldconfig (because it can never do anything >> useful), we don't want ld.so.cache and we don't want ld.so.conf >> because they would cause ld.so to do extra file loads and computation >> but end up with the same result that it would anyway. >> >> 2. Immutable filesystem but some libraries are in places that ld.so >> wouldn't automatically know about. In this case we do want ld.so.conf >> and ld.so.cache, but we still don't want ldconfig. >> >> 3. Mutable filesystem where arbitrary binaries can be installed in >> arbitrary places. It's probably debatable whether ldconfig is needed >> in all these cases, but clearly it's needed in some and I think at >> this point it can be left to a DISTRO decision how exactly they want >> to optimize things. >> >> I think right now oe-core supports #1 and #3. The proposed patch >> seems to be aimed at #2, which is a completely valid usecase, > > It seems like a weird corner case to me. Where do these libraries come > from and why can't they be moved or symlinked into a standard path? Pre-compiled proprietary binaries/libraries that go in /opt//..., just as /opt is meant for (and no, we can't just install them to /usr/lib, /usr/bin etc - partly because they contain hard-coded absolute paths, but also due to some not-entirely-techical reasons). So there's an /etc/ld.so.conf.d/.conf that needs to be picked up. > If they are somehow special and only used by special applications then > setting rpath or using LD_LIBRARY_PATH just for those applications > would seem to be a better solution than enabling them globally. We don't control the compilation, so rpath is out. We did consider LD_LIBRARY_PATH, but figured that the ld.conf solution is more robust (no problems with setuid binaries or an application that misguidedly sanitizes the environment for subprocesses). Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139149): https://lists.openembedded.org/g/openembedded-core/message/139149 Mute This Topic: https://lists.openembedded.org/mt/74625854/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
On 02/06/2020 23.15, Phil Blundell wrote: > On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote: >> I understand the concern, I am a little torn on this as adding in too >> many different controls and options complicates the test matrix and >> makes things harder for users. >> >> You're effectively suggesting a new DISTRO_FEATURE to control this? or >> maybe better, perhaps a glibc PACKAGECONFIG? > > Yes, right. I don't think it need be a DISTRO_FEATURE because nothing > outside glibc needs to care. > > Here are the scenarios that I think matter, ordered from simplest to > most complex: > > 1. Immutable filesystem, every library installed in the place where > ld.so would first look for it anyway (i.e. everything in {/usr}/lib). > In this case, we don't want ldconfig (because it can never do anything > useful), we don't want ld.so.cache and we don't want ld.so.conf > because they would cause ld.so to do extra file loads and computation > but end up with the same result that it would anyway. > > I think right now oe-core supports #1 and #3. The proposed patch > seems to be aimed at #2, which is a completely valid usecase, but > my concern is that we don't want to pessimize #1 in the process. With or without my patch, there's an ldconfig invocation at image build time, and lacking an /etc/ld.so.conf in the -r filesystem, ldconfig just looks at the default locations (/lib and /usr/lib, perhaps a few more depending on $arch). But you still get an ld.so.cache generated listing the libs in /lib and /usr/lib. So I don't understand this performance concern at all - at the very most, the presence of the config file causes a microscopic performance hit at image build time when ldconfig goes looking for the /etc/ld.so.conf.d/* files and (usually) find none. Also, IME, the cache file usually speed things up since ld.so avoids a filesystem access to figure out that /{usr,}/lib//libfoo.so doesn't exist (of course there's an upfront cost of mmapping and parsing the ld.so.cache, so for programs with very few dynlib dependencies, it might be faster to not have the cache). See below. If you really don't want the ld.so.cache, ldconfig has the -N option. But adding some knob to put that in is completely orthogonal to this patch. Rasmus Here's the first few lines of "strace -o /tmp/cache.txt factor 12345": execve("/usr/bin/factor", ["factor", "12345"], 0xbff6cf3c /* 14 vars */) = 0 brk(NULL) = 0x10173000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=6527, ...}) = 0 mmap2(NULL, 6527, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7ef2000 close(3)= 0 openat(AT_FDCWD, "/lib/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3 read(3, "\177ELF\1\2\1\3\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\36\300\0\0\0004"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=38596, ...}) = 0 mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ef mmap2(0xffdf000, 65768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffdf000 mprotect(0xffe7000, 28672, PROT_NONE) = 0 mmap2(0xffee000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0xffee000 close(3)= 0 openat(AT_FDCWD, "/lib/libcap.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3 read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\21P\0\0\0004"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=22060, ...}) = 0 mmap2(0xffbe000, 65820, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffbe000 mprotect(0xffc3000, 40960, PROT_NONE) = 0 mmap2(0xffcd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0xffcd000 close(3)= 0 openat(AT_FDCWD, "/usr/lib/libgmp.so.10", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3 read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\226 \0\0\0004"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=551740, ...}) = 0 mmap2(0xff1d000, 591268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xff1d000 mprotect(0xffa3000, 36864, PROT_NONE) = 0 mmap2(0xffac000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x85000) = 0xffac000 close(3)= 0 Compare that to what happens when I move the ld.so.cache out of the way: execve("/usr/bin/factor", ["factor", "12345"], 0xbfc7c3bc /* 14 vars */) = 0 brk(NULL) = 0x10173000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/lib/tls/ppc603/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat64("/lib/tls/ppc603", 0xbf98df78) = -1 ENOENT (No such file or
Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
On 02/06/2020 19.58, Khem Raj wrote: > On Tue, Jun 2, 2020 at 5:17 AM Rasmus Villemoes > wrote: >> >> There are cases where one doesn't want ldconfig on target (e.g. for >> read-only root filesystems, it's rather pointless), yet one still >> needs ld.so.conf to be present at image build time: >> >> When some recipe installs libraries to a non-standard location, and >> dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the >> ld.so.conf containing the >> >> include /etc/ld.so.conf.d/*.conf >> >> stanza to get those other locations picked up. >> >> So change the packaging logic so that there's always an ld.so.conf >> present when the build-time ldconfig runs. >> >> The ld.so.conf and ld.so.conf.d/*.conf files don't take up much >> room (at least not compared to the 700K binary ldconfig), and they >> might be needed in case ldconfig is installable, so leave them >> alone. >> >> In case of a read-only rootfs, one could add some logic to remove them >> if one really wants to shave those few dozens of bytes off. >> >> While here, fix typos in the bb.note (add spaces) so one can just >> copy-paste the line from the log-file and redo the command. >> > > The patch, in essence, looks ok, have you considered the feeds based > distro's case > will it keep working for them, or if there are changes e.g. they must > include ldconfig > compulsory etc. then it should be documented. I don't think I understand the question. All I'm doing (now) is making sure the rootfs always (well, it comes bundled with glibc, so for all glibc-based systems) has a ld.so.conf. If there's an update of that ld.so.conf, that will change the hash of the glibc recipe (it's a SRC_URI) and hence of both the glibc and ldconfig packages. A change of the provided ld.so.conf will lead to glibc being rebuilt (no?), which I expect in turn will cause glibc on target to (need to) be updated (no?). That seems to be the case with or without this patch. But maybe I have no clue what a "feeds based distro" is, I've only ever done wholesale replace-the-entire-image (in an A/B setup). Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139119): https://lists.openembedded.org/g/openembedded-core/message/139119 Mute This Topic: https://lists.openembedded.org/mt/74625854/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3] glibc: move ld.so.conf back to main package
There are cases where one doesn't want ldconfig on target (e.g. for read-only root filesystems, it's rather pointless), yet one still needs ld.so.conf to be present at image build time: When some recipe installs libraries to a non-standard location, and dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the ld.so.conf containing the include /etc/ld.so.conf.d/*.conf stanza to get those other locations picked up. So change the packaging logic so that there's always an ld.so.conf present when the build-time ldconfig runs. The ld.so.conf and ld.so.conf.d/*.conf files don't take up much room (at least not compared to the 700K binary ldconfig), and they might be needed in case ldconfig is installable, so leave them alone. In case of a read-only rootfs, one could add some logic to remove them if one really wants to shave those few dozens of bytes off. While here, fix typos in the bb.note (add spaces) so one can just copy-paste the line from the log-file and redo the command. Signed-off-by: Rasmus Villemoes --- v3: rebase to master v2: don't delete the configuration file(s) - that's also a much simpler patch. meta/lib/oe/rootfs.py | 2 +- meta/recipes-core/glibc/glibc-package.inc | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index cd65e62030..a0ac33ada6 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -297,7 +297,7 @@ class Rootfs(object, metaclass=ABCMeta): def _run_ldconfig(self): if self.d.getVar('LDCONFIGDEPEND'): -bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v") +bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v") self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c', 'new', '-v']) diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc index ff25fd4187..5f1ab60dc6 100644 --- a/meta/recipes-core/glibc/glibc-package.inc +++ b/meta/recipes-core/glibc/glibc-package.inc @@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1" libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}" INSANE_SKIP_${PN}_append_aarch64 = " libdir" -FILES_${PN} = "${libc_baselibs} ${libexecdir}/*" +FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf" RRECOMMENDS_${PN} = "${@bb.utils.contains('DISTRO_FEATURES', 'ldconfig', '${MLPREFIX}ldconfig', '', d)}" -FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf" +FILES_ldconfig = "${base_sbindir}/ldconfig" FILES_ldd = "${bindir}/ldd" FILES_libsegfault = "${base_libdir}/libSegFault*" FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*" -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139091): https://lists.openembedded.org/g/openembedded-core/message/139091 Mute This Topic: https://lists.openembedded.org/mt/74625854/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] glibc: move ld.so.conf back to main package
There are cases where one doesn't want ldconfig on target (e.g. for read-only root filesystems, it's rather pointless), yet one still needs ld.so.conf to be present at image build time: When some recipe installs libraries to a non-standard location, and dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the ld.so.conf containing the include /etc/ld.so.conf.d/*.conf stanza to get those other locations picked up. So change the packaging logic so that there's always an ld.so.conf present when the build-time ldconfig runs. The ld.so.conf and ld.so.conf.d/*.conf files don't take up much room (at least not compared to the 700K binary ldconfig), and they might be needed in case ldconfig is installable, so leave them alone. In case of a read-only rootfs, one could add some logic to remove them if one really wants to shave those few dozens of bytes off. While here, fix typos in the bb.note (add spaces) so one can just copy-paste the line from the log-file and redo the command. Signed-off-by: Rasmus Villemoes --- v2: don't delete the configuration file(s) - that's also a much simpler patch. meta/lib/oe/rootfs.py | 2 +- meta/recipes-core/glibc/glibc-package.inc | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index cd65e62030..a0ac33ada6 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -297,7 +297,7 @@ class Rootfs(object, metaclass=ABCMeta): def _run_ldconfig(self): if self.d.getVar('LDCONFIGDEPEND'): -bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v") +bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v") self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c', 'new', '-v']) diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc index aa8e059216..387e90a0ab 100644 --- a/meta/recipes-core/glibc/glibc-package.inc +++ b/meta/recipes-core/glibc/glibc-package.inc @@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1" libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}" INSANE_SKIP_${PN}_append_aarch64 = " libdir" -FILES_${PN} = "${libc_baselibs} ${libexecdir}/*" +FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf" RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}" -FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf" +FILES_ldconfig = "${base_sbindir}/ldconfig" FILES_ldd = "${bindir}/ldd" FILES_libsegfault = "${base_libdir}/libSegFault*" FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*" -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139087): https://lists.openembedded.org/g/openembedded-core/message/139087 Mute This Topic: https://lists.openembedded.org/mt/74624263/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] glibc: move ld.so.conf back to main package
On 18/05/2020 14.29, Andreas Oberritter wrote: > Hello Rasmus, > > On Mon, 18 May 2020 14:12:43 +0200 > Rasmus Villemoes wrote: > >> I'm certainly open to other ways of solving this. But can we agree that >> it is a bug that the ldconfig done at build-time does not take >> /etc/ld.so.conf.d/* into account, and that that should not depend on >> whether one has ldconfig-the-binary on target? > > have you tried installing ldconfig and adding it to ROOTFS_RO_UNNEEDED? It > might be an improvement to include it in ROOTFS_RO_UNNEEDED by default. Thanks, but I don't think that will work: self._uninstall_unneeded() if self.progress_reporter: self.progress_reporter.next_stage() self._insert_feed_uris() self._run_ldconfig() so if the ldconfig package (including the ld.so.conf file) is just added to ROOTFS_RO_UNNEEDED, it will be gone by the time we get to doing the build-time ld.so.cache generation. And I think it has to be done in this order - if some of the packages removed by _uninstall_unneeded remove shared libraries, one doesn't want stale entries in ld.so.cache referring to those. > Please consider use-cases where writable filesystems ship without ldconfig, > but a user installs it from a package feed when needed. Do you mean I should leave /etc/ld.so.conf.d/ alone? I can do that. But I'd say that also in that case the current behaviour is buggy - if any package in the rootfs ships with an /etc/ld.so.conf.d/ entry, that entry should be taken into account at image build time, regardless of whether ldconfig is there at image build time or can usefully be added later via a package installer. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#138408): https://lists.openembedded.org/g/openembedded-core/message/138408 Mute This Topic: https://lists.openembedded.org/mt/74289052/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] glibc: move ld.so.conf back to main package
On 18/05/2020 13.55, Martin Jansa wrote: > This won't work, because as soon as glibc is upgraded with package > management, the ldconfig files will be restored and there won't be any > do_rootfs hook to remove it. > > Why isn't ldconfig installed in your setup in first place? RRECOMMENDS > should be enough to pull it into every image by default. [At the risk of sounding like the lady in the Monty Python spam sketch:] I don't _want_ it in my setup - it's a gigantic 700K binary that is useless on a squashfs rootfs where for obvious reasons no new libraries (or new versions) will ever appear. So no, I don't have ldconfig in my distro features (so it's not even rrecommended), and I also don't install recommended packages. I just need a ld.so.cache built at image build time. Which works just fine, except that the drop-in dir gets ignored due to lack of ld.so.conf. After ld.so.cache has been built, the configuration files are welcome to go the way of the dodo. And I'm not removing the ldconfig files if ldconfig is in distro features (or if someone has explicitly RDEPEND'ed on the ldconfig package), so there's no issue of those files getting "restored" - if you have ldconfig on target, they never disappear, so them getting replaced during a package upgrade is not a problem. I'm certainly open to other ways of solving this. But can we agree that it is a bug that the ldconfig done at build-time does not take /etc/ld.so.conf.d/* into account, and that that should not depend on whether one has ldconfig-the-binary on target? Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#138403): https://lists.openembedded.org/g/openembedded-core/message/138403 Mute This Topic: https://lists.openembedded.org/mt/74289052/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] glibc: move ld.so.conf back to main package
There are cases where one doesn't want ldconfig on target (e.g. for read-only root filesystems, it's rather pointless), yet one still needs ld.so.conf to be present at image build time: When some recipe installs libraries to a non-standard location, and dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the ld.so.conf containing the include /etc/ld.so.conf.d/*.conf stanza to get those other locations picked up. So change the packaging logic so that there's always an ld.so.conf present when the build-time ldconfig runs, then delete it (and the .d directory) when they would serve no purpose at run-time. While here, fix a typo in the bb.note so one can just copy-paste the line from the log-file and redo the command. Signed-off-by: Rasmus Villemoes --- meta/lib/oe/rootfs.py | 11 ++- meta/recipes-core/glibc/glibc-package.inc | 4 ++-- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index cd65e62030..0ff4278d15 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -297,9 +297,18 @@ class Rootfs(object, metaclass=ABCMeta): def _run_ldconfig(self): if self.d.getVar('LDCONFIGDEPEND'): -bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v") +bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v") self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c', 'new', '-v']) +# If DISTRO_FEATURES doesn't contain ldconfig, and the +# ldconfig binary hasn't been pulled in explicitly, remove the +# configuration file(s) from the image. +keep_conf = bb.utils.contains("DISTRO_FEATURES", "ldconfig", True, False, self.d) +keep_conf = keep_conf or os.path.exists(os.path.join(self.image_rootfs, self.d.getVar('base_sbindir'), 'ldconfig')) +if not keep_conf: +sysconfdir = os.path.join(self.image_rootfs, self.d.getVar('sysconfdir')) +bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf')) +bb.utils.remove(os.path.join(sysconfdir, 'ld.so.conf.d'), recurse=True) def _check_for_kernel_modules(self, modules_dir): for root, dirs, files in os.walk(modules_dir, topdown=True): diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc index aa8e059216..387e90a0ab 100644 --- a/meta/recipes-core/glibc/glibc-package.inc +++ b/meta/recipes-core/glibc/glibc-package.inc @@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1" libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}" INSANE_SKIP_${PN}_append_aarch64 = " libdir" -FILES_${PN} = "${libc_baselibs} ${libexecdir}/*" +FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf" RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}" -FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf" +FILES_ldconfig = "${base_sbindir}/ldconfig" FILES_ldd = "${bindir}/ldd" FILES_libsegfault = "${base_libdir}/libSegFault*" FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*" -- 2.23.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#138401): https://lists.openembedded.org/g/openembedded-core/message/138401 Mute This Topic: https://lists.openembedded.org/mt/74289052/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] Hard-linking to u-boot source files considered harmful
I was wondering why my U-Boot build would always report as coming from a -dirty directory. The /git source directory was completely clean and matched the desired commit. But the -dirty is appended by the scripts/setlocalversion (linux uses the same script) if git diff-index --name-only HEAD reports anything (mind you, you have to run that command before running "git status" - the latter will update git's index so the diff-index may no longer print anything). Indeed, that prints Licenses/gpl-2.0.txt The reason diff-index thinks that might be changed is that, since the git checkout, that file has had its nlinks (and consequently ctime) modified: File: Licenses/gpl-2.0.txt Size: 18092 Blocks: 40 IO Block: 4096 regular file Device: 811h/2065d Inode: 41445215Links: 3 Access: (0644/-rw-r--r--) Uid: ( 1000/ravi) Gid: ( 1001/ abcdef) Access: 2020-05-01 12:04:47.708081401 +0200 Modify: 2020-05-01 12:04:45.756078017 +0200 Change: 2020-05-01 12:04:48.032081963 +0200 Birth: - And who creates those extra links? do_populate_lic (and perhaps some other task later yet). Since that is just "after patch before build", it is almost certain to get run before the compile task gets around to invoking setlocalversion and thus create the include/config/uboot.release file. A crude hack that just forces do_populate_lic after do_install indeed makes the -dirty go away. Or, an even cruder hack, prepending a "git status" done in ${S} to do_compile causes git to refresh its index. I have tons of ways of working around this, but the question is, is the optimization of using hard links really worth it? If it is, can we at least have a "do not ever create hard links to stuff under ${S}" per-recipe variable, defaulting to 1 if the "primary" SRC_URI (yeah, that concept probably doesn't exist) is a git:// one? do_populate_lic is not the only offender, btw., there's apparently also some create-debug-package-with-sources that will do the same thing, but I think that will by its nature always run after any configure/compile step that uses git to create some version. Rasmus -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#137702): https://lists.openembedded.org/g/openembedded-core/message/137702 Mute This Topic: https://lists.openembedded.org/mt/73394646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/leave/8023207/1426099254/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2] glibc: don't ignore global LDFLAGS settings
ping On 20/01/2020 16.42, Khem Raj wrote: > this patch is ok but I have reservations since += now means global > ldflags will be applied > so it would need some testing to ensure it works well. > > On Mon, Jan 20, 2020 at 1:23 AM Rasmus Villemoes > wrote: >> >> Two things: The -Wl,-rpath-link comment is stale (due to per-recipe >> staging), so we no longer need to set our own LDFLAGS to avoid >> -Wl,-rpath-link being in there. Second, overriding LDFLAGS should be >> done at the recipe level so "bitbake -e" can show what is going on. >> Otherwise debugging why one's global LDFLAGS tweaks are being ignored >> is needlessly painful. >> >> So pull out the LDFLAGS setting from do_compile, and change it to an >> append instead of assignment. >> >> For the benefit of future git blame: the -fuse-ld=bfd setting was >> added by ac64c3b96b (glibc: always use bfd linker). >> >> Signed-off-by: Rasmus Villemoes >> --- >> v2: Rebase to real upstream master. Note to self: "git pull" before rebasing >> to master. >> >> meta/recipes-core/glibc/glibc_2.31.bb | 3 +-- >> 1 file changed, 1 insertion(+), 2 deletions(-) >> >> diff --git a/meta/recipes-core/glibc/glibc_2.31.bb >> b/meta/recipes-core/glibc/glibc_2.31.bb >> index b8c570db52..cfba4de49b 100644 >> --- a/meta/recipes-core/glibc/glibc_2.31.bb >> +++ b/meta/recipes-core/glibc/glibc_2.31.bb >> @@ -93,9 +93,8 @@ do_configure () { >> CPPFLAGS="" oe_runconf >> } >> >> +LDFLAGS += "-fuse-ld=bfd" >> do_compile () { >> - # -Wl,-rpath-link /lib in LDFLAGS can cause breakage if >> another glibc is in staging >> - LDFLAGS="-fuse-ld=bfd" >> base_do_compile >> echo "Adjust ldd script" >> if [ -n "${RTLDLIST}" ] >> -- >> 2.23.0 >> -- Rasmus Villemoes Software Developer Prevas A/S Hedeager 3 DK-8200 Aarhus N +45 51210274 rasmus.villem...@prevas.dk www.prevas.dk -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2] glibc: don't ignore global LDFLAGS settings
On 20/01/2020 16.42, Khem Raj wrote: > this patch is ok but I have reservations since += now means global > ldflags will be applied Indeed, that's half the point of the patch. > so it would need some testing to ensure it works well. Yes, but I don't think there's any way to figure out if it breaks something other than trying it. If it does, I can certainly make it just LDFLAGS = "..." outside do_compile, and then I'll have to .bbappend or _pn-glibc_append my custom LDFLAGS, which is ok for a single rather special recipe, but would be a hassle to maintain if this was a common thing. Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2] glibc: don't ignore global LDFLAGS settings
Two things: The -Wl,-rpath-link comment is stale (due to per-recipe staging), so we no longer need to set our own LDFLAGS to avoid -Wl,-rpath-link being in there. Second, overriding LDFLAGS should be done at the recipe level so "bitbake -e" can show what is going on. Otherwise debugging why one's global LDFLAGS tweaks are being ignored is needlessly painful. So pull out the LDFLAGS setting from do_compile, and change it to an append instead of assignment. For the benefit of future git blame: the -fuse-ld=bfd setting was added by ac64c3b96b (glibc: always use bfd linker). Signed-off-by: Rasmus Villemoes --- v2: Rebase to real upstream master. Note to self: "git pull" before rebasing to master. meta/recipes-core/glibc/glibc_2.31.bb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/recipes-core/glibc/glibc_2.31.bb b/meta/recipes-core/glibc/glibc_2.31.bb index b8c570db52..cfba4de49b 100644 --- a/meta/recipes-core/glibc/glibc_2.31.bb +++ b/meta/recipes-core/glibc/glibc_2.31.bb @@ -93,9 +93,8 @@ do_configure () { CPPFLAGS="" oe_runconf } +LDFLAGS += "-fuse-ld=bfd" do_compile () { - # -Wl,-rpath-link /lib in LDFLAGS can cause breakage if another glibc is in staging - LDFLAGS="-fuse-ld=bfd" base_do_compile echo "Adjust ldd script" if [ -n "${RTLDLIST}" ] -- 2.23.0 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] glibc: don't ignore global LDFLAGS settings
Two things: The -Wl,-rpath-link comment is stale (due to per-recipe staging), so we no longer need to set our own LDFLAGS to avoid -Wl,-rpath-link being in there. Second, overriding LDFLAGS should be done at the recipe level so "bitbake -e" can show what is going on. Otherwise debugging why one's global LDFLAGS tweaks are being ignored is needlessly painful. So pull out the LDFLAGS setting from do_compile, and change it to an append instead of assignment. For the benefit of future git blame: the -fuse-ld=bfd setting was added by ac64c3b96b (glibc: always use bfd linker). Signed-off-by: Rasmus Villemoes --- meta/recipes-core/glibc/glibc_2.30.bb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/recipes-core/glibc/glibc_2.30.bb b/meta/recipes-core/glibc/glibc_2.30.bb index 7913bc2812..c6dd78ca8b 100644 --- a/meta/recipes-core/glibc/glibc_2.30.bb +++ b/meta/recipes-core/glibc/glibc_2.30.bb @@ -93,9 +93,8 @@ do_configure () { CPPFLAGS="" oe_runconf } +LDFLAGS += "-fuse-ld=bfd" do_compile () { - # -Wl,-rpath-link /lib in LDFLAGS can cause breakage if another glibc is in staging - LDFLAGS="-fuse-ld=bfd" base_do_compile echo "Adjust ldd script" if [ -n "${RTLDLIST}" ] -- 2.23.0 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] glibc and LDFLAGS
I've added a custom setting to TARGET_LDFLAGS, but then I noticed that the glibc build ignored that. The glibc do_compile did LDFLAGS = "" for a very long time, with the explanation that "# -Wl,-rpath-link /lib in LDFLAGS can cause breakage if another glibc is in staging". Is that actually still relevant with per-recipe staging? If that is indeed moot, can we do CPPFLAGS="" oe_runconf } +LDFLAGS += "-fuse-ld=bfd" do_compile () { - # -Wl,-rpath-link /lib in LDFLAGS can cause breakage if another glibc is in staging - LDFLAGS="-fuse-ld=bfd" base_do_compile echo "Adjust ldd script" if [ -n "${RTLDLIST}" ] ? And if not, at least move the LDFLAGS definition out to recipe level so "bitbake -e" will show the value actually in effect? (And that would also allow a .bbappend to tweak glibc's LDFLAGS with my custom setting, though it's annoying to have to maintain that, so the best is if glibc can just be built with the global LDFLAGS setting). Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2] iproute2: drop pointless configure-cross.patch
The configure script has already found IPT_LIB_DIR via pkgconfig, so the configure-cross.patch really just introduces dead and broken fallback code. Broken, because the SYSROOT variable does not actually get set to a sensible value - the argument $1 passed to the configure script when invoked from do_configure is ${STAGING_INCDIR}. Obviously that directory does not have /lib or /usr subdirectories, so we're not really helping the fallback logic in check_ipt_lib_dir() - in fact, we're more or less guaranteeing that we won't find those .so files. Signed-off-by: Rasmus Villemoes --- v2: apply to master instead of warrior, update patch description .../iproute2/iproute2/configure-cross.patch | 39 --- .../iproute2/iproute2_5.2.0.bb| 1 - 2 files changed, 40 deletions(-) delete mode 100644 meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch diff --git a/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch b/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch deleted file mode 100644 index b0c4a088fa..00 --- a/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch +++ /dev/null @@ -1,39 +0,0 @@ -From 3835b1cc4a55361443c2b37dea688da652217635 Mon Sep 17 00:00:00 2001 -From: Koen Kooi -Date: Sun, 31 Mar 2019 17:32:07 +0200 -Subject: [PATCH] make configure cross compile safe - -According to Kevin Tian: -Upstream-Status: Pending - -Signed-off-by: Koen Kooi -Signed-off-by: Shane Wang - - configure | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -diff --git a/configure b/configure -index 45fcffb6..0c5c9146 100755 a/configure -+++ b/configure -@@ -3,6 +3,7 @@ - # This is not an autoconf generated configure - # - INCLUDE=${1:-"$PWD/include"} -+SYSROOT=$1 - - # Output file which is input to Makefile - CONFIG=config.mk -@@ -162,7 +163,7 @@ check_ipt_lib_dir() - return - fi - -- for dir in /lib /usr/lib /usr/local/lib; do -+ for dir in $SYSROOT/lib $SYSROOT/usr/lib $SYSROOT/usr/local/lib; do - for file in "xtables" "iptables"; do - file="$dir/$file/lib*t_*so" - if [ -f $file ]; then --- -2.17.1 - diff --git a/meta/recipes-connectivity/iproute2/iproute2_5.2.0.bb b/meta/recipes-connectivity/iproute2/iproute2_5.2.0.bb index 8ef529d91a..1728cd69a1 100644 --- a/meta/recipes-connectivity/iproute2/iproute2_5.2.0.bb +++ b/meta/recipes-connectivity/iproute2/iproute2_5.2.0.bb @@ -1,7 +1,6 @@ require iproute2.inc SRC_URI = "${KERNELORG_MIRROR}/linux/utils/net/${BPN}/${BP}.tar.xz \ - file://configure-cross.patch \ file://0001-libc-compat.h-add-musl-workaround.patch \ " -- 2.20.1 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] iproute2: drop pointless configure-cross.patch
The configure script has already found IPT_LIB_DIR via pkgconfig, so the configure-cross.patch really just introduces dead and broken fallback code. Broken, because the SYSROOT variable does not actually get set to a sensible value - the argument $1 passed to the configure script when invoked from the Makefile is KERNEL_INCLUDE, which we set to ${STAGING_INCDIR} in EXTRA_OEMAKE. Obviously that directory does not have /lib or /usr subdirectories, so we're not really helping the fallback logic in check_ipt_lib_dir() - in fact, we're more or less guaranteeing that we won't find those .so files. Signed-off-by: Rasmus Villemoes --- .../iproute2/iproute2/configure-cross.patch | 32 --- .../iproute2/iproute2_4.19.0.bb | 1 - 2 files changed, 33 deletions(-) delete mode 100644 meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch diff --git a/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch b/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch deleted file mode 100644 index 8b75a2ada4..00 --- a/meta/recipes-connectivity/iproute2/iproute2/configure-cross.patch +++ /dev/null @@ -1,32 +0,0 @@ -From 85b0589b4843c03e8e6fd9416d71ea449a73c5c0 Mon Sep 17 00:00:00 2001 -From: Koen Kooi -Date: Thu, 3 Nov 2011 10:46:16 +0100 -Subject: [PATCH] make configure cross compile safe - -According to Kevin Tian: -Upstream-Status: Pending - -Signed-off-by: Koen Kooi -Signed-off-by: Shane Wang - -Index: iproute2-4.14.1/configure -=== iproute2-4.14.1.orig/configure -+++ iproute2-4.14.1/configure -@@ -2,6 +2,7 @@ - # This is not an autoconf generated configure - # - INCLUDE=${1:-"$PWD/include"} -+SYSROOT=$1 - - # Output file which is input to Makefile - CONFIG=config.mk -@@ -195,7 +196,7 @@ check_ipt_lib_dir() - return - fi - -- for dir in /lib /usr/lib /usr/local/lib -+ for dir in $SYSROOT/lib $SYSROOT/usr/lib $SYSROOT/usr/local/lib - do - for file in $dir/{xtables,iptables}/lib*t_*so ; do - if [ -f $file ]; then diff --git a/meta/recipes-connectivity/iproute2/iproute2_4.19.0.bb b/meta/recipes-connectivity/iproute2/iproute2_4.19.0.bb index 6db4062d68..b1db40885a 100644 --- a/meta/recipes-connectivity/iproute2/iproute2_4.19.0.bb +++ b/meta/recipes-connectivity/iproute2/iproute2_4.19.0.bb @@ -1,7 +1,6 @@ require iproute2.inc SRC_URI = "${KERNELORG_MIRROR}/linux/utils/net/${BPN}/${BP}.tar.xz \ - file://configure-cross.patch \ file://0001-libc-compat.h-add-musl-workaround.patch \ file://0001-ip-Remove-unneed-header.patch \ " -- 2.20.1 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/2] openssh: usable sshd depends on rngd from rng-tools
On 08/05/2019 16.22, mikko.rap...@bmw.de wrote: > On Wed, May 08, 2019 at 05:07:08PM +0300, Adrian Bunk wrote: >> On Wed, May 08, 2019 at 04:26:09PM +0300, Mikko Rapeli wrote: >>> Since openssl 1.1.1 and openssh which uses it, sshd >>> startup is delayed. The delays range from few seconds >>> to minutes and even to hours. The delays are visible >>> in host keys generation and when sshd process is started >>> in response to incoming TCP connection but is failing >>> to provide SSH version string and clients or tests time out. >>> >>> In all cases traces show that sshd is waiting for getentropy() >>> system call to return from Linux kernel, which returns only >>> after kernel side random number pool is initialized. The pool >>> is initialized via various entropy source which may be >>> missing on embedded development boards or via rngd from >>> rng-tools package from userspace. HW random number generation >>> and kernel support help but rngd is till needed to feed that data >>> back to the Linux kernel. >>> >>> Example from an NXP imx8 board shows that kernel random number pool >>> initialization can take over 400 seconds without rngd, >>> and with rngd it is initialized at around 4 seconds after boot. >>> The completion of initialization is visible in kernel dmesg with line >>> "random: crng init done". >>> ... >>> --- a/meta/recipes-connectivity/openssh/openssh_7.9p1.bb >>> +++ b/meta/recipes-connectivity/openssh/openssh_7.9p1.bb >>> @@ -148,6 +148,7 @@ FILES_${PN}-keygen = "${bindir}/ssh-keygen" >>> >>> RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen" >>> RDEPENDS_${PN}-sshd += "${PN}-keygen >>> ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit >>> pam-plugin-loginuid', '', d)}" >>> +RDEPENDS_${PN}-sshd += "rng-tools" >>> ... >> >> This should only be an RRECOMMENDS so that people can opt out of it. >> >> E.g. CONFIG_RANDOM_TRUST_CPU in the kernel can solve the same >> problem without using rng-tools on some platforms. > > I think this is a stronger dependency than just RRECOMMENDS. We build > images and disable recommends but we care that sshd starts as fast as in > sumo and earlier yocto releases for testing etc purposes. But why should boards without a hwrng be forced to spend disk space and run-time resources on a daemon which they don't benefit from at all? I don't think RANDOM_TRUST_CPU works, though. That's just for stuff like rdrand(), i.e. instructions built into the CPU - not for some other on-chip hwrng. Whether those are used for seeding early on (i.e., without rngd doing its thing) depends on the ->quality parameter set by the individual hwrng drivers. Very few set one, so they get assigned the default_quality, which is a module parameter that defaults to 0. IOW, I think (but I haven't got around to testing this) one should set rng_core.default_quality=512 (or something) on the kernel command line to make the kernel start the hwrng_fill thread that will seed the entropy pool from the hwrng. At least if I'm reading drivers/char/hw_random/core.c correctly. Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] rng-tools: Fix crazy defaults
On 09/11/2018 09.54, Hongxu Jia wrote: > Since commit [f1dc9ac rng-tools: Fix crazy defaults] fixed > init based on sysvinit, this fix rngd.service based on systemd. > > Signed-off-by: Hongxu Jia > --- > meta/recipes-support/rng-tools/rng-tools/rngd.service | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-support/rng-tools/rng-tools/rngd.service > b/meta/recipes-support/rng-tools/rng-tools/rngd.service > index cb81024..f0355db 100644 > --- a/meta/recipes-support/rng-tools/rng-tools/rngd.service > +++ b/meta/recipes-support/rng-tools/rng-tools/rngd.service > @@ -5,7 +5,7 @@ After=systemd-udev-settle.service > Before=sysinit.target > > [Service] > -ExecStart=@SBINDIR@/rngd -f -r /dev/urandom > +ExecStart=@SBINDIR@/rngd -f -r /dev/hwrng > SuccessExitStatus=66 > > [Install] > This has been applied to master, but please apply this to thud as well - feeding the kernel's entropy pool from that same entropy pool is of course utter nonsense, but since rngd is also _crediting those bytes with providing real bits of entropy_ (ioctl RNDADDENTROPY) makes it a security issue. Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] rng-tools: Fix crazy defaults
On 09/11/2018 09.54, Hongxu Jia wrote: > Since commit [f1dc9ac rng-tools: Fix crazy defaults] fixed > init based on sysvinit, this fix rngd.service based on systemd. > > Signed-off-by: Hongxu Jia > --- > meta/recipes-support/rng-tools/rng-tools/rngd.service | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-support/rng-tools/rng-tools/rngd.service > b/meta/recipes-support/rng-tools/rng-tools/rngd.service > index cb81024..f0355db 100644 > --- a/meta/recipes-support/rng-tools/rng-tools/rngd.service > +++ b/meta/recipes-support/rng-tools/rng-tools/rngd.service > @@ -5,7 +5,7 @@ After=systemd-udev-settle.service > Before=sysinit.target > > [Service] > -ExecStart=@SBINDIR@/rngd -f -r /dev/urandom > +ExecStart=@SBINDIR@/rngd -f -r /dev/hwrng > SuccessExitStatus=66 > > [Install] > This has been applied to master, but please apply this to thud as well - feeding the kernel's entropy pool from that same entropy pool is of course utter nonsense, but since rngd is also _crediting those bytes with providing real bits of entropy_ (ioctl RNDADDENTROPY) makes it a security issue. Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH resend^2] wic: allow bitbake variables in kickstarter files
ping ping On 29/12/2018 01.06, Rasmus Villemoes wrote: > image_types_wic.bbclass has a mechanism for doing variable substitution > on .wks files by simply letting the input file be called > .wks.in. However, that doesn't allow using variables in files included > via the include directive. This is unfortunate, because lacking either > the ability to include other files or variable substitution leads to > fragile and error-prone duplication between kickstarter files and > recipes/configuration files used for various boards. > > This adds (somewhat naive) support for variable substitution in all > files parsed by wic. The user should add all required variables to > WICVARS to get them exported appropriately. > > Signed-off-by: Rasmus Villemoes > --- > scripts/lib/wic/ksparser.py | 17 + > 1 file changed, 17 insertions(+) > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py > index 7e5a9c5092..08baf76123 100644 > --- a/scripts/lib/wic/ksparser.py > +++ b/scripts/lib/wic/ksparser.py > @@ -28,14 +28,30 @@ > import os > import shlex > import logging > +import re > > from argparse import ArgumentParser, ArgumentError, ArgumentTypeError > > from wic.engine import find_canned > from wic.partition import Partition > +from wic.misc import get_bitbake_var > > logger = logging.getLogger('wic') > > +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") > + > +def expand_line(line): > +while True: > +m = __expand_var_regexp__.search(line) > +if not m: > +return line > +key = m.group()[2:-1] > +val = get_bitbake_var(key) > +if val is None: > +logger.warning("cannot expand variable %s" % key) > +return line > +line = line[:m.start()] + val + line[m.end():] > + > class KickStartError(Exception): > """Custom exception.""" > pass > @@ -190,6 +206,7 @@ class KickStart(): > line = line.strip() > lineno += 1 > if line and line[0] != '#': > +line = expand_line(line) > try: > line_args = shlex.split(line) > parsed = parser.parse_args(line_args) > -- Rasmus Villemoes Software Developer Prevas A/S Hedeager 3 DK-8200 Aarhus N +45 51210274 rasmus.villem...@prevas.dk www.prevas.dk -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH resend^2] wic: allow bitbake variables in kickstarter files
ping On 29/12/2018 01.06, Rasmus Villemoes wrote: > image_types_wic.bbclass has a mechanism for doing variable substitution > on .wks files by simply letting the input file be called > .wks.in. However, that doesn't allow using variables in files included > via the include directive. This is unfortunate, because lacking either > the ability to include other files or variable substitution leads to > fragile and error-prone duplication between kickstarter files and > recipes/configuration files used for various boards. > > This adds (somewhat naive) support for variable substitution in all > files parsed by wic. The user should add all required variables to > WICVARS to get them exported appropriately. > > Signed-off-by: Rasmus Villemoes > --- > scripts/lib/wic/ksparser.py | 17 + > 1 file changed, 17 insertions(+) > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py > index 7e5a9c5092..08baf76123 100644 > --- a/scripts/lib/wic/ksparser.py > +++ b/scripts/lib/wic/ksparser.py > @@ -28,14 +28,30 @@ > import os > import shlex > import logging > +import re > > from argparse import ArgumentParser, ArgumentError, ArgumentTypeError > > from wic.engine import find_canned > from wic.partition import Partition > +from wic.misc import get_bitbake_var > > logger = logging.getLogger('wic') > > +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") > + > +def expand_line(line): > +while True: > +m = __expand_var_regexp__.search(line) > +if not m: > +return line > +key = m.group()[2:-1] > +val = get_bitbake_var(key) > +if val is None: > +logger.warning("cannot expand variable %s" % key) > +return line > +line = line[:m.start()] + val + line[m.end():] > + > class KickStartError(Exception): > """Custom exception.""" > pass > @@ -190,6 +206,7 @@ class KickStart(): > line = line.strip() > lineno += 1 > if line and line[0] != '#': > +line = expand_line(line) > try: > line_args = shlex.split(line) > parsed = parser.parse_args(line_args) > -- Rasmus Villemoes Software Developer Prevas A/S Hedeager 3 DK-8200 Aarhus N +45 51210274 rasmus.villem...@prevas.dk www.prevas.dk -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH resend^2] wic: allow bitbake variables in kickstarter files
image_types_wic.bbclass has a mechanism for doing variable substitution on .wks files by simply letting the input file be called .wks.in. However, that doesn't allow using variables in files included via the include directive. This is unfortunate, because lacking either the ability to include other files or variable substitution leads to fragile and error-prone duplication between kickstarter files and recipes/configuration files used for various boards. This adds (somewhat naive) support for variable substitution in all files parsed by wic. The user should add all required variables to WICVARS to get them exported appropriately. Signed-off-by: Rasmus Villemoes --- scripts/lib/wic/ksparser.py | 17 + 1 file changed, 17 insertions(+) diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index 7e5a9c5092..08baf76123 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -28,14 +28,30 @@ import os import shlex import logging +import re from argparse import ArgumentParser, ArgumentError, ArgumentTypeError from wic.engine import find_canned from wic.partition import Partition +from wic.misc import get_bitbake_var logger = logging.getLogger('wic') +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") + +def expand_line(line): +while True: +m = __expand_var_regexp__.search(line) +if not m: +return line +key = m.group()[2:-1] +val = get_bitbake_var(key) +if val is None: +logger.warning("cannot expand variable %s" % key) +return line +line = line[:m.start()] + val + line[m.end():] + class KickStartError(Exception): """Custom exception.""" pass @@ -190,6 +206,7 @@ class KickStart(): line = line.strip() lineno += 1 if line and line[0] != '#': +line = expand_line(line) try: line_args = shlex.split(line) parsed = parser.parse_args(line_args) -- 2.19.1.6.gbde171bbf5 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 2/2] lib/oe/package.py: use bb.utils.break_hardlinks helper
This does the same thing, but is more efficient in case st_nlinks is (already) 1. Depends on bitbake commit 7ae93cf40ab91965147055100432961436bce46c . Signed-off-by: Rasmus Villemoes --- meta/lib/oe/package.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 21c80aaa38..efd36b3758 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -159,7 +159,7 @@ def strip_execs(pn, dstdir, strip_cmd, libdir, base_libdir, d, qa_already_stripp else: # break hardlinks so that we do not strip the original. inodes[inodecache[file]] = file -bb.utils.copyfile(file, file) +bb.utils.break_hardlinks(file) elffiles[file] = elf_file # -- 2.16.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 1/2] package.bbclass: use bb.utils.break_hardlinks helper
This does the same thing, but is more efficient in case st_nlinks is (already) 1. Depends on bitbake commit 7ae93cf40ab91965147055100432961436bce46c . Signed-off-by: Rasmus Villemoes --- meta/classes/package.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass index 4ce9de2f57..8dca3b0191 100644 --- a/meta/classes/package.bbclass +++ b/meta/classes/package.bbclass @@ -1029,7 +1029,7 @@ python split_and_strip_files () { else: inodes[file_reference] = [file] # break hardlink -bb.utils.copyfile(file, file) +bb.utils.break_hardlinks(file) elffiles[file] = elf_file # Modified the file so clear the cache cpath.updatecache(file) -- 2.16.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [pseudo] [RFC 1/2] pseudo_ipc.c: Use MSG_NOSIGNAL if available
On 2018-07-11 16:30, Rasmus Villemoes wrote: > MSG_NOSIGNAL has been in Linux since 2.2, and has been standardized in > POSIX 2008. Using that when available avoids the overhead of the two > syscalls to set and restore the SIGPIPE handler. Moreover, we can > eliminate one write() call by making use of sendmsg() to do > scatter-gather I/O. Ping. The README in the pseudo repo says to use the openembedded-core list for pseudo patches, so I assume this is the right place. I have other ideas for improving pseudo's performance on both client and server side, but there's no point sending patches to /dev/null. Rasmus -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: improve -dbg and -src package ordering
On 2018-07-17 10:13, Rasmus Villemoes wrote: > ping... > ping2 > On 2018-07-11 13:38, Rasmus Villemoes wrote: >> nativesdk-gpgme fails package_qa when setting PACKAGE_DEBUG_SPLIT_STYLE >> = "debug-with-srcpkg". >> >> ERROR: nativesdk-gpgme-1.10.0-r0 do_package_qa: QA Issue: non debug package >> contains .debug directory: nativesdk-python3-gpg path >> /work/x86_64-nativesdk-oesdk-linux/nativesdk-gpgme/1.10.0-r0/packages-split/nativesdk-python3-gpg/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/lib/python3.5/site-packages/gpg/.debug/_gpgme.cpython-35m-x86_64-linux-gnu.so >> [debug-files] >> >> This turns out to be because the automatic moving of the -dbg package to >> the beginning of the package list is disabled in that case, so the >> python3-gpg packages that the recipe prepends to PACKAGES ends up before >> the -dbg package. >> >> It's not clear why the "and not split_source_package" was added when >> debug-with-srcpkg was introduced. Presumably the intention was to >> prevent the -dbg package to end up before the -src package, which we of >> course need to. But at the same time, we still need -dbg packages to end >> up before all other packages. >> >> Using list.insert(0, ...) also means that if there happens to more than >> one -dbg package, their relative ordering gets inverted in the new list. >> >> This tries to fix these issues by sorting the packages by (priority, >> original position), where priority is 10 for -src, 30 for -dbg and 50 >> for everything else. That guarantees that packages of the same "type" >> preserve their relative ordering, while also ensuring that -dbg always >> preceed other packages. This scheme is also quite extensible, and, >> should the need arise, one could even expose the priorities as a knob >> the recipe author could use to ensure specific orderings of packages >> instead of the somewhat fragile and coarse-grained method of "prepend or >> append, and ensure you do that in a proper order". >> >> Probably the autodebug condition needs to stay, but I think the >> split_source_package condition in the preceding elif should be removed, >> so that that logic applies to all packages called -src, not just the one >> we might have created a few lines above. >> >> Signed-off-by: Rasmus Villemoes >> --- >> meta/classes/package.bbclass | 15 --- >> 1 file changed, 8 insertions(+), 7 deletions(-) >> >> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass >> index 8276046369..72b9c66c3b 100644 >> --- a/meta/classes/package.bbclass >> +++ b/meta/classes/package.bbclass >> @@ -1151,21 +1151,22 @@ python populate_packages () { >> >> # Sanity check PACKAGES for duplicates >> # Sanity should be moved to sanity.bbclass once we have the >> infrastructure >> -package_list = [] >> +package_dict = {} >> >> -for pkg in packages.split(): >> -if pkg in package_list: >> +for i, pkg in enumerate(packages.split()): >> +if pkg in package_dict: >> msg = "%s is listed in PACKAGES multiple times, this leads to >> packaging errors." % pkg >> package_qa_handle_error("packages-list", msg, d) >> # If debug-with-srcpkg mode is enabled then the src package will >> have >> # priority over dbg package when assigning the files. >> # This allows src package to include source files and remove them >> from dbg. >> elif split_source_package and pkg.endswith("-src"): >> -package_list.insert(0, pkg) >> -elif autodebug and pkg.endswith("-dbg") and not >> split_source_package: >> -package_list.insert(0, pkg) >> +package_dict[pkg] = (10, i) >> +elif autodebug and pkg.endswith("-dbg"): >> +package_dict[pkg] = (30, i) >> else: >> -package_list.append(pkg) >> +package_dict[pkg] = (50, i) >> +package_list = sorted(package_dict.keys(), key=package_dict.get) >> d.setVar('PACKAGES', ' '.join(package_list)) >> pkgdest = d.getVar('PKGDEST') >> >> > -- Rasmus Villemoes Software Developer Prevas A/S Hedeager 3 DK-8200 Aarhus N +45 51210274 rasmus.villem...@prevas.dk www.prevas.dk -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] busybox: udhcpd: fix not dying on SIGTERM
busybox 1.27.2 udhcpd does not respond to SIGTERM. Backport the upstream fix. I've verified that this makes our local automated test suite pass again. Signed-off-by: Rasmus Villemoes --- .../busybox/udhcpd-fix-not-dying-on-SIGTERM.patch | 273 + meta/recipes-core/busybox/busybox_1.27.2.bb| 1 + 2 files changed, 274 insertions(+) create mode 100644 meta/recipes-core/busybox/busybox/udhcpd-fix-not-dying-on-SIGTERM.patch diff --git a/meta/recipes-core/busybox/busybox/udhcpd-fix-not-dying-on-SIGTERM.patch b/meta/recipes-core/busybox/busybox/udhcpd-fix-not-dying-on-SIGTERM.patch new file mode 100644 index 00..1ea0385a8f --- /dev/null +++ b/meta/recipes-core/busybox/busybox/udhcpd-fix-not-dying-on-SIGTERM.patch @@ -0,0 +1,273 @@ +From 1384459d88cb0f5f47b6a36b8346dcf425a643f5 Mon Sep 17 00:00:00 2001 +From: Denys Vlasenko +Date: Sat, 10 Mar 2018 19:01:48 +0100 +Subject: [PATCH] udhcpd: fix "not dying on SIGTERM" + +Upstream-status: Backport [https://git.busybox.net/busybox/commit/?id=3293bc146985c56790033409facc0ad64a471084] + +Fixes: + commit 52a515d18724bbb34e3ccbbb0218efcc4eccc0a8 + "udhcp: use poll() instead of select()" + Feb 16 2017 + +udhcp_sp_read() is meant to check whether signal pipe indeed has some data to read. +In the above commit, it was changed as follows: + +- if (!FD_ISSET(signal_pipe.rd, rfds)) ++ if (!pfds[0].revents) + return 0; + +The problem is, the check was working for select() purely by accident. +Caught signal interrupts select()/poll() syscalls, they return with EINTR +(regardless of SA_RESTART flag in sigaction). _Then_ signal handler is invoked. +IOW: they can't see any changes to fd state caused by signal haldler +(in our case, signal handler makes signal pipe ready to be read). + +For select(), it means that rfds[] bit array is unmodified, bit of signal +pipe's read fd is still set, and the above check "works": it thinks select() +says there is data to read. + +This accident does not work for poll(): .revents stays clear, and we do not +try reading signal pipe as we should. In udhcpd, we fall through and block +in socket read. Further SIGTERM signals simply cause socket read to be +interrupted and then restarted (since SIGTERM handler has SA_RESTART=1). + +Fixing this as follows: remove the check altogether. Set signal pipe read fd +to nonblocking mode. Always read it in udhcp_sp_read(). +If read fails, assume it's EAGAIN and return 0 ("no signal seen"). + +udhcpd avoids reading signal pipe on every recvd packet by looping if EINTR +(using safe_poll()) - thus ensuring we have correct .revents for all fds - +and calling udhcp_sp_read() only if pfds[0].revents!=0. + +udhcpc performs much fewer reads (typically it sleeps >99.999% of the time), +there is no need to optimize it: can call udhcp_sp_read() after each poll +unconditionally. + +To robustify socket reads, unconditionally set pfds[1].revents=0 +in udhcp_sp_fd_set() (which is before poll), and check it before reading +network socket in udhcpd. + +TODO: +This might still fail: if pfds[1].revents=POLLIN, socket read may still block. +There are rare cases when select/poll indicates that data can be read, +but then actual read still blocks (one such case is UDP packets with +wrong checksum). General advise is, if you use a poll/select loop, +keep all your fds nonblocking. +Maybe we should also do that to our network sockets? + +function old new delta +udhcp_sp_setup55 65 +10 +udhcp_sp_fd_set 54 60 +6 +udhcp_sp_read 46 36 -10 +udhcpd_main 14511437 -14 +udhcpc_main 27232708 -15 +-- +(add/remove: 0/0 grow/shrink: 2/3 up/down: 16/-39)Total: -23 bytes + +Signed-off-by: Denys Vlasenko +--- + examples/var_service/dhcpd_if/run | 4 +-- + .../dhcpd_if/{udhcpc.conf => udhcpd.conf} | 0 + networking/udhcp/common.h | 2 +- + networking/udhcp/d6_dhcpc.c| 9 +++--- + networking/udhcp/dhcpc.c | 9 +++--- + networking/udhcp/dhcpd.c | 34 -- + networking/udhcp/signalpipe.c | 11 +++ + 7 files changed, 35 insertions(+), 34 deletions(-) + rename examples/var_service/dhcpd_if/{udhcpc.conf => udhcpd.conf} (100%) + +diff --git a/examples/var_service/dhcpd_if/run b/examples/var_service/dhcpd_if/run +index de85dece0..a603bdc71 100755 +--- a/examples/var_service/dhcpd_if/run b/examples/var_service/dhcpd_if/run +@@ -11,13 +11,13 @@ echo "* Upping iface $if&q
Re: [OE-core] [PATCH resend] wic: allow bitbake variables in kickstarter files
On 2018-07-03 14:54, Rasmus Villemoes wrote: > image_types_wic.bbclass has a mechanism for doing variable substitution > on .wks files by simply letting the input file be called > .wks.in. However, that doesn't allow using variables in files included > via the include directive. This is unfortunate, because lacking either > the ability to include other files or variable substitution leads to > fragile and error-prone duplication between kickstarter files and > recipes/configuration files used for various boards. > > This adds (somewhat naive) support for variable substitution in all > files parsed by wic. The user should add all required variables to > WICVARS to get them exported appropriately. ping^n -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC PATCH] package.bbclass: improve -dbg and -src package ordering
ping... On 2018-07-11 13:38, Rasmus Villemoes wrote: > nativesdk-gpgme fails package_qa when setting PACKAGE_DEBUG_SPLIT_STYLE > = "debug-with-srcpkg". > > ERROR: nativesdk-gpgme-1.10.0-r0 do_package_qa: QA Issue: non debug package > contains .debug directory: nativesdk-python3-gpg path > /work/x86_64-nativesdk-oesdk-linux/nativesdk-gpgme/1.10.0-r0/packages-split/nativesdk-python3-gpg/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/lib/python3.5/site-packages/gpg/.debug/_gpgme.cpython-35m-x86_64-linux-gnu.so > [debug-files] > > This turns out to be because the automatic moving of the -dbg package to > the beginning of the package list is disabled in that case, so the > python3-gpg packages that the recipe prepends to PACKAGES ends up before > the -dbg package. > > It's not clear why the "and not split_source_package" was added when > debug-with-srcpkg was introduced. Presumably the intention was to > prevent the -dbg package to end up before the -src package, which we of > course need to. But at the same time, we still need -dbg packages to end > up before all other packages. > > Using list.insert(0, ...) also means that if there happens to more than > one -dbg package, their relative ordering gets inverted in the new list. > > This tries to fix these issues by sorting the packages by (priority, > original position), where priority is 10 for -src, 30 for -dbg and 50 > for everything else. That guarantees that packages of the same "type" > preserve their relative ordering, while also ensuring that -dbg always > preceed other packages. This scheme is also quite extensible, and, > should the need arise, one could even expose the priorities as a knob > the recipe author could use to ensure specific orderings of packages > instead of the somewhat fragile and coarse-grained method of "prepend or > append, and ensure you do that in a proper order". > > Probably the autodebug condition needs to stay, but I think the > split_source_package condition in the preceding elif should be removed, > so that that logic applies to all packages called -src, not just the one > we might have created a few lines above. > > Signed-off-by: Rasmus Villemoes > --- > meta/classes/package.bbclass | 15 --- > 1 file changed, 8 insertions(+), 7 deletions(-) > > diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass > index 8276046369..72b9c66c3b 100644 > --- a/meta/classes/package.bbclass > +++ b/meta/classes/package.bbclass > @@ -1151,21 +1151,22 @@ python populate_packages () { > > # Sanity check PACKAGES for duplicates > # Sanity should be moved to sanity.bbclass once we have the > infrastructure > -package_list = [] > +package_dict = {} > > -for pkg in packages.split(): > -if pkg in package_list: > +for i, pkg in enumerate(packages.split()): > +if pkg in package_dict: > msg = "%s is listed in PACKAGES multiple times, this leads to > packaging errors." % pkg > package_qa_handle_error("packages-list", msg, d) > # If debug-with-srcpkg mode is enabled then the src package will have > # priority over dbg package when assigning the files. > # This allows src package to include source files and remove them > from dbg. > elif split_source_package and pkg.endswith("-src"): > -package_list.insert(0, pkg) > -elif autodebug and pkg.endswith("-dbg") and not split_source_package: > -package_list.insert(0, pkg) > +package_dict[pkg] = (10, i) > +elif autodebug and pkg.endswith("-dbg"): > +package_dict[pkg] = (30, i) > else: > -package_list.append(pkg) > +package_dict[pkg] = (50, i) > +package_list = sorted(package_dict.keys(), key=package_dict.get) > d.setVar('PACKAGES', ' '.join(package_list)) > pkgdest = d.getVar('PKGDEST') > > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core