Re: [OE-core] [PATCH] openssl: fix pkgconfig path problems
Hi, On Fri, May 03, 2024 at 10:59:21AM +, Ross Burton via lists.openembedded.org wrote: > OpenSSL 3.3.0 introduced new pkgconfig generators which interact badly > with our incorrect setting of --libdir, which is documented as being > the name of the directory _under $prefix_, not an absolute path. This > resulted in the pkgconfig files have libdir=/usr which mostly works as > the actual library directory is on the search path, but can break other > recipes (such as tpm2-openssl). This fixes tpm2-openssl build for me. Thanks! Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199002): https://lists.openembedded.org/g/openembedded-core/message/199002 Mute This Topic: https://lists.openembedded.org/mt/105885512/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] openssl: fix pkg-config file libdir from /usr back to /usr/lib
Hi,
On Fri, May 03, 2024 at 12:43:29PM +0200, Martin Jansa wrote:
> On Fri, May 3, 2024 at 12:37 PM Mikko Rapeli via
> lists.openembedded.org
> wrote:
> >
> > Hi,
> >
> > On Fri, May 03, 2024 at 12:32:42PM +0200, Alexander Kanavin wrote:
> > > I’m sorry, this doesn’t look right. Shouldn’t we rather ensure $(LIBDIR)
> > > is
> > > correctly set? Besides, hardcoding to lib is certainly incorrect as it
> > > will
> > > break multilib. We constantly have to fix upstreams in the opposite
> > > direction.
> >
> > Yes, you and Ross on #yocto are right. This is not right. I tried also
> > setting
> > LIBDIR=${BASELIB} in do_install() and do_configure but those did not work
> > either. openssl builds scripts are quite hard to work with...
>
> Isn't it easier to fix tpm2-openssl then?
>
> I had similar issue in chip which was using "pkg-config
> --variable=prefix openssl" and didn't expect it to be empty in "if
> prefix[-4] == '/usr'" and rather patched chip than changing openssl.
To me this is a bug/regression in openssl pkg-config since libdir changed
from /usr/lib to /usr which doesn't work for same use cases anymore.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198973):
https://lists.openembedded.org/g/openembedded-core/message/198973
Mute This Topic: https://lists.openembedded.org/mt/105885236/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] openssl: fix pkg-config file libdir from /usr back to /usr/lib
Hi,
On Fri, May 03, 2024 at 12:32:42PM +0200, Alexander Kanavin wrote:
> I’m sorry, this doesn’t look right. Shouldn’t we rather ensure $(LIBDIR) is
> correctly set? Besides, hardcoding to lib is certainly incorrect as it will
> break multilib. We constantly have to fix upstreams in the opposite
> direction.
Yes, you and Ross on #yocto are right. This is not right. I tried also setting
LIBDIR=${BASELIB} in do_install() and do_configure but those did not work
either. openssl builds scripts are quite hard to work with...
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198971):
https://lists.openembedded.org/g/openembedded-core/message/198971
Mute This Topic: https://lists.openembedded.org/mt/105885236/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] openssl: fix pkg-config file libdir from /usr back to /usr/lib
openssl update from 3.2.1 to 3.3.0 changed pkg-config generators
and libdir changed from /usr/lib to /lib. This breaks builds
of recipes which use openssl pkg-config files for library install
etc paths. For example meta-security tpm2-openssl fails to build:
ERROR: mc:trs-qemuarm64:tpm2-openssl-1.1.1-r0 do_package: QA Issue:
tpm2-openssl: Files/directories were installed but not shipped in any package:
/usr/ossl-modules/tpm2.so
Affected pkg-config files are: libssl.pc, openssl.pc and libcrypto.pc
Fix this by hard coding LIBDIR variable to "lib". Prefix should
be correctly used as before.
Signed-off-by: Mikko Rapeli
---
.../openssl/openssl_pkg-config_libdir.patch | 31 +++
.../openssl/openssl_3.3.0.bb | 1 +
2 files changed, 32 insertions(+)
create mode 100644
meta/recipes-connectivity/openssl/openssl/openssl_pkg-config_libdir.patch
diff --git
a/meta/recipes-connectivity/openssl/openssl/openssl_pkg-config_libdir.patch
b/meta/recipes-connectivity/openssl/openssl/openssl_pkg-config_libdir.patch
new file mode 100644
index 00..e3b4249a69
--- /dev/null
+++ b/meta/recipes-connectivity/openssl/openssl/openssl_pkg-config_libdir.patch
@@ -0,0 +1,31 @@
+openssl: fix pkg-config file libdir from /usr back to /usr/lib
+
+openssl update from 3.2.1 to 3.3.0 changed pkg-config generators
+and libdir changed from /usr/lib to /lib. This breaks builds
+of recipes which use openssl pkg-config files for library install
+etc paths. For example meta-security tpm2-openssl fails to build:
+
+ERROR: mc:trs-qemuarm64:tpm2-openssl-1.1.1-r0 do_package: QA Issue:
tpm2-openssl: Files/directories were installed but not shipped in any package:
+ /usr/ossl-modules/tpm2.so
+
+Affected pkg-config files are: libssl.pc, openssl.pc and libcrypto.pc
+
+Fix this by hard coding LIBDIR variable to "lib". Prefix should
+be correctly used as before.
+
+Upstream-Status: Inappropriate [configuration change]
+
+Signed-off-by: Mikko Rapeli
+
+--- a/exporters/build.info
b/exporters/build.info
+@@ -19,7 +19,7 @@ DEPEND[openssl.pc]=libcrypto.pc libssl.pc
+ DEPEND[""]=openssl.pc
+
+ GENERATE[../installdata.pm]=../util/mkinstallvars.pl \
+-"PREFIX=$(INSTALLTOP)" BINDIR=bin "LIBDIR=$(LIBDIR)" \
++"PREFIX=$(INSTALLTOP)" BINDIR=bin LIBDIR=lib \
+ INCLUDEDIR=include APPLINKDIR=include/openssl \
+ "ENGINESDIR=$(ENGINESDIR)" "MODULESDIR=$(MODULESDIR)" \
+ "PKGCONFIGDIR=$(PKGCONFIGDIR)" "CMAKECONFIGDIR=$(CMAKECONFIGDIR)" \
+
diff --git a/meta/recipes-connectivity/openssl/openssl_3.3.0.bb
b/meta/recipes-connectivity/openssl/openssl_3.3.0.bb
index 2cdaf4c75d..5ee8375ebc 100644
--- a/meta/recipes-connectivity/openssl/openssl_3.3.0.bb
+++ b/meta/recipes-connectivity/openssl/openssl_3.3.0.bb
@@ -13,6 +13,7 @@ SRC_URI = "http://www.openssl.org/source/openssl-${PV}.tar.gz
\
file://0001-Configure-do-not-tweak-mips-cflags.patch \
file://0001-Added-handshake-history-reporting-when-test-fails.patch
\
file://bti.patch \
+ file://openssl_pkg-config_libdir.patch \
"
SRC_URI:append:class-nativesdk = " \
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198969):
https://lists.openembedded.org/g/openembedded-core/message/198969
Mute This Topic: https://lists.openembedded.org/mt/105885236/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/2] curl: disable ca-certificates.crt path setting for native build
Hi,
On Thu, Apr 04, 2024 at 03:13:08PM +0100, Richard Purdie wrote:
> On Thu, 2024-04-04 at 16:29 +0300, Mikko Rapeli wrote:
> > If linux-yocto-dev is compiled without specific SRCREV, it uses
> > AUTOREV which tries to update to latest available commit. This is
> > currently failing with these steps:
> >
> > $ rm -rf tmp*/work/*/linux-yocto-dev && \
> > bitbake -c do_configure mc:machine:linux-yocto-dev ; \
> > bitbake -c do_clean mc:machine:linux-yocto-dev
> > [...]
>
> > The variable dependency chain for the failure is:
> > fetcher_hashes_dummyfunc[vardepvalue]
> >
> > ERROR: Parsing halted due to errors, see error messages above
> >
> > Summary: There were 6 WARNING messages.
> > Summary: There were 2 ERROR messages, returning a non-zero exit code.
> >
> > This state is not recoverable with bitbake calls. All of them fail from now
> > on.
> > "rm -rf tmp/work/*/linux-yocto-dev" recovers the situation
> > and bitbake commands work again.
> >
> > Root cause is curl-native, dependency of git-native, which
> > has --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt
> > which for native build target is wrong and points to curl-native build
> > directory path
> > /home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
> >
> > Since git is a build time host package dependency listed in
> > https://docs.yoctoproject.org/dev/singleindex.html#build-host-packages
> > then its dependencies like curl and ca-certificates are too, it should
> > be safe for curl-native to use the default host ca-certificates path
> > instead of the one in recipe specific sysroots which would need to be set
> > with complicated
> > environment variables. Set non-default ca-certificates path only for
> > target and nativesdk builds.
> >
> > Reported-by: Mathieu Poirier
> > Signed-off-by: Mikko Rapeli
> > ---
> > meta/recipes-support/curl/curl_8.6.0.bb | 7 ++-
> > 1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/recipes-support/curl/curl_8.6.0.bb
> > b/meta/recipes-support/curl/curl_8.6.0.bb
> > index 49ba0cb4a7..da5571ca14 100644
> > --- a/meta/recipes-support/curl/curl_8.6.0.bb
> > +++ b/meta/recipes-support/curl/curl_8.6.0.bb
> > @@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd"
> > EXTRA_OECONF = " \
> > --disable-libcurl-option \
> > --disable-ntlm-wb \
> > - --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
> > --without-libpsl \
> > --enable-optimize \
> > ${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls
> > mbedtls openssl', d) == '') else ''} \
> > "
> > +EXTRA_OECONF:class-target = " \
> > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
> > +"
> > +EXTRA_OECONF:class-nativesdk = " \
> > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
> > +"
> >
> > fix_absolute_paths () {
> > # cleanup buildpaths from curl-config
>
> This change is fraught with danger :(.
>
> I have a feeling we've gone around in circles as in some cases you
> don't have the ca-certs on the host, or they're in unusual paths so the
> previous conclusion was we should always have them present in the
> sysroot if curl-native is being used. Yes, that does mean we have to
> set the environment correctly to relocate curl's paths appropriately.
>
> Certainly at this point in the release cycle I'm very nervous about
> changing this around.
No worries, 421083c46c97bf758496b8c58402aea5d74aa097 already on master fixes
the issue. We're lagging few weeks behind master branch but trying
to catch up.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197968):
https://lists.openembedded.org/g/openembedded-core/message/197968
Mute This Topic: https://lists.openembedded.org/mt/105328588/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/2] curl: disable ca-certificates.crt path setting for native build
Hi, On Thu, Apr 04, 2024 at 02:57:20PM +0100, Jose Quaresma wrote: > Hi Mikko, > > Did you test with the below change already merged on master? > Looks like it was to fix the same issue. > > git: git-replacement-native: depend on ca-certificate > https://git.yoctoproject.org/poky/commit/meta/recipes-devtools/git?id=421083c46c97bf758496b8c58402aea5d74aa097 No, I was on slighlty older commit 47c201da56155f80e18ee1269096ffec1864bce0. 421083c46c97bf758496b8c58402aea5d74aa097 fixes the issue too, thanks for the hint! This patch can be ignored. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#197967): https://lists.openembedded.org/g/openembedded-core/message/197967 Mute This Topic: https://lists.openembedded.org/mt/105328588/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] curl: disable ca-certificates.crt path setting for native build
p_poky/hosttools";
export HOME="/home/builder"; git -c gc.autoDetach=false -c core.pager=cat -c
safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto-dev.git failed with exit code 128,
output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto-dev.git/':
error setting certificate file:
/home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]
ERROR: Parsing halted due to errors, see error messages above
Summary: There were 6 WARNING messages.
Summary: There were 2 ERROR messages, returning a non-zero exit code.
This state is not recoverable with bitbake calls. All of them fail from now on.
"rm -rf tmp/work/*/linux-yocto-dev" recovers the situation
and bitbake commands work again.
Root cause is curl-native, dependency of git-native, which
has --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt
which for native build target is wrong and points to curl-native build
directory path
/home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
Since git is a build time host package dependency listed in
https://docs.yoctoproject.org/dev/singleindex.html#build-host-packages
then its dependencies like curl and ca-certificates are too, it should
be safe for curl-native to use the default host ca-certificates path
instead of the one in recipe specific sysroots which would need to be set with
complicated
environment variables. Set non-default ca-certificates path only for
target and nativesdk builds.
Reported-by: Mathieu Poirier
Signed-off-by: Mikko Rapeli
---
meta/recipes-support/curl/curl_8.6.0.bb | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-support/curl/curl_8.6.0.bb
b/meta/recipes-support/curl/curl_8.6.0.bb
index 49ba0cb4a7..da5571ca14 100644
--- a/meta/recipes-support/curl/curl_8.6.0.bb
+++ b/meta/recipes-support/curl/curl_8.6.0.bb
@@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd"
EXTRA_OECONF = " \
--disable-libcurl-option \
--disable-ntlm-wb \
---with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
--without-libpsl \
--enable-optimize \
${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls mbedtls
openssl', d) == '') else ''} \
"
+EXTRA_OECONF:class-target = " \
+--with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
+"
+EXTRA_OECONF:class-nativesdk = " \
+--with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \
+"
fix_absolute_paths () {
# cleanup buildpaths from curl-config
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197964):
https://lists.openembedded.org/g/openembedded-core/message/197964
Mute This Topic: https://lists.openembedded.org/mt/105328588/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] linux-yocto-dev: remove duplicate DEPENDS
They are already set in linux-yocto.inc
Signed-off-by: Mikko Rapeli
---
meta/recipes-kernel/linux/linux-yocto-dev.bb | 5 -
1 file changed, 5 deletions(-)
diff --git a/meta/recipes-kernel/linux/linux-yocto-dev.bb
b/meta/recipes-kernel/linux/linux-yocto-dev.bb
index 9bd219825e..d5cba42a5c 100644
--- a/meta/recipes-kernel/linux/linux-yocto-dev.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-dev.bb
@@ -34,11 +34,6 @@ PV = "${LINUX_VERSION}+git"
LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
-DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
-DEPENDS += "${@bb.utils.contains('ARCH', 'powerpc', 'elfutils-native', '', d)}"
-DEPENDS += "openssl-native util-linux-native"
-DEPENDS += "gmp-native libmpc-native"
-
# yaml and dtschema are required for 5.16+ device tree validation, libyaml is
checked
# via pkgconfig, so must always be present, but we can wrap the others to make
them
# conditional
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197963):
https://lists.openembedded.org/g/openembedded-core/message/197963
Mute This Topic: https://lists.openembedded.org/mt/105328586/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [master] [PATCH] cve-check: Add provision to exclude classes
Hi, On Fri, Mar 15, 2024 at 07:52:00PM +, Dhairya Nagodra via lists.openembedded.org wrote: > > > >-Original Message- > >From: Ross Burton > >Sent: Friday, March 15, 2024 9:39 PM > >To: Dhairya Nagodra -X (dnagodra - E-INFO CHIPS INC at Cisco) > > > >Cc: openembedded-core@lists.openembedded.org; xe-linux-external(mailer > >list) > >Subject: Re: [OE-core] [master] [PATCH] cve-check: Add provision to exclude > >classes > > > >On 3 Mar 2024, at 17:53, Dhairya Nagodra via lists.openembedded.org > > wrote: > >> > >> From: Dhairya Nagodra > >> > >> - There are times when exluding a package that inherits a particular > >> class/classes may be desired. > >> - This provides the framework for that via the variable: > >> CVE_CHECK_CLASS_EXCLUDELIST > > > >What’s the use-case for this? Note that you can control whether cve-check > >runs per-layer already, if that’s useful. > > Currently, the CVE report is generated for all packages associated with the > build. > However, not all of them might be getting used in the target device. > The package associated with native, nativesdk, cross classes are examples of > such. > This patch would provide a way to exclude these packages in the CVE report. > So, if the variable is set like CVE_CHECK_CLASS_EXCLUDELIST = "native", > The report would not have the entries for these packages: > gnupg-native, nasm-native, binutils-native (and so on) > > This is helpful when one wants to concentrate their CVE fixing efforts to the > specific packages going into the target device. CVE check generates report summaries for all images already. Doesn't that cover this usecase? And many build tools end up talking to servers in the Internet so detecting and fixing CVEs in them is also quite important. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#197293): https://lists.openembedded.org/g/openembedded-core/message/197293 Mute This Topic: https://lists.openembedded.org/mt/104706824/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] wic partition.py: add --apparent-size to du calls
Hi,
On Tue, Feb 27, 2024 at 12:33:15PM +, Richard Purdie wrote:
> On Tue, 2024-02-27 at 14:23 +0200, Mikko Rapeli wrote:
> > On Tue, Feb 27, 2024 at 12:06:41PM +, Richard Purdie wrote:
> > > Should we break it for others so it works for you?
> > >
> > > If I merge this patch and then I get other reports in of issues,
> > > I'm
> > > going to be left to somehow fix those other problems. Since your
> > > case
> > > will work you will object if I revert this but you likely won't
> > > have
> > > time to work on the resulting other issues, or be able to reproduce
> > > them since you're building on zfs.
> > >
> > > So you can see my dilemma? :/
> > >
> > > Most people don't build on zfs which is probably what I need to
> > > balance
> > > this on.
> >
> > What if I make this optional for zfs? If build directory type is zfs,
> > then --apparent-size is added, else not.
>
> Wouldn't that cause some really interesting bug reports?
Well, if anyone builds on zfs they will hit bugs like this.
zfs usage can be detected and build can fail nicely or apply
the workaround/fix in wic.
> Asking the user to report their host system filesystem type to debug
> anything isn't going to be great.
>
> I don't like the idea of the non-determinism that introduces but I
> agree we do already have a determinism issue.
>
> What we probably need is the absolute files size, plus a number of
> files and a number of directories, then you can round up to the worst
> cases for the block size.
>
> Currently that size number is just including the block sizes.
FWIW, I tried to move this workaround into image recipe with
IMAGE_CMD:wic:prepend() {
alias df="df --apparent-size"
}
but wic python code doesn't seem to take this into account. Tried also
changing the du calls to have exec_cmd(..., as_shell=True) but this didn't
help either. Something in wic causes the use of another shell or different
default
environment for processes.
So I'm stuck with this change in poky to build on a machine with zfs.
Any hints how to apply this simple workaround are welcome.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196417):
https://lists.openembedded.org/g/openembedded-core/message/196417
Mute This Topic: https://lists.openembedded.org/mt/104600633/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] wic partition.py: add --apparent-size to du calls
Hi, On Tue, Feb 27, 2024 at 12:06:41PM +, Richard Purdie wrote: > Should we break it for others so it works for you? > > If I merge this patch and then I get other reports in of issues, I'm > going to be left to somehow fix those other problems. Since your case > will work you will object if I revert this but you likely won't have > time to work on the resulting other issues, or be able to reproduce > them since you're building on zfs. > > So you can see my dilemma? :/ > > Most people don't build on zfs which is probably what I need to balance > this on. What if I make this optional for zfs? If build directory type is zfs, then --apparent-size is added, else not. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196269): https://lists.openembedded.org/g/openembedded-core/message/196269 Mute This Topic: https://lists.openembedded.org/mt/104600633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] wic partition.py: add --apparent-size to du calls
Hi,
On Tue, Feb 27, 2024 at 11:48:27AM +, Richard Purdie wrote:
> On Tue, 2024-02-27 at 13:23 +0200, Mikko Rapeli wrote:
> > If build happens on zfs filesystem with compression enabled,
> > then image size calculations in do_image_wic task can fail:
> >
> > output: mke2fs 1.47.0 (5-Feb-2023)
> > Discarding device blocks: done
> > Creating filesystem with 351999 4k blocks and 176000 inodes
> > Filesystem UUID: 6091b3a4-ce08-3020-93a6-f755a22ef03b
> > Superblock backups stored on blocks:
> > 32768, 98304, 163840, 229376, 294912
> >
> > Allocating group tables: done
> > Writing inode tables: done
> > Creating journal (8192 blocks): done
> > Copying files into the device: __populate_fs: Could not allocate
> > block in ext2 filesystem while writing file "service-2.json"
> > mkfs.ext4: Could not allocate block in ext2 filesystem while
> > populating file system
> >
> > du --help says:
> >
> > --apparent-size print apparent sizes, rather than disk usage;
> > although
> > the apparent size is usually smaller, it
> > may be
> > larger due to holes in ('sparse') files,
> > internal
> > fragmentation, indirect blocks, and the
> > like
> >
> > du -b already includes --apparent-size.
> >
> > Same issue reported also in
> > https://lists.yoctoproject.org/g/poky/message/12389
> >
> > Signed-off-by: Mikko Rapeli
>
>
> From memory we've gone around in circles on this. In some cases we do
> really want the size in bytes but in others we want it rounded up to
> block size for each file. I think this was why what bug was never fixed
> as you can address one set of issues but create a new set.
>
> So for that reason I'm extremely cautious about this, particularly this
> late in the release cycle. I don't really know what to do with this
> change.
I have a build machine with zfs and this issue pops up ever now and then.
Right now I can't build there without this change. I don't see a way to
fix this issue in my build/host specific configuration.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196267):
https://lists.openembedded.org/g/openembedded-core/message/196267
Mute This Topic: https://lists.openembedded.org/mt/104600633/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] wic partition.py: add --apparent-size to du calls
If build happens on zfs filesystem with compression enabled,
then image size calculations in do_image_wic task can fail:
output: mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 351999 4k blocks and 176000 inodes
Filesystem UUID: 6091b3a4-ce08-3020-93a6-f755a22ef03b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Copying files into the device: __populate_fs: Could not allocate block in ext2
filesystem while writing file "service-2.json"
mkfs.ext4: Could not allocate block in ext2 filesystem while populating file
system
du --help says:
--apparent-size print apparent sizes, rather than disk usage; although
the apparent size is usually smaller, it may be
larger due to holes in ('sparse') files, internal
fragmentation, indirect blocks, and the like
du -b already includes --apparent-size.
Same issue reported also in https://lists.yoctoproject.org/g/poky/message/12389
Signed-off-by: Mikko Rapeli
---
scripts/lib/wic/partition.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 795707ec5d..4690ddaa4d 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -254,7 +254,7 @@ class Partition():
# Bitbake variable ROOTFS_SIZE is not defined so compute it
# from the rootfs_dir size using the same logic found in
# get_rootfs_size() from meta/classes/image.bbclass
-du_cmd = "du -ks %s" % rootfs_dir
+du_cmd = "du -ks --apparent-size %s" % rootfs_dir
out = exec_cmd(du_cmd)
self.size = int(out.split()[0])
@@ -273,7 +273,7 @@ class Partition():
"""
Prepare content for an ext2/3/4 rootfs partition.
"""
-du_cmd = "du -ks %s" % rootfs_dir
+du_cmd = "du -ks --apparent-size %s" % rootfs_dir
out = exec_cmd(du_cmd)
actual_rootfs_size = int(out.split()[0])
@@ -349,7 +349,7 @@ class Partition():
"""
Prepare content for a btrfs rootfs partition.
"""
-du_cmd = "du -ks %s" % rootfs_dir
+du_cmd = "du -ks --apparent-size %s" % rootfs_dir
out = exec_cmd(du_cmd)
actual_rootfs_size = int(out.split()[0])
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196261):
https://lists.openembedded.org/g/openembedded-core/message/196261
Mute This Topic: https://lists.openembedded.org/mt/104600633/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [Openembedded-architecture] [RFC PATCH] Add genericarm64 MACHINE using upstream defconfig
Hi, On Wed, Feb 21, 2024 at 03:23:48PM +0200, Mikko Rapeli via lists.openembedded.org wrote: > FWIW, we have been using upstream kernel.org aarch64 defconfig plus > few board specific fragments and few extra features for our testing needs. > I have been very happy that several major kernel version updates have already > been done this way and zero adaptations needed on our side for the > ARM SystemReady boards and firmware which we support. A simple CI run to show > passing test results was sufficient for a poky update with new kernel > major version. > > The implementation here looks pretty much like ours. Looks good, thanks Ross! > > More details of our setup: > > https://gitlab.com/Linaro/trustedsubstrate/meta-ledge-secure/-/blob/main/meta-ledge-secure/recipes-kernel/linux/linux-ledge-common.inc?ref_type=heads > https://gitlab.com/Linaro/trustedsubstrate/meta-ledge-secure/-/tree/main/meta-ledge-secure/recipes-kernel/linux/ledgearm64-kmeta?ref_type=heads > https://trs.readthedocs.io/en/latest/ Our machine is based on poky qemuarm64 machine and thus uses KBRANCH:qemuarm64 so I don't see a need for new branch from linux-yocto recipe maintainer. https://git.yoctoproject.org/linux-yocto/log/?h=v6.5/standard/qemuarm64 But maybe I don't see the full picture. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#195974): https://lists.openembedded.org/g/openembedded-core/message/195974 Mute This Topic: https://lists.openembedded.org/mt/104486073/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [Openembedded-architecture] [RFC PATCH] Add genericarm64 MACHINE using upstream defconfig
Hi, On Wed, Feb 21, 2024 at 11:21:39AM +, Richard Purdie wrote: > On Wed, 2024-02-21 at 10:57 +, Ross Burton wrote: > > From: Ross Burton > > > > This is a new 64-bit "generic" Arm machine, that expects the hardware to > > be SystemReady IR compatible. This is slightly forward-leaning as there's > > not a _lot_ of SystemReady hardware in the wild, but most modern boards > > are and the number will only grow. Also, this is the only way to have a > > 'generic' machine as without standardised bootloaders and firmware it > > would be impossible. > > > > The base machine configuration isn't that exciting: it's a fully featured > > machine that supports most things, booting via UEFI and an initramfs. > > > > However, the kernel is more interesting. This RFC uses the upstream > > defconfig > > because unlike some other platforms, the arm64 defconfig is actively > > maintained with the goal of being a 'boots on most hardware' configuration. > > My argument is: why would we duplicate that effort? > > Can you point at the policy/process which decides how a config option > makes it in there? > > > The "linux-yocto way" is configuration fragments and after a week of > > hair-pulling I do actually have fragments that boot on a BeaglePlay, but > > to say this was a tiresome and frustrating exercise would be understating > > it. > > > > So, a request for comments: is it acceptable to use the upstream defconfig > > in > > a reference BSP? Personally I'm torn: the Yocto way is fragments not > > monolithic > > configs, but repeating the effort to fragmentise the configuration and then > > also have it sufficiently modular that it can be used in pieces - instead of > > just being a large file split up into smaller files - is a lot of effort for > > what might end up being minimal gain. My fear is we end up with a > > fragmented > > configuration that can't be easily modified without breaking some platforms, > > and badly copies what the defconfig already does. > > Let me play devils advocate. > > I do understand it is a pain, equally, once you do have it working, it > is something you rarely have to touch again for a given board. > > In the context of linux-yocto, I suspect we'd end up with a few board > specific lists of config options (fragments) that board *really* needs. > genericarm64 would end up as the sum of all those configs, plus any > other bits which you really want in a generic machine, presumably the > genericx86 machines have an idea of that. > > The advantage here is that you give the users three options: > > a) the kitchen sink upstream "systemready" defconfig > b) the genericarm64 machine defconfig which works most places and >is 'guaranteed' on our target test platforms > c) a machine targeted defconfig with only what a board needs > > Developers/users end up in different places at different points in > there lifecycles. > > For in depth development, you might like to be able to cut the kernel > down so you'd not rebuilding tons of stuff you don't need/use/care > about each time, nor transferring it to a device during updates. > > For a specific product release, you again might want to trim it down to > what it needs. > > If you're shipping demo software, a full systemready image would be > much more appropriate. > > I think it comes down to whether the fragments are usable and testable. > We have a list of targets we want this new machine to run on so lets > start with those, define genericarm64 as that set of fragments combined > plus the generic pieces linux-yocto adds, then go from there. If you > add a new machine to the test matrix, we add a new fragment. If someone > wants to add new config, they need to show a machine using it. FWIW, we have been using upstream kernel.org aarch64 defconfig plus few board specific fragments and few extra features for our testing needs. I have been very happy that several major kernel version updates have already been done this way and zero adaptations needed on our side for the ARM SystemReady boards and firmware which we support. A simple CI run to show passing test results was sufficient for a poky update with new kernel major version. The implementation here looks pretty much like ours. Looks good, thanks Ross! More details of our setup: https://gitlab.com/Linaro/trustedsubstrate/meta-ledge-secure/-/blob/main/meta-ledge-secure/recipes-kernel/linux/linux-ledge-common.inc?ref_type=heads https://gitlab.com/Linaro/trustedsubstrate/meta-ledge-secure/-/tree/main/meta-ledge-secure/recipes-kernel/linux/ledgearm64-kmeta?ref_type=heads https://trs.readthedocs.io/en/latest/ Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#195971): https://lists.openembedded.org/g/openembedded-core/message/195971 Mute This Topic: https://lists.openembedded.org/mt/104486073/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe:
Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params
Hi, On Fri, Jan 26, 2024 at 02:04:18PM +, Joakim Tjernlund via lists.openembedded.org wrote: > On Fri, 2024-01-26 at 14:33 +0100, Alexander Kanavin via > lists.openembedded.org wrote: > > On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund > > wrote: > > > I will try workaround these somehow but will not go so far as creating a > > > new MACHINE just to > > > run these tests. > > > > Why would you need a new machine? You only need a new configuration > > template in addition to existing one (e.g. > > conf/templates/repro-test/local.conf.sample and bblayer.conf.sample). > > Then initialize the build with TEMPLATECONF pointing to that. > > Ah, there is an idea. But this will still change the original machine so one > cannot switch > between normal builds and repro builds in he same tree. FWIW, I ran into the same issues. Many selftests also have dependencies which only work on pure poky distro, machine and build configs. The dependencies to various features which break tests are not clearly marked inside the tests. At least with repro test it is now possible to define different image and recipe build targets instead of world builds now but sadly these tests can not be run on real product machine configurations and need a vanilla poky machine and/or distro configs. I guess patches welcome but not sure into which direction... Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194373): https://lists.openembedded.org/g/openembedded-core/message/194373 Mute This Topic: https://lists.openembedded.org/mt/103973969/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
Hi,
I think this patch should be applied as it aligns oeqa .py and .text file
searches
across layers. Without this layers need to add "addpylib ${LAYERDIR}/lib oeqa"
in layer.conf which doesn't have any other uses and debugging this is really
hard.
There may be addition things wrong in bitbake python environment when oeqa
tests run
but this is already an improvement.
Ross, do you agree?
Cheers,
-Mikko
On Wed, Jan 10, 2024 at 04:23:22PM +0200, Mikko Rapeli via
lists.openembedded.org wrote:
> Hi,
>
> On Wed, Jan 10, 2024 at 02:01:36PM +, Ross Burton wrote:
> > On 10 Jan 2024, at 13:03, Mikko Rapeli via lists.openembedded.org
> > wrote:
> > > python import.resources open_text() loads files from the module paths
> > > but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> > > which is not needed to find the plain .py test files to run the tests.
> > > Also an empty __init__.py file in a layer will break the resource file
> > > loading completely as only that path with __init__.py file will
> > > be used to search the resource files. Then open_text() is marked
> > > as deprecated from python 3.11 onwards
> > > https://docs.python.org/3/library/importlib.resources.html
> >
> > Deprecated but replaceable with files(package).joinpath(resource).open()
> > (which is all open_text does internally).
>
> Yes, but that seems to break if __init__.py file is found from
> ${LAYERDIR}/lib/oeqa/runtime/cases.
>
> > > So replace open_text() by iterating over sys.path to find the ignore
> > > files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> > > already in sys.path. Add debug prints for found and not found files
> > > while at it.
> >
> > I’m confused why lib/oeqa/runtime/cases is on sys.path directly, that
> > sounds… unusual. Do you have any idea where that comes from? I’m guessing
> > oeqa is throwing it on the path whilst searching for test cases, but that
> > sounds like bad behaviour that should be removed to me.
>
> No idea where this comes from, but it's there. addpylib is doing similar
> things,
> adding paths to sys.path and for this use case this not really necessary as
> ${LAYERDIR}/lib should not need to be, it only has oeqa directory. I presume
> test case loading via testimage.bbclass or OERuntim are adding it. These are
> kind
> of hard to debug since "bitbake -e" show any of this.
>
> It's a bit odd that layer.conf would need "addpylib ${LAYERDIR}/lib oeqa" for
> loading
> .txt files but that's not needed for finding and using the oeqa .py test
> files.
> With this change addpylib magic is not needed, at least for finding these
> .txt files.
>
> Cheers,
>
> -Mikko
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194016):
https://lists.openembedded.org/g/openembedded-core/message/194016
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] runqemu: ahci option
Hi, On Thu, Jan 11, 2024 at 11:42:58AM +0100, Alexander Kanavin wrote: > Generally runqemu is inteded for (and tested with) specifically > qemux86_64 MACHINE (or any other qemu* machine). So if testing your > particular target bootloader with qemu isn't actually important, you > can rebuild the image for qemux86_64, and in that configuration there > is no bootloader: qemu boots the kernel directly. True, yocto has a lot of history with the assumption that every board even with common and compatible SoC's needs a specific machine configuration. I think this is a bit too old fashioned and would like to move towards generic and compatible builds where a single binary build works on a large number of instruction set compatible SoCs and boards. Thus I'd be happy to have qemu testing part of this and the hard coded assumptions of qemu configs to be removed or made optional so that they can be overwritten if needed. So patches welcome :) Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193540): https://lists.openembedded.org/g/openembedded-core/message/193540 Mute This Topic: https://lists.openembedded.org/mt/103658327/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
Hi,
On Wed, Jan 10, 2024 at 02:01:36PM +, Ross Burton wrote:
> On 10 Jan 2024, at 13:03, Mikko Rapeli via lists.openembedded.org
> wrote:
> > python import.resources open_text() loads files from the module paths
> > but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
> > which is not needed to find the plain .py test files to run the tests.
> > Also an empty __init__.py file in a layer will break the resource file
> > loading completely as only that path with __init__.py file will
> > be used to search the resource files. Then open_text() is marked
> > as deprecated from python 3.11 onwards
> > https://docs.python.org/3/library/importlib.resources.html
>
> Deprecated but replaceable with files(package).joinpath(resource).open()
> (which is all open_text does internally).
Yes, but that seems to break if __init__.py file is found from
${LAYERDIR}/lib/oeqa/runtime/cases.
> > So replace open_text() by iterating over sys.path to find the ignore
> > files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
> > already in sys.path. Add debug prints for found and not found files
> > while at it.
>
> I’m confused why lib/oeqa/runtime/cases is on sys.path directly, that sounds…
> unusual. Do you have any idea where that comes from? I’m guessing oeqa is
> throwing it on the path whilst searching for test cases, but that sounds like
> bad behaviour that should be removed to me.
No idea where this comes from, but it's there. addpylib is doing similar things,
adding paths to sys.path and for this use case this not really necessary as
${LAYERDIR}/lib should not need to be, it only has oeqa directory. I presume
test case loading via testimage.bbclass or OERuntim are adding it. These are
kind
of hard to debug since "bitbake -e" show any of this.
It's a bit odd that layer.conf would need "addpylib ${LAYERDIR}/lib oeqa" for
loading
.txt files but that's not needed for finding and using the oeqa .py test files.
With this change addpylib magic is not needed, at least for finding these
.txt files.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193513):
https://lists.openembedded.org/g/openembedded-core/message/193513
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] oeqa parselogs.py: load ignore files from sys.path
python import.resources open_text() loads files from the module paths
but this requires layers to set "addpylib ${LAYERDIR}/lib oeqa"
which is not needed to find the plain .py test files to run the tests.
Also an empty __init__.py file in a layer will break the resource file
loading completely as only that path with __init__.py file will
be used to search the resource files. Then open_text() is marked
as deprecated from python 3.11 onwards
https://docs.python.org/3/library/importlib.resources.html
So replace open_text() by iterating over sys.path to find the ignore
files. This works since paths like ${LAYERDIR}/lib/oeqa/runtime/cases are
already in sys.path. Add debug prints for found and not found files
while at it.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/runtime/cases/parselogs.py | 29 +++-
1 file changed, 13 insertions(+), 16 deletions(-)
diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py
b/meta/lib/oeqa/runtime/cases/parselogs.py
index 6966923c94..47583dbb5d 100644
--- a/meta/lib/oeqa/runtime/cases/parselogs.py
+++ b/meta/lib/oeqa/runtime/cases/parselogs.py
@@ -6,27 +6,19 @@
import collections
import os
+import pathlib
import sys
from shutil import rmtree
from oeqa.runtime.case import OERuntimeTestCase
from oeqa.core.decorator.depends import OETestDepends
-# importlib.resources.open_text in Python <3.10 doesn't search all directories
-# when a package is split across multiple directories. Until we can rely on
-# 3.10+, reimplement the searching logic.
-if sys.version_info < (3, 10):
-def _open_text(package, resource):
-import importlib, pathlib
-module = importlib.import_module(package)
-for path in module.__path__:
-candidate = pathlib.Path(path) / resource
-if candidate.exists():
-return candidate.open(encoding='utf-8')
-raise FileNotFoundError
-else:
-from importlib.resources import open_text as _open_text
-
+def open_syspath_text(resource):
+for path in sys.path:
+candidate = pathlib.Path(path) / resource
+if candidate.exists():
+return candidate.open(encoding='utf-8')
+raise FileNotFoundError
class ParseLogsTest(OERuntimeTestCase):
@@ -61,11 +53,16 @@ class ParseLogsTest(OERuntimeTestCase):
for candidate in ["common", cls.td.get("TARGET_ARCH")] +
cls.td.get("MACHINEOVERRIDES").split(":"):
try:
name = f"parselogs-ignores-{candidate}.txt"
-for line in _open_text("oeqa.runtime.cases", name):
+print_once = True
+for line in open_syspath_text(name):
+if print_once:
+bb.debug(1, "parselogs: ignore file %s found" % (name))
+print_once = False
line = line.strip()
if line and not line.startswith("#"):
cls.ignore_errors.append(line.casefold())
except FileNotFoundError:
+bb.debug(1, "parselogs: ignore file %s not found" % (name))
pass
# Go through the log locations provided and if it's a folder
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193499):
https://lists.openembedded.org/g/openembedded-core/message/193499
Mute This Topic: https://lists.openembedded.org/mt/103639917/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 0/1] rng-tools: remove unneeded build time dependency
Hi, On Mon, Jan 08, 2024 at 04:09:02PM -0600, david zuhn via lists.openembedded.org wrote: > rng-tools used to depend on libsysfs, but has since been changed > to read directly from the /sys filesystem. The build time > dependency on sysfsutils is no longer needed. This is a good change log entry but should be in the git commit directly to be recorded in git history. No need for extra cover letter if change itself documents everything. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193443): https://lists.openembedded.org/g/openembedded-core/message/193443 Mute This Topic: https://lists.openembedded.org/mt/103608803/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] oeqa systemd.py: settle() using "running" or "degraded" state
systemd boot has completed when system is in "running" or "degraded"
(some services failed) state. Check for that in the systemd settle()
function instead of listing all services and checking their activation
state since some services are in activation state even when whole
system is already in "running" state. Examples of services which can be
in activation state are rootfs auto mounting related generated services.
Without this patch systemd test_systemd_list (systemd.SystemdBasicTests) times
out on an image with dm-verity /usr partition and systemd generated
rootfs:
NOTE: ... FAIL
Traceback (most recent call last):
File
"/home/builder/src/base/build/../poky/meta/lib/oeqa/core/decorator/__init__.py",
line 35, in wrapped_f
return func(*args, **kwargs)
File "/home/builder/src/base/poky/meta/lib/oeqa/runtime/cases/systemd.py",
line 97, in test_systemd_failed
self.assertTrue(settled, msg=msg)
AssertionError: False is not true : Timed out waiting for systemd to settle:
UNIT
LOAD ACTIVE SUB
DESCRIPTION
dev-disk-by\x2did-dm\x2dname\x2droot.device
loaded activating tentativ
e/dev/disk/by-id/dm-name-root
dev-disk-by\x2did-dm\x2dname\x2dusr.device
loaded activating tentativ
e/dev/disk/by-id/dm-name-usr
dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS2\x2df2b944f394174eb5918cb6af2c6b4cb2\x2droot.device
loaded activating tentativ
e/dev/disk/by-id/dm-uuid-CRYPT-LUKS2-f2b944f394174eb5918cb6af2c6b4cb2-root
dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dVERITY\x2d3dd703c88f1946658697a6d57617473b\x2dusr.device
loaded activating tentativ
e/dev/disk/by-id/dm-uuid-CRYPT-VERITY-3dd703c88f1946658697a6d57617473b-usr
dev-disk-by\x2duuid-bfbf856e\x2d3c65\x2d4eb2\x2d9ffb\x2d8e0b11641d85.device
loaded activating tentativ
e/dev/disk/by-uuid/bfbf856e-3c65-4eb2-9ffb-8e0b11641d85
dev-dm\x2d0.device
loaded activating tentativ
e/dev/dm-0
dev-dm\x2d1.device
loaded activating tentativ
e/dev/dm-1
...
Fix is to check for the systemd global "running" or "degraded" state.
Note that it would be possible to use a blocking call
"systemctl is-system-running --wait" to exit after system enters "running"
or "degraded" state but using the existing loop for a 2 minute timeout.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/runtime/cases/systemd.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/runtime/cases/systemd.py
b/meta/lib/oeqa/runtime/cases/systemd.py
index 37f295492d..58e94b62ff 100644
--- a/meta/lib/oeqa/runtime/cases/systemd.py
+++ b/meta/lib/oeqa/runtime/cases/systemd.py
@@ -68,8 +68,8 @@ class SystemdBasicTests(SystemdTest):
"""
endtime = time.time() + (60 * 2)
while True:
-status, output = self.target.run('SYSTEMD_BUS_TIMEOUT=240s
systemctl --state=activating')
-if "0 loaded units listed" in output:
+status, output = self.target.run('SYSTEMD_BUS_TIMEOUT=240s
systemctl is-system-running')
+if "running" in output or "degraded" in output:
return (True, '')
if time.time() >= endtime:
return (False, output)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193323):
https://lists.openembedded.org/g/openembedded-core/message/193323
Mute This Topic: https://lists.openembedded.org/mt/103522170/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] runqemu: match .rootfs. in addition to -image- for rootfs
Also change path.exists() and !path.isdir() to a single
path.isfile() which should be equal.
Enables running tests against image recipes which are not called
"bla-image" but plain "bla". Currently they fail with do_testimage/runqemu
error:
runqemu - ERROR - Unknown path arg
/home/builder/src/base/build/tmp_qemuarm64/deploy/images/qemuarm64/img-qemuarm64.rootfs.wic
Suggested-by: Richard Purdie
Signed-off-by: Mikko Rapeli
---
scripts/runqemu | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
v2: check for both -image- and .rootfs. in file names as suggested by
Richard Purdie , and using
isfile() instead of both exists() and !isdir()
v1: https://lists.openembedded.org/g/openembedded-core/message/193280
diff --git a/scripts/runqemu b/scripts/runqemu
index 18aeb7f5f0..9ed3ab17ac 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -373,7 +373,7 @@ class BaseConfig(object):
re.search('zImage', p) or re.search('vmlinux', p) or \
re.search('fitImage', p) or re.search('uImage', p):
self.kernel = p
-elif os.path.exists(p) and (not os.path.isdir(p)) and '-image-' in
os.path.basename(p):
+elif os.path.isfile(p) and ('-image-' in os.path.basename(p) or
'.rootfs.' in os.path.basename(p)):
self.rootfs = p
# Check filename against self.fstypes can handle .cpio.gz,
# otherwise, its type would be "gz", which is incorrect.
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193313):
https://lists.openembedded.org/g/openembedded-core/message/193313
Mute This Topic: https://lists.openembedded.org/mt/103519834/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] runqemu: match .rootfs. in image names instead of -image-
Hi, On Thu, Jan 04, 2024 at 08:34:03AM +, Richard Purdie wrote: > On Thu, 2024-01-04 at 09:50 +0200, Mikko Rapeli wrote: > > Hi, > > > > On Thu, Jan 04, 2024 at 08:59:03AM +0200, Mikko Rapeli via > > lists.openembedded.org wrote: > > > Hi, > > > > > > On Wed, Jan 03, 2024 at 03:54:02PM +0100, Martin Jansa wrote: > > > > On Wed, Jan 3, 2024 at 11:16 AM Mikko Rapeli > > > > wrote: > > > > > > > > > Don't expect image file names to include "-image-" but > > > > > expect them to have the ".rootfs." postfix which is the > > > > > default. > > > > > > > > > > > > > It's the current default, but whole runqemu won't work when someone > > > > changes: > > > > > > > > meta/classes-recipe/image-artifact-names.bbclass:IMAGE_NAME_SUFFIX ??= > > > > ".rootfs" > > > > > > > > So you should probably read IMAGE_NAME_SUFFIX variable first, but the > > > > most > > > > likely change someone will have in local.conf would be to drop this > > > > suffix > > > > and set this to empty and then the check for empty string in basename > > > > won't > > > > be very useful as well. > > > > > > So drop this file name check completely? Fine for me. I'll send an update. > > > > As you said too, this doesn't work. runqemu has a bunch of heuristics to > > detect different > > kinds of file arguments which break too easily. Test for .rootfs. in file > > name is IMO > > better than than -image- in recipe/file name, but also not perfect. Using > > IMAGE_NAME_SUFFIX > > would be nice but the indirection with qemuboot.conf makes this hard too, > > and if that variable > > is set to empty then a lot of things break again on runqemu side > > heuristics, gah... > > > > I have a dm-verity image recipe and a separate recipe which bakes that into > > a .wic image > > file which I'd like to test with oeqa, testimage.bbclass and qemu. > > > > It's either this patch or I rename all image recipes to have -image in > > their names. > > You could change the check to check for -image- or .rootfs. ? > > Not perfect either but might be good enough... Good idea, I'll do that. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193308): https://lists.openembedded.org/g/openembedded-core/message/193308 Mute This Topic: https://lists.openembedded.org/mt/103499438/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] runqemu: match .rootfs. in image names instead of -image-
Hi, On Thu, Jan 04, 2024 at 08:59:03AM +0200, Mikko Rapeli via lists.openembedded.org wrote: > Hi, > > On Wed, Jan 03, 2024 at 03:54:02PM +0100, Martin Jansa wrote: > > On Wed, Jan 3, 2024 at 11:16 AM Mikko Rapeli > > wrote: > > > > > Don't expect image file names to include "-image-" but > > > expect them to have the ".rootfs." postfix which is the > > > default. > > > > > > > It's the current default, but whole runqemu won't work when someone changes: > > > > meta/classes-recipe/image-artifact-names.bbclass:IMAGE_NAME_SUFFIX ??= > > ".rootfs" > > > > So you should probably read IMAGE_NAME_SUFFIX variable first, but the most > > likely change someone will have in local.conf would be to drop this suffix > > and set this to empty and then the check for empty string in basename won't > > be very useful as well. > > So drop this file name check completely? Fine for me. I'll send an update. As you said too, this doesn't work. runqemu has a bunch of heuristics to detect different kinds of file arguments which break too easily. Test for .rootfs. in file name is IMO better than than -image- in recipe/file name, but also not perfect. Using IMAGE_NAME_SUFFIX would be nice but the indirection with qemuboot.conf makes this hard too, and if that variable is set to empty then a lot of things break again on runqemu side heuristics, gah... I have a dm-verity image recipe and a separate recipe which bakes that into a .wic image file which I'd like to test with oeqa, testimage.bbclass and qemu. It's either this patch or I rename all image recipes to have -image in their names. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193305): https://lists.openembedded.org/g/openembedded-core/message/193305 Mute This Topic: https://lists.openembedded.org/mt/103499438/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] runqemu: match .rootfs. in image names instead of -image-
Hi, On Wed, Jan 03, 2024 at 03:54:02PM +0100, Martin Jansa wrote: > On Wed, Jan 3, 2024 at 11:16 AM Mikko Rapeli > wrote: > > > Don't expect image file names to include "-image-" but > > expect them to have the ".rootfs." postfix which is the > > default. > > > > It's the current default, but whole runqemu won't work when someone changes: > > meta/classes-recipe/image-artifact-names.bbclass:IMAGE_NAME_SUFFIX ??= > ".rootfs" > > So you should probably read IMAGE_NAME_SUFFIX variable first, but the most > likely change someone will have in local.conf would be to drop this suffix > and set this to empty and then the check for empty string in basename won't > be very useful as well. So drop this file name check completely? Fine for me. I'll send an update. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#193304): https://lists.openembedded.org/g/openembedded-core/message/193304 Mute This Topic: https://lists.openembedded.org/mt/103499438/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] runqemu: match .rootfs. in image names instead of -image-
Don't expect image file names to include "-image-" but
expect them to have the ".rootfs." postfix which is the
default.
Enables running tests against image recipes which are not called
"bla-image" but plain "bla". Currently they fail with do_testimage/runqemu
error:
runqemu - ERROR - Unknown path arg
/home/builder/src/base/build/tmp_qemuarm64/deploy/images/qemuarm64/img-qemuarm64.rootfs.wic
Signed-off-by: Mikko Rapeli
---
scripts/runqemu | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/runqemu b/scripts/runqemu
index 18aeb7f5f0..1575677c45 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -373,7 +373,7 @@ class BaseConfig(object):
re.search('zImage', p) or re.search('vmlinux', p) or \
re.search('fitImage', p) or re.search('uImage', p):
self.kernel = p
-elif os.path.exists(p) and (not os.path.isdir(p)) and '-image-' in
os.path.basename(p):
+elif os.path.exists(p) and (not os.path.isdir(p)) and '.rootfs.' in
os.path.basename(p):
self.rootfs = p
# Check filename against self.fstypes can handle .cpio.gz,
# otherwise, its type would be "gz", which is incorrect.
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193280):
https://lists.openembedded.org/g/openembedded-core/message/193280
Mute This Topic: https://lists.openembedded.org/mt/103499438/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/2] qemurunner: Impove stdout logging handling
Hi,
On Tue, Dec 19, 2023 at 12:03:07PM +, Richard Purdie wrote:
> On Mon, 2023-12-18 at 23:01 +, Richard Purdie via
> lists.openembedded.org wrote:
> > On Mon, 2023-12-18 at 10:07 -0800, Khem Raj wrote:
> > > On Mon, Dec 18, 2023 at 9:58 AM Richard Purdie
> > > wrote:
> > > >
> > > > On Mon, 2023-12-18 at 09:45 -0800, Khem Raj wrote:
> > > > > I tried the two patches in this series. It did improve the situation
> > > > > but I am still getting SSH timeouts. But this time its 13 tests
> > > > > earlier it used to be 40+
> > > > > btw. my images are using systemd. So it might be good to see if we see
> > > > > this with poky-altcfg as well or not.
> > > >
> > > > Do you have the log.do_testimage and the ${WORKDIR}/testimage/ files?
> > >
> > > yes, further I ran the failing tests in loop one after another still
> > > one test gzip fails with ssh timeouts
> > >
> > > https://busybox.net/~kraj/log.do_testimage.503
> > > https://busybox.net/~kraj/testimage/
> > >
> > > there are two runs in the testimages folder. In one you see the RCU
> > > stall and in second you do not
> > > but it fails with same ssh timeout issue.
> > >
> > > >
> > > > Did you still see rcu stalls in the logs?
> >
> > What is interesting is there is ~3MB of nulls in the .2 serial log. The
> > rcu stall is also:
> >
> > [ 88.261687] serial8250_tx_chars+0xea/0x2b0
> > [ 88.261689] serial8250_handle_irq+0x1e9/0x330
> > [ 88.261691] serial8250_default_handle_irq+0x4a/0x90
> > [ 88.261693] serial8250_interrupt+0x66/0xc0
> > [ 88.261696] __handle_irq_event_percpu+0x54/0x1c0
> > [ 88.261701] handle_irq_event+0x3d/0x80
> >
> > i.e. it is stalled in the serial TX path.
> >
> > The big question is why is there so many nulls on the serial port. I
> > see a few on my local x86 test runs but only ~4kb, not megabytes of
> > them. I hadn't worked out where/what they are from yet.
> >
> > I suspect something in the serial/kernel/qemu space isn't interacting
> > correctly.
> >
> > Find the cause of the nulls and we might make progress on this.
>
> I went digging and it is a mess. That 3MB file is actually a wtmp file
> from /var/log. The reason is a failure of the "target_dumper" commands
> and that the simplistic serial command handling doesn't like binary
> files.
>
> I've a few patches which should improve the situation but we're clearly
> multiple issues at play here where issues are stacking on top of other
> issues. My rough list of issues:
>
> a) stdout logging from qemu wasn't being read and could overflow
> buffers locking qemu
> b) serial data from commands wasn't being fully read and could overflow
> buffers locking qemu
> c) the dumper log reading command is reading binary files like wtmp
> d) the dumper logs overwrite each other so are useless anyway
> e) the dumper logs run on every failed command
> f) qmp monitor dump command also run on every failed command
>
> I think we'd be much better off if we drop the dumper/monitor stuff and
> qemu runs might be quite a bit faster too.
I agree to this. If target has serious issues like a full lockup, then dumper
and
monitor commands don't help anyway. Capturing the full serial logs would be
sufficient,
though I'd like to see a live capture of /var/log/message and systemd journal
in case
services started by init have major issues.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192725):
https://lists.openembedded.org/g/openembedded-core/message/192725
Mute This Topic: https://lists.openembedded.org/mt/103244414/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH][kirkstone] openssh: drop sudo from ptest dependencies
The tests don't actually need sudo on core-image-ptest-openssh.
Based on logs seen in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178 it seems
that socket errors from sudo are creeping into stderr which are failing
the banner ptest from openssh. Removing sudo should help removing
the stderr messages and possibly cure the banner test failures.
(From OE-Core rev: 47e754f483b674b207bfddcc8d4c5d9a3008e102)
Signed-off-by: Mikko Rapeli
Signed-off-by: Richard Purdie
---
meta/recipes-connectivity/openssh/openssh/run-ptest | 2 +-
meta/recipes-connectivity/openssh/openssh_8.9p1.bb | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
v1: Issue https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178 was still
seen on kirkstone
branch so cherry-pick the master/nanbield commit
diff --git a/meta/recipes-connectivity/openssh/openssh/run-ptest
b/meta/recipes-connectivity/openssh/openssh/run-ptest
index 8a9b770d59..9a406e9b65 100755
--- a/meta/recipes-connectivity/openssh/openssh/run-ptest
+++ b/meta/recipes-connectivity/openssh/openssh/run-ptest
@@ -5,7 +5,7 @@ export SKIP_UNIT=1
cd regress
sed -i "/\t\tagent-ptrace /d" Makefile
-make -k BUILDDIR=`pwd`/.. .OBJDIR=`pwd` .CURDIR=`pwd` SUDO="sudo" tests \
+make -k BUILDDIR=`pwd`/.. .OBJDIR=`pwd` .CURDIR=`pwd` SUDO="" tests \
| sed -u -e 's/^skipped/SKIP: /g' -e 's/^ok /PASS: /g' -e
's/^failed/FAIL: /g'
SSHAGENT=`which ssh-agent`
diff --git a/meta/recipes-connectivity/openssh/openssh_8.9p1.bb
b/meta/recipes-connectivity/openssh/openssh_8.9p1.bb
index da7ab7716c..32761b8bb8 100644
--- a/meta/recipes-connectivity/openssh/openssh_8.9p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_8.9p1.bb
@@ -170,7 +170,7 @@ RDEPENDS:${PN}-sshd += "${PN}-keygen
${@bb.utils.contains('DISTRO_FEATURES', 'pa
# conflict with each other
RDEPENDS:${PN}-dev = ""
# gdb would make attach-ptrace test pass rather than skip but not worth the
build dependencies
-RDEPENDS:${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make sed sudo
coreutils"
+RDEPENDS:${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make sed
coreutils"
RPROVIDES:${PN}-ssh = "ssh"
RPROVIDES:${PN}-sshd = "sshd"
--
2.39.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192203):
https://lists.openembedded.org/g/openembedded-core/message/192203
Mute This Topic: https://lists.openembedded.org/mt/103129959/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] systemd-boot: Add recipe to compile native
Hi,
On Fri, Dec 01, 2023 at 12:23:36PM +0530, Viswanath Kraleti wrote:
> systemd has ukify, a native tool, which will combine the kernel/initrd/stub
> components to build the UKI (unified kernel images). systemd-boot recipe
> isn't providing this tool.
>
> To use ukify, one need systemd-boot native recipe so that a dependency can
> be added on populate_sysroot task to get the tool under sysroot-native.
Why not call the recipe systemd-ukify then? systemd-boot doesn't provide more
than that.
Then, when using ukify, one needs to have the .efi binaries from target systemd
recipe which
for aarch64 are:
/usr/lib/systemd/boot/efi/addonaa64.efi.stub
/usr/lib/systemd/boot/efi/linuxaa64.efi.stub
/usr/lib/systemd/boot/efi/systemd-bootaa64.efi
and these could IMO be in a systemd-boot binary package because these files are
not needed
in the main systemd binary package and rootfs.
The recipe itself looks neat and small, thanks!
For secure boot sbsign-native would be needed, but maybe a separate issue.
Cheers,
-Mikko
> Signed-off-by: Viswanath Kraleti
> ---
> .../systemd/systemd-boot-native_254.4.bb | 15 +++
> 1 file changed, 15 insertions(+)
> create mode 100644 meta/recipes-core/systemd/systemd-boot-native_254.4.bb
>
> diff --git a/meta/recipes-core/systemd/systemd-boot-native_254.4.bb
> b/meta/recipes-core/systemd/systemd-boot-native_254.4.bb
> new file mode 100644
> index 00..6d89e493b6
> --- /dev/null
> +++ b/meta/recipes-core/systemd/systemd-boot-native_254.4.bb
> @@ -0,0 +1,15 @@
> +require systemd.inc
> +
> +inherit native
> +
> +RRECOMMENDS:${PN} += "python3-pefile-native"
> +
> +COMPATIBLE_HOST = "(aarch64.*|arm.*|x86_64.*|i.86.*)-linux"
> +
> +do_configure[noexec] = "1"
> +do_compile[noexec] = "1"
> +
> +do_install () {
> + install -d ${D}${bindir}
> + install -m 0755 ${S}/src/ukify/ukify.py ${D}${bindir}/ukify
> +}
> --
> 2.25.1
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191559):
https://lists.openembedded.org/g/openembedded-core/message/191559
Mute This Topic: https://lists.openembedded.org/mt/102910738/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] uki: Add support for building Unified Kernel Images
Hi,
On Tue, Nov 28, 2023 at 02:32:14PM +0200, Dmitry Baryshkov wrote:
> On Sat, 2 Sept 2023 at 02:32, Michelle Lin wrote:
> >
> > Currently, there is not a class to support the building of unified kernel
> > images. Adding a uki.bbclass to support the creation of UKIs. This class
> > calls
> > the systemd Ukify tool, which will combine the kernel/initrd/stub
> > components to
> > build the UKI. To sign the UKI (i.e. SecureBoot, TPM PCR signing), the
> > keys/cert
> > files are to be specified in a separate configuration file, and the path to
> > the
> > file is passed to the Ukify tool. UKIs are supported by UEFI and can improve
> > security through predicted TPM PCR states, and reduce the build burden due
> > to
> > its single PE binary format.
> >
> > Signed-off-by: Michelle Lin
> > ---
> > meta/classes/uki.bbclass | 140 +++
> > meta/recipes-core/systemd/systemd_254.bb | 23
> > 2 files changed, 163 insertions(+)
> > create mode 100644 meta/classes/uki.bbclass
> >
> > diff --git a/meta/classes/uki.bbclass b/meta/classes/uki.bbclass
> > new file mode 100644
> > index 00..2eff387c75
> > --- /dev/null
> > +++ b/meta/classes/uki.bbclass
> > @@ -0,0 +1,140 @@
> > +#
> > +# Unified kernel image (UKI) class
> > +#
> > +#
> > +# This bbclass is designed to repack an Overlake image as a UKI, to be
> > booted on a qemuarm64 with SecureBoot
> > +# signing and embedded with TPM PCR measurements.
> > +#
> > +# The UKI is composed by:
> > +# - an UEFI stub
> > +# The linux kernel can generate a UEFI stub, however the one from
> > systemd-boot can fetch
> > +# the command line from a separate section of the EFI application,
> > avoiding the need to
> > +# rebuild the kernel.
> > +# - the kernel
> > +# - an initramfs
> > +# - other metadata (e.g. PCR measurements)
> > +#
> > +#
> > +#
> > +
> > +# List build time dependencies
> > +DEPENDS += "systemd-native \
> > +sbsigntool-native \
> > +virtual/${TARGET_PREFIX}binutils \
> > +"
> > +
> > +REQUIRED_DISTRO_FEATURES += "usrmerge systemd"
> > +
> > +inherit features_check
> > +require ../conf/image-uefi.conf
> > +
> > +INITRD_IMAGE ?= "core-image-minimal-initramfs"
> > +
> > +INITRD_LIVE ?= "${@ ('${DEPLOY_DIR_IMAGE}/' + d.getVar('INITRD_IMAGE') +
> > '-${MACHINE}.cpio.gz') if d.getVar('INITRD_IMAGE') else ''}"
> > +
> > +UKI_CONFIG_FILE ?= "${WORKDIR}/core-image-minimal-uki.conf"
> > +UKI_FILENAME ?= "${@ 'UKI.signed.efi' if d.getVar('UKI_CONFIG_FILE') else
> > 'UKI.unsigned.efi'}"
> > +
> > +do_uki[depends] += " \
> > +systemd-boot:do_deploy \
> > +virtual/kernel:do_deploy \
> > + "
> > +
> > +# INITRD_IMAGE is added to INITRD_LIVE, which we use to create our initrd,
> > so depend on it if it is set
> > +# So we want to generate the initrd image if INITRD_IMAGE exists
> > +do_uki[depends] += "${@ '${INITRD_IMAGE}:do_image_complete' if
> > d.getVar('INITRD_IMAGE') else ''}"
> > +
> > +# ensure that the build directory is empty everytime we generate a
> > newly-created uki
> > +do_uki[cleandirs] = "${B}"
> > +# influence the build directory at the start of the builds
> > +do_uki[dirs] = "${B}"
> > +
> > +# we want to allow specifying files in SRC_URI, such as for signing the UKI
> > +python () {
> > +d.delVarFlag("do_fetch","noexec")
> > +d.delVarFlag("do_unpack","noexec")
> > +}
> > +
> > +# main task
> > +python do_uki() {
> > +import glob
> > +import subprocess
> > +
> > +# Construct the ukify command
> > +ukify_cmd = ("ukify build")
> > +
> > +# Handle the creation of an initrd image by reading and concatenating
> > multiple cpio files.
> > +# If the INITRD_LIVE variable is defined and not empty, it opens the
> > necessary files, reads their contents,
> > +# and constructs a list.
> > +if d.getVar('INITRD_LIVE'):
> > +initrd_list = ""
> > +for cpio in d.getVar('INITRD_LIVE').split():
> > +# get a list of initrds
> > +initrd_list += cpio + ' '
> > +
> > +ukify_cmd += " --initrd=%s" % initrd_list
> > +else:
> > +bb.fatal("ERROR - Required argument: INITRD")
> > +
> > +deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
> > +
> > +# Kernel
> > +if d.getVar('KERNEL_IMAGETYPE'):
> > +kernel = "%s/%s" % (deploy_dir_image, d.getVar('KERNEL_IMAGETYPE'))
> > +kernel_version = d.getVar('KERNEL_VERSION')
> > +if not os.path.exists(kernel):
> > +bb.fatal(f"ERROR: cannot find {kernel}.")
> > +
> > +ukify_cmd += " --linux=%s --uname %s" % (kernel, kernel_version)
> > +else:
> > +bb.fatal("ERROR - Required argument: KERNEL")
> > +
> > +# Architecture
> > +target_arch = d.getVar('EFI_ARCH')
> > +ukify_cmd += " --efi-arch %s" % target_arch
> > +
> > +# Stub
> > +stub = "%s/linux%s.efi.stub" % (deploy_dir_image,
Re: [OE-core] [PATCH v2] cve-check.bbclass: support embedded SW components with different version number
Hi Richard, I see master-next has both v1 and v2 of this change. If there is a rebase, maybe you can squash them into a single commit which is v2. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189702): https://lists.openembedded.org/g/openembedded-core/message/189702 Mute This Topic: https://lists.openembedded.org/mt/102076964/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2] cve-check.bbclass: support embedded SW components with different version number
Hi, On Fri, Oct 20, 2023 at 08:54:14AM -0700, Khem Raj wrote: > On 10/20/23 12:49 AM, Mikko Rapeli wrote: > > Many recipes embed other SW components. The name and version of the > > embedded SW component differs from the main recipe. To detect CVEs in the > > embedded SW component, it needs to be added to CVE_PRODUCT list using > > name of the SW product in CVE database or with "vendor:product" syntax. > > Then the version of the embedded SW component can be set using > > CVE_VERSION_product variable. > > > > For example in meta-arm, trusted-firmware-a embeds mbed_tls SW component. > > Thus trusted-firmware-a can add CVE_PRODUCT for it since CVE database > > uses product name "mbed_tls": > > > > CVE_PRODUCT += "mbed_tls" > > > > and set the version of mbed_tls: > > > > CVE_VERSION_mbed_tls = "2.28.4" > > > > (Real patches for both are a bit more complex due to conditional build > > enabling mbed_tls support and due to mbed_tls version being set in an > > .inc file.) > > > > Now trusted-firmware-a CVE check output shows: > > > > NOTE: recipe > > trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: > > task do_cve_check: Started > > WARNING: > > trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0 > > do_cve_check: Found unpatched CVE (CVE-2021-36647 CVE-2021-43666 > > CVE-2021-45451 CVE-2023-43615), for more information check > > /home/builder/src/base/build/tmp/work/arm64-poky-linux/trusted-firmware-a/2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122/temp/cve.log > > NOTE: recipe > > trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: > > task do_cve_check: Succeeded > > > > Here CVE-2023-43615 is a newly added and fixed CVE in version 2.28.5 and > > the CVEs > > from 2021 need to be checked but are likely fixed in 2.28.3 and newer > > 2.28.y releases. > > > > Note that CVE-2023-43615 does not impact trusted-firmware-a since it > > doesn't use > > TLS or null or RC4 ciphers, but I think it's a good idea to extend > > CVE checker for this use case. I hope the "CVE_VERSION_vendor:product" > > does not cause odd breakages. > > > > This is a good improvement. There is one more kink to it, where the vendored > subpackage might be there in source but we might have configured the recipe > to use the system version of the package instead, so how do we cater to such > situation ? Very good point. I think yocto project maintainer should recommend recipe maintainers what to do in these cases. I think safest option is to remove/delete such code paths from upstream sources in do_patch(). Debian does something similar also due to license compliance with their pristine/dfsg source package format https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#repackaged-upstream-source when they can't distributed certain files inside the source package. This issue goes way beyond CVE related data though. LICENSE, PN and PV are affected. AFAIK, currently recipe maintainers decide on what goes into LICENSE for example, based on how they have configured the SW component build and dependencies. These may not be correct though and for complex SW components this really is an issue where users who create real products need to check a lot of details. High level SW components like javascript engines and browser are very eager to build their own versions SW components which are already available as recipes in the build environment. At least for CVE scanning, this patch enables telling the tooling "this recipe embeds a custom version of tool/lib xyz" so recipe maintainers can fill in the data if they know and care about the details. For license compliance, I hope LICENSE is up to date :) though I know that the deeper on looks the more problems will pop up... Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189700): https://lists.openembedded.org/g/openembedded-core/message/189700 Mute This Topic: https://lists.openembedded.org/mt/102076964/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] cve-check.bbclass: support embedded SW components with different version number
On Fri, Oct 20, 2023 at 08:56:43AM +0100, Jose Quaresma wrote:
> Mikko Rapeli escreveu no dia quinta, 19/10/2023
> à(s) 13:45:
>
> > Hi,
> >
> > Could something like this work?
> >
> > --- a/meta/lib/oe/cve_check.py
> > +++ b/meta/lib/oe/cve_check.py
> > @@ -140,15 +140,14 @@ def get_patched_cves(d):
> > return patched_cves
> >
> >
> > -def get_cpe_ids(cve_product, version):
> > +def get_cpe_ids(cve_product, cve_version):
> > """
> > Get list of CPE identifiers for the given product and version
> > """
> >
> > -version = version.split("+git")[0]
> > -
> > cpe_ids = []
> > for product in cve_product.split():
> > +version = (d.getVar("CVE_VERSION_%s" % product) or
> > cve_version).split("+git")[0]
> >
>
> Looks like your patch fixes the remaining issue
> but don't know if it will be better to get the CVE_VERSION_ after
> splitting the vendor from the product
This is now in v2. For the CVE_VERSION_%s, it uses what ever product was defined
in CVE_PRODUCT space separated list so it is used before vendor and product
split.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189504):
https://lists.openembedded.org/g/openembedded-core/message/189504
Mute This Topic: https://lists.openembedded.org/mt/101991269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] cve-check.bbclass: support embedded SW components with different version number
Many recipes embed other SW components. The name and version of the
embedded SW component differs from the main recipe. To detect CVEs in the
embedded SW component, it needs to be added to CVE_PRODUCT list using
name of the SW product in CVE database or with "vendor:product" syntax.
Then the version of the embedded SW component can be set using
CVE_VERSION_product variable.
For example in meta-arm, trusted-firmware-a embeds mbed_tls SW component.
Thus trusted-firmware-a can add CVE_PRODUCT for it since CVE database
uses product name "mbed_tls":
CVE_PRODUCT += "mbed_tls"
and set the version of mbed_tls:
CVE_VERSION_mbed_tls = "2.28.4"
(Real patches for both are a bit more complex due to conditional build
enabling mbed_tls support and due to mbed_tls version being set in an
.inc file.)
Now trusted-firmware-a CVE check output shows:
NOTE: recipe
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: task
do_cve_check: Started
WARNING:
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0
do_cve_check: Found unpatched CVE (CVE-2021-36647 CVE-2021-43666 CVE-2021-45451
CVE-2023-43615), for more information check
/home/builder/src/base/build/tmp/work/arm64-poky-linux/trusted-firmware-a/2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122/temp/cve.log
NOTE: recipe
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: task
do_cve_check: Succeeded
Here CVE-2023-43615 is a newly added and fixed CVE in version 2.28.5 and the
CVEs
from 2021 need to be checked but are likely fixed in 2.28.3 and newer 2.28.y
releases.
Note that CVE-2023-43615 does not impact trusted-firmware-a since it doesn't use
TLS or null or RC4 ciphers, but I think it's a good idea to extend
CVE checker for this use case. I hope the "CVE_VERSION_vendor:product"
does not cause odd breakages.
Signed-off-by: Mikko Rapeli
---
meta/classes/create-spdx-2.2.bbclass | 2 +-
meta/classes/cve-check.bbclass | 28 +++-
meta/lib/oe/cve_check.py | 5 ++---
3 files changed, 22 insertions(+), 13 deletions(-)
v1: https://lists.openembedded.org/g/openembedded-core/message/189260
v2: adapt SPDX too
diff --git a/meta/classes/create-spdx-2.2.bbclass
b/meta/classes/create-spdx-2.2.bbclass
index b0aef80db1..5b842e67ee 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -590,7 +590,7 @@ python do_create_spdx() {
if patched_cves:
recipe.sourceInfo = "CVEs fixed: " + patched_cves
-cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"),
d.getVar("CVE_VERSION"))
+cpe_ids = oe.cve_check.get_cpe_ids(d, d.getVar("CVE_PRODUCT"),
d.getVar("CVE_VERSION"))
if cpe_ids:
for cpe_id in cpe_ids:
cpe = oe.spdx.SPDXExternalReference()
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index b55f4299da..9c41d54188 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -309,7 +309,16 @@ def check_cves(d, patched_cves):
# If this has been unset then we're not scanning for CVEs here (for
example, image recipes)
if not products:
return ([], [], [], [])
-pv = d.getVar("CVE_VERSION").split("+git")[0]
+
+# Version is PV, CVE_VERSION or CVE_VERSION_%s where %s is one of the
entries in CVE_PRODUCT.
+# Enables checking embedded SW component CVEs provided that CVE_PRODUCT
contains the embedded SW
+# component and that version of that component is set via
CVE_VERSION_embedded_component variable.
+pv = {}
+for product in products:
+version = (d.getVar("CVE_VERSION_%s" % product) or "").split("+git")[0]
+if version == "":
+version = d.getVar("CVE_VERSION").split("+git")[0]
+pv[product] = version
# If the recipe has been skipped/ignored we return empty lists
if pn in d.getVar("CVE_CHECK_SKIP_RECIPE").split():
@@ -329,6 +338,7 @@ def check_cves(d, patched_cves):
# For each of the known product names (e.g. curl has CPEs using curl and
libcurl)...
for product in products:
+full_product = product
cves_in_product = False
if ":" in product:
vendor, product = product.split(":", 1)
@@ -341,7 +351,7 @@ def check_cves(d, patched_cves):
cve = cverow[0]
if cve in cve_ignore:
-bb.note("%s-%s ignores %s" % (product, pv, cve))
+bb.note("%s-%s ignores %s" % (product, pv[full_product], cve))
cves_ignored.append(cve)
continue
elif cve in patched_cves:
@@ -366,27 +376,27 @@ def check_cves(d, patched_cves):
version_start = convert_cve_version(version
Re: [OE-core] [PATCH] cve-check.bbclass: support embedded SW components with different version number
Hi,
Could something like this work?
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -140,15 +140,14 @@ def get_patched_cves(d):
return patched_cves
-def get_cpe_ids(cve_product, version):
+def get_cpe_ids(cve_product, cve_version):
"""
Get list of CPE identifiers for the given product and version
"""
-version = version.split("+git")[0]
-
cpe_ids = []
for product in cve_product.split():
+version = (d.getVar("CVE_VERSION_%s" % product) or
cve_version).split("+git")[0]
# CVE_PRODUCT in recipes may include vendor information for CPE
identifiers. If not,
# use wildcard for vendor.
if ":" in product:
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189455):
https://lists.openembedded.org/g/openembedded-core/message/189455
Mute This Topic: https://lists.openembedded.org/mt/101991269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] cve-check.bbclass: support embedded SW components with different version number
Hi, On Thu, Oct 19, 2023 at 12:54:44PM +0100, Jose Quaresma wrote: > Hi > > This change will need some adaptations in the create-spdx.bbclass to handle > this new variable with _PN Good point. How does SPDX tooling handle embedded SW components in recipe sources? I presume it does not because recipe and license don't handle it either. Should there be a more generic PN_subpn, PV_subpn, LICENSE_subpn and matching CVE_PRODUCT and CVE_VERSION? I don't have use cases for these currently. I would like to fix the CVE reporting issues with embedded SW components though. mbedtls being one good example. Or would it be better to convert mbedtls users to use the meta-oe side recipe for it? Additionally I don't currently read the SDPX output. I don't have use cases for it. I do check recipes and their metadata like LICENSE though. Feels like the SDPX data is used as reporting/export data format which is fed to some other tools which are not open source. Can of worms... Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189450): https://lists.openembedded.org/g/openembedded-core/message/189450 Mute This Topic: https://lists.openembedded.org/mt/101991269/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] cve-check.bbclass: support embedded SW components with different version number
Hi, On Thu, Oct 19, 2023 at 10:19:53AM +0200, Marta Rybczynska wrote: > On Mon, Oct 16, 2023 at 9:01 AM Mikko Rapeli wrote: > > > > Many recipes embed other SW components. The name and version of the > > embedded SW component differs from the main recipe. To detect CVEs in the > > embedded SW component, it needs to be added to CVE_PRODUCT list using > > name of the SW product in CVE database or with "vendor:product" syntax. > > Then the version of the embedded SW component can be set using > > CVE_VERSION_product variable. > > > > For example in meta-arm, trusted-firmware-a embeds mbed_tls SW component. > > Thus trusted-firmware-a can add CVE_PRODUCT for it since CVE database > > uses product name "mbed_tls": > > > > CVE_PRODUCT += "mbed_tls" > > > > and set the version of mbed_tls: > > > > CVE_VERSION_mbed_tls = "2.28.4" > > > > (Real patches for both are a bit more complex due to conditional build > > enabling mbed_tls support and due to mbed_tls version being set in an > > .inc file.) > > > > I like the support for embedded software. In this approach, I'm wondering > how it would work for packages like curl that have multiple CPEs. Would we > need to duplicate the list of CPEs? The current approach of listing multiple CPEs in CVE_PRODUCT still works. It's just possible to include a different version for an entry in CVE_PRODUCT via CVE_VERSION_swcomponent variable. It will fall back to PV. > There are layers/recipes where we have a very long list of embedded > components, > meta-zephyr is probably the best example. Yes, I think this embedding of SW components is very common. I think some of the LICENSE data does reflect this but not in all cases. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189437): https://lists.openembedded.org/g/openembedded-core/message/189437 Mute This Topic: https://lists.openembedded.org/mt/101991269/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: Kernel 6.5 ttyS1 hang with qemu (was Re: [OE-core] Summary of the remaining 6.5 kernel serial issue (and 6.5 summary)
Hi, On Mon, Oct 16, 2023 at 10:23:52AM +0300, Tony Lindgren wrote: > * Mikko Rapeli [231016 07:16]: > > On Mon, Oct 16, 2023 at 09:35:01AM +0300, Tony Lindgren wrote: > > > * Richard Purdie [231015 21:30]: > > > > On Sun, 2023-10-15 at 17:31 +0200, Greg Kroah-Hartman wrote: > > > > > Can you try the patch below? I just sent it to Linus and it's from > > > > > Tony > > > > > to resolve some other pm issues with the serial port code. > > > > > > > > Thanks for the pointer to this. I've put it through some testing and > > > > had one failure so far so I suspect this isn't enough unfortunately. > > > > > > > > FWIW I was looping the testing on the complete removal of the > > > > conditions and didn't see any failures with that. > > > > > > Care to clarify what's the failing test now? > > > > > > Is the issue still the second port not always coming up after boot or > > > something else? > > > > Yes, data from the ttyS1 getty is not coming through from kernel and qemu to > > the test framework looking for login prompt after qemu machine boot. > > Workarounds like sending "\n\n" from the test framework through qemu to > > ttyS1 > > or "echo helloB > /dev/ttyS1" via working ttyS0 don't seem to help and wake > > it up. > > OK so for trying to reproduce this with qemu, is this with the default uarts > or with some -device pci-serial-2x type options? Here is how yocto test frameworks starts qemu: qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:35:07 -netdev user,id=net0,hostfwd=tcp:127.0.0.1:2225-:22,hostfwd=tcp:127.0.0.1:2325-:23,tftp=/home/builder/src/base/build_test/tmp/deploy/images/qemux86-64 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -drive file=/home/builder/src/base/build_test/tmp/deploy/images/qemux86-64/core-image-ptest-openssh-qemux86-64.rootfs.ext4,if=virtio,format=raw -usb -device usb-tablet -usb -device usb-kbd -cpu IvyBridge -machine q35,i8042=off -smp 4 -enable-kvm -m 1024 -serial tcp:127.0.0.1:46313 -serial tcp:127.0.0.1:53891 -pidfile /home/builder/src/base/build_test/pidfile_1926373 -S -qmp unix:./.lzkynxe1,server,wait -qmp unix:./.qcpdyz4j,server,nowait -snapshot -nographic -kernel /home/builder/src/base/build_test/tmp/deploy/images/qemux86-64/bzImage -append 'root=/dev/vda rw ip=dhcp console=ttyS0 console=ttyS1 oprofile.timer=1 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 swiotlb=1 printk.time=1' Example boot log when this ttyS1 failure happens is here: https://pastebin.com/raw/jRRa2CwW Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189265): https://lists.openembedded.org/g/openembedded-core/message/189265 Mute This Topic: https://lists.openembedded.org/mt/101849211/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: Kernel 6.5 ttyS1 hang with qemu (was Re: [OE-core] Summary of the remaining 6.5 kernel serial issue (and 6.5 summary)
Hi, On Mon, Oct 16, 2023 at 09:35:01AM +0300, Tony Lindgren wrote: > * Richard Purdie [231015 21:30]: > > On Sun, 2023-10-15 at 17:31 +0200, Greg Kroah-Hartman wrote: > > > Can you try the patch below? I just sent it to Linus and it's from Tony > > > to resolve some other pm issues with the serial port code. > > > > Thanks for the pointer to this. I've put it through some testing and > > had one failure so far so I suspect this isn't enough unfortunately. > > > > FWIW I was looping the testing on the complete removal of the > > conditions and didn't see any failures with that. > > Care to clarify what's the failing test now? > > Is the issue still the second port not always coming up after boot or > something else? Yes, data from the ttyS1 getty is not coming through from kernel and qemu to the test framework looking for login prompt after qemu machine boot. Workarounds like sending "\n\n" from the test framework through qemu to ttyS1 or "echo helloB > /dev/ttyS1" via working ttyS0 don't seem to help and wake it up. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189261): https://lists.openembedded.org/g/openembedded-core/message/189261 Mute This Topic: https://lists.openembedded.org/mt/101849211/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] cve-check.bbclass: support embedded SW components with different version number
Many recipes embed other SW components. The name and version of the
embedded SW component differs from the main recipe. To detect CVEs in the
embedded SW component, it needs to be added to CVE_PRODUCT list using
name of the SW product in CVE database or with "vendor:product" syntax.
Then the version of the embedded SW component can be set using
CVE_VERSION_product variable.
For example in meta-arm, trusted-firmware-a embeds mbed_tls SW component.
Thus trusted-firmware-a can add CVE_PRODUCT for it since CVE database
uses product name "mbed_tls":
CVE_PRODUCT += "mbed_tls"
and set the version of mbed_tls:
CVE_VERSION_mbed_tls = "2.28.4"
(Real patches for both are a bit more complex due to conditional build
enabling mbed_tls support and due to mbed_tls version being set in an
.inc file.)
Now trusted-firmware-a CVE check output shows:
NOTE: recipe
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: task
do_cve_check: Started
WARNING:
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0
do_cve_check: Found unpatched CVE (CVE-2021-36647 CVE-2021-43666 CVE-2021-45451
CVE-2023-43615), for more information check
/home/builder/src/base/build/tmp/work/arm64-poky-linux/trusted-firmware-a/2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122/temp/cve.log
NOTE: recipe
trusted-firmware-a-2.9.0+gitd3e71ead6ea5bc3555ac90a446efec84ef6c6122-r0: task
do_cve_check: Succeeded
Here CVE-2023-43615 is a newly added and fixed CVE in version 2.28.5 and the
CVEs
from 2021 need to be checked but are likely fixed in 2.28.3 and newer 2.28.y
releases.
Note that CVE-2023-43615 does not impact trusted-firmware-a since it doesn't use
TLS or null or RC4 ciphers, but I think it's a good idea to extend
CVE checker for this use case. I hope the "CVE_VERSION_vendor:product"
does not cause odd breakages.
Signed-off-by: Mikko Rapeli
---
meta/classes/cve-check.bbclass | 28 +++-
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index b55f4299da..9c41d54188 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -309,7 +309,16 @@ def check_cves(d, patched_cves):
# If this has been unset then we're not scanning for CVEs here (for
example, image recipes)
if not products:
return ([], [], [], [])
-pv = d.getVar("CVE_VERSION").split("+git")[0]
+
+# Version is PV, CVE_VERSION or CVE_VERSION_%s where %s is one of the
entries in CVE_PRODUCT.
+# Enables checking embedded SW component CVEs provided that CVE_PRODUCT
contains the embedded SW
+# component and that version of that component is set via
CVE_VERSION_embedded_component variable.
+pv = {}
+for product in products:
+version = (d.getVar("CVE_VERSION_%s" % product) or "").split("+git")[0]
+if version == "":
+version = d.getVar("CVE_VERSION").split("+git")[0]
+pv[product] = version
# If the recipe has been skipped/ignored we return empty lists
if pn in d.getVar("CVE_CHECK_SKIP_RECIPE").split():
@@ -329,6 +338,7 @@ def check_cves(d, patched_cves):
# For each of the known product names (e.g. curl has CPEs using curl and
libcurl)...
for product in products:
+full_product = product
cves_in_product = False
if ":" in product:
vendor, product = product.split(":", 1)
@@ -341,7 +351,7 @@ def check_cves(d, patched_cves):
cve = cverow[0]
if cve in cve_ignore:
-bb.note("%s-%s ignores %s" % (product, pv, cve))
+bb.note("%s-%s ignores %s" % (product, pv[full_product], cve))
cves_ignored.append(cve)
continue
elif cve in patched_cves:
@@ -366,27 +376,27 @@ def check_cves(d, patched_cves):
version_start = convert_cve_version(version_start)
version_end = convert_cve_version(version_end)
-if (operator_start == '=' and pv == version_start) or
version_start == '-':
+if (operator_start == '=' and pv[full_product] ==
version_start) or version_start == '-':
vulnerable = True
else:
if operator_start:
try:
-vulnerable_start = (operator_start == '>=' and
Version(pv,suffix) >= Version(version_start,suffix))
-vulnerable_start |= (operator_start == '>' and
Version(pv,suffix) > Version(version_start,suffix))
+vulnerable_start = (operator_start == '>=' and
Version(pv[full_product],suffix) >= Version(version_start,suffix))
+vulnerable_start
Re: Kernel 6.5 ttyS1 hang with qemu (was Re: [OE-core] Summary of the remaining 6.5 kernel serial issue (and 6.5 summary)
; serial_core_remove_one_port > 84a9582fd203 serial: core: Start managing serial controllers to enable > runtime PM *** < FAIL (33/1436) > ae62c49c0cef serial: 8250: omap: convert to modern PM ops > < PASS (0/1593) > c53aab207622 serial: 8250: omap: Move uart_write() inside PM section > d2d4bd217ccd serial: 8250-fsl: Expand description of the MPC83xx UART's > misbehaviour > 1b997aef4f54 serial: Indicate fintek option may also be required for RS232 > support > 20ec397d694b tty: serial: fsl_lpuart: Check the return value of > dmaengine_tx_status < PASS (0/316) > 0d07703be74f serial: Switch i2c drivers back to use .probe() > d0b309a5d3f4 serial: 8250: synchronize and annotate UART_IER access > 25614735a647 serial: 8250: lock port for UART_IER access in omap8250_irq() > 8b45503776b6 serial: 8250: lock port for omap8250_restore_regs() > 87660fb4041f serial: 8250: lock port for rx_dma() callback > 51e45fba14bf serial: core: lock port for start_rx() in uart_resume_port() > ca73a892c5be serial: 8250: lock port for stop_rx() in omap8250_irq() > abcb0cf1f5b2 serial: core: lock port for stop_rx() in uart_suspend_port() > b1207d86169d serial: 8250: lock port in startup() callbacks > 0e4daea31d83 Merge 6.4-rc3 into tty-next > d5b3d02d0b10 serial: Make uart_remove_one_port() return void > 6bd6cd29c924 serial: stm32: Ignore return value of uart_remove_one_port() in > .remove() > 72fc578b220c serial: 8250_rt288x: Remove unnecessary UART_REG_UNMAPPED > 33e3b0eb7550 serial: 8250_rt288x: Name non-standard divisor latch reg > b334214ea08d serial: 8250: RT288x/Au1xxx code away from core > 30c61f53fdf2 serial: 8250: Add dl_read/write, bugs and mapsize into > plat_serial8250_port > 98658ae8f392 serial: 8250: Document uart_8250_port's ->dl_read/write() > b245aa0cc583 serial: 8250: Change dl_read/write to handle value as u32 > 9d86719f8769 serial: 8250: Allow using ports higher than > SERIAL_8250_RUNTIME_UARTS > db86bb6ed497 serial: 8250: omap: Shut down on remove for console uart > fef4f600319e serial: 8250: omap: Fix life cycle issues for interrupt handlers > 398cecc24846 serial: 8250: omap: Fix imprecise external abort for > omap_8250_pm() > b9ab22c2bc86 serial: 8250: omap: Fix freeing of resources on failed register > 6d8c1fca0a6e serial: pl011: set UART011_CR_RXE in pl011_set_termios after > port shutdown > cf9aa72d2f91 tty: serial: fsl_lpuart: optimize the timer based EOP logic > > Baseline of tty queue for v6.5 was: > $git show --oneline --stat cf9aa72d2f91~ > ac9a78681b92 (tag: v6.4-rc1) Linux 6.4-rc1 > Makefile | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > Have also 1000+ runs on Yocto's v6.4.16 based kernel with no fails. On Mon, Oct 09, 2023 at 01:05:23PM +0300, Mikko Rapeli via lists.openembedded.org wrote: > Hi linux-serial and Greg, > > Yocto Linux distro maintainer Richard Purdie is seeing a regression or > behavior > change after updating kernel from 6.4 to 6.5. Yocto runs a lot of automated > tests with qemu > where a python test framework configures and spawns qemu (version 8.1) with > two > serial ports and boots a newly built kernel and rootfs there. The python test > framework > tries to detect getty login prompt from ttyS1 while boot time logging > from qemu, kernel, userspace etc happens on serial console ttyS0. This has > worked > reliably for years and across multiple kernel versions. Now with 6.5 kernel > there are > suddenly frequent failures where ttyS1 doesn't show the login prompt in the > python code > which listens on the qemu socket from target system ttyS1. The target system > ttyS0 is working > and boot time logs and getty prompt are visible there. The ttyS1 logs from > test framework show > that only two characters CR and LF make it through. This same happens on > x86_64 host > running qemu machine for x86_64 and arm64 targets. > > The kernel boot logs from target system with 6.5.5 don't show any errors and > getty > processes are started correctly and they don't see any errors either. Data > from > target system userspace is being written, but seems to sometimes hang inside > kernel > or qemu serial port buffers, and only on the ttyS1 while ttyS0 works reliably > all > the time. > > Example target logs with 6.5.5 via ttyS0: > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853 > or https://pastebin.com/raw/jRRa2CwW > > ttyS1 logs from the same run: > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853.2 > > Kernel config: > https://pastebin.com/raw/Lv9Hfeuh > > Do you ha
Re: [OE-core] [PATCH] oeqa/qemurunner: Add extra logging when console doesn't appear
Reviewed-by: Mikko Rapeli -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188875): https://lists.openembedded.org/g/openembedded-core/message/188875 Mute This Topic: https://lists.openembedded.org/mt/101871106/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/2] oeqa/qemurunner: Add newlines serial workaround
Reviewed-by: Mikko Rapeli -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188874): https://lists.openembedded.org/g/openembedded-core/message/188874 Mute This Topic: https://lists.openembedded.org/mt/101856880/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Kernel 6.5 ttyS1 hang with qemu (was Re: [OE-core] Summary of the remaining 6.5 kernel serial issue (and 6.5 summary)
Hi linux-serial and Greg, Yocto Linux distro maintainer Richard Purdie is seeing a regression or behavior change after updating kernel from 6.4 to 6.5. Yocto runs a lot of automated tests with qemu where a python test framework configures and spawns qemu (version 8.1) with two serial ports and boots a newly built kernel and rootfs there. The python test framework tries to detect getty login prompt from ttyS1 while boot time logging from qemu, kernel, userspace etc happens on serial console ttyS0. This has worked reliably for years and across multiple kernel versions. Now with 6.5 kernel there are suddenly frequent failures where ttyS1 doesn't show the login prompt in the python code which listens on the qemu socket from target system ttyS1. The target system ttyS0 is working and boot time logs and getty prompt are visible there. The ttyS1 logs from test framework show that only two characters CR and LF make it through. This same happens on x86_64 host running qemu machine for x86_64 and arm64 targets. The kernel boot logs from target system with 6.5.5 don't show any errors and getty processes are started correctly and they don't see any errors either. Data from target system userspace is being written, but seems to sometimes hang inside kernel or qemu serial port buffers, and only on the ttyS1 while ttyS0 works reliably all the time. Example target logs with 6.5.5 via ttyS0: https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853 or https://pastebin.com/raw/jRRa2CwW ttyS1 logs from the same run: https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853.2 Kernel config: https://pastebin.com/raw/Lv9Hfeuh Do you have any ideas or hints how to debug or possibly fix this? tty layer has seen quite a few major changes between 6.4.14 and 6.5.5 and we're not able to pinpoint or bisect the issue, since it so annoyingly rare. Some more details from Richard below. Cheers, -Mikko On Sun, Oct 08, 2023 at 09:23:57AM +0100, Richard Purdie wrote: > On Sat, 2023-10-07 at 23:05 +0100, Richard Purdie via > lists.openembedded.org wrote: > > I thought I'd summarise where things are at with the 6.5 kernel. > > > > We've fixed: > > * the ARM LTP OOM lockup (kernel patch) > > * the locale ARM selftest failure which was OOM due to silly buffer > > allocations in 6.5 (kernel commandline) > > * the ARM jitterentropy errors (kernel patch) > > * the cryptodev build failures (recipe updated) > > > > We've also: > > * disabled the strace tests that fail with 6.5. > > * made sure the serial ports and getty counts match > > * added ttyrun which wraps serial consoles and avoids hacks > > * made the qemurunner logging save all the port logs > > * made the qemurunner write the binary data it is sent verbatim > > * made sure to use nodelay on qemu's tcpserial > > > > This leaves an annoying serial console problem where ttyS1 never has > > the getty login prompt appear. > > > > What we know: > > > > * We've only seen this on x86 more recently (yesterday/today) but have > > seen it on ARM in the days before that. > > > > * It affects both sysvinit and systemd images. > > > > * Systemd does print that it started a getty on ttyS0 and ttyS1 when > > the failure occurs > > > > * There is a getty running according to "ps" when the failure occurs > > > > * There are only ever one or three characters received to ttyS1 in the > > failure case (0x0d and 0x0a chars, i.e. CR and LF) > > > > * It can't be any kind of utf-8 conversion issue since the login prompt > > isn't visible in the binary log > > > > * the kernel boot logs do show the serial port created with the same > > ioport and irq on x86. > > > > Previously we did see some logs with timing issues on the ttyS0 port > > but the nodelay parameter may have helped with that. > > > > There are debug patches in master-next against qemurunner which try and > > poke around to gather more debug when things fail using ttyS0. > > > > The best failure log we have is now this one: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5874/steps/14/logs/stdio > > > > where I've saved the logs: > > > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853 > > and > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/j/qemu_boot_log.20231007084853.2 > > > > You can see ttyS1 times out after 1000 seconds and the port only has a > > single byte (in the .2 file). The other log shows ps output showing the > > getty running for ttyS1. > > > > Ideas welcome on where from here. > > > > I've tweaked master-next to keep reading the ttyS1 port after we poke > > it from ttyS0 to see if that reveals anything next time it fails (build > > running). > > Testing overnight with the new debug yielded: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5895/steps/14/logs/stdio > > The interesting bit being: >
[OE-core] [PATCH] openssh: drop sudo from ptest dependencies
The tests don't actually need sudo on core-image-ptest-openssh.
Based on logs seen in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178 it seems
that socket errors from sudo are creeping into stderr which are failing
the banner ptest from openssh. Removing sudo should help removing
the stderr messages and possibly cure the banner test failures.
Signed-off-by: Mikko Rapeli
---
meta/recipes-connectivity/openssh/openssh/run-ptest | 2 +-
meta/recipes-connectivity/openssh/openssh_9.4p1.bb | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/recipes-connectivity/openssh/openssh/run-ptest
b/meta/recipes-connectivity/openssh/openssh/run-ptest
index 1e6eec5799..b2244d725a 100755
--- a/meta/recipes-connectivity/openssh/openssh/run-ptest
+++ b/meta/recipes-connectivity/openssh/openssh/run-ptest
@@ -19,7 +19,7 @@ output_failed_logs() {
trap output_failed_logs 0
sed -i "/\t\tagent-ptrace /d" Makefile
-make -k BUILDDIR=`pwd`/.. .OBJDIR=`pwd` .CURDIR=`pwd` SUDO="sudo" tests \
+make -k BUILDDIR=`pwd`/.. .OBJDIR=`pwd` .CURDIR=`pwd` SUDO="" tests \
| sed -u -e 's/^skipped/SKIP: /g' -e 's/^ok /PASS: /g' -e
's/^failed/FAIL: /g'
SSHAGENT=`which ssh-agent`
diff --git a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
index a38d9c2b81..e2508aa63a 100644
--- a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
@@ -158,7 +158,7 @@ FILES:${PN}-keygen = "${bindir}/ssh-keygen"
RDEPENDS:${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen
${PN}-sftp-server"
RDEPENDS:${PN}-sshd += "${PN}-keygen ${@bb.utils.contains('DISTRO_FEATURES',
'pam', 'pam-plugin-keyinit pam-plugin-loginuid', '', d)}"
# gdb would make attach-ptrace test pass rather than skip but not worth the
build dependencies
-RDEPENDS:${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make sed sudo
coreutils openssl-bin"
+RDEPENDS:${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make sed
coreutils openssl-bin"
RPROVIDES:${PN}-ssh = "ssh"
RPROVIDES:${PN}-sshd = "sshd"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188775):
https://lists.openembedded.org/g/openembedded-core/message/188775
Mute This Topic: https://lists.openembedded.org/mt/101799351/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 5/5] qemurunner: Show both the login console log and all logging upon failure
Hi, On Thu, Oct 05, 2023 at 04:47:16PM +0100, Richard Purdie wrote: > On Thu, 2023-10-05 at 18:27 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Wed, Oct 04, 2023 at 09:35:20PM +0100, Richard Purdie wrote: > > > On Wed, 2023-10-04 at 22:34 +0300, Mikko Rapeli wrote: > > > > Hi, > > > > > > > > On Wed, Oct 04, 2023 at 02:29:40PM +0100, Richard Purdie wrote: > > > > > On Wed, 2023-10-04 at 16:07 +0300, Mikko Rapeli wrote: > > > > > > Hi, > > > > > > > > > > > > Acked-by: Mikko Rapeli > > > > > > > > > > > > for the full series. I hope our discussion over #yocto irc channel > > > > > > helped resolve these and the test failures seen with master-next. > > > > > > > > > > Yes, thanks! It was useful to have someone to talk the issues through > > > > > with and much appreciated. > > > > > > > > > > Sadly I don't think we've found all the issues yet. > > > > > > > > > > On the build N-1 we have a selftest failure on debian12-ty-1: > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5803/steps/14/logs/stdio > > > > > > > > Can I get from this test run stdio to the more detailed task, > > > > qemurunner or qemu boot logs somehow? > > > > Or do you have to dig into the system manually to capture them, like in > > > > the example below? > > > > > > > > I would like to check into the detailed logs from both passing and > > > > failing test runs. > > > > Of course kernel dmesg would need to be there so maybe after your patch > > > > to the runqemu > > > > side at least. > > > > > > > > If there is a regression in kernel to the serial device detection and > > > > numbering, > > > > then we need some way to see e.g. 8250 uart/tty0/ttyAMA0 related dmesg > > > > messages. > > > > > > I can't capture everything in time in each case but I did grab: > > > > > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/e/ > > > > > > which is from > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7863/steps/21/logs/stdio > > > > > > which has this threads patches applied. > > > > > > I've put two sets of logs there from the same builds, one success, one > > > is missing the getty on one log. > > > > > > > > > I also have the logs from: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5848/steps/14/logs/stdio > > > > > > which I've saved here: > > > > > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/f/ > > > > > > noting that: > > > > > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/f/qemu_boot_log.20231004134427 > > > > > > is 3 bytes in size. > > > > All this is very confusing indeed. I sent another version of the detect > > login prompt without > > utf-8 conversion, with some luck that could help. At least on some of the > > logs the login prompt > > is there but with lots of garbage around it. Serial port core got > > refactored in 6.5 and diff to > > 6.4 is quite large so there could be bugs there, and those could be > > triggering new > > errors visible to userspace too. > > With the changes I've made to the logging and the raw log files we have > without any utf-8 conversion, I'm now 99% certain this isn't any utf > conversion issue since the login prompts are not in the raw binary > files and those are written with no conversion. It is much more likely > the getty isn't running or isn't communicating somehow. > > I'm trying to work out how to add some further debug/testing when this > fails in a way it will help us debug. Alright, I was confused by logs from working login detection. I presume qemu is configured with two serial ports to avoid mixing kernel messages with the login prompt. It would be easy to grep for the login prompt also from the first serial output though it would not explain the root cause for these issues but could allow testing to move forward to the ssh connection. In all of the logs I saw, the first serial port had a login prompt. The serial ports are not used in testing, AFAIK. Oh now there are the debug commands in master-next. Nice, I was hoping for this kind of data in logs when things go horribly wrong. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188749): https://lists.openembedded.org/g/openembedded-core/message/188749 Mute This Topic: https://lists.openembedded.org/mt/101753837/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 5/5] qemurunner: Show both the login console log and all logging upon failure
Hi, On Wed, Oct 04, 2023 at 09:35:20PM +0100, Richard Purdie wrote: > On Wed, 2023-10-04 at 22:34 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Wed, Oct 04, 2023 at 02:29:40PM +0100, Richard Purdie wrote: > > > On Wed, 2023-10-04 at 16:07 +0300, Mikko Rapeli wrote: > > > > Hi, > > > > > > > > Acked-by: Mikko Rapeli > > > > > > > > for the full series. I hope our discussion over #yocto irc channel > > > > helped resolve these and the test failures seen with master-next. > > > > > > Yes, thanks! It was useful to have someone to talk the issues through > > > with and much appreciated. > > > > > > Sadly I don't think we've found all the issues yet. > > > > > > On the build N-1 we have a selftest failure on debian12-ty-1: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5803/steps/14/logs/stdio > > > > Can I get from this test run stdio to the more detailed task, qemurunner or > > qemu boot logs somehow? > > Or do you have to dig into the system manually to capture them, like in the > > example below? > > > > I would like to check into the detailed logs from both passing and failing > > test runs. > > Of course kernel dmesg would need to be there so maybe after your patch to > > the runqemu > > side at least. > > > > If there is a regression in kernel to the serial device detection and > > numbering, > > then we need some way to see e.g. 8250 uart/tty0/ttyAMA0 related dmesg > > messages. > > I can't capture everything in time in each case but I did grab: > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/e/ > > which is from > > https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7863/steps/21/logs/stdio > > which has this threads patches applied. > > I've put two sets of logs there from the same builds, one success, one > is missing the getty on one log. > > > I also have the logs from: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5848/steps/14/logs/stdio > > which I've saved here: > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/f/ > > noting that: > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/f/qemu_boot_log.20231004134427 > > is 3 bytes in size. All this is very confusing indeed. I sent another version of the detect login prompt without utf-8 conversion, with some luck that could help. At least on some of the logs the login prompt is there but with lots of garbage around it. Serial port core got refactored in 6.5 and diff to 6.4 is quite large so there could be bugs there, and those could be triggering new errors visible to userspace too. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188725): https://lists.openembedded.org/g/openembedded-core/message/188725 Mute This Topic: https://lists.openembedded.org/mt/101753837/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH][master-next] qemurunner.py: detect login prompt without without utf-8 conversion
In case utf-8 conversion of the serial console data is flaky due to bad characters in the stream. Signed-off-by: Mikko Rapeli --- meta/lib/oeqa/utils/qemurunner.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py index 9aa2e7b3be..5fed705977 100644 --- a/meta/lib/oeqa/utils/qemurunner.py +++ b/meta/lib/oeqa/utils/qemurunner.py @@ -480,8 +480,7 @@ class QemuRunner: self.log(data, extension = ".2") data = b'' -decodedlog = self.decode_qemulog(bootlog) -if self.boot_patterns['search_reached_prompt'] in decodedlog: +if bytes(self.boot_patterns['search_reached_prompt'], 'utf-8') in bootlog: self.server_socket.close() self.server_socket = qemusock stopread = True -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188724): https://lists.openembedded.org/g/openembedded-core/message/188724 Mute This Topic: https://lists.openembedded.org/mt/101778733/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 5/5] qemurunner: Show both the login console log and all logging upon failure
Hi, On Wed, Oct 04, 2023 at 02:29:40PM +0100, Richard Purdie wrote: > On Wed, 2023-10-04 at 16:07 +0300, Mikko Rapeli wrote: > > Hi, > > > > Acked-by: Mikko Rapeli > > > > for the full series. I hope our discussion over #yocto irc channel > > helped resolve these and the test failures seen with master-next. > > Yes, thanks! It was useful to have someone to talk the issues through > with and much appreciated. > > Sadly I don't think we've found all the issues yet. > > On the build N-1 we have a selftest failure on debian12-ty-1: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5803/steps/14/logs/stdio Can I get from this test run stdio to the more detailed task, qemurunner or qemu boot logs somehow? Or do you have to dig into the system manually to capture them, like in the example below? I would like to check into the detailed logs from both passing and failing test runs. Of course kernel dmesg would need to be there so maybe after your patch to the runqemu side at least. If there is a regression in kernel to the serial device detection and numbering, then we need some way to see e.g. 8250 uart/tty0/ttyAMA0 related dmesg messages. Cheers, -Mikko > with these logs saved: > > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/d/qemurunner_log.20231004022851 > https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/d/qemu_boot_log.20231004022851 > > which suggest that ttyS1 never got any output but a getty did appear on > ttyS0. I confirmed there were only two serial devices in that > configuration. The exact image downloaded off that worker and run > locally worked so it isn't the image, it has to be some kind of race. > The image does have gettys configured for both serial ports. The kernel > boot log isn't present which a patch in this series attempts to fix. > > There was also: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5528/steps/13/logs/stdio > > which looks again like a getty didn't show up on a serial port. We > don't have more logs for that one but "Running postinst /etc/rpm- > postinsts/100-sysvinit-inittab" worries Ross and I a bit. It may or may > not be related. > > That brings me to: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2199 > > which has two failures, I think both in commands run over the serial > interface. These ones appear to be more reproducible on ARM. > > I'm trying a new build run with this series applied to see if that > helps shed any light on what is going on. > > Cheers, > > Richard > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188698): https://lists.openembedded.org/g/openembedded-core/message/188698 Mute This Topic: https://lists.openembedded.org/mt/101753837/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 5/5] qemurunner: Show both the login console log and all logging upon failure
Hi, Acked-by: Mikko Rapeli for the full series. I hope our discussion over #yocto irc channel helped resolve these and the test failures seen with master-next. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188685): https://lists.openembedded.org/g/openembedded-core/message/188685 Mute This Topic: https://lists.openembedded.org/mt/101753837/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/2] qemurunner.py: capture raw boot log
Hi, These are for master-next. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188632): https://lists.openembedded.org/g/openembedded-core/message/188632 Mute This Topic: https://lists.openembedded.org/mt/101731194/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] qemurunner.py: capture raw boot log
Don't process captured boot log in case it contains invalid utf8 characters etc which may filter out important log entries. Signed-off-by: Mikko Rapeli --- meta/lib/oeqa/utils/qemurunner.py | 1 - 1 file changed, 1 deletion(-) diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py index 3cd656df82..df290fc091 100644 --- a/meta/lib/oeqa/utils/qemurunner.py +++ b/meta/lib/oeqa/utils/qemurunner.py @@ -115,7 +115,6 @@ class QemuRunner: def log(self, msg): if self.logfile: -self.msg += self.decode_qemulog(msg) with codecs.open(self.logfile, "ab") as f: f.write(msg) -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188630): https://lists.openembedded.org/g/openembedded-core/message/188630 Mute This Topic: https://lists.openembedded.org/mt/101731194/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] qemurunner.py: detect login prompt from byte stream
Converting boot log to utf-8 strings may drop content
breaking the prompt detection resulting in timeouts.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/utils/qemurunner.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/utils/qemurunner.py
b/meta/lib/oeqa/utils/qemurunner.py
index df290fc091..27166282ea 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -482,9 +482,7 @@ class QemuRunner:
data = b''
-decodedlog = self.decode_qemulog(bootlog)
-self.logger.debug('Searching in %s chars' %
len(decodedlog))
-if self.boot_patterns['search_reached_prompt'] in
decodedlog:
+if bytes(self.boot_patterns['search_reached_prompt'],
'utf-8') in bootlog:
self.server_socket.close()
self.server_socket = qemusock
stopread = True
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188631):
https://lists.openembedded.org/g/openembedded-core/message/188631
Mute This Topic: https://lists.openembedded.org/mt/101731195/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa qemurunner.py: switch utf8 error handling from ignore to replace
Hi, This is actually for the issues seen by Richard on master-next with 6.5 kernel. Somehow login prompt detection is failing but raw log data shows it correctly. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#188613): https://lists.openembedded.org/g/openembedded-core/message/188613 Mute This Topic: https://lists.openembedded.org/mt/101712722/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] oeqa qemurunner.py: switch utf8 error handling from ignore to replace
It seems like "ignore" is stopping decoding and failing to detect
login prompts. Maybe "replace" works more reliably when reads from
serial console are incomplete and we're trying to detect the login
prompt. See logs like:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5833/steps/14/logs/stdio
https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/log.do_testimage.3051530
https://autobuilder.yocto.io/pub/failed-builds-data/6.5%20kernel/qemu_boot_log.20231001114436
Python documentation is not clear on how "ignore" and "replace" differ
and both should continue processing if there are errors in the stream:
https://docs.python.org/3/library/codecs.html#error-handlers
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/utils/qemurunner.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/utils/qemurunner.py
b/meta/lib/oeqa/utils/qemurunner.py
index b24091e212..5c18ccbd3e 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -109,7 +109,7 @@ class QemuRunner:
def decode_qemulog(self, todecode):
# Sanitize the data received from qemu as it may contain control
characters
-msg = todecode.decode("utf-8", errors='ignore')
+msg = todecode.decode("utf-8", errors='replace')
msg = re_control_char.sub('', msg)
return msg
@@ -474,7 +474,7 @@ class QemuRunner:
self.logger.error('Invalid file type: %s\n%s' % (file))
read = b''
-self.logger.debug2('Partial boot log:\n%s' %
(read.decode('utf-8', errors='ignore')))
+self.logger.debug2('Partial boot log:\n%s' %
(read.decode('utf-8', errors='replace')))
data = data + read
if data:
bootlog += data
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188612):
https://lists.openembedded.org/g/openembedded-core/message/188612
Mute This Topic: https://lists.openembedded.org/mt/101712722/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] openssh: capture logs in run-ptest
Drop patch to improve logging since upstream rejected it
but capture failure logs in run-ptests with similar code
as what upstream uses when running the tests via
https://github.com/openssh/openssh-portable/blob/master/.github/run_test.sh#L23
Signed-off-by: Mikko Rapeli
---
...ss-Makefile-print-logs-if-test-fails.patch | 34 ---
.../openssh/openssh/run-ptest | 14
.../openssh/openssh_9.4p1.bb | 1 -
3 files changed, 14 insertions(+), 35 deletions(-)
delete mode 100644
meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
diff --git
a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
deleted file mode 100644
index 3355ac4f23..00
---
a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
+++ /dev/null
@@ -1,34 +0,0 @@
-From 554f7baed050f89ffc2a7192d3071e8c5420f6d3 Mon Sep 17 00:00:00 2001
-From: Mikko Rapeli
-Date: Fri, 25 Aug 2023 10:35:28 +
-Subject: [PATCH] openssh regress/Makefile: print logs if test fails
-
-Some tests are failing in CI runs and reproduction has failed. Print
-the captured sshd and ssh client logs if test fails. This should
-help to fix the root causes.
-
-Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
-
-Signed-off-by: Mikko Rapeli
- regress/Makefile | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-Upstream-Status: Denied [https://github.com/openssh/openssh-portable/pull/437]
-
-diff --git a/regress/Makefile b/regress/Makefile
-index d80bf59..a972dff 100644
a/regress/Makefile
-+++ b/regress/Makefile
-@@ -229,7 +229,7 @@ t-exec:${LTESTS:=.sh}
- done; \
- if [ "x$${skip}" = "xno" ]; then \
- echo "run test $${TEST}" ... 1>&2; \
-- (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || exit $$?; \
-+ (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || (echo return value:
$$?; echo capturing logs; cat *.log; exit 1); \
- else \
- echo skip test $${TEST} 1>&2; \
- fi; \
---
-2.34.1
-
diff --git a/meta/recipes-connectivity/openssh/openssh/run-ptest
b/meta/recipes-connectivity/openssh/openssh/run-ptest
index 8a9b770d59..1e6eec5799 100755
--- a/meta/recipes-connectivity/openssh/openssh/run-ptest
+++ b/meta/recipes-connectivity/openssh/openssh/run-ptest
@@ -4,6 +4,20 @@ export TEST_SHELL=sh
export SKIP_UNIT=1
cd regress
+
+# copied from openssh-portable/.github/run_test.sh
+output_failed_logs() {
+for i in failed*.log; do
+if [ -f "$i" ]; then
+echo
-
+echo LOGFILE $i
+cat $i
+echo
-
+fi
+done
+}
+trap output_failed_logs 0
+
sed -i "/\t\tagent-ptrace /d" Makefile
make -k BUILDDIR=`pwd`/.. .OBJDIR=`pwd` .CURDIR=`pwd` SUDO="sudo" tests \
| sed -u -e 's/^skipped/SKIP: /g' -e 's/^ok /PASS: /g' -e
's/^failed/FAIL: /g'
diff --git a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
index 1cf6937038..a38d9c2b81 100644
--- a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
@@ -24,7 +24,6 @@ SRC_URI =
"http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
file://fix-potential-signed-overflow-in-pointer-arithmatic.patch \
file://sshd_check_keys \
file://add-test-support-for-busybox.patch \
- file://0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
\
file://0001-regress-banner.sh-log-input-and-output-files-on-erro.patch \
"
SRC_URI[sha256sum] =
"3608fd9088db2163ceb3e600c85ab79d0de3d221e59192ea1923e23263866a85"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187630):
https://lists.openembedded.org/g/openembedded-core/message/187630
Mute This Topic: https://lists.openembedded.org/mt/101355525/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/3] testimage.bbclass: remove QEMU_USE_SLIRP variable
QEMU_USE_SLIRP is replaced by TEST_RUNQEMUPARAMS with "slirp" and
possibly other arguments to runqemu script.
Signed-off-by: Mikko Rapeli
---
meta/classes-recipe/testimage.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes-recipe/testimage.bbclass
b/meta/classes-recipe/testimage.bbclass
index 8a944f5809..7c56fe9674 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -322,7 +322,7 @@ def testimage_main(d):
ovmf = d.getVar("QEMU_USE_OVMF")
slirp = False
-if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS',
'slirp', True, False, d):
+if bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
slirp = True
# TODO: We use the current implementation of qemu runner because of
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187629):
https://lists.openembedded.org/g/openembedded-core/message/187629
Mute This Topic: https://lists.openembedded.org/mt/101355524/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/3] oeqa/selftest/context.py: check git command return values
Don't ignore return values from the git command lines. If something goes
wrong, fail the test right away.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/context.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index bd03e765b0..7dab5614a9 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -104,7 +104,7 @@ class OESelftestTestContext(OETestContext):
oe.path.copytree(builddir + "/cache", newbuilddir + "/cache")
oe.path.copytree(selftestdir, newselftestdir)
-subprocess.check_output("git init; git add *; git commit -a -m
'initial'", cwd=newselftestdir, shell=True)
+subprocess.check_output("git init && git add * && git commit -a -m
'initial'", cwd=newselftestdir, shell=True)
# Tried to used bitbake-layers add/remove but it requires recipe
parsing and hence is too slow
subprocess.check_output("sed %s/conf/bblayers.conf -i -e 's#%s#%s#g'"
% (newbuilddir, selftestdir, newselftestdir), cwd=newbuilddir, shell=True)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187631):
https://lists.openembedded.org/g/openembedded-core/message/187631
Mute This Topic: https://lists.openembedded.org/mt/101355526/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [Openembedded-architecture] Security processes: YP needs
Hi, On Wed, Sep 13, 2023 at 01:52:19PM +0200, Marta Rybczynska wrote: > Hello, > I've been working recently on collecting what works and what doesn't > in YP security processes. The goal is to go forward and define an > actionable strategy! > > Today, I'd like to share with you the summary of what I have heard as > needs from several people (those in Cc:). > > I want the community to comment and tell us what you find important > and what you'd like to see added or changed from this list. Since most users take poky reference distro and combine it with a number of open source and closed source BSP and other meta layers and build systems to produce SW for products, they also need documentation and tooling so that they can replicate the Yocto Project security processes and use the available tools. In the best case downstream users are on poky master branch or one of the maintained LTS branches, but they can also be stuck on a non-LTS branch due to BSP or other technical, contractual or even purely political reasons. Having a description of the processes and tools and best practices helps, and if needed they can for example backport the needed tooling changes to their version to help or kick off in the maintenance effort, just like how upstream LTS branches are managed. I think most of the documentation around the tools and processes is in place already. Having maintained and shipped from a non-maintained poky branch, I can just say thank you to all who participated in the upstream work to get security vulnerability detection and fixing possible! poky seems well maintained and serves as an example to everyone wether open source or not. That being said, extending the CVE scanning and status tracking work to include more open source layers would be nice both for the maintainers and for the users of those layers. Using some random old branch of meta-foo may not be so safe. Maybe add this data to layer-index? Cheers, -Mikko > * CVEs: Visibility if YP is vulnerable or not > > People want to be able to check/look up a specific CVE; it might be a > CVE unrelated to YP > (eg. package not included, Windows issue). The cve-checker result is a > part of the solution, but people also want to know which CVEs do not > apply. > > * CVEs: synchronization of the work on fixes > > Currently, there is no synchronization; multiple parties might be > working on the same fix while nobody is working on another. There > might be duplication of work. > Ross has https://wiki.yoctoproject.org/wiki/CVE_Status > > * Triaging of security issues > > Related to CVE fixes and includes issues reported directly to the YP. > Some issues are more likely to be serious for embedded products > (attack by network), so not all has the same priority. > > * Private security communication > > A way to send a notification of a non-public security issue. For > researchers, other projects etc. > The security alias exists, but only some people know about its existence. > > * Visibility of the security work of the YP > > There is much work on security in the YP, but it lacks visibility. > > * Documentation > > Related to visibility. We need easy-to-find documentation of subjects > like submitting a CVE fix, > reporting a private issue, and how our processes work... This > documentation should address people who are not regular contributors. > > * Additional tooling > > We could add additional tooling: a template on how to add cve-check to > the CI (possibly > a different one than the autobuilder), analyze the result, and extend > our tooling to their layers... > It is also related to the "Architecture" topic below. > > * Architecture work > > Security if more than CVE fixes. We also have what is happening in > meta-security: hardening, compiler option, > secure package configuration, use of code coverage tools, and so on > > * SRTool > > We might decide to use it again. It allows one to do much but requires > constant commitment. > > * Presence on pre-notification lists and receiving information before > the vulnerability gets public > > YP currently depends on public data. Principal distributions receive > the information before > a vulnerability becomes public. It requires (in short) private > reporting, a security team, and a track > of excellent security record. > > * Becoming a CNA (be able to assign CVEs) > > Needed if we want to assign CVEs to the software of the YP, like > autobuilder, Toaster etc. > > Kind regards, > Marta > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187588): https://lists.openembedded.org/g/openembedded-core/message/187588 Mute This Topic: https://lists.openembedded.org/mt/101335537/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bitbake: fetch2: git: revert path checks
Hi, Sorry, wrong list. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187555): https://lists.openembedded.org/g/openembedded-core/message/187555 Mute This Topic: https://lists.openembedded.org/mt/101333061/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] bitbake: fetch2: git: revert path checks
Commit 10f38157a069cb6938323cadd5e523037a29ace9
triggers re-clones of uptodate repositories when their
access path changes. This is wrong. It is enough that
needed commits are found from the git repo. For example
when same DL_DIR is accessed using two different paths
due to a symlink, then the resulting absolute paths will
differ but the content is the same. Trust the content.
Avoids extra downloads of large repositories like linux-yocto
for kernel when the local cache was already uptodate.
Signed-off-by: Mikko Rapeli
---
bitbake/lib/bb/fetch2/git.py | 14 +-
1 file changed, 1 insertion(+), 13 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e11271b757..0c547ea303 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -371,19 +371,7 @@ class Git(FetchMethod):
# The directory may exist, but not be the top level of a bare git
# repository in which case it needs to be deleted and re-cloned.
try:
-# Since clones can be bare, use --absolute-git-dir instead of
--show-toplevel
-output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir"
% ud.basecmd, d, workdir=ud.clonedir)
-
-toplevel = os.path.abspath(output.rstrip())
-abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/')
-# The top level Git directory must either be the clone
directory
-# or a child of the clone directory. Any ancestor directory of
-# the clone directory is not valid as the Git directory (and
-# probably belongs to some other unrelated repository), so a
-# clone is required
-if os.path.commonprefix([abs_clonedir, toplevel]) !=
abs_clonedir:
-logger.warning("Top level directory '%s' doesn't match
expected '%s'. Re-cloning", toplevel, ud.clonedir)
-needs_clone = True
+runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" %
ud.basecmd, d, workdir=ud.clonedir)
except bb.fetch2.FetchError as e:
logger.warning("Unable to get top level for %s (not a git
directory?): %s", ud.clonedir, e)
needs_clone = True
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187554):
https://lists.openembedded.org/g/openembedded-core/message/187554
Mute This Topic: https://lists.openembedded.org/mt/101333061/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa/selftest/runtime_test: No need to use append with TEST_RUNQEMUPARAMS
Hi, On Tue, Sep 12, 2023 at 12:50:26PM +0100, Richard Purdie wrote: > On Tue, 2023-09-12 at 14:46 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Tue, Sep 12, 2023 at 12:32:42PM +0100, Richard Purdie wrote: > > > A simple += operation is fine for working with TEST_RUNQEMUPARAMS so > > > remove > > > the heavier append override usage. > > > > Ok, but I was being consistent with the other variables in the tests. > > Fair enough, I did kind of guess that. In general we want to try and > reduce the amount of :append use over time though so where we have a > choice, we should just use +=. I fully agree to this but I wasn't sure what kind of local.conf is used together with the test so an :append made sure that the needed content was added, always. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187511): https://lists.openembedded.org/g/openembedded-core/message/187511 Mute This Topic: https://lists.openembedded.org/mt/101312801/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] oeqa/selftest/runtime_test: No need to use append with TEST_RUNQEMUPARAMS
Hi,
On Tue, Sep 12, 2023 at 12:32:42PM +0100, Richard Purdie wrote:
> A simple += operation is fine for working with TEST_RUNQEMUPARAMS so remove
> the heavier append override usage.
Ok, but I was being consistent with the other variables in the tests.
Cheers,
-Mikko
> Signed-off-by: Richard Purdie
> ---
> meta/lib/oeqa/selftest/cases/runtime_test.py | 8
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py
> b/meta/lib/oeqa/selftest/cases/runtime_test.py
> index 57d3852d579..12000aac16e 100644
> --- a/meta/lib/oeqa/selftest/cases/runtime_test.py
> +++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
> @@ -138,7 +138,7 @@ class TestImage(OESelftestTestCase):
> IMAGE_CLASSES:append = " testimage"
> IMAGE_FEATURES:append = " ssh-server-dropbear"
> IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES",
> "testimage", " + 5120", "", d)}"
> -TEST_RUNQEMUPARAMS:append = " slirp"
> +TEST_RUNQEMUPARAMS += " slirp"
> '''
> self.write_config(features)
>
> @@ -252,11 +252,11 @@ TEST_RUNQEMUPARAMS:append = " slirp"
> features += 'TEST_SUITES = "ping ssh virgl"\n'
> features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
> features += 'IMAGE_INSTALL:append = " kmscube"\n'
> -features_gtk = features + 'TEST_RUNQEMUPARAMS:append = " gtk gl"\n'
> +features_gtk = features + 'TEST_RUNQEMUPARAMS += " gtk gl"\n'
> self.write_config(features_gtk)
> bitbake('core-image-minimal')
> bitbake('-c testimage core-image-minimal')
> -features_sdl = features + 'TEST_RUNQEMUPARAMS:append = " sdl gl"\n'
> +features_sdl = features + 'TEST_RUNQEMUPARAMS += " sdl gl"\n'
> self.write_config(features_sdl)
> bitbake('core-image-minimal')
> bitbake('-c testimage core-image-minimal')
> @@ -284,7 +284,7 @@ TEST_RUNQEMUPARAMS:append = " slirp"
> features += 'TEST_SUITES = "ping ssh virgl"\n'
> features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
> features += 'IMAGE_INSTALL:append = " kmscube"\n'
> -features += 'TEST_RUNQEMUPARAMS:append = " egl-headless"\n'
> +features += 'TEST_RUNQEMUPARAMS += " egl-headless"\n'
> self.write_config(features)
> bitbake('core-image-minimal')
> bitbake('-c testimage core-image-minimal')
> --
> 2.39.2
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187509):
https://lists.openembedded.org/g/openembedded-core/message/187509
Mute This Topic: https://lists.openembedded.org/mt/101312801/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
On Tue, Sep 12, 2023 at 11:30:57AM +, Ross Burton wrote:
> On 12 Sep 2023, at 12:29, Mikko Rapeli wrote:
> >
> > Hi,
> >
> > On Tue, Sep 12, 2023 at 11:25:28AM +, Ross Burton wrote:
> >> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org
> >> wrote:
> >>> -if d.getVar("QEMU_USE_SLIRP"):
> >>> +if d.getVar("QEMU_USE_SLIRP") or
> >>> bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >>>slirp = True
> >>
> >> This means we can remove the QEMU_USE_SLIRP variable too, right?
> >
> > Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with
> > testimage.bbclass.
>
> Can you send a followup to do that shortly?
Yes, I can do that.
In my developer builds I have following changes from this patch series:
oeqa selftest runtime_test.py: append to TEST_RUNQEMUPARAMS
oeqa dnf_runtime.py: fix HTTP server IP address and port
testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Rest are already applied or rejected.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187508):
https://lists.openembedded.org/g/openembedded-core/message/187508
Mute This Topic: https://lists.openembedded.org/mt/100910036/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi,
On Tue, Sep 12, 2023 at 11:25:28AM +, Ross Burton wrote:
> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org
> wrote:
> > -if d.getVar("QEMU_USE_SLIRP"):
> > +if d.getVar("QEMU_USE_SLIRP") or
> > bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> > slirp = True
>
> This means we can remove the QEMU_USE_SLIRP variable too, right?
Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with
testimage.bbclass.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187505):
https://lists.openembedded.org/g/openembedded-core/message/187505
Mute This Topic: https://lists.openembedded.org/mt/100910036/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] openssh: improve banner ptest failure logging
Log the input and output banner files. Output seems to
contain more lines than input which fails the test but
it's not clear what is in there from the ssh command
stderr. So print them out to dig deeper into the root
cause.
Upstream rejected previous logging patch so they will likely
do the same for this:
https://github.com/openssh/openssh-portable/pull/437
Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
Signed-off-by: Mikko Rapeli
---
...h-log-input-and-output-files-on-erro.patch | 61 +++
.../openssh/openssh_9.4p1.bb | 1 +
2 files changed, 62 insertions(+)
create mode 100644
meta/recipes-connectivity/openssh/openssh/0001-regress-banner.sh-log-input-and-output-files-on-erro.patch
diff --git
a/meta/recipes-connectivity/openssh/openssh/0001-regress-banner.sh-log-input-and-output-files-on-erro.patch
b/meta/recipes-connectivity/openssh/openssh/0001-regress-banner.sh-log-input-and-output-files-on-erro.patch
new file mode 100644
index 00..2c14014fed
--- /dev/null
+++
b/meta/recipes-connectivity/openssh/openssh/0001-regress-banner.sh-log-input-and-output-files-on-erro.patch
@@ -0,0 +1,61 @@
+From f5a4dacc987ca548fc86577c2dba121c86da3c34 Mon Sep 17 00:00:00 2001
+From: Mikko Rapeli
+Date: Mon, 11 Sep 2023 09:55:21 +0100
+Subject: [PATCH] regress/banner.sh: log input and output files on error
+
+Some test environments like yocto with qemu are seeing these
+tests failing. There may be additional error messages in the
+stderr of ssh cloent command. busybox cmp shows this error when
+first input file has less new line characters then second
+input file:
+
+cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
+
+Logging the full banner.out will show what other error messages
+are captured in addition of the expected banner.
+
+Full log of a failing banner test runs is:
+
+run test banner.sh ...
+test banner: missing banner file
+test banner: size 0
+cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
+banner size 0 mismatch
+test banner: size 10
+test banner: size 100
+cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
+banner size 100 mismatch
+test banner: size 1000
+test banner: size 1
+test banner: size 10
+test banner: suppress banner (-q)
+FAIL: banner
+return value: 1
+
+See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
+
+Signed-off-by: Mikko Rapeli
+---
+ regress/banner.sh | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+Upstream-Status: Denied [https://github.com/openssh/openssh-portable/pull/437]
+
+diff --git a/regress/banner.sh b/regress/banner.sh
+index a84feb5a..de84957a 100644
+--- a/regress/banner.sh
b/regress/banner.sh
+@@ -32,7 +32,9 @@ for s in 0 10 100 1000 1 10 ; do
+ verbose "test $tid: size $s"
+ ( ${SSH} -F $OBJ/ssh_proxy otherhost true 2>$OBJ/banner.out && \
+ cmp $OBJ/banner.in $OBJ/banner.out ) || \
+- fail "banner size $s mismatch"
++ ( verbose "Contents of $OBJ/banner.in:"; cat $OBJ/banner.in; \
++verbose "Contents of $OBJ/banner.out:"; cat $OBJ/banner.out; \
++fail "banner size $s mismatch" )
+ done
+
+ trace "test suppress banner (-q)"
+--
+2.34.1
+
diff --git a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
index 2c85780e4d..1cf6937038 100644
--- a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
@@ -25,6 +25,7 @@ SRC_URI =
"http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
file://sshd_check_keys \
file://add-test-support-for-busybox.patch \
file://0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
\
+
file://0001-regress-banner.sh-log-input-and-output-files-on-erro.patch \
"
SRC_URI[sha256sum] =
"3608fd9088db2163ceb3e600c85ab79d0de3d221e59192ea1923e23263866a85"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187491):
https://lists.openembedded.org/g/openembedded-core/message/187491
Mute This Topic: https://lists.openembedded.org/mt/101292700/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] openssh: update Upstream-Status to Denied in test logging patch
Upstream rejected the change: https://github.com/openssh/openssh-portable/pull/437 Signed-off-by: Mikko Rapeli --- ...0001-openssh-regress-Makefile-print-logs-if-test-fails.patch | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch index baa68dc6ff..3355ac4f23 100644 --- a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch +++ b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch @@ -14,7 +14,7 @@ Signed-off-by: Mikko Rapeli regress/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -Upstream-Status: Submitted [https://github.com/openssh/openssh-portable/pull/437] +Upstream-Status: Denied [https://github.com/openssh/openssh-portable/pull/437] diff --git a/regress/Makefile b/regress/Makefile index d80bf59..a972dff 100644 -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187490): https://lists.openembedded.org/g/openembedded-core/message/187490 Mute This Topic: https://lists.openembedded.org/mt/101292699/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] classes: Drop ';' delimiter from ROOTFS/IMAGE*COMMAND variables
Hi, On Thu, Sep 07, 2023 at 11:11:20PM +0100, Richard Purdie wrote: > Originally these were shell functions but they have long since been processed > by > bb.build.exec_func(). Since we no longer need shell syntax, we can drop the > ';' > delimiters and just use a space separated string. > > This cleans up the variable and quietly removes any stray ';' that do happen > to > still make it in. I'm afraid many layers and recipes may be breaking with this because of missing space in appends, e.g. IMAGE_PREPROCESS_COMMAND:append = "foo;" used to work, and now will fail. Fix is to use IMAGE_PREPROCESS_COMMAND:append = " foo" but I wonder if we could help detect these? And I have dislike for :append when a plain += would work and would add the needed space chars and still allow overriding the variable elsewhere when it is needed. But maybe that's just me. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187408): https://lists.openembedded.org/g/openembedded-core/message/187408 Mute This Topic: https://lists.openembedded.org/mt/101225689/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] openssh: capture ptest regression test failure logs
When tests fail, capture the sshd and ssh client logs from
the failing test run. These are needed to investigate
the root cause.
Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
Signed-off-by: Mikko Rapeli
---
...ss-Makefile-print-logs-if-test-fails.patch | 34 +++
.../openssh/openssh_9.4p1.bb | 1 +
2 files changed, 35 insertions(+)
create mode 100644
meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
v2: Upstream-Status from Pending to Submitted,
https://github.com/openssh/openssh-portable/pull/437
v1: https://lists.openembedded.org/g/openembedded-core/message/187315
diff --git
a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
new file mode 100644
index 00..baa68dc6ff
--- /dev/null
+++
b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
@@ -0,0 +1,34 @@
+From 554f7baed050f89ffc2a7192d3071e8c5420f6d3 Mon Sep 17 00:00:00 2001
+From: Mikko Rapeli
+Date: Fri, 25 Aug 2023 10:35:28 +
+Subject: [PATCH] openssh regress/Makefile: print logs if test fails
+
+Some tests are failing in CI runs and reproduction has failed. Print
+the captured sshd and ssh client logs if test fails. This should
+help to fix the root causes.
+
+Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
+
+Signed-off-by: Mikko Rapeli
+---
+ regress/Makefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+Upstream-Status: Submitted
[https://github.com/openssh/openssh-portable/pull/437]
+
+diff --git a/regress/Makefile b/regress/Makefile
+index d80bf59..a972dff 100644
+--- a/regress/Makefile
b/regress/Makefile
+@@ -229,7 +229,7 @@ t-exec:${LTESTS:=.sh}
+ done; \
+ if [ "x$${skip}" = "xno" ]; then \
+ echo "run test $${TEST}" ... 1>&2; \
+- (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || exit $$?; \
++ (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || (echo return value:
$$?; echo capturing logs; cat *.log; exit 1); \
+ else \
+ echo skip test $${TEST} 1>&2; \
+ fi; \
+--
+2.34.1
+
diff --git a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
index 3f232a839d..2c85780e4d 100644
--- a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
@@ -24,6 +24,7 @@ SRC_URI =
"http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
file://fix-potential-signed-overflow-in-pointer-arithmatic.patch \
file://sshd_check_keys \
file://add-test-support-for-busybox.patch \
+ file://0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
\
"
SRC_URI[sha256sum] =
"3608fd9088db2163ceb3e600c85ab79d0de3d221e59192ea1923e23263866a85"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187317):
https://lists.openembedded.org/g/openembedded-core/message/187317
Mute This Topic: https://lists.openembedded.org/mt/101193789/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] openssh: capture ptest regression test failure logs
When tests fail, capture the sshd and ssh client logs from
the failing test run. These are needed to investigate
the root cause.
Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
Signed-off-by: Mikko Rapeli
---
...ss-Makefile-print-logs-if-test-fails.patch | 34 +++
.../openssh/openssh_9.4p1.bb | 1 +
2 files changed, 35 insertions(+)
create mode 100644
meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
diff --git
a/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
new file mode 100644
index 00..bcadbbed0b
--- /dev/null
+++
b/meta/recipes-connectivity/openssh/openssh/0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
@@ -0,0 +1,34 @@
+From 554f7baed050f89ffc2a7192d3071e8c5420f6d3 Mon Sep 17 00:00:00 2001
+From: Mikko Rapeli
+Date: Fri, 25 Aug 2023 10:35:28 +
+Subject: [PATCH] openssh regress/Makefile: print logs if test fails
+
+Some tests are failing in CI runs and reproduction has failed. Print
+the captured sshd and ssh client logs if test fails. This should
+help to fix the root causes.
+
+Reference: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
+
+Upstream-Status: Pending
+
+Signed-off-by: Mikko Rapeli
+---
+ regress/Makefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/regress/Makefile b/regress/Makefile
+index d80bf59..a972dff 100644
+--- a/regress/Makefile
b/regress/Makefile
+@@ -229,7 +229,7 @@ t-exec:${LTESTS:=.sh}
+ done; \
+ if [ "x$${skip}" = "xno" ]; then \
+ echo "run test $${TEST}" ... 1>&2; \
+- (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || exit $$?; \
++ (env SUDO="${SUDO}" TEST_ENV=${TEST_ENV} ${TEST_SHELL}
${.CURDIR}/test-exec.sh ${.OBJDIR} ${.CURDIR}/$${TEST}) || (echo return value:
$$?; echo capturing logs; cat *.log; exit 1); \
+ else \
+ echo skip test $${TEST} 1>&2; \
+ fi; \
+--
+2.34.1
+
diff --git a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
index 3f232a839d..2c85780e4d 100644
--- a/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_9.4p1.bb
@@ -24,6 +24,7 @@ SRC_URI =
"http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
file://fix-potential-signed-overflow-in-pointer-arithmatic.patch \
file://sshd_check_keys \
file://add-test-support-for-busybox.patch \
+ file://0001-openssh-regress-Makefile-print-logs-if-test-fails.patch
\
"
SRC_URI[sha256sum] =
"3608fd9088db2163ceb3e600c85ab79d0de3d221e59192ea1923e23263866a85"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187315):
https://lists.openembedded.org/g/openembedded-core/message/187315
Mute This Topic: https://lists.openembedded.org/mt/101192324/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] uki: Add support for building Unified Kernel Images
Hi, On Fri, Sep 01, 2023 at 11:32:31PM +, Michelle Lin wrote: > Currently, there is not a class to support the building of unified kernel > images. Adding a uki.bbclass to support the creation of UKIs. This class calls > the systemd Ukify tool, which will combine the kernel/initrd/stub components > to > build the UKI. To sign the UKI (i.e. SecureBoot, TPM PCR signing), the > keys/cert > files are to be specified in a separate configuration file, and the path to > the > file is passed to the Ukify tool. UKIs are supported by UEFI and can improve > security through predicted TPM PCR states, and reduce the build burden due to > its single PE binary format. Thanks, I'm interesting in using this. Could you add a oeqa selftest for this class too? Something which builds a UKI image and then does something to verify it is what's expected, maybe full boot with e.g. qemu is too hard to do? Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#187160): https://lists.openembedded.org/g/openembedded-core/message/187160 Mute This Topic: https://lists.openembedded.org/mt/101106095/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Dilemma on changes - merge or not to merge (e.g. 6.4)
Hi, On Fri, Aug 25, 2023 at 07:34:25AM +0100, Richard Purdie wrote: > On Fri, 2023-08-25 at 09:27 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Thu, Aug 24, 2023 at 09:18:03PM +0100, Richard Purdie wrote: > > > On Thu, 2023-08-24 at 15:04 +0100, Richard Purdie via > > > lists.openembedded.org wrote: > > > > On Wed, 2023-08-23 at 22:16 +0100, Richard Purdie via > > > > lists.openembedded.org wrote: > > > > > On Tue, 2023-08-22 at 23:01 +0100, Richard Purdie via > > > > > lists.openembedded.org wrote: > > > > > > so the commands are stopping mid flow for unknown reasons or the ssh > > > > > > connection fails. I can't tell if this coincides with an rcu stall > > > > > > or > > > > > > not. Both logs do have rcu stalls in. > > > > > > > > > > > > After these failures the system does continue to otherwise work > > > > > > normally and subsequent tests pass. > > > > > > > > > > > > I wonder if the slow emulation might be causing the networking to > > > > > > glitch and break the ssh connection. > > > > > > > > > > > > I'm at a bit of a loss on where from here. > > > > > > > > > > I thought I'd update the thread with new information. > > > > > > > > > > I went back to the start with this and looked again and what is going > > > > > on. Interestingly, I found one of the autobuilder workers would > > > > > consistently fail the qemuppc-alt configuration for core-image-sato- > > > > > sdk. I paused the worker and experimented. > > > > > > > > > > I saw two different failures (included below). One shows systemd-udevd > > > > > timing out on it's watchdog after 3 minutes and resetting, including > > > > > taking out an ssh session running the cpio configure command. There > > > > > was > > > > > no RCU stall reported. > > > > > > > > > > The second failure shows systemd-logind as well as systemd-udevd with > > > > > the 3 minute time out, the kernel complaining about missed IRQs, an > > > > > RCU > > > > > stall and lots of breakage following including cut ssh commands. > > > > > > > > > > I could not get the cpio build test to complete. > > > > > > > > > > Interestingly, I came back to the same image/worker later this evening > > > > > and now it all works fine. The difference is earlier there was a world > > > > > build running on the worker, which continued to wind down even after I > > > > > paused the worker. By the evening, that background load was no longer > > > > > present and the ppc image works in isolation. This tells us the issue > > > > > is system load dependent and only occurs on loaded systems. > > > > > > > > > > I suspect I need to replicate the load and retry locally, see if I can > > > > > reliably reproduce the hang. The watchdog won't be present on sysvinit > > > > > systems which also show the issues but I'd guess there is still some > > > > > other starvation/timeout occurring. > > > > > > > > I've now seen the failure on the autobuilder: > > > > > > > > * with linux-yocto 6.1.38 > > > > * with linux-yocto 6.1.46 > > > > * with qemu 8.0.4 > > > > * with qemu 8.0.3 > > > > * with qemu 8.0.0 > > > > > > > > I was a little suspicious of: > > > > > > > > "hw/ppc: Fix clock update drift" > > > > https://gitlab.com/qemu-project/qemu/-/commit/73d6ac24c81f1aeae554d469616c9181511e6523 > > > > > > > > but we've tested with and without that. > > > > > > > > qemu has just released 8.1.0 so perhaps we should try that next. > > > > > > qemu 8.1.0 brings with it a new set of problems but I've reproduced the > > > hang with 8.1.0 so it does not solve that. > > > > > > I'm really struggling to understand which change brought in these > > > issues for qemuppc. > > > > Are these issues visible on mickledore branch? Maybe mickledore with kernel > > 6.1 stable update or > > qemu 7.2 update to 8.y.x could be tested too. At least then kernel or qemu > > could be blamed > > for the issues. > > Not th
Re: [OE-core] Dilemma on changes - merge or not to merge (e.g. 6.4)
Hi, On Thu, Aug 24, 2023 at 09:18:03PM +0100, Richard Purdie wrote: > On Thu, 2023-08-24 at 15:04 +0100, Richard Purdie via > lists.openembedded.org wrote: > > On Wed, 2023-08-23 at 22:16 +0100, Richard Purdie via > > lists.openembedded.org wrote: > > > On Tue, 2023-08-22 at 23:01 +0100, Richard Purdie via > > > lists.openembedded.org wrote: > > > > so the commands are stopping mid flow for unknown reasons or the ssh > > > > connection fails. I can't tell if this coincides with an rcu stall or > > > > not. Both logs do have rcu stalls in. > > > > > > > > After these failures the system does continue to otherwise work > > > > normally and subsequent tests pass. > > > > > > > > I wonder if the slow emulation might be causing the networking to > > > > glitch and break the ssh connection. > > > > > > > > I'm at a bit of a loss on where from here. > > > > > > I thought I'd update the thread with new information. > > > > > > I went back to the start with this and looked again and what is going > > > on. Interestingly, I found one of the autobuilder workers would > > > consistently fail the qemuppc-alt configuration for core-image-sato- > > > sdk. I paused the worker and experimented. > > > > > > I saw two different failures (included below). One shows systemd-udevd > > > timing out on it's watchdog after 3 minutes and resetting, including > > > taking out an ssh session running the cpio configure command. There was > > > no RCU stall reported. > > > > > > The second failure shows systemd-logind as well as systemd-udevd with > > > the 3 minute time out, the kernel complaining about missed IRQs, an RCU > > > stall and lots of breakage following including cut ssh commands. > > > > > > I could not get the cpio build test to complete. > > > > > > Interestingly, I came back to the same image/worker later this evening > > > and now it all works fine. The difference is earlier there was a world > > > build running on the worker, which continued to wind down even after I > > > paused the worker. By the evening, that background load was no longer > > > present and the ppc image works in isolation. This tells us the issue > > > is system load dependent and only occurs on loaded systems. > > > > > > I suspect I need to replicate the load and retry locally, see if I can > > > reliably reproduce the hang. The watchdog won't be present on sysvinit > > > systems which also show the issues but I'd guess there is still some > > > other starvation/timeout occurring. > > > > I've now seen the failure on the autobuilder: > > > > * with linux-yocto 6.1.38 > > * with linux-yocto 6.1.46 > > * with qemu 8.0.4 > > * with qemu 8.0.3 > > * with qemu 8.0.0 > > > > I was a little suspicious of: > > > > "hw/ppc: Fix clock update drift" > > https://gitlab.com/qemu-project/qemu/-/commit/73d6ac24c81f1aeae554d469616c9181511e6523 > > > > but we've tested with and without that. > > > > qemu has just released 8.1.0 so perhaps we should try that next. > > qemu 8.1.0 brings with it a new set of problems but I've reproduced the > hang with 8.1.0 so it does not solve that. > > I'm really struggling to understand which change brought in these > issues for qemuppc. Are these issues visible on mickledore branch? Maybe mickledore with kernel 6.1 stable update or qemu 7.2 update to 8.y.x could be tested too. At least then kernel or qemu could be blamed for the issues. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186690): https://lists.openembedded.org/g/openembedded-core/message/186690 Mute This Topic: https://lists.openembedded.org/mt/100733646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails
Hi, On Wed, Aug 23, 2023 at 08:37:47PM +0200, Alexander Kanavin wrote: > On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli wrote: > > > > If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will > > fail to start and errors are rather cryptic like: > > > > qemu-system-x86_64: egl: render node init failed > > > > To fix this, users likely need to > > > > * modprobe vgem > > * add their user to "render" group to write to /dev/dri/renderD128 > > > > If access is not available due to missing HW, driver or failing access, > > then skip the test: > > This should be a failure rather than a skip. Otherwise, autobuilder > testing will silently regress and no one will notice. The error > message should include information from IOError as well so that it's > clear that the issue is device file permissions. Then this change is not needed at all. Test already fails if render device is not available or if access fails. In my experience very few environments have vgem driver loaded and "render" group permissions for the user running the builds and tests. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186638): https://lists.openembedded.org/g/openembedded-core/message/186638 Mute This Topic: https://lists.openembedded.org/mt/100910043/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test
Hi, On Wed, Aug 23, 2023 at 08:45:49PM +0200, Alexander Kanavin wrote: > I guess the extra space stuff should be going here instead? Or some other image which is supposed to be compatible with oeqa runtime tests and testimage.bbclass. I think it's better to just ignore this patch. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186637): https://lists.openembedded.org/g/openembedded-core/message/186637 Mute This Topic: https://lists.openembedded.org/mt/100910039/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test
Hi,
On Wed, Aug 23, 2023 at 08:44:33PM +0200, Alexander Kanavin wrote:
> Either you need to use a different image in testing, or add the needed
> pieces to this image in the same commit, explaining why.
>
> Adding extra space because you have a local, custom configuration
> where it is required is not correct.
Sorry I thought upstream would be testing core-image-minimal as well.
If that's not the case, then just drop this. Some other images need
to be tested instead, but I don't know which.
Cheers,
-Mikko
> Alex
>
> On Wed, 23 Aug 2023 at 08:10, Mikko Rapeli wrote:
> >
> > testimage.bbclass requires that ssh communication with target works
> > and then it runs oeqa runtime tests with that. Tests like df.py
> > check that there is more than 5Mb free space on the rootfs on target. Sadly
> > latest core-image-minimal only has 1.5 Mb free space if
> > ssh-server-dropbear is added to it. Thus by default, core-image-minimal
> > is now failing oeqa rutime df.py test.
> >
> > Fix this by increasing core-image-minimal rootfs size by 5Mb if
> > testimage.bbclass is used which implies adding either
> > ssh-server-dropbear or ssh-server-openssh to the target rootfs.
> >
> > Signed-off-by: Mikko Rapeli
> > ---
> > meta/recipes-core/images/core-image-minimal.bb | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > v2: no changes
> >
> > v1: https://lists.openembedded.org/g/openembedded-core/message/186423
> >
> > diff --git a/meta/recipes-core/images/core-image-minimal.bb
> > b/meta/recipes-core/images/core-image-minimal.bb
> > index 84343adcd8..c04612bc15 100644
> > --- a/meta/recipes-core/images/core-image-minimal.bb
> > +++ b/meta/recipes-core/images/core-image-minimal.bb
> > @@ -10,3 +10,6 @@ inherit core-image
> >
> > IMAGE_ROOTFS_SIZE ?= "8192"
> > IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES",
> > "systemd", " + 4096", "", d)}"
> > +
> > +# a bit more space needed for sshd to actually run the tests
> > +IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES",
> > "testimage", " + 5120", "", d)}"
> > --
> > 2.34.1
> >
> >
> >
> >
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186636):
https://lists.openembedded.org/g/openembedded-core/message/186636
Mute This Topic: https://lists.openembedded.org/mt/100910038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )
Hi, On Wed, Aug 23, 2023 at 03:24:43PM +0200, Alexander Kanavin wrote: > I've just seen this too. I think to trigger this you need to run the > test via bitbake -c testimage rather than directly over console - the > output is then piped over ssh, and something goes wrong due to the > large volume of banner data perhaps? Would *really* appreciate if you > can get to the bottom of it. Ok, I'll try to have a look. I also just found out that: https://wiki.yoctoproject.org/wiki/Weekly_Status -> https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT -> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178 Reported:2023-08-02 14:32 UTC by Alexandre Belloni So it's a known issue. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186578): https://lists.openembedded.org/g/openembedded-core/message/186578 Mute This Topic: https://lists.openembedded.org/mt/100914026/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
openssh ptest banner failure (was Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS )
Hi,
On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via
lists.openembedded.org wrote:
> The tests passed with one warning, which I'll try to check:
>
> WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing
> ptests.
> Traceback (most recent call last):
> File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py",
> line 35, in wrapped_f
> return func(*args, **kwargs)
> File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py",
> line 35, in wrapped_f
> return func(*args, **kwargs)
> File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py",
> line 35, in wrapped_f
> return func(*args, **kwargs)
> File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line
> 27, in test_ptestrunner_expectfail
> self.do_ptestrunner()
> File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line
> 120, in do_ptestrunner
> self.fail(failmsg)
> AssertionError:
> Failed ptests:
> {'openssh': ['banner']}
Has this openssh banner test on x86_64 been failing on yocto autobuilders?
Log
tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
shows:
PASS: environment passing
run test transfer.sh ...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
banner size 1000 mismatch
test banner: size 1
test banner: size 10
test banner: suppress banner (-q)
FAIL: banner
make: *** [Makefile:223: t-exec] Error 1
On manual execution this passes:
$ runqemu slirp nographic novga core-image-ptest-openssh
...
root@qemux86-64:~# ptest-runner
START: ptest-runner
2023-08-23T12:04
BEGIN: /usr/lib/openssh/ptest
...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
test banner: size 1
test banner: size 10
test banner: suppress banner (-q)
PASS: banner
...
client rekey tests are still running.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186574):
https://lists.openembedded.org/g/openembedded-core/message/186574
Mute This Topic: https://lists.openembedded.org/mt/100914026/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi,
On Wed, Aug 23, 2023 at 11:49:39AM +0100, Richard Purdie wrote:
> On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> > On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > > Hi,
> > > >
> > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > > will this work when running multiple instances of qemu ?
> > > > > e.g. try bitbake core-image-ptest-all
> > > >
> > > > I was not aware of core-image-ptest-all. Tried to build it but it
> > > > doesn't
> > > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear"
> > > > which is
> > > > needed to test core-image-minimal:
> > > >
> > > > Error:
> > > > Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from
> > > > oe-repo requires dropbear, but none of the providers can be installed
> > > > - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with
> > > > openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > > > - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with
> > > > dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > > > - conflicting requests
> > > > (try to add '--allowerasing' to command line to replace conflicting
> > > > packages or '--skip-broken' to skip uninstallable packages)
> > > >
> > > > oeqa runtime testing core-image-minimal without ssh server doesn't make
> > > > sense as all tests will
> > > > just be skipped.
> > >
> > > The autobuilder actually does that, the minimal image is just tested
> > > with the small number of non-network tests. The main thing was to test
> > > it does actually boot to a login prompt. We have other tests which test
> > > the other areas with other images.
> >
> > Yes, granted it's enough to test that boot to serial console login works.
> >
> > > The reason for the above is that there will be ptest openssh images
> > > which conflict with the dropbear ones. You can likely avoid that by
> > > using:
> > >
> > > IMAGE_FEATURES:append:pn-core-image-minimal = " ssh-server-dropbear"
> > >
> > > The ptest images are designed to only include the ptest in question so
> > > in theory are otherwise as minimal as the dependencies allow.
> >
> > Alright, this I could try. But I fear there is a log more missing from my
> > plain poky and default machine target to get the selftests and tests
> > running.
>
> There is no secret magic config the autobuilder uses. You keep asking
> me for this and there isn't anything. It is actually starting to annoy
> me a bit as there isn't anything "hidden".
If I can't run the tests for poky the way yocto upstream does, then I'm afraid
I can't really help with the issues you've seeing with the tests.
> The configurations used are all from this file:
>
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json
Ok so https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1948
for example uses yocto-autobuilder-helper at commit
af5d072a654a060c3ee61b5f394f52632e20200b. And the full repo
URL is visible not in the snippet there but in the full stdio download
of "Fetch yocto-autobuilder-helper" step. It was the web UI confusing me.
I'm sorry if these are all obvious to you and other developers but not
to me.
> Yes, there is a block of high level config around numbers of threads,
> disk space monitoring, pressure regulation values and so on but we
> purposefully keep the config to be as close to standard poky as we can.
>
> When we run selftest we do a couple of things. Firstly we split the
> machine and toolchain targets into separate areas. We also split
> reproducibility to it's own target and test mirroring elsewhere too.
> This results in a slightly more complex selftest invocation:
>
> OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1
> oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg
> buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine
> -T toolchain-user -T toolchain-system -j 15
>
> The only test which I don't think we run anywhere any more is the
> test_checkpkg target.
>
> You can see all this from the logs buildbot shows from it's UI on the
> autobuilder too.
Thanks, this explains alot!
I'm still wondering how the build hosts are setup with "modprobe vgem" a
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi,
On Wed, Aug 23, 2023 at 11:57:50AM +0200, Alexander Kanavin wrote:
> On Wed, 23 Aug 2023 at 11:48, Mikko Rapeli wrote:
>
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> > QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > RESULTS:
> > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> > RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED
> > (0.00s)
> > SUMMARY:
> > core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> > core-image-ptest-libtry-tiny-perl - OK - All required tests passed
> > (successes=3, skipped=1, failures=0, errors=0)
> >
> > The ptest execution seems to be skipped for all images.
>
> This is actually correct and ptest did run. We have a soft and hard
> fail options for ptests, and it defaults to soft fail
> (PTEST_EXPECT_FAILURE = "1" in core-image-ptest), i.e. warnings on the
> autobuilder.
OK, that is good to know. In this case with local.conf additions for "slirp":
IMAGE_CLASSES += "testimage"
#IMAGE_FEATURES += "ssh-server-dropbear"
TEST_RUNQEMUPARAMS += "slirp"
# update kernel to latest available in poky
PREFERRED_VERSION_linux-yocto = ""
SANITY_TESTED_DISTROS = ""
The tests passed with one warning, which I'll try to check:
WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing
ptests.
Traceback (most recent call last):
File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line
35, in wrapped_f
return func(*args, **kwargs)
File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line
35, in wrapped_f
return func(*args, **kwargs)
File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line
35, in wrapped_f
return func(*args, **kwargs)
File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27,
in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120,
in do_ptestrunner
self.fail(failmsg)
AssertionError:
Failed ptests:
{'openssh': ['banner']}
...
SUMMARY:
core-image-ptest-lttng-tools () - Ran 5 tests in 2289.072s
core-image-ptest-lttng-tools - OK - All required tests passed (successes=3,
skipped=1, failures=0, errors=0)
NOTE: recipe core-image-ptest-lttng-tools-1.0-r0: task do_testimage: Succeeded
NOTE: Running noexec task 1405 of 1405
(/home/builder/src/poky/meta/recipes-core/images/core-image-ptest-all.bb:do_testimage)
NOTE: Tasks Summary: Attempted 1405 tasks of which 1304 didn't need to be rerun
and all succeeded.
Summary: There was 1 WARNING message.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186571):
https://lists.openembedded.org/g/openembedded-core/message/186571
Mute This Topic: https://lists.openembedded.org/mt/100910036/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi, On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote: > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote: > > > will this work when running multiple instances of qemu ? > > > e.g. try bitbake core-image-ptest-all > > > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is > > needed to test core-image-minimal: > > > > Error: > > Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo > > requires dropbear, but none of the providers can be installed > > - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with > > openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo > > - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear > > provided by dropbear-2022.83-r0.core2_64 from oe-repo > > - conflicting requests > > (try to add '--allowerasing' to command line to replace conflicting > > packages or '--skip-broken' to skip uninstallable packages) > > > > oeqa runtime testing core-image-minimal without ssh server doesn't make > > sense as all tests will > > just be skipped. > > The autobuilder actually does that, the minimal image is just tested > with the small number of non-network tests. The main thing was to test > it does actually boot to a login prompt. We have other tests which test > the other areas with other images. Yes, granted it's enough to test that boot to serial console login works. > The reason for the above is that there will be ptest openssh images > which conflict with the dropbear ones. You can likely avoid that by > using: > > IMAGE_FEATURES:append:pn-core-image-minimal = " ssh-server-dropbear" > > The ptest images are designed to only include the ptest in question so > in theory are otherwise as minimal as the dependencies allow. Alright, this I could try. But I fear there is a log more missing from my plain poky and default machine target to get the selftests and tests running. This magic is somewhere in the autobuilder related git repositories, but from plain poky checkout with a specific commit from master branch I don't know which versions and repos to use so that the tests would be passing. With these modifications in local.conf: IMAGE_CLASSES += "testimage" TEST_RUNQEMUPARAMS += "slirp" IMAGE_FEATURES += "ssh-server-dropbear" # update kernel to latest available in poky PREFERRED_VERSION_linux-yocto = "" SANITY_TESTED_DISTROS = "" at least runtime_test.TestImage are passing with slirp now. Without MAGE_FEATURES += "ssh-server-dropbear", "bitbake core-image-ptest-all" now succeeds and "bitbake -c testimage core-image-ptest-all" is running the tests, seeminly in series. At least there are no multiple qemu instances running in parallel and no failures related to slirp ssh port being reserved by a single qemu instance. But the tests are reporting only skips so maybe the autobuilder scripts have some settings which I don't have correctly set: Cannot run ptests without @expectedFailure as ptests are expected to fail QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect Cannot run ptests without @expectedFailure as ptests are expected to fail QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect Cannot run ptests without @expectedFailure as ptests are expected to fail RESULTS: RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s) RESULTS - ping.PingTest.test_ping: PASSED (0.04s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s) RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s) SUMMARY: core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0) The ptest execution seems to be skipped for all images. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186564): https://lists.openembedded.org/g/openembedded-core/message/186564 Mute This Topic: https://lists.openembedded.org/mt/100910036/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi, On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote: > will this work when running multiple instances of qemu ? > e.g. try bitbake core-image-ptest-all I was not aware of core-image-ptest-all. Tried to build it but it doesn't seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is needed to test core-image-minimal: Error: Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo - conflicting requests (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages) oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will just be skipped. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186561): https://lists.openembedded.org/g/openembedded-core/message/186561 Mute This Topic: https://lists.openembedded.org/mt/100910036/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi, On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote: > On Tue, Aug 22, 2023 at 11:10 PM Mikko Rapeli wrote: > > > > Also set target and server IP addresses correctly to 127.0.0.1 > > so that TEST_TARGET_IP and TEST_SERVER_IP don't > > need to be set manually with slirp. > > > > Users run qemu images with slirp networking like this: > > > > $ bitbake core-image-minimal > > $ ../poky/scripts/runqemu slirp > > > > slirp networking means that only one TCP port is forwarded from host > > system to the qemu machine so that ssh into the machine works. ping > > and other low level networking from host to target machine do not work, > > but networking from target via host does work. This is much more easy to > > setup than the bridge networking used by default, which I've never > > managed to get running on various host machines. > > > > To use slirp networking wita qemu and core-image-minimal to run > > testimage.bbclass tests, add > > these to local.conf: > > > > IMAGE_CLASSES += "testimage" > > IMAGE_FEATURES += "ssh-server-dropbear" > > TEST_RUNQEMUPARAMS += "slirp" > > > > Then image can be compiled and tested with: > > > > $ bitbake core-image-minimal > > $ bitbake -c testimage core-image-minimal > > > > Note that only few of the current oeqa runtime tests work against > > core-image-minimal, but test results look like: > > > > RESULTS: > > RESULTS - date.DateTest.test_date: PASSED (3.64s) > > RESULTS - df.DfTest.test_df: PASSED (0.55s) > > RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s) > > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s) > > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s) > > RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s) > > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s) > > RESULTS - ping.PingTest.test_ping: PASSED (0.05s) > > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s) > > RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s) > > RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s) > > RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s) > > RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s) > > RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s) > > RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s) > > RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s) > > RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s) > > RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s) > > RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s) > > RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s) > > RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s) > > RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s) > > RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s) > > RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s) > > RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s) > > RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s) > > RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s) > > RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s) > > RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s) > > RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s) > > RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s) > > RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s) > > RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s) > > RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s) > > RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s) > > RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s) > > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_s
[OE-core] [PATCH v2 8/9] oeqa selftest runtime_test.py: append to TEST_RUNQEMUPARAMS
Don't overwrite TEST_RUNQEMUPARAMS since user may have added "slirp"
or similar things to it in site.conf, auto.conf or local.conf. Use
:append to add to it. Fixes failing tests when only "slirp" networking
works with qemu and tun based one is broken on the host machine.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/cases/runtime_test.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py
b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 19c8f0e9ec..5f90bc658f 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -251,11 +251,11 @@ TEST_RUNQEMUPARAMS:append = " slirp"
features += 'TEST_SUITES = "ping ssh virgl"\n'
features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
features += 'IMAGE_INSTALL:append = " kmscube"\n'
-features_gtk = features + 'TEST_RUNQEMUPARAMS = "gtk gl"\n'
+features_gtk = features + 'TEST_RUNQEMUPARAMS:append = " gtk gl"\n'
self.write_config(features_gtk)
bitbake('core-image-minimal')
bitbake('-c testimage core-image-minimal')
-features_sdl = features + 'TEST_RUNQEMUPARAMS = "sdl gl"\n'
+features_sdl = features + 'TEST_RUNQEMUPARAMS:append = " sdl gl"\n'
self.write_config(features_sdl)
bitbake('core-image-minimal')
bitbake('-c testimage core-image-minimal')
@@ -283,7 +283,7 @@ TEST_RUNQEMUPARAMS:append = " slirp"
features += 'TEST_SUITES = "ping ssh virgl"\n'
features += 'IMAGE_FEATURES:append = " ssh-server-dropbear"\n'
features += 'IMAGE_INSTALL:append = " kmscube"\n'
-features += 'TEST_RUNQEMUPARAMS = "egl-headless"\n'
+features += 'TEST_RUNQEMUPARAMS:append = " egl-headless"\n'
self.write_config(features)
bitbake('core-image-minimal')
bitbake('-c testimage core-image-minimal')
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186553):
https://lists.openembedded.org/g/openembedded-core/message/186553
Mute This Topic: https://lists.openembedded.org/mt/100910042/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 7/9] oeqa selftest context.py: fix git commands and set branch name
Check return values of each git command and set initial branch name
to avoid this warning on every test run:
NOTE: Starting bitbake server...
hint: Using 'master' as the name for the initial branch. This default branch
name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/context.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index bd03e765b0..bc3915a3aa 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -104,7 +104,7 @@ class OESelftestTestContext(OETestContext):
oe.path.copytree(builddir + "/cache", newbuilddir + "/cache")
oe.path.copytree(selftestdir, newselftestdir)
-subprocess.check_output("git init; git add *; git commit -a -m
'initial'", cwd=newselftestdir, shell=True)
+subprocess.check_output("git init -b main && git add * && git commit
-a -m 'initial'", cwd=newselftestdir, shell=True)
# Tried to used bitbake-layers add/remove but it requires recipe
parsing and hence is too slow
subprocess.check_output("sed %s/conf/bblayers.conf -i -e 's#%s#%s#g'"
% (newbuilddir, selftestdir, newselftestdir), cwd=newbuilddir, shell=True)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186552):
https://lists.openembedded.org/g/openembedded-core/message/186552
Mute This Topic: https://lists.openembedded.org/mt/100910041/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 9/9] oeqa selftest runtime_test.py: skip virgl_headless test if /dev/dri/renderD128 access fails
If access to /dev/dri/renderD128 fails, then qemu with 3d graphics will
fail to start and errors are rather cryptic like:
qemu-system-x86_64: egl: render node init failed
To fix this, users likely need to
* modprobe vgem
* add their user to "render" group to write to /dev/dri/renderD128
If access is not available due to missing HW, driver or failing access,
then skip the test:
2023-08-22 14:41:20,591 - oe-selftest - INFO - test_testimage_virgl_headless
(runtime_test.TestImage)
2023-08-22 14:41:20,603 - oe-selftest - INFO - ... skipped 'Can not open
"/dev/dri/renderD128" device'
2023-08-22 14:41:20,603 - oe-selftest - INFO - Can not open
"/dev/dri/renderD128" device
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/cases/runtime_test.py | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py
b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 5f90bc658f..e72504773a 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -264,7 +264,8 @@ TEST_RUNQEMUPARAMS:append = " slirp"
def test_testimage_virgl_headless(self):
"""
Summary: Check host-assisted accelerate OpenGL functionality in qemu
with egl-headless frontend
-Expected: 1. Check that virgl kernel driver is loaded and 3d
acceleration is enabled
+Expected: 1. Check that virgl kernel driver is loaded (modprobe vgem,
user part of "render" group and
+ can open /dev/dri/renderD* device) and 3d acceleration is
enabled
2. Check that kmscube demo runs without crashing.
Product: oe-core
Author: Alexander Kanavin
@@ -276,6 +277,14 @@ TEST_RUNQEMUPARAMS:append = " slirp"
distro.startswith('almalinux') or distro.startswith('rocky')):
self.skipTest('virgl headless cannot be tested with %s' %(distro))
+# test requires vgem driver and possibly "render" group rights to
access device file
+render_dev = "/dev/dri/renderD128"
+try:
+with open(render_dev, "w") as f:
+f.close()
+except IOError:
+self.skipTest('Can not open "%s" device' % (render_dev))
+
qemu_distrofeatures = get_bb_var('DISTRO_FEATURES',
'qemu-system-native')
features = 'IMAGE_CLASSES += "testimage"\n'
if 'opengl' not in qemu_distrofeatures:
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186554):
https://lists.openembedded.org/g/openembedded-core/message/186554
Mute This Topic: https://lists.openembedded.org/mt/100910043/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 5/9] oeqa selftest context.py: whitespace fix
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/context.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index f2a5ba792f..4b33245758 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -226,14 +226,14 @@ class
OESelftestTestContextExecutor(OETestContextExecutor):
machines = []
bbpath = self.tc_kwargs['init']['td']['BBPATH'].split(':')
-
+
for path in bbpath:
found_machines = glob.glob(os.path.join(path, 'conf', 'machine',
'*.conf'))
if found_machines:
for i in found_machines:
# eg:
'/home//poky/meta-intel/conf/machine/intel-core2-32.conf'
machines.append(os.path.splitext(os.path.basename(i))[0])
-
+
return machines
def _get_cases_paths(self, bbpath):
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186546):
https://lists.openembedded.org/g/openembedded-core/message/186546
Mute This Topic: https://lists.openembedded.org/mt/100910035/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
mpile: SKIPPED (0.00s)
RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED
(0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED
(0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor:
SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
SUMMARY:
core-image-minimal () - Ran 70 tests in 22.173s
core-image-minimal - OK - All required tests passed (successes=14, skipped=56,
failures=0, errors=0)
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun
and all succeeded.
Signed-off-by: Mikko Rapeli
---
meta/classes-recipe/testimage.bbclass | 14 +++---
1 file changed, 11 insertions(+), 3 deletions(-)
v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
so that testimage selftests pass
v1: https://lists.openembedded.org/g/openembedded-core/message/186424
diff --git a/meta/classes-recipe/testimage.bbclass
b/meta/classes-recipe/testimage.bbclass
index e3068348ff..7340996788 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -322,7 +322,7 @@ def testimage_main(d):
ovmf = d.getVar("QEMU_USE_OVMF")
slirp = False
-if d.getVar("QEMU_USE_SLIRP"):
+if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS',
'slirp', True, False, d):
slirp = True
# TODO: We use the current implementation of qemu runner because of
@@ -369,10 +369,18 @@ def testimage_main(d):
# runtime use network for download projects for build
export_proxies(d)
+if slirp:
+target_ip = "127.0.0.1"
+# from qemu target to host with default DHCP server
+server_ip = "10.0.2.2"
+else:
+target_ip = d.getVar("TEST_TARGET_IP")
+server_ip = d.getVar("TEST_SERVER_IP")
+
# the robot dance
target = OERuntimeTestContextExecutor.getTarget(
-d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
-d.getVar("TEST_SERVER_IP"), **target_kwargs)
+d.getVar("TEST_TARGET"), logger, target_ip,
+server_ip, **target_kwargs)
# test context
tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186547):
https://lists.openembedded.org/g/openembedded-core/message/186547
Mute This Topic: https://lists.openembedded.org/mt/100910036/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 4/9] oeqa dnf_runtime.py: fix HTTP server IP address and port
Use correct HTTPService parameters like apt.py when setting up the repo server. These work with qemu tun and slirp networking. Fixes test failure with slirp networking when executing testimage.bbclass selftests "oe-selftest -r runtime_test.TestImage". Signed-off-by: Mikko Rapeli --- meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py index 64a3502370..d2f0f88f7d 100644 --- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py +++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py @@ -10,7 +10,8 @@ class DnfSelftest(DnfTest): import tempfile cls.temp_dir = tempfile.TemporaryDirectory(prefix="oeqa-remotefeeds-") cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'), - cls.tc.target.server_ip) + '0.0.0.0', port=cls.tc.target.server_port, + logger=cls.tc.logger) cls.repo_server.start() @classmethod -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186548): https://lists.openembedded.org/g/openembedded-core/message/186548 Mute This Topic: https://lists.openembedded.org/mt/100910037/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test
testimage.bbclass requires that ssh communication with target works
and then it runs oeqa runtime tests with that. Tests like df.py
check that there is more than 5Mb free space on the rootfs on target. Sadly
latest core-image-minimal only has 1.5 Mb free space if
ssh-server-dropbear is added to it. Thus by default, core-image-minimal
is now failing oeqa rutime df.py test.
Fix this by increasing core-image-minimal rootfs size by 5Mb if
testimage.bbclass is used which implies adding either
ssh-server-dropbear or ssh-server-openssh to the target rootfs.
Signed-off-by: Mikko Rapeli
---
meta/recipes-core/images/core-image-minimal.bb | 3 +++
1 file changed, 3 insertions(+)
v2: no changes
v1: https://lists.openembedded.org/g/openembedded-core/message/186423
diff --git a/meta/recipes-core/images/core-image-minimal.bb
b/meta/recipes-core/images/core-image-minimal.bb
index 84343adcd8..c04612bc15 100644
--- a/meta/recipes-core/images/core-image-minimal.bb
+++ b/meta/recipes-core/images/core-image-minimal.bb
@@ -10,3 +10,6 @@ inherit core-image
IMAGE_ROOTFS_SIZE ?= "8192"
IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES",
"systemd", " + 4096", "", d)}"
+
+# a bit more space needed for sshd to actually run the tests
+IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES",
"testimage", " + 5120", "", d)}"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186549):
https://lists.openembedded.org/g/openembedded-core/message/186549
Mute This Topic: https://lists.openembedded.org/mt/100910038/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 6/9] oeqa selftest context.py: remove warning from missing meta-selftest
It's not a warning but a handled case and layer gets added
automatically. Very few build configs have this layer enabled
by default.
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/context.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index 4b33245758..bd03e765b0 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -306,7 +306,7 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
os.chdir(builddir)
if not "meta-selftest" in self.tc.td["BBLAYERS"]:
-self.tc.logger.warning("meta-selftest layer not found in
BBLAYERS, adding it")
+self.tc.logger.info("meta-selftest layer not found in
BBLAYERS, adding it")
meta_selftestdir = os.path.join(
self.tc.td["BBLAYERS_FETCH_DIR"], 'meta-selftest')
if os.path.isdir(meta_selftestdir):
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186551):
https://lists.openembedded.org/g/openembedded-core/message/186551
Mute This Topic: https://lists.openembedded.org/mt/100910040/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 3/9] selftest runtime_test.py: add testimage.bbclass slirp test
Builds core-image-minimal with testimage.bbclass and an ssh server
and then runs tests with slirp networking. Default test suite must
pass.
With full caches on x86_64 with x86_64 target the test passes in 416 seconds:
2023-08-21 09:01:12,295 - oe-selftest - INFO - Ran 1 test in 464.955s
2023-08-21 09:01:12,295 - oe-selftest - INFO - OK
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS:
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS -
runtime_test.TestImage.test_testimage_slirp: PASSED (416.62s)
2023-08-21 09:01:19,711 - oe-selftest - INFO - SUMMARY:
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest () - Ran 1 test in
464.956s
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest - OK - All required
tests passed (successes=1, skipped=0, failures=0, errors=0)
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/cases/runtime_test.py | 15 +++
1 file changed, 15 insertions(+)
v2: use :append instead of overwriting variables
v1: https://lists.openembedded.org/g/openembedded-core/message/186427
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py
b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 463679dd32..19c8f0e9ec 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -129,6 +129,21 @@ class TestImage(OESelftestTestCase):
bitbake('core-image-full-cmdline socat')
bitbake('-c testimage core-image-full-cmdline')
+def test_testimage_slirp(self):
+"""
+Summary: Check basic testimage functionality with qemu and slirp
networking.
+"""
+
+features = '''
+IMAGE_CLASSES:append = " testimage"
+IMAGE_FEATURES:append = " ssh-server-dropbear"
+TEST_RUNQEMUPARAMS:append = " slirp"
+'''
+self.write_config(features)
+
+bitbake('core-image-minimal')
+bitbake('-c testimage core-image-minimal')
+
def test_testimage_dnf(self):
"""
Summary: Check package feeds functionality for dnf
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186550):
https://lists.openembedded.org/g/openembedded-core/message/186550
Mute This Topic: https://lists.openembedded.org/mt/100910039/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Dilemma on changes - merge or not to merge (e.g. 6.4)
Hi, On Tue, Aug 22, 2023 at 12:47:04PM +0100, Richard Purdie wrote: > so between 6.1.38 and 6.1.39? Maybe: commit b1cdc56bc177c2e182c204bb08ad4e87bfd67942 Author: Paul E. McKenney AuthorDate: Wed Apr 26 11:11:29 2023 -0700 Commit: Greg Kroah-Hartman CommitDate: Wed Jul 19 16:21:01 2023 +0200 rcu-tasks: Stop rcu_tasks_invoke_cbs() from using never-onlined CPUs and commit d58f0f0ce6332ffeb406540295cc49732c26fb51 Author: Paul E. McKenney AuthorDate: Thu Apr 27 10:50:47 2023 -0700 Commit: Greg Kroah-Hartman CommitDate: Wed Jul 19 16:21:01 2023 +0200 rcu: Make rcu_cpu_starting() rely on interrupts being disabled ? master branch seems to have larger set of changes to rcu. Maybe locking debugging options could help to find this on every boot. Then wasn't commit 77cc52f1b8d76c995648cb4286e57142cac8ce0a Author: Wen Yang AuthorDate: Fri May 5 00:12:53 2023 +0800 Commit: Greg Kroah-Hartman CommitDate: Wed Jul 19 16:20:59 2023 +0200 tick/rcu: Fix bogus ratelimit condition [ Upstream commit a7e282c77785c7eabf98836431b1f029481085ad ] causing some issues too? Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186505): https://lists.openembedded.org/g/openembedded-core/message/186505 Mute This Topic: https://lists.openembedded.org/mt/100733646/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/3] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
Hi, On Mon, Aug 21, 2023 at 01:00:51PM +0300, Mikko Rapeli wrote: > Also set target and server IP addresses correctly to 127.0.0.1 > so that TEST_TARGET_IP and TEST_SERVER_IP don't > need to be set manually with slirp. > > Users run qemu images with slirp networking like this: > > $ bitbake core-image-minimal > $ ../poky/scripts/runqemu slirp > > slirp networking means that only one TCP port is forwarded from host > system to the qemu machine so that ssh into the machine works. ping > and other low level networking from host to target machine do not work, > but networking from target via host does work. This is much more easy to > setup than the bridge networking used by default, which I've never > managed to get running on various host machines. > > To use slirp networking wita qemu and core-image-minimal to run > testimage.bbclass tests, add > these to local.conf: > > IMAGE_CLASSES += "testimage" > IMAGE_FEATURES += "ssh-server-dropbear" > TEST_RUNQEMUPARAMS += "slirp" > > Then image can be compiled and tested with: > > $ bitbake core-image-minimal > $ bitbake -c testimage core-image-minimal > > Note that only few of the current oeqa runtime tests work against > core-image-minimal, but test results look like: > > RESULTS: > RESULTS - date.DateTest.test_date: PASSED (3.64s) > RESULTS - df.DfTest.test_df: PASSED (0.55s) > RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s) > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s) > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s) > RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s) > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s) > RESULTS - ping.PingTest.test_ping: PASSED (0.05s) > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s) > RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s) > RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s) > RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s) > RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s) > RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s) > RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s) > RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s) > RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s) > RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s) > RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s) > RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s) > RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s) > RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s) > RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s) > RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s) > RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s) > RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s) > RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s) > RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s) > RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s) > RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s) > RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s) > RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s) > RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s) > RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s) > RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s) > RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s) > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED > (0.00s) > RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s) > RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s) > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s) > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED > (0.00s) > RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s) > RESULTS - r
Re: [OE-core] [PATCH 00/10] linux-yocto: consolidated pull request
Hi, On Tue, Aug 22, 2023 at 09:13:44AM +0200, Luca Ceresoli via lists.openembedded.org wrote: > Hello Bruce, > > On Mon, 21 Aug 2023 11:44:35 -0400 > "Bruce Ashfield" wrote: > > > From: Bruce Ashfield > > > > Richard, > > > > While we track down the remaining 6.4 boot issues, here is the > > set of -stable and specific fixes that I've queued in my kernel > > testing tree. > > > > It is possible that some of the 6.4-stable's have fixes for our > > boot issues, but I didn't see anything obvious. > > > > In particular we have the -rt warning fix from PaulG and also > > a fixup on our isocpus patch. > > > > Cheers, > > > > Bruce > > You're probably already aware of the build failure triggered by this > patch series: > > stdio: ERROR: linux-yocto-6.4.11+gitAUTOINC+d9483a24c9_9753b44979-r0 > do_fetch: Fetcher failure: Unable to find revision > d9483a24c9ee33b6e566baa5233cbd9feaeb5212 in branch yocto-6.4 even from > upstream > > This one is less visible: > > WARNING: linux-yocto-6.4.11+gitAUTOINC+d9483a24c9_9753b44979-r0 > do_kernel_configcheck: [kernel config]: This BSP contains fragments with > warnings: > > [INFO]: the following symbols were not found in the active configuration: >- CONFIG_COMMON_CLK_NPCM8XX Fix for this warning has been submitted upstream: https://lists.infradead.org/pipermail/linux-arm-kernel/2023-August/859760.html and meta-arm has applied it separately with: https://git.yoctoproject.org/meta-arm/commit/?id=5b8329a99f5fb6497575612ae677e03decf2e3b2 I hope upstream would pick this up for master and all 6.x stable branches. No feedback so far. Cheers, -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#186495): https://lists.openembedded.org/g/openembedded-core/message/186495 Mute This Topic: https://lists.openembedded.org/mt/100875501/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/3] selftest runtime_test.py: add testimage.bbclass slirp test
Builds core-image-minimal with testimage.bbclass and an ssh server
and then runs tests with slirp networking. Default test suite must
pass.
With full caches on x86_64 with x86_64 target the test passes in 416 seconds:
2023-08-21 09:01:12,295 - oe-selftest - INFO - Ran 1 test in 464.955s
2023-08-21 09:01:12,295 - oe-selftest - INFO - OK
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS:
2023-08-21 09:01:19,648 - oe-selftest - INFO - RESULTS -
runtime_test.TestImage.test_testimage_slirp: PASSED (416.62s)
2023-08-21 09:01:19,711 - oe-selftest - INFO - SUMMARY:
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest () - Ran 1 test in
464.956s
2023-08-21 09:01:19,711 - oe-selftest - INFO - oe-selftest - OK - All required
tests passed (successes=1, skipped=0, failures=0, errors=0)
Signed-off-by: Mikko Rapeli
---
meta/lib/oeqa/selftest/cases/runtime_test.py | 15 +++
1 file changed, 15 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py
b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 463679dd32..d8dca7f671 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -129,6 +129,21 @@ class TestImage(OESelftestTestCase):
bitbake('core-image-full-cmdline socat')
bitbake('-c testimage core-image-full-cmdline')
+def test_testimage_slirp(self):
+"""
+Summary: Check basic testimage functionality with qemu and slirp
networking.
+"""
+
+features = '''
+IMAGE_CLASSES += "testimage"
+IMAGE_FEATURES += "ssh-server-dropbear"
+TEST_RUNQEMUPARAMS += "slirp"
+'''
+self.write_config(features)
+
+bitbake('core-image-minimal')
+bitbake('-c testimage core-image-minimal')
+
def test_testimage_dnf(self):
"""
Summary: Check package feeds functionality for dnf
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186427):
https://lists.openembedded.org/g/openembedded-core/message/186427
Mute This Topic: https://lists.openembedded.org/mt/100869975/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/3] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS
mpile: SKIPPED (0.00s)
RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED
(0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED
(0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor:
SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
SUMMARY:
core-image-minimal () - Ran 70 tests in 22.173s
core-image-minimal - OK - All required tests passed (successes=14, skipped=56,
failures=0, errors=0)
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun
and all succeeded.
Signed-off-by: Mikko Rapeli
---
meta/classes-recipe/testimage.bbclass | 13 ++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/meta/classes-recipe/testimage.bbclass
b/meta/classes-recipe/testimage.bbclass
index e3068348ff..ad3adceca9 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -322,7 +322,7 @@ def testimage_main(d):
ovmf = d.getVar("QEMU_USE_OVMF")
slirp = False
-if d.getVar("QEMU_USE_SLIRP"):
+if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS',
'slirp', True, False, d):
slirp = True
# TODO: We use the current implementation of qemu runner because of
@@ -369,10 +369,17 @@ def testimage_main(d):
# runtime use network for download projects for build
export_proxies(d)
+if slirp:
+target_ip = "127.0.0.1"
+server_ip = target_ip
+else:
+target_ip = d.getVar("TEST_TARGET_IP")
+server_ip = d.getVar("TEST_SERVER_IP")
+
# the robot dance
target = OERuntimeTestContextExecutor.getTarget(
-d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
-d.getVar("TEST_SERVER_IP"), **target_kwargs)
+d.getVar("TEST_TARGET"), logger, target_ip,
+server_ip, **target_kwargs)
# test context
tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186424):
https://lists.openembedded.org/g/openembedded-core/message/186424
Mute This Topic: https://lists.openembedded.org/mt/100869824/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] core-image-minimal: increase extra space to pass df.py oeqa runtime test
testimage.bbclass requires that ssh communication with target works
and then it runs oeqa runtime tests with that. Tests like df.py
check that there is more than 5Mb free space on the rootfs on target. Sadly
latest core-image-minimal only has 1.5 Mb free space if
ssh-server-dropbear is added to it. Thus by default, core-image-minimal
is now failing oeqa rutime df.py test.
Fix this by increasing core-image-minimal rootfs size by 5Mb if
testimage.bbclass is used which implies adding either
ssh-server-dropbear or ssh-server-openssh to the target rootfs.
Signed-off-by: Mikko Rapeli
---
meta/recipes-core/images/core-image-minimal.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/recipes-core/images/core-image-minimal.bb
b/meta/recipes-core/images/core-image-minimal.bb
index 84343adcd8..c04612bc15 100644
--- a/meta/recipes-core/images/core-image-minimal.bb
+++ b/meta/recipes-core/images/core-image-minimal.bb
@@ -10,3 +10,6 @@ inherit core-image
IMAGE_ROOTFS_SIZE ?= "8192"
IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("DISTRO_FEATURES",
"systemd", " + 4096", "", d)}"
+
+# a bit more space needed for sshd to actually run the tests
+IMAGE_ROOTFS_EXTRA_SPACE:append = "${@bb.utils.contains("IMAGE_CLASSES",
"testimage", " + 5120", "", d)}"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186423):
https://lists.openembedded.org/g/openembedded-core/message/186423
Mute This Topic: https://lists.openembedded.org/mt/100869823/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core][PATCH 1/2] systemd-boot: enable verbose compilation
Hi,
On Wed, Aug 09, 2023 at 08:47:05AM +0200, Alexander Kanavin wrote:
> On Wed, 9 Aug 2023 at 01:06, Khem Raj wrote:
> >>> I think this is only useful during debugging. So perhaps a comment to
> >>> enable it is better than enabling it always.
> >>
> >>
> >> The meson bbclass also compile with verbose so in my opinion it is better
> >> to do the same
> >> and make it consistent with the main systemd recipe
> >>
> >> meson_do_compile() {
> >> meson compile -v ${PARALLEL_MAKE}
> >> }
> >
> >
> > Ideally we only want the verbosity when something goes wrong and meson
> > ninja etc are good at that with their defaults perhaps that -v should be
> > turned off by default as well
>
> Absolutely not. As someone who spends a lot of time fixing build and
> runtime failures across all of oe-core I beg to differ. I want
> verbosity all the time, everywhere. I also do not want to spend a
> second figuring out how to switch it on (it's different everywhere
> too), or to run another build just to get the verbose logs out of it.
I prefer verbose-by-default too. It is really annoying to investigate
build race conditions in various CI environments without enough details in
logs. Had to trigger a lot test builds because of this in the past.
Same goes for testing. I know logging infra can be abused and on the limit too,
but I don't think bitbake task log sizes increase that much compared to
other stuff remaining in build tmp directories after builds.
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185693):
https://lists.openembedded.org/g/openembedded-core/message/185693
Mute This Topic: https://lists.openembedded.org/mt/100625045/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
