Re: [OE-core] [PATCH v3] rootfs-postcommands: Set vardeps for do_write_test_data
Hello,
This causes oe-selftest failures on the autobuilders:
File
"/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/core/decorator/__init__.py",
line 35, in wrapped_f
return func(*args, **kwargs)
File
"/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-2280027/meta-selftest/lib/oeqa/runtime/cases/selftest.py",
line 18, in test_install_package
self.assertEqual(status, 0, msg="socat is not installed")
AssertionError: 126 != 0 : socat is not installed
Test case selftest.Selftest.test_verify_uninstall depends on
selftest.Selftest.test_install_package but it didn't pass/run.
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5450/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5417/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1691/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5373/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5424/steps/14/logs/stdio
On 29/06/2023 16:26:54+0100, Peter Hoyes wrote:
> From: Peter Hoyes
>
> The testdata.json file generated by testdata.bbclass currently contains
> almost all Bitbake variables. However only a small number of variables
> are actually used and the testdata.json is not automatically updated
> when the variables are updated.
>
> Introduce the testdata_vars bbclass which defines TESTDATA_EXPORT_VARS
> to explicitly set the variables to be collected for testdata.json and
> populate with all variables used by runtime and sdk tests in OE-core.
> Use this variable to set the vardeps of do_write_test_data.
>
> Modify export2json to take an explicit list of keys to export, instead
> of exporting everything in the datastore.
>
> Adapt write_sdk_test_data and meta-ide-support to use the same list of
> testdata variables.
>
> Add a demonstrative OE selftest to rootfspostcommandstests.
>
> Signed-off-by: Peter Hoyes
> ---
>
> Changes from v2:
> * Move TESTDATA_EXPORT_VARS to common testdata_vars.bbclass
> * Update other export2json call sites (eSDK and meta-ide-support)
>
> Tested with:
> * oe-selftest -r rootfspostcommandstests
> * oe-selftest -r sstatetests.SStateHashSameSigs
> * bitbake core-image-sato -c testimage
> * bitbake core-image-sato -c testsdkext
> * bitbake buildtools-tarball buildtools-extended-tarball
> * bitbake meta-ide-support
>
> meta/classes-recipe/populate_sdk_base.bbclass | 5 +++-
> .../rootfs-postcommands.bbclass | 6 +++-
> meta/classes-recipe/testdata_vars.bbclass | 30 +++
> meta/lib/oe/data.py | 17 +--
> .../selftest/cases/rootfspostcommandstests.py | 19
> meta/recipes-core/meta/meta-ide-support.bb| 5 +++-
> 6 files changed, 63 insertions(+), 19 deletions(-)
> create mode 100644 meta/classes-recipe/testdata_vars.bbclass
>
> diff --git a/meta/classes-recipe/populate_sdk_base.bbclass
> b/meta/classes-recipe/populate_sdk_base.bbclass
> index b23ea26d40..ffb06f2062 100644
> --- a/meta/classes-recipe/populate_sdk_base.bbclass
> +++ b/meta/classes-recipe/populate_sdk_base.bbclass
> @@ -134,11 +134,14 @@ sdk_prune_dirs () {
> done
> }
>
> +inherit testdata_vars
> +
> python write_sdk_test_data() {
> from oe.data import export2json
> testdata = "%s/%s.testdata.json" % (d.getVar('SDKDEPLOYDIR'),
> d.getVar('TOOLCHAIN_OUTPUTNAME'))
> bb.utils.mkdirhier(os.path.dirname(testdata))
> -export2json(d, testdata)
> +keys = d.getVar('TESTDATA_EXPORT_VARS').split()
> +export2json(d, keys, testdata)
> }
>
> python write_host_sdk_manifest () {
> diff --git a/meta/classes-recipe/rootfs-postcommands.bbclass
> b/meta/classes-recipe/rootfs-postcommands.bbclass
> index 652601b95f..26eddc8c35 100644
> --- a/meta/classes-recipe/rootfs-postcommands.bbclass
> +++ b/meta/classes-recipe/rootfs-postcommands.bbclass
> @@ -454,6 +454,8 @@ rootfs_sysroot_relativelinks () {
> }
>
> # Generated test data json file
> +inherit testdata_vars
> +
> python write_image_test_data() {
> from oe.data import export2json
>
> @@ -461,8 +463,9 @@ python write_image_test_data() {
> link_name = d.getVar('IMAGE_LINK_NAME')
> testdata_name = os.path.join(deploy_dir, "%s.testdata.json" %
> d.getVar('IMAGE_NAME'))
>
> +keys = d.getVar('TESTDATA_EXPORT_VARS').split()
> searchString = "%s/"%(d.getVar("TOPDIR")).replace("//","/")
> -export2json(d, testdata_name, searchString=searchString,
> replaceString="")
> +export2json(d, keys, testdata_name, searchString=searchString,
> replaceString="")
>
> if os.path.exists(testdata_name) and link_name:
> testdata_link = os.path.join(deploy_dir, "%s.testdata.json" %
> link_name)
> @@ -471,6 +474,7 @@ python write_image_test_data() {
> os.remove(testdata_link)
> os.symlink(os.path.basename(testdata_name),
[OE-core] [PATCH v3] rootfs-postcommands: Set vardeps for do_write_test_data
From: Peter Hoyes
The testdata.json file generated by testdata.bbclass currently contains
almost all Bitbake variables. However only a small number of variables
are actually used and the testdata.json is not automatically updated
when the variables are updated.
Introduce the testdata_vars bbclass which defines TESTDATA_EXPORT_VARS
to explicitly set the variables to be collected for testdata.json and
populate with all variables used by runtime and sdk tests in OE-core.
Use this variable to set the vardeps of do_write_test_data.
Modify export2json to take an explicit list of keys to export, instead
of exporting everything in the datastore.
Adapt write_sdk_test_data and meta-ide-support to use the same list of
testdata variables.
Add a demonstrative OE selftest to rootfspostcommandstests.
Signed-off-by: Peter Hoyes
---
Changes from v2:
* Move TESTDATA_EXPORT_VARS to common testdata_vars.bbclass
* Update other export2json call sites (eSDK and meta-ide-support)
Tested with:
* oe-selftest -r rootfspostcommandstests
* oe-selftest -r sstatetests.SStateHashSameSigs
* bitbake core-image-sato -c testimage
* bitbake core-image-sato -c testsdkext
* bitbake buildtools-tarball buildtools-extended-tarball
* bitbake meta-ide-support
meta/classes-recipe/populate_sdk_base.bbclass | 5 +++-
.../rootfs-postcommands.bbclass | 6 +++-
meta/classes-recipe/testdata_vars.bbclass | 30 +++
meta/lib/oe/data.py | 17 +--
.../selftest/cases/rootfspostcommandstests.py | 19
meta/recipes-core/meta/meta-ide-support.bb| 5 +++-
6 files changed, 63 insertions(+), 19 deletions(-)
create mode 100644 meta/classes-recipe/testdata_vars.bbclass
diff --git a/meta/classes-recipe/populate_sdk_base.bbclass
b/meta/classes-recipe/populate_sdk_base.bbclass
index b23ea26d40..ffb06f2062 100644
--- a/meta/classes-recipe/populate_sdk_base.bbclass
+++ b/meta/classes-recipe/populate_sdk_base.bbclass
@@ -134,11 +134,14 @@ sdk_prune_dirs () {
done
}
+inherit testdata_vars
+
python write_sdk_test_data() {
from oe.data import export2json
testdata = "%s/%s.testdata.json" % (d.getVar('SDKDEPLOYDIR'),
d.getVar('TOOLCHAIN_OUTPUTNAME'))
bb.utils.mkdirhier(os.path.dirname(testdata))
-export2json(d, testdata)
+keys = d.getVar('TESTDATA_EXPORT_VARS').split()
+export2json(d, keys, testdata)
}
python write_host_sdk_manifest () {
diff --git a/meta/classes-recipe/rootfs-postcommands.bbclass
b/meta/classes-recipe/rootfs-postcommands.bbclass
index 652601b95f..26eddc8c35 100644
--- a/meta/classes-recipe/rootfs-postcommands.bbclass
+++ b/meta/classes-recipe/rootfs-postcommands.bbclass
@@ -454,6 +454,8 @@ rootfs_sysroot_relativelinks () {
}
# Generated test data json file
+inherit testdata_vars
+
python write_image_test_data() {
from oe.data import export2json
@@ -461,8 +463,9 @@ python write_image_test_data() {
link_name = d.getVar('IMAGE_LINK_NAME')
testdata_name = os.path.join(deploy_dir, "%s.testdata.json" %
d.getVar('IMAGE_NAME'))
+keys = d.getVar('TESTDATA_EXPORT_VARS').split()
searchString = "%s/"%(d.getVar("TOPDIR")).replace("//","/")
-export2json(d, testdata_name, searchString=searchString, replaceString="")
+export2json(d, keys, testdata_name, searchString=searchString,
replaceString="")
if os.path.exists(testdata_name) and link_name:
testdata_link = os.path.join(deploy_dir, "%s.testdata.json" %
link_name)
@@ -471,6 +474,7 @@ python write_image_test_data() {
os.remove(testdata_link)
os.symlink(os.path.basename(testdata_name), testdata_link)
}
+write_image_test_data[vardeps] += "${TESTDATA_EXPORT_VARS}"
write_image_test_data[vardepsexclude] += "TOPDIR"
# Check for unsatisfied recommendations (RRECOMMENDS)
diff --git a/meta/classes-recipe/testdata_vars.bbclass
b/meta/classes-recipe/testdata_vars.bbclass
new file mode 100644
index 00..cecbc6d7ca
--- /dev/null
+++ b/meta/classes-recipe/testdata_vars.bbclass
@@ -0,0 +1,30 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# The variables exported to testdata.json for testimage and other similar tasks
+TESTDATA_EXPORT_VARS ?= "\
+ARCH \
+DEPLOY_DIR \
+DEPLOY_DIR_DEB \
+DISTRO_FEATURES \
+DL_DIR \
+IMAGE \
+IMAGE_FEATURES \
+libdir \
+MACHINE \
+MULTILIB_VARIANTS \
+PACKAGE_FEED_GPG_NAME \
+PTEST_EXPECT_FAILURE \
+QEMU_USE_KVM \
+SDK_DEPLOY \
+SDKPATH \
+T \
+TEST_LOG_DIR \
+TOOLCHAINEXT_OUTPUTNAME \
+TUNE_PKGARCH \
+VIRTUAL-RUNTIME_init_manager \
+WORKDIR \
+"
diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
index 37121cfad2..731f2b5b46 100644
--- a/meta/lib/oe/data.py
+++ b/meta/lib/oe/data.py
@@ -23,23 +23,8 @@ def typed_value(key, d):
except (TypeError, ValueError) as exc:
bb.msg.fatal("Data", "%s: %s" % (key, str(exc)))
-def
