Re: [OE-core] [langdale][kirkstone][PATCH] go-mod.bbclass: Allow network in do_compile
Hi Martin, Am 26.12.2022 um 10:59 schrieb Martin Jansa via lists.openembedded.org: On Mon, Dec 26, 2022 at 10:38 AM Vyacheslav Yurkov wrote: Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something? That's not my understanding. The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet). Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete). go fetcher in bitbake is another possibility for future, but generating the long deps in SRC_URI is much better solution then enabling network in every go-mod recipe. Does an agreement exist if we should use a package manager class which creates a wget fetcher URI (ex. pipi) or a specific fetcher (ex. crate and npm)? At the moment both specific fetchers have different URI styles, create their specific tar commands and doesn't reuse the wget fetcher. Furthermore, the crate fetcher doesn't check the integrity during fetch. The fetcher inside bitbake have the advantage that you can fetch the latest version (ex. AUTOREV), but it brings the risk of code duplication and missing features. If we have an agreement and a common style its easy to replace the function with a specific fetcher. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#175317): https://lists.openembedded.org/g/openembedded-core/message/175317 Mute This Topic: https://lists.openembedded.org/mt/95827446/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 01/13] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
Hi Alex, Am 09.12.2022 um 13:09 schrieb Alexander Kanavin: On Fri, 9 Dec 2022 at 11:39, Stefan Herbrechtsmeier wrote: Sorry for the harsh tone but you ignore existing tools and add just another tool to oe-core without mention any reason or document it. The reason to add the class was to be able to update existing rust-based recipes without having to manually update long lists of crates in those recipes, which is a rather urgent issue, more urgent than having rust support in recipetool. The class does not replace recipetool, in fact it can, and should be used by devtool/recipetool once someone finds time and motivation to sit down and add (currently missing) rust support to it (class functionality can be executed from those tools via tinfoil.build_targets()). I do not have that time, and I am not getting paid for doing that either. Oh, and we do not have a maintainer for those tools. Would you like to volunteer for that? Yes, I'm happy to maintain recipetool. We need to go on with our npm und go support anyway. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#174492): https://lists.openembedded.org/g/openembedded-core/message/174492 Mute This Topic: https://lists.openembedded.org/mt/94683148/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 01/13] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
Hi Richard, Am 09.12.2022 um 13:01 schrieb Richard Purdie: On Fri, 2022-12-09 at 11:39 +0100, Stefan Herbrechtsmeier wrote: Sorry for the harsh tone but you ignore existing tools and add just another tool to oe-core without mention any reason or document it. Do you really expect that somebody will patch existing tools if main developers ignore existing tools and provide new tool for their use case? I'm not sure that is entirely fair criticism. We have some challenges with some of the new languages. Rust support was in a separate layer. We took some of those recipes into core, then ended up making significant changes to them. We never took the external tool. The hope in doing that was that we'd find a better way to make things work. I think what Alex has done is an improvement over that external tool and it experiments with a different way of handling things. I've had generally quite positive feedback on the approach itself. Yes, there are some issues with documentation and some people using it have struggled with some usability issues. None of those look like they're unfixable. Why should somebody improve an existing tool, extend the documentation, add tests or even upstream its work if these same requirements don’t exist for the main developers. It looks like the requirements for foreign and main contributors are different and this doesn't encourage people to participant. I don't see us treating developers differently and I am concerned you think we/I do. Any given solution that is proposed is evaluated on it's pros and cons. In this case the solution is quite self contained and allowed the approach to be experimented with whilst solving a real world issue. If it doesn't work out we can easily drop it. The risk from taking it is therefore low. Yes, I should probably insist on documentation. In this case it is relatively simple code which is relatively easily understood so I've been less worried about that up front. If it worked out well, we can integrate it further and document it. If it doesn't it can be removed. FWIW I have heard people saying they like the approach and that we should use it for some of the other languages with challenges like this. Maybe this is only my personal feeling and I apologize my harsh tone, but the acceptance of patches should be comprehensible, and expectations should be the same for everyone. I do try to ensure that. My "algorithm" for accepting patches is probably not easily documented but I think the factors here which are the standalone nature of the change and the easy with which we could drop it if needed. As such it is in my "low risk" category of patches. I'd note Alex has another patch which has been sitting for months unmerged as it is in my "high risk" to the project category. I suspect that one will not actually merge but I need to find the time to explain why, right now it is more based on a feeling it is the wrong direction. Should others answer to your comments that their solution doesn’t preclude your suggestion and that they welcome patches? For sure it takes more time to add rust support to recipetool but I think a second tool without a clear reason in oe-core hurts more in long term because its now unclear if new features (like checksums or licenses support) should be added to this tool or if this tool is only a temporary solution and should be replaced by recipetool in long term. Furthermore, this class is marked as a class for a recipe but shouldn’t be inherit by a recipe and manipulates a file inside the meta data. We do have precedence for classes that help updates like this. Both python and perl have code that adds tasks that function a bit like this, in those cases for the core recipe. Personally, I would ultimately like to see these operations handled by recipetool and I suspect natural evolution of the code may head that way. Both devtool and recipetool have used classes as a way to help them perform operations so in that sense, this is actually a logical development path for those tools. Thanks for you feedback and the clarification. This helps me a lot. Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#174461): https://lists.openembedded.org/g/openembedded-core/message/174461 Mute This Topic: https://lists.openembedded.org/mt/94683148/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 01/13] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
Hi Alex, Am 07.12.2022 um 17:21 schrieb Alexander Kanavin: On Wed 7. Dec 2022 at 16.28, Stefan Herbrechtsmeier wrote: Hi Alex, Am 07.12.2022 um 09:53 schrieb Alexander Kanavin via lists.openembedded.org <http://lists.openembedded.org>: > On Wed, 7 Dec 2022 at 01:21, Sergey Bostandzhyan wrote: >> I think this should be documented more prominently, as missing information >> on how to get going was the biggest obstacle, at least for me. The benefit of >> this workflow is, that no additional layer is needed and that everything seems >> to more or less work out of the box within the usual checkout. > Thanks for the report, I totally agree that the rust recipe workflow > should be documented somewhere in the official manuals, I'm just not > sure where. Perhaps the 'common tasks' in > https://docs.yoctoproject.org/dev-manual/index.html ? > > Can you write and propose a patch to the documentation repository? Is the recipetool obsolete or why we have yust an other tool to create a recipe in oe-core? Which another tool are you referring to? Cargo bitbake is a 3rd party project, one that I am not recommending to anyone in any way. And having a class to update or create a list of crates does not preclude its use in recipetool. Patches welcome. A bit of politeness would be welcome too, Stefan, seriously. Watch your tone. Sorry for the harsh tone but you ignore existing tools and add just another tool to oe-core without mention any reason or document it. Do you really expect that somebody will patch existing tools if main developers ignore existing tools and provide new tool for their use case? Why should somebody improve an existing tool, extend the documentation, add tests or even upstream its work if these same requirements don’t exist for the main developers. It looks like the requirements for foreign and main contributors are different and this doesn't encourage people to participant. Maybe this is only my personal feeling and I apologize my harsh tone, but the acceptance of patches should be comprehensible, and expectations should be the same for everyone. Should others answer to your comments that their solution doesn’t preclude your suggestion and that they welcome patches? For sure it takes more time to add rust support to recipetool but I think a second tool without a clear reason in oe-core hurts more in long term because its now unclear if new features (like checksums or licenses support) should be added to this tool or if this tool is only a temporary solution and should be replaced by recipetool in long term. Furthermore, this class is marked as a class for a recipe but shouldn’t be inherit by a recipe and manipulates a file inside the meta data. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17): https://lists.openembedded.org/g/openembedded-core/message/17 Mute This Topic: https://lists.openembedded.org/mt/94683148/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 01/13] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
Hi Alex, Am 07.12.2022 um 09:53 schrieb Alexander Kanavin via lists.openembedded.org: On Wed, 7 Dec 2022 at 01:21, Sergey Bostandzhyan wrote: I think this should be documented more prominently, as missing information on how to get going was the biggest obstacle, at least for me. The benefit of this workflow is, that no additional layer is needed and that everything seems to more or less work out of the box within the usual checkout. Thanks for the report, I totally agree that the rust recipe workflow should be documented somewhere in the official manuals, I'm just not sure where. Perhaps the 'common tasks' in https://docs.yoctoproject.org/dev-manual/index.html ? Can you write and propose a patch to the documentation repository? Is the recipetool obsolete or why we have yust an other tool to create a recipe in oe-core? Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#174359): https://lists.openembedded.org/g/openembedded-core/message/174359 Mute This Topic: https://lists.openembedded.org/mt/94683148/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Easyish reproducibility issue for someone?
Hi Markus,
Am 22.06.2022 um 16:18 schrieb Markus Volk via lists.openembedded.org:
Hi,
couldn't we just do it like this?
PACKAGECONFIG ?= ""
PACKAGECONFIG[optviewer] =
"-DLLVM_HAVE_OPT_VIEWER_MODULES=ON,-DLLVM_HAVE_OPT_VIEWER_MODULES=OFF,,python3-pygments
python3-pyyaml"
Depending on wether the PACKAGECONFIG option is taken it looks like this
in CMakeCache.txt:
//No help, variable specified on the command line.
LLVM_HAVE_OPT_VIEWER_MODULES:UNINITIALIZED=OFF
//No help, variable specified on the command line.
LLVM_HAVE_OPT_VIEWER_MODULES:UNINITIALIZED=ON
Have you test it? The config-ix.cmake file sets the
LLVM_HAVE_OPT_VIEWER_MODULES variable [1].
Instead of LLVM_HAVE_OPT_VIEWER_MODULES I would set PY_PYGMENTS_FOUND,
PY_PYGMENTS_LEXERS_C_CPP_FOUND and PY_YAML_FOUND.
[1]
https://github.com/llvm/llvm-project/blob/main/llvm/cmake/config-ix.cmake#L693
Regards
Stefan
Am 22.06.22 um 15:54 schrieb Stefan Herbrechtsmeier:
Hi Richard,
Am 22.06.2022 um 15:10 schrieb Richard Purdie via lists.openembedded.org:
There is a reproducibility issue which shouldn't be too hard to fix if
someone had a few minutes to help out:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14845
It is in llvm and is a cmake configuration issue. Rather than setting
that option based on the presence of python modules (with python from
the host), we should add a configuration option to force it to be
disabled in our case. Should be upstreamable with the right patch.
I'm not able to test it at the moment but it should be possible to set
the variables (PY_PYGMENTS_FOUND, PY_PYGMENTS_LEXERS_C_CPP_FOUND and
PY_YAML_FOUND) to OFF via EXTRA_OECMAKE:
-DPY_PYGMENTS_FOUND=OFF
The find_python_module function in config-ix.cmake returns early if
the variable PY_${module_upper}_FOUND is defined.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#167241):
https://lists.openembedded.org/g/openembedded-core/message/167241
Mute This Topic: https://lists.openembedded.org/mt/91921251/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Easyish reproducibility issue for someone?
Hi Richard,
Am 22.06.2022 um 15:10 schrieb Richard Purdie via lists.openembedded.org:
There is a reproducibility issue which shouldn't be too hard to fix if
someone had a few minutes to help out:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14845
It is in llvm and is a cmake configuration issue. Rather than setting
that option based on the presence of python modules (with python from
the host), we should add a configuration option to force it to be
disabled in our case. Should be upstreamable with the right patch.
I'm not able to test it at the moment but it should be possible to set
the variables (PY_PYGMENTS_FOUND, PY_PYGMENTS_LEXERS_C_CPP_FOUND and
PY_YAML_FOUND) to OFF via EXTRA_OECMAKE:
-DPY_PYGMENTS_FOUND=OFF
The find_python_module function in config-ix.cmake returns early if the
variable PY_${module_upper}_FOUND is defined.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#167237):
https://lists.openembedded.org/g/openembedded-core/message/167237
Mute This Topic: https://lists.openembedded.org/mt/91921251/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/5] recipetool-create: add ensure_native_cmd function
Hi Luca, Lukas will check the logs and fix the problem. Regards Stefan Am 09.05.2022 um 23:46 schrieb Luca Ceresoli: Hello Stefan, Il giorno Fri, 6 May 2022 08:59:13 +0200 "Stefan Herbrechtsmeier" ha scritto: From: Lukas Funke Signed-off-by: Lukas Funke Signed-off-by: Stefan Herbrechtsmeier Testing builds with your series trigger many build failures related to recipetool. Can you check these logs? https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3557/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3506/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3527/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3566/steps/14/logs/stdio -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#165487): https://lists.openembedded.org/g/openembedded-core/message/165487 Mute This Topic: https://lists.openembedded.org/mt/90928682/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 5/5] oe-selftest: add go recipe create selftest
From: Lukas Funke
Signed-off-by: Lukas Funke
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/recipetool.py | 88 ++
1 file changed, 88 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 510dae6bad..bb36e7b6d5 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -530,6 +530,94 @@ class RecipetoolTests(RecipetoolBase):
libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib',
'recipetool')
sys.path.insert(0, libpath)
+def test_recipetool_create_go(self):
+# Basic test to check go recipe generation
+def urifiy(url, version, path = None, subdir = None):
+path = ",path='%s'" % path if path else ''
+subdir = ",subdir='%s'" % subdir if subdir else ''
+return "${@go_src_uri('%s','%s'%s%s)}" % (url, path, subdir)
+
+temprecipe = os.path.join(self.tempdir, 'recipe')
+os.makedirs(temprecipe)
+recipefile = os.path.join(temprecipe, 'edgex-go_git.bb')
+srcuri = 'https://github.com/edgexfoundry/edgex-go.git'
+srcrev = "v2.2.0-dev.54"
+result = runCmd('recipetool create -o %s %s -S %s' % (temprecipe,
srcuri, srcrev))
+self.assertTrue(os.path.isfile(recipefile))
+checkvars = {}
+src_uri = ['git://${GO_IMPORT};nobranch=1;name=${BPN}']
+checkvars['LIC_FILES_CHKSUM'] =
set(['file://src/${GO_IMPORT}/LICENSE;md5=71a6955f3cd81a809549da266346dc59'])
+checkvars['GO_IMPORT'] = "github.com/edgexfoundry/edgex-go"
+inherits = ['go-vendor']
+dependencies = \
+[
+('bitbucket.org/bertimus9/systemstat'),
+
('github.com/edgexfoundry/go-mod-bootstrap','github.com/edgexfoundry/go-mod-bootstrap/v2'),
+
('github.com/edgexfoundry/go-mod-core-contracts''github.com/edgexfoundry/go-mod-core-contracts/v2'),
+
('github.com/edgexfoundry/go-mod-messaging','github.com/edgexfoundry/go-mod-messaging/v2'),
+
('github.com/edgexfoundry/go-mod-registry','github.com/edgexfoundry/go-mod-registry/v2'),
+
('github.com/edgexfoundry/go-mod-secrets','github.com/edgexfoundry/go-mod-secrets/v2'),
+('github.com/fxamacker/cbor','github.com/fxamacker/cbor/v2'),
+('github.com/golang-jwt/jwt','github.com/golang-jwt/jwt/v4'),
+('github.com/gomodule/redigo'),
+('github.com/google/uuid'),
+('github.com/gorilla/mux'),
+('github.com/lib/pq'),
+('github.com/pelletier/go-toml'),
+
('github.com/spiffe/go-spiffe','github.com/spiffe/go-spiffe/v2'),
+('github.com/stretchr/testify'),
+('go.googlesource.com/crypto','golang.org/x/crypto'),
+('gopkg.in/eapache/queue.v1'),
+('gopkg.in/yaml.v3'),
+('github.com/armon/go-metrics'),
+('github.com/cenkalti/backoff'),
+('github.com/davecgh/go-spew'),
+('github.com/eclipse/paho.mqtt.golang'),
+
('github.com/edgexfoundry/go-mod-configuration','github.com/edgexfoundry/go-mod-configuration/v2'),
+('github.com/fatih/color'),
+('github.com/go-kit/log'),
+('github.com/go-logfmt/logfmt'),
+('github.com/go-playground/locales'),
+('github.com/go-playground/universal-translator'),
+
('github.com/go-playground/validator','github.com/go-playground/validator/v10'),
+('github.com/go-redis/redis','github.com/go-redis/redis/v7'),
+('github.com/golang/protobuf'),
+('github.com/gorilla/websocket'),
+('github.com/hashicorp/consul','api'),
+('github.com/hashicorp/errwrap'),
+('github.com/hashicorp/go-cleanhttp'),
+('github.com/hashicorp/go-hclog'),
+('github.com/hashicorp/go-immutable-radix'),
+('github.com/hashicorp/go-multierror'),
+('github.com/hashicorp/go-rootcerts'),
+('github.com/hashicorp/golang-lru'),
+('github.com/hashicorp/serf'),
+('github.com/leodido/go-urn'),
+('github.com/mattn/go-colorable'),
+('github.com/mattn/go-isatty'),
+('github.com/mitchellh/consulstructure'),
+('github.com/mitchellh/copystructure'),
+('github.com/mitchellh/go-homedir'),
+('github.com/mitchellh/mapstructure'),
+('github.com/mitchellh/reflectwalk'),
+('github.com/pebbe/zmq4'),
+('github.com/pmezard/go-difflib'),
+('github.com/stretchr/
[OE-core] [PATCH 4/5] recipetool: add go recipe generator
From: Lukas Funke
Signed-off-by: Lukas Funke
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/recipetool/create_go.py | 394
1 file changed, 394 insertions(+)
create mode 100644 scripts/lib/recipetool/create_go.py
diff --git a/scripts/lib/recipetool/create_go.py
b/scripts/lib/recipetool/create_go.py
new file mode 100644
index 00..4552e9b470
--- /dev/null
+++ b/scripts/lib/recipetool/create_go.py
@@ -0,0 +1,394 @@
+# Recipe creation tool - go support plugin
+#
+# Copyright (C) 2022 Weidmueller GmbH & Co KG
+# Author: Lukas Funke
+#
+# Copyright (c) 2009 The Go Authors. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause AND GPL-2.0-only
+#
+import bb.utils
+from collections import namedtuple
+from enum import Enum
+from html.parser import HTMLParser
+import json
+import logging
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import shutil
+from urllib.error import URLError, HTTPError
+import urllib.parse
+import urllib.request
+
+from recipetool.create import RecipeHandler, handle_license_vars,
ensure_native_cmd
+
+GoImport = namedtuple('GoImport', 'reporoot vcs repourl suffix')
+logger = logging.getLogger('recipetool')
+
+tinfoil = None
+
+re_pseudo_semver =
re.compile(r"v([0-9]+)\.([0-9]+).([0-9]+|\([0-9]+\+1\))-(pre\.[0-9]+\.)?([0-9]+\.)?(?P[0-9]+)-(?P[0-9Aa-zA-Z]+)")
+re_semver =
re.compile(r"^v(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+def tinfoil_init(instance):
+global tinfoil
+tinfoil = instance
+
+class GoRecipeHandler(RecipeHandler):
+
+def _resolve_repository_static(self, modulepath):
+_rootpath = None
+_vcs = None
+_repourl = None
+_suffix = None
+
+host, _, path = modulepath.partition('/')
+
+class vcs(Enum):
+pathprefix = "pathprefix"
+regexp = "regexp"
+vcs = "vcs"
+repo = "repo"
+check = "check"
+schemelessRepo = "schemelessRepo"
+
+# GitHub
+vcsGitHub = {}
+vcsGitHub[vcs.pathprefix] = "github.com"
+vcsGitHub[vcs.regexp] =
re.compile(r'^(?Pgithub\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$')
+vcsGitHub[vcs.vcs] = "git"
+vcsGitHub[vcs.repo] = "https://\g"
+
+# Bitbucket
+vcsBitbucket = {}
+vcsBitbucket[vcs.pathprefix] = "bitbucket.org"
+vcsBitbucket[vcs.regexp] =
re.compile(r'^(?Pbitbucket\.org/(?P[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/(?P[A-Za-z0-9_.\-]+))*$')
+vcsBitbucket[vcs.vcs] = "git"
+vcsBitbucket[vcs.repo] = "https://\g"
+
+# IBM DevOps Services (JazzHub)
+vcsIBMDevOps = {}
+vcsIBMDevOps[vcs.pathprefix] = "hub.jazz.net/git"
+vcsIBMDevOps[vcs.regexp] =
re.compile(r'^(?Phub\.jazz\.net/git/[a-z0-9]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$')
+vcsIBMDevOps[vcs.vcs] = "git"
+vcsIBMDevOps[vcs.repo] = "https://\g"
+
+# Git at Apache
+vcsApacheGit = {}
+vcsApacheGit[vcs.pathprefix] = "git.apache.org"
+vcsApacheGit[vcs.regexp] =
re.compile(r'^(?Pgit\.apache\.org/[a-z0-9_.\-]+\.git)(/(?P[A-Za-z0-9_.\-]+))*$')
+vcsApacheGit[vcs.vcs] = "git"
+vcsApacheGit[vcs.repo] = "https://\g"
+
+# Git at OpenStack
+vcsOpenStackGit = {}
+vcsOpenStackGit[vcs.pathprefix] = "git.openstack.org"
+vcsOpenStackGit[vcs.regexp] =
re.compile(r'^(?Pgit\.openstack\.org/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(\.git)?(/(?P[A-Za-z0-9_.\-]+))*$')
+vcsOpenStackGit[vcs.vcs] = "git"
+vcsOpenStackGit[vcs.repo] = "https://\g"
+
+# chiselapp.com for fossil
+vcsChiselapp = {}
+vcsChiselapp[vcs.pathprefix] = "chiselapp.com"
+vcsChiselapp[vcs.regexp] =
re.compile(r'^(?Pchiselapp\.com/user/[A-Za-z0-9]+/repository/[A-Za-z0-9_.\-]+)$')
+vcsChiselapp[vcs.vcs] = "fossil"
+vcsChiselapp[vcs.repo] = "https://\g"
+
+# General syntax for any server.
+# Must be last.
+vcsGeneralServer = {}
+vcsGeneralServer[vcs.regexp] =
re.compile("(?P(?P([a-z0-9.\-]+\.)+[a-z0-9.\-]+(:[0-9]+)?(/~?[A-Za-z0-9_.\-]+)+?)\.(?Pbzr|fossil|git|hg|svn))(/~?(?P[A-Za-z0-9_.\-]+))*$")
+vcsGeneralServer[vcs.schemelessRepo] = True
+
+vcsPaths = [vcsGitHub, vcsBitbucket, vcsIBMDevOps, vcsApacheGit,
vcsOpenStackGit, vcsChiselapp, vcsGeneralServer]
+
+if modulepath.startswith("example.net") or modulepath == "rsc.io":
+logger.warning
[OE-core] [PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py
From: Lukas Funke
Signed-off-by: Lukas Funke
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/recipetool/create_npm.py | 28 ++--
1 file changed, 2 insertions(+), 26 deletions(-)
diff --git a/scripts/lib/recipetool/create_npm.py
b/scripts/lib/recipetool/create_npm.py
index 3394a89970..3aed59252c 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -18,6 +18,7 @@ from recipetool.create import RecipeHandler
from recipetool.create import get_license_md5sums
from recipetool.create import guess_license
from recipetool.create import split_pkg_licenses
+from recipetool.create import ensure_native_cmd
logger = logging.getLogger('recipetool')
TINFOIL = None
@@ -54,31 +55,6 @@ class NpmRecipeHandler(RecipeHandler):
return registry
-@staticmethod
-def _ensure_npm():
-"""Check if the 'npm' command is available in the recipes"""
-if not TINFOIL.recipes_parsed:
-TINFOIL.parse_recipes()
-
-try:
-d = TINFOIL.parse_recipe("nodejs-native")
-except bb.providers.NoProvider:
-bb.error("Nothing provides 'nodejs-native' which is required for
the build")
-bb.note("You will likely need to add a layer that provides nodejs")
-sys.exit(14)
-
-bindir = d.getVar("STAGING_BINDIR_NATIVE")
-npmpath = os.path.join(bindir, "npm")
-
-if not os.path.exists(npmpath):
-TINFOIL.build_targets("nodejs-native", "addto_recipe_sysroot")
-
-if not os.path.exists(npmpath):
-bb.error("Failed to add 'npm' to sysroot")
-sys.exit(14)
-
-return bindir
-
@staticmethod
def _npm_global_configs(dev):
"""Get the npm global configuration"""
@@ -190,7 +166,7 @@ class NpmRecipeHandler(RecipeHandler):
# npm version is high enough to ensure an efficient dependency tree
# resolution and avoid issue with the shrinkwrap file format.
# Moreover the native npm is mandatory for the build.
-bindir = self._ensure_npm()
+bindir = ensure_native_cmd(TINFOIL, "npm")
d = bb.data.createCopy(TINFOIL.config_data)
d.prependVar("PATH", bindir + ":")
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#165325):
https://lists.openembedded.org/g/openembedded-core/message/165325
Mute This Topic: https://lists.openembedded.org/mt/90928684/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/5] poky-meta: add go vendor class for offline builds
From: Lukas Funke
Signed-off-by: Lukas Funke
Signed-off-by: Stefan Herbrechtsmeier
---
meta/classes/go-vendor.bbclass | 68 ++
1 file changed, 68 insertions(+)
create mode 100644 meta/classes/go-vendor.bbclass
diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass
new file mode 100644
index 00..a1a740a1fc
--- /dev/null
+++ b/meta/classes/go-vendor.bbclass
@@ -0,0 +1,68 @@
+# Copyright 2022 (C) Weidmueller GmbH & Co KG
+# Author: Lukas Funke
+#
+# Handle Go vendor support for offline builds
+#
+# When importing Go modules, Go downloads the imported module using
+# a network (proxy) connection ahead of the compile stage. This contradicts
+# the yocto build concept of fetching every source ahead of build-time
+# and supporting offline builds.
+#
+# To support offline builds, we use Go 'vendoring': module dependencies are
+# downloaded during the fetch-phase and unpacked into the modules 'vendor'
+# folder. Additinally a manifest file is generated for the 'vendor' folder
+#
+
+inherit go-mod
+
+def go_src_uri(repo, path=None, subdir=None, vcs='git', destsuffix_prefix =
'git/src/import/vendor.fetch'):
+module_path = repo if not path else path
+src_uri = "{}://{};name={};destsuffix={}/{}".format(vcs, repo, \
+module_path.replace('/', '.'), \
+destsuffix_prefix, module_path)
+
+src_uri += ";subdir={}".format(subdir) if subdir else ""
+src_uri += ";nobranch=1;protocol=https" if vcs == "git" else ""
+
+return src_uri
+
+def go_generate_vendor_manifest(d):
+
+vendor_dir = os.path.join(os.path.basename(d.getVar('S')),
+'src', d.getVar('GO_IMPORT'), "vendor")
+dst = os.path.join(vendor_dir, "modules.txt")
+
+go_modules = d.getVarFlags("GO_MODULE_PATH")
+with open(dst, "w") as manifest:
+for go_module in go_modules:
+module_path = d.getVarFlag("GO_MODULE_PATH", go_module)
+module_version = d.getVarFlag("GO_MODULE_VERSION", go_module)
+if module_path and module_version:
+manifest.write("# %s %s\n" % (module_path, module_version))
+manifest.write("## explicit\n")
+exclude = set(['vendor'])
+for subdir, dirs, files in os.walk(os.path.join(vendor_dir,
module_path), topdown=True):
+dirs[:] = [d for d in dirs if d not in exclude]
+for file in files:
+if file.endswith(".go"):
+manifest.write(subdir[len(vendor_dir)+1:] + "\n")
+break
+
+python go_do_unpack:append() {
+src_uri = (d.getVar('SRC_URI') or "").split()
+if len(src_uri) == 0:
+return
+
+try:
+fetcher = bb.fetch2.Fetch(src_uri, d)
+src_folder = os.path.join(os.path.basename(d.getVar('S')),
+'src', d.getVar('GO_IMPORT'))
+vendor_src = os.path.join(src_folder, "vendor")
+vendor_dst = os.path.join(d.getVar('S'), "src", "import",
"vendor.fetch")
+
+os.symlink(os.path.relpath(vendor_dst, src_folder), vendor_src)
+go_generate_vendor_manifest(d)
+
+except bb.fetch2.BBFetchException as e:
+raise bb.build.FuncFailed(e)
+}
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#165324):
https://lists.openembedded.org/g/openembedded-core/message/165324
Mute This Topic: https://lists.openembedded.org/mt/90928683/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/5] recipetool-create: add ensure_native_cmd function
From: Lukas Funke
Signed-off-by: Lukas Funke
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/recipetool/create.py | 23 +++
1 file changed, 23 insertions(+)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 824ac6350d..efcb82173e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1375,6 +1375,29 @@ def convert_rpm_xml(xmlfile):
values[varname] = child[0].text
return values
+def ensure_native_cmd(tinfoil, cmd):
+"""Check if the command is available in the recipes"""
+if not tinfoil.recipes_parsed:
+tinfoil.parse_recipes()
+
+try:
+d = tinfoil.parse_recipe("%s-native" % cmd)
+except bb.providers.NoProvider:
+bb.error("Nothing provides '%s-native' which is required for the
build" % cmd)
+bb.note("You will likely need to add a layer that provides %s" % cmd)
+sys.exit(14)
+
+bindir = d.getVar("STAGING_BINDIR_NATIVE")
+cmdpath = os.path.join(bindir, cmd)
+
+if not os.path.exists(cmdpath):
+tinfoil.build_targets("%s-native" % cmd, "addto_recipe_sysroot")
+
+if not os.path.exists(cmdpath):
+bb.error("Failed to add '%s' to sysroot" % cmd)
+sys.exit(14)
+
+return bindir
def register_commands(subparsers):
parser_create = subparsers.add_parser('create',
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#165323):
https://lists.openembedded.org/g/openembedded-core/message/165323
Mute This Topic: https://lists.openembedded.org/mt/90928682/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2] bluez5: Add fix for startup issues under systemd
Hi Richard,
should I update your patch or only the Upstream-Status or will you
prefer to do it yourself?
Regards
Stefan
Am 14.04.2022 um 22:52 schrieb Richard Purdie via lists.openembedded.org:
The systemd bluetooth service failed to start. Add a workaround for this whilst
the
final fix is discussed upstream, https://github.com/bluez/bluez/issues/329.
Signed-off-by: Richard Purdie
---
v2: Add ProtectSystem=strict
meta/recipes-connectivity/bluez5/bluez5.inc | 1 +
.../bluez5/bluez5/fix_service.patch | 30 +++
2 files changed, 31 insertions(+)
create mode 100644 meta/recipes-connectivity/bluez5/bluez5/fix_service.patch
diff --git a/meta/recipes-connectivity/bluez5/bluez5.inc
b/meta/recipes-connectivity/bluez5/bluez5.inc
index 79d4645ca89..22dd07b3480 100644
--- a/meta/recipes-connectivity/bluez5/bluez5.inc
+++ b/meta/recipes-connectivity/bluez5/bluez5.inc
@@ -53,6 +53,7 @@ SRC_URI =
"${KERNELORG_MIRROR}/linux/bluetooth/bluez-${PV}.tar.xz \
${@bb.utils.contains('DISTRO_FEATURES', 'systemd', '',
'file://0001-Allow-using-obexd-without-systemd-in-the-user-sessio.patch', d)} \
file://0001-tests-add-a-target-for-building-tests-without-runnin.patch \
file://0001-test-gatt-Fix-hung-issue.patch \
+ file://fix_service.patch \
"
S = "${WORKDIR}/bluez-${PV}"
diff --git a/meta/recipes-connectivity/bluez5/bluez5/fix_service.patch b/meta/recipes-connectivity/bluez5/bluez5/fix_service.patch
new file mode 100644
index 000..96fdf6b299c
--- /dev/null
+++ b/meta/recipes-connectivity/bluez5/bluez5/fix_service.patch
@@ -0,0 +1,30 @@
+The systemd bluetooth service failed to start because the /var/lib/bluetooth
+path of ReadWritePaths= is created by the bluetooth daemon itself.
+
+The commit systemd: Add more filesystem lockdown (442d211) add
ReadWritePaths=/etc/bluetooth
+and ReadOnlyPaths=/var/lib/bluetooth options to the bluetooth systemd service.
+The existing ProtectSystem=full option mounts the /usr, the boot loader
+directories and /etc read-only. This means the two option are useless and
could be removed.
+
+Upstream-Status: Submitted [https://github.com/bluez/bluez/issues/329]
+
+Index: bluez-5.64/src/bluetooth.service.in
+===
+--- bluez-5.64.orig/src/bluetooth.service.in
bluez-5.64/src/bluetooth.service.in
+@@ -15,12 +15,12 @@ LimitNPROC=1
+
+ # Filesystem lockdown
+ ProtectHome=true
+-ProtectSystem=full
++ProtectSystem=strict
+ PrivateTmp=true
+ ProtectKernelTunables=true
+ ProtectControlGroups=true
+-ReadWritePaths=@statedir@
+-ReadOnlyPaths=@confdir@
++ConfigurationDirectory=bluetooth
++StateDirectory=bluetooth
+
+ # Execute Mappings
+ MemoryDenyWriteExecute=true
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164813):
https://lists.openembedded.org/g/openembedded-core/message/164813
Mute This Topic: https://lists.openembedded.org/mt/90474022/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] recipetool: Do not use mutable default arguments in Python
From: Stefan Herbrechtsmeier
Remove mutable default arguments in Python because they can lead to all
sorts of nasty and horrible bugs.
https://florimond.dev/en/posts/2018/08/python-mutable-defaults-are-the-source-of-all-evil/
Revert `recipetool: Change default paramter fallback_licenses of
function split_pkg_licenses from None to []` and instead check
fallback_licenses before use.
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/recipetool/create.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 220465ed2f..824ac6350d 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1235,7 +1235,7 @@ def guess_license(srctree, d):
return licenses
-def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=[],
pn='${PN}'):
+def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None,
pn='${PN}'):
"""
Given a list of (license, path, md5sum) as returned by guess_license(),
a dict of package name to path mappings, write out a set of
@@ -1258,7 +1258,7 @@ def split_pkg_licenses(licvalues, packages, outlines,
fallback_licenses=[], pn='
for pkgname in packages:
# Assume AND operator between license files
license = ' & '.join(list(set(pkglicenses.get(pkgname, ['Unknown']
or 'Unknown'
-if license == 'Unknown' and pkgname in fallback_licenses:
+if license == 'Unknown' and fallback_licenses and pkgname in
fallback_licenses:
license = fallback_licenses[pkgname]
licenses = tidy_licenses(license)
license = ' & '.join(licenses)
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164407):
https://lists.openembedded.org/g/openembedded-core/message/164407
Mute This Topic: https://lists.openembedded.org/mt/90461493/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bluez5: install /var/lib/bluetooth directory
Hi, Am 11.04.2022 um 09:17 schrieb Stefan Herbrechtsmeier via lists.openembedded.org: Am 08.04.2022 um 22:03 schrieb Markus Volk via lists.openembedded.org: also had this issue and found out, that the bluetooth service did come up, once bluetooth was enabled My fix was to edit like this ReadWritePaths=:/var/lib/bluetooth The colon ensures that the service won't fail if the directory doesn't exist Regarding the documentation the ReadOnlyPaths and ReadWritePaths makes not sense because ProtectSystem=full mounts /usr, boot loader and /etc directories read-only. Have somebody report the problem to the bluez project? I have open an issue: https://github.com/bluez/bluez/issues/329 I will post a patch when we come to a solution. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#164308): https://lists.openembedded.org/g/openembedded-core/message/164308 Mute This Topic: https://lists.openembedded.org/mt/90331367/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bluez5: install /var/lib/bluetooth directory
Am 08.04.2022 um 22:03 schrieb Markus Volk via lists.openembedded.org:
also had this issue and found out, that the bluetooth service did come
up, once bluetooth was enabled
My fix was to edit like this
ReadWritePaths=:/var/lib/bluetooth
The colon ensures that the service won't fail if the directory doesn't
exist
Regarding the documentation the ReadOnlyPaths and ReadWritePaths makes
not sense because ProtectSystem=full mounts /usr, boot loader and /etc
directories read-only.
Have somebody report the problem to the bluez project?
Regards
Stefan
Am 08.04.22 um 13:01 schrieb Yi Zhao:
On 4/8/22 18:51, Stefan Herbrechtsmeier wrote:
Am 08.04.2022 um 11:23 schrieb Yi Zhao:
On 4/8/22 16:34, Stefan Herbrechtsmeier wrote:
Am 08.04.2022 um 10:15 schrieb Yi Zhao via lists.openembedded.org:
There is a bluetooth service startup failure:
bluetooth.service: Failed at step NAMESPACE spawning
/usr/libexec/bluetooth/bluetoothd: No such file or directory
bluetooth.service: Failed to set up mount namespacing:
/run/systemd/unit-root/var/lib/bluetooth: No such file or directory
This is because the directory /var/lib/bluetooth has been listed in
ReadWritePaths= in the service unit file but does not exist. We
need to
create it before service startup.
Signed-off-by: Yi Zhao
---
meta/recipes-connectivity/bluez5/bluez5.inc | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-connectivity/bluez5/bluez5.inc
b/meta/recipes-connectivity/bluez5/bluez5.inc
index 79d4645ca8..ee2cdbcc59 100644
--- a/meta/recipes-connectivity/bluez5/bluez5.inc
+++ b/meta/recipes-connectivity/bluez5/bluez5.inc
@@ -102,6 +102,8 @@ do_install:append() {
# Patch python tools to use Python 3; they should be source
compatible, but
# still refer to Python 2 in the shebang
sed -i -e '1s,#!.*python.*,#!${bindir}/python3,'
${D}${libdir}/bluez/test/*
+
+ install -d ${D}${localstatedir}/lib/bluetooth
Please use systemd StateDirectory= [1]. Maybe it is reasonable to
replace the ReadWritePaths= with StateDirectory=.
[1]
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
Thanks Stefank, but the ReadWritePaths= is set by bluez:
https://github.com/bluez/bluez/commit/442d211b5f30f00d5ddd69b43385a03c1428ac45
And who creates the directory before the patch?
There is no need this directory before the patch. The service can
startup without this directory.
The systemd solution to create a state directory is StateDirectory=
(or tmpfiles.d for older versions of systemd). Systemd supports
stateless system [1] and therefore every service must work with an
empty /var/lib.
I'll try adding StateDirecotry to service unit file to see if this
can solve the problem. Thanks.
//Yi
This patch hide a bug in bluez and the problem should be fixed in bluez.
[1] https://0pointer.net/blog/projects/stateless.html
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164217):
https://lists.openembedded.org/g/openembedded-core/message/164217
Mute This Topic: https://lists.openembedded.org/mt/90331367/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bluez5: install /var/lib/bluetooth directory
Am 08.04.2022 um 11:23 schrieb Yi Zhao:
On 4/8/22 16:34, Stefan Herbrechtsmeier wrote:
Am 08.04.2022 um 10:15 schrieb Yi Zhao via lists.openembedded.org:
There is a bluetooth service startup failure:
bluetooth.service: Failed at step NAMESPACE spawning
/usr/libexec/bluetooth/bluetoothd: No such file or directory
bluetooth.service: Failed to set up mount namespacing:
/run/systemd/unit-root/var/lib/bluetooth: No such file or directory
This is because the directory /var/lib/bluetooth has been listed in
ReadWritePaths= in the service unit file but does not exist. We need to
create it before service startup.
Signed-off-by: Yi Zhao
---
meta/recipes-connectivity/bluez5/bluez5.inc | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-connectivity/bluez5/bluez5.inc
b/meta/recipes-connectivity/bluez5/bluez5.inc
index 79d4645ca8..ee2cdbcc59 100644
--- a/meta/recipes-connectivity/bluez5/bluez5.inc
+++ b/meta/recipes-connectivity/bluez5/bluez5.inc
@@ -102,6 +102,8 @@ do_install:append() {
# Patch python tools to use Python 3; they should be source
compatible, but
# still refer to Python 2 in the shebang
sed -i -e '1s,#!.*python.*,#!${bindir}/python3,'
${D}${libdir}/bluez/test/*
+
+ install -d ${D}${localstatedir}/lib/bluetooth
Please use systemd StateDirectory= [1]. Maybe it is reasonable to
replace the ReadWritePaths= with StateDirectory=.
[1]
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
Thanks Stefank, but the ReadWritePaths= is set by bluez:
https://github.com/bluez/bluez/commit/442d211b5f30f00d5ddd69b43385a03c1428ac45
And who creates the directory before the patch?
The systemd solution to create a state directory is StateDirectory= (or
tmpfiles.d for older versions of systemd). Systemd supports stateless
system [1] and therefore every service must work with an empty /var/lib.
This patch hide a bug in bluez and the problem should be fixed in bluez.
[1] https://0pointer.net/blog/projects/stateless.html
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164158):
https://lists.openembedded.org/g/openembedded-core/message/164158
Mute This Topic: https://lists.openembedded.org/mt/90331367/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] bluez5: install /var/lib/bluetooth directory
Am 08.04.2022 um 10:15 schrieb Yi Zhao via lists.openembedded.org:
There is a bluetooth service startup failure:
bluetooth.service: Failed at step NAMESPACE spawning
/usr/libexec/bluetooth/bluetoothd: No such file or directory
bluetooth.service: Failed to set up mount namespacing:
/run/systemd/unit-root/var/lib/bluetooth: No such file or directory
This is because the directory /var/lib/bluetooth has been listed in
ReadWritePaths= in the service unit file but does not exist. We need to
create it before service startup.
Signed-off-by: Yi Zhao
---
meta/recipes-connectivity/bluez5/bluez5.inc | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-connectivity/bluez5/bluez5.inc
b/meta/recipes-connectivity/bluez5/bluez5.inc
index 79d4645ca8..ee2cdbcc59 100644
--- a/meta/recipes-connectivity/bluez5/bluez5.inc
+++ b/meta/recipes-connectivity/bluez5/bluez5.inc
@@ -102,6 +102,8 @@ do_install:append() {
# Patch python tools to use Python 3; they should be source compatible,
but
# still refer to Python 2 in the shebang
sed -i -e '1s,#!.*python.*,#!${bindir}/python3,'
${D}${libdir}/bluez/test/*
+
+ install -d ${D}${localstatedir}/lib/bluetooth
Please use systemd StateDirectory= [1]. Maybe it is reasonable to
replace the ReadWritePaths= with StateDirectory=.
[1]
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164156):
https://lists.openembedded.org/g/openembedded-core/message/164156
Mute This Topic: https://lists.openembedded.org/mt/90331367/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH] bitbake.conf: Add base package version (BPV) variable
Am 30.03.2022 um 19:32 schrieb Khem Raj:
On Wed, Mar 30, 2022 at 1:16 AM Stefan Herbrechtsmeier
wrote:
From: Stefan Herbrechtsmeier
Add a base package version (BPV) variable and use it as default for the
package version (PV) variable. The BPV variable contains the base
package version of the recipe read from the recipe filename.
The base package version variable supports an expansion of the version
from the recipe filename without the need of an immediate variable
expansion:
PV = "${BPV}+git${SRCPV}"
It allows the inclusion of include files inside a recipe file with the
same version in the recipe and include filename even if the package
version is different to the base package version in the filename:
require linux-yocto-${BPV}.inc
how many such usecase do we have ?
4x require binutils-${PV}.inc
9x require recipes-devtools/gcc/gcc-${PV}.inc
6x require go-${PV}.inc
4x require libtool-${PV}.inc
2x require m4-${PV}.inc
1x require boost-${PV}.inc
Thereby binutils and gcc set the real PV inside the .inc file.
seems to be quite errorprone too.
The current use cases in oe core is much more error-prone because they
use ${PV} and set the PV inside the .inc file. This means a second .inc
file must use the new PV. Furthermore, if somebody in oe-core add a `PV
.= "+git${SRCPV}"` it breaks includes in other layers.
A MAIN_VERSION (MAJOR.MINOR) and a BPV (MAJOR.MINOR.PATCH) would make
the use case explicit:
require binutils-${MAIN_VERSION}.inc
require recipes-devtools/gcc/gcc-${MAIN_VERSION}.inc
require go-${BPV}.inc
require libtool-${BPV}.inc
require m4-${BPV}.inc
require boost-${BPV}.inc
At the moment oe-core have recipe specific names and code for the same
value:
MAJOR.MINOR.PATCH:
MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:2])}"
CMAKE_MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:2])}"
MAJOR.MINOR:
WESTON_MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:1])}"
SHRT_VER =
"${@d.getVar('PV').split('.')[0]}.${@d.getVar('PV').split('.')[1]}"
VIMDIR =
"vim${@d.getVar('PV').split('.')[0]}${@d.getVar('PV').split('.')[1]}"
SHRT_VER = "${@oe.utils.trim_version("${PV}", 2)}"
NAMEVER = "${@oe.utils.trim_version("${PV}", 2)}"
SHRT_VER = "${@oe.utils.trim_version("${PV}", 2)}"
MAJ_VER = "${@oe.utils.trim_version("${PV}", 2)}"
MAJ_VER = "${@oe.utils.trim_version("${PV}", 2)}"
MAJ_VER = "${@oe.utils.trim_version("${PV}", 2)}"
MAJOR:
MAJ_VER = "${@oe.utils.trim_version("${PV}", 2).split('.')[0]}"
ICU_MAJOR_VER = "${@d.getVar('PV').split('.')[0]}"
MINOR:
MIN_VER = "${@oe.utils.trim_version("${PV}", 2).split('.')[1]}"
Sometimes the same name is used for different values:
MAJOR_VERSION = "${@oe.utils.trim_version("${PV}", 1)}"
WESTON_MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:1])}"
MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:2])}"
Maybe the bitbake.conf isn't the ride place but I think a common name
for the base / short version (MAJOR.MINOR), base package version
(MAJOR.MINOR.PATCH) and major, minor and patch would be helpful.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163804):
https://lists.openembedded.org/g/openembedded-core/message/163804
Mute This Topic: https://lists.openembedded.org/mt/90127138/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH] bitbake.conf: Add base package version (BPV) variable
Hi Richard,
Am 30.03.2022 um 13:51 schrieb Richard Purdie:
On Wed, 2022-03-30 at 10:16 +0200, Stefan Herbrechtsmeier wrote:
From: Stefan Herbrechtsmeier
Add a base package version (BPV) variable and use it as default for the
package version (PV) variable. The BPV variable contains the base
package version of the recipe read from the recipe filename.
The base package version variable supports an expansion of the version
from the recipe filename without the need of an immediate variable
expansion:
PV = "${BPV}+git${SRCPV}"
It allows the inclusion of include files inside a recipe file with the
same version in the recipe and include filename even if the package
version is different to the base package version in the filename:
require linux-yocto-${BPV}.inc
Signed-off-by: Stefan Herbrechtsmeier
---
meta/conf/bitbake.conf | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7705415a4f..d56cf811ba 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -232,7 +232,8 @@ ASSUME_PROVIDED = "\
##
PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
-PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+BPV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+PV = "${BPV}"
PR = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
PE = ""
PF = "${PN}-${EXTENDPE}${PV}-${PR}"
I'm not really seeing a compelling use case for what is a fairly core change,
you've not really said why we need changes and what the big win is for users.
binutils_2.32.bb:
require binutils-${PV}.inc
binutils-2.32.inc:
PV = "2.32.0"
CVE_VERSION = "2.32"
cdrtools-native_3.01.bb:
PV = "3.01a31+really3.01"
REALPV = "3.01"
S = "${WORKDIR}/${BPN}-${REALPV}"
gcc-cross_8.3.bb:
require recipes-devtools/gcc/gcc-${PV}.inc
gcc-8.3.inc:
PV = "8.3.0"
gdk-pixbuf_2.38.0.bb:
MAJ_VER = "${@oe.utils.trim_version("${PV}", 2)}"
util-linux.inc:
MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:2])}"
Does the binutils and gcc-cross trick works inside a bbappend? The other
examples could benefit from a variable.
I'm also quite worried that this behaves quite differently to BPN. "base" in the
case of PN means strip off prefixes and suffixes yet here, BPV doesn't actually
strip anything off PV, you're just expected to set it differently yourself as
the user. The different usage semantics will likely confuse people and that
confusion isn't worth it IMO.
BPV = "${@oe.utils.trim_version("${PV}", 3)}"
This would make it inline with the BPN but disallow the usage of the
variable inside the PV itself.
Until now I don't know the oe.utils.trim_version function.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163789):
https://lists.openembedded.org/g/openembedded-core/message/163789
Mute This Topic: https://lists.openembedded.org/mt/90127138/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH] bitbake.conf: Add base package version (BPV) variable
Hi Alex,
Am 30.03.2022 um 10:41 schrieb Alexander Kanavin:
Mangling PV is more flexibly done with a lambda function in the
recipe, where you can trim and tweak PV to your heart's desire, e.g.
gnomebase.bbclass:
def gnome_verdir(v):
return ".".join(v.split(".")[:-1])
SRC_URI =
"${GNOME_MIRROR}/${GNOMEBN}/${@gnome_verdir("${PV}")}/${GNOMEBN}-${PV}.tar.${GNOME_COMPRESS_TYPE};name=archive"
What is the intention of your comment? Do you recommend a function per
recipe and thereby code duplication?
Or is your objection that the base package version isn't always the file
name version and the variable name is wrong?
The advantage of the BPV is that you can change the PV outside of the
recipe file:
PV:pn- = "${BPV}+git${SRCPV}"
The `PV .= "+git${SRCPV}"` assume that the PV is the base package
version but couldn't ensure it.
Regards
Stefan
On Wed, 30 Mar 2022 at 10:16, Stefan Herbrechtsmeier
wrote:
From: Stefan Herbrechtsmeier
Add a base package version (BPV) variable and use it as default for the
package version (PV) variable. The BPV variable contains the base
package version of the recipe read from the recipe filename.
The base package version variable supports an expansion of the version
from the recipe filename without the need of an immediate variable
expansion:
PV = "${BPV}+git${SRCPV}"
It allows the inclusion of include files inside a recipe file with the
same version in the recipe and include filename even if the package
version is different to the base package version in the filename:
require linux-yocto-${BPV}.inc
Signed-off-by: Stefan Herbrechtsmeier
---
meta/conf/bitbake.conf | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7705415a4f..d56cf811ba 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -232,7 +232,8 @@ ASSUME_PROVIDED = "\
##
PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or
'defaultpkgname'}"
-PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+BPV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+PV = "${BPV}"
PR = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
PE = ""
PF = "${PN}-${EXTENDPE}${PV}-${PR}"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163782):
https://lists.openembedded.org/g/openembedded-core/message/163782
Mute This Topic: https://lists.openembedded.org/mt/90127138/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH] bitbake.conf: Add base package version (BPV) variable
From: Stefan Herbrechtsmeier
Add a base package version (BPV) variable and use it as default for the
package version (PV) variable. The BPV variable contains the base
package version of the recipe read from the recipe filename.
The base package version variable supports an expansion of the version
from the recipe filename without the need of an immediate variable
expansion:
PV = "${BPV}+git${SRCPV}"
It allows the inclusion of include files inside a recipe file with the
same version in the recipe and include filename even if the package
version is different to the base package version in the filename:
require linux-yocto-${BPV}.inc
Signed-off-by: Stefan Herbrechtsmeier
---
meta/conf/bitbake.conf | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7705415a4f..d56cf811ba 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -232,7 +232,8 @@ ASSUME_PROVIDED = "\
##
PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or
'defaultpkgname'}"
-PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+BPV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+PV = "${BPV}"
PR = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
PE = ""
PF = "${PN}-${EXTENDPE}${PV}-${PR}"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163774):
https://lists.openembedded.org/g/openembedded-core/message/163774
Mute This Topic: https://lists.openembedded.org/mt/90127138/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3] zstd: add libzstd package
From: Stefan Herbrechtsmeier
Add a libzstd package to remove libstdc++ runtime dependency from
library users and reduce the dependencies and size of the library
package.
Add a lib package instead of a bin package to use a similar name as
other buildtools and be backward compatible.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v3:
- Add libzstd package instead of zstd-bin package
Changes in v2:
- Add zstd-bin package instead of libzstd package
meta/recipes-extended/zstd/zstd_1.5.2.bb | 4
1 file changed, 4 insertions(+)
diff --git a/meta/recipes-extended/zstd/zstd_1.5.2.bb
b/meta/recipes-extended/zstd/zstd_1.5.2.bb
index 0ddff469b8..3887f31ecf 100644
--- a/meta/recipes-extended/zstd/zstd_1.5.2.bb
+++ b/meta/recipes-extended/zstd/zstd_1.5.2.bb
@@ -36,4 +36,8 @@ do_install () {
oe_runmake install 'DESTDIR=${D}' PREFIX=${prefix} -C contrib/pzstd
}
+PACKAGE_BEFORE_PN = "libzstd"
+
+FILES:libzstd = "${libdir}/libzstd${SOLIBS}"
+
BBCLASSEXTEND = "native nativesdk"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163689):
https://lists.openembedded.org/g/openembedded-core/message/163689
Mute This Topic: https://lists.openembedded.org/mt/90080665/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2] zstd: add zstd-bin package
Hi, Am 25.03.2022 um 20:00 schrieb Alexandre Belloni: On 25/03/2022 13:29:50+0100, Stefan Herbrechtsmeier wrote: From: Stefan Herbrechtsmeier Add a zstd-bin package to remove libstdc++ runtime dependency from library users and reduce the dependencies and size of the library package. Signed-off-by: Stefan Herbrechtsmeier I believe this is the cause of: https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/4954/steps/11/logs/stdio Can you check? The debian.bbclass renames the packages to libzstd-*. What is the correct way to handle this? Should I rename nativesdk-zstd to nativesdk-libzstd-bin in buildtools-tarball.bb or should we take the first patch version which add a libzstd package only. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#163685): https://lists.openembedded.org/g/openembedded-core/message/163685 Mute This Topic: https://lists.openembedded.org/mt/90021107/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] zstd: add libzstd package to minimize runtime dependencies
Am 25.03.2022 um 14:54 schrieb Alexander Kanavin: On Fri, 25 Mar 2022 at 14:47, Khem Raj wrote: The standard way to do this is to inherit lib_package. that is appropriate when main package is a library but it also has some utilities accompanying it. This case seems reverse, where its tryin to create a fine grained packaging and separating the lib out instead. No, it's the same here: most (or all) of the consumers link to the library, and utilities are mostly for interactive development. We have examples for both solutions. xz has a liblzma package and brotli a brotli-bin package. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#163642): https://lists.openembedded.org/g/openembedded-core/message/163642 Mute This Topic: https://lists.openembedded.org/mt/90019254/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] zstd: add zstd-bin package
From: Stefan Herbrechtsmeier
Add a zstd-bin package to remove libstdc++ runtime dependency from
library users and reduce the dependencies and size of the library
package.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Add zstd-bin package instead of libzstd package
meta/recipes-extended/zstd/zstd_1.5.2.bb | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-extended/zstd/zstd_1.5.2.bb
b/meta/recipes-extended/zstd/zstd_1.5.2.bb
index 0ddff469b8..2fdb9b6950 100644
--- a/meta/recipes-extended/zstd/zstd_1.5.2.bb
+++ b/meta/recipes-extended/zstd/zstd_1.5.2.bb
@@ -18,6 +18,8 @@ CVE_PRODUCT = "zstandard"
S = "${WORKDIR}/git"
+inherit lib_package
+
PACKAGECONFIG ??= ""
PACKAGECONFIG[lz4] = "HAVE_LZ4=1,HAVE_LZ4=0,lz4"
PACKAGECONFIG[lzma] = "HAVE_LZMA=1,HAVE_LZMA=0,xz"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163639):
https://lists.openembedded.org/g/openembedded-core/message/163639
Mute This Topic: https://lists.openembedded.org/mt/90021107/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] zstd: add libzstd package to minimize runtime dependencies
Am 25.03.2022 um 11:48 schrieb Alexander Kanavin:
The standard way to do this is to inherit lib_package.
Do you propose to move the binaries to a new zstd-bin package and break
backward compatibility?
Regards
Stefan
On Fri, 25 Mar 2022 at 11:19, Stefan Herbrechtsmeier
wrote:
From: Stefan Herbrechtsmeier
Add a libzstd package to remove libstdc++ runtime dependency from
library users and reduce the size of the library package and its
dependencies.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-extended/zstd/zstd_1.5.2.bb | 4
1 file changed, 4 insertions(+)
diff --git a/meta/recipes-extended/zstd/zstd_1.5.2.bb
b/meta/recipes-extended/zstd/zstd_1.5.2.bb
index 0ddff469b8..3887f31ecf 100644
--- a/meta/recipes-extended/zstd/zstd_1.5.2.bb
+++ b/meta/recipes-extended/zstd/zstd_1.5.2.bb
@@ -36,4 +36,8 @@ do_install () {
oe_runmake install 'DESTDIR=${D}' PREFIX=${prefix} -C contrib/pzstd
}
+PACKAGE_BEFORE_PN = "libzstd"
+
+FILES:libzstd = "${libdir}/libzstd${SOLIBS}"
+
BBCLASSEXTEND = "native nativesdk"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163634):
https://lists.openembedded.org/g/openembedded-core/message/163634
Mute This Topic: https://lists.openembedded.org/mt/90019254/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] zstd: add libzstd package to minimize runtime dependencies
From: Stefan Herbrechtsmeier
Add a libzstd package to remove libstdc++ runtime dependency from
library users and reduce the size of the library package and its
dependencies.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-extended/zstd/zstd_1.5.2.bb | 4
1 file changed, 4 insertions(+)
diff --git a/meta/recipes-extended/zstd/zstd_1.5.2.bb
b/meta/recipes-extended/zstd/zstd_1.5.2.bb
index 0ddff469b8..3887f31ecf 100644
--- a/meta/recipes-extended/zstd/zstd_1.5.2.bb
+++ b/meta/recipes-extended/zstd/zstd_1.5.2.bb
@@ -36,4 +36,8 @@ do_install () {
oe_runmake install 'DESTDIR=${D}' PREFIX=${prefix} -C contrib/pzstd
}
+PACKAGE_BEFORE_PN = "libzstd"
+
+FILES:libzstd = "${libdir}/libzstd${SOLIBS}"
+
BBCLASSEXTEND = "native nativesdk"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#163632):
https://lists.openembedded.org/g/openembedded-core/message/163632
Mute This Topic: https://lists.openembedded.org/mt/90019254/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 1/2] classes: rootfs-postcommands: include /etc/fstab in overlayfs_qa_check
From: Stefan Herbrechtsmeier
The systemd init manager support mount point configuration via mount
units and /etc/fstab. 'Mounts listed in /etc/fstab will be converted
into native units dynamically at boot and when the configuration of
the system manager is reloaded. In general, configuring mount points
through /etc/fstab is the preferred approach.' [1]
Read mount points from /etc/fstab to determine dynamic mount units.
[1] https://www.freedesktop.org/software/systemd/man/systemd.mount.html
Signed-off-by: Stefan Herbrechtsmeier
---
meta/classes/rootfs-postcommands.bbclass | 38 +--
meta/lib/oeqa/selftest/cases/overlayfs.py | 4 +--
2 files changed, 31 insertions(+), 11 deletions(-)
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index cd8986d8a9..e1276a220a 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -376,25 +376,45 @@ rootfs_reproducible () {
fi
}
+# Perform a dumb check for unit existence, not its validity
python overlayfs_qa_check() {
from oe.overlayfs import mountUnitName
-# this is a dumb check for unit existence, not its validity
overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") or {}
imagepath = d.getVar("IMAGE_ROOTFS")
-searchpaths = [oe.path.join(imagepath, d.getVar("sysconfdir"), "systemd",
"system"),
+sysconfdir = d.getVar("sysconfdir")
+searchpaths = [oe.path.join(imagepath, sysconfdir, "systemd", "system"),
oe.path.join(imagepath, d.getVar("systemd_system_unitdir"))]
+fstabpath = oe.path.join(imagepath, sysconfdir, "fstab")
+
+if not any(os.path.exists(path) for path in [*searchpaths, fstabpath]):
+return
+
+fstabDevices = []
+if os.path.isfile(fstabpath):
+with open(fstabpath, 'r') as f:
+for line in f:
+if line[0] == '#':
+continue
+path = line.split(maxsplit=2)
+if len(path) > 2:
+fstabDevices.append(path[1])
allUnitExist = True;
for mountPoint in overlayMountPoints:
-path = d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint)
-unit = mountUnitName(path)
+mountPath = d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint)
+if mountPath in fstabDevices:
+continue
+
+mountUnit = mountUnitName(mountPath)
+if any(os.path.isfile(oe.path.join(dirpath, mountUnit))
+ for dirpath in searchpaths):
+continue
-if not any(os.path.isfile(oe.path.join(dirpath, unit))
- for dirpath in searchpaths):
-bb.warn('Unit name %s not found in systemd unit directories' %
unit)
-allUnitExist = False;
+bb.warn('Mount path %s not found in fstat and unit %s not found '
+'in systemd unit directories' % (mountPath, mountUnit))
+allUnitExist = False;
if not allUnitExist:
-bb.fatal('Not all mount units are installed by the BSP')
+bb.fatal('Not all mount paths and units are installed in the image')
}
diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py
b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 82007fade7..79d36fa93c 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -61,9 +61,9 @@ DISTRO_FEATURES += "systemd overlayfs"
self.add_overlay_conf_to_machine()
res = bitbake('core-image-minimal', ignore_status=True)
-line = getline(res, "Unit name mnt-overlay.mount not found in systemd
unit directories")
+line = getline(res, " Mount path /mnt/overlay not found in fstat and
unit mnt-overlay.mount not found in systemd unit directories")
self.assertTrue(line and line.startswith("WARNING:"), msg=res.output)
-line = getline(res, "Not all mount units are installed by the BSP")
+line = getline(res, "Not all mount paths and units are installed in
the image")
self.assertTrue(line and line.startswith("ERROR:"), msg=res.output)
def test_mount_unit_not_set(self):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162612):
https://lists.openembedded.org/g/openembedded-core/message/162612
Mute This Topic: https://lists.openembedded.org/mt/89502037/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 2/2] oeqa: selftest: overlayfs: add test for image with fstab entry
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
.../base-files/base-files_%.bbappend | 2 +
meta/lib/oeqa/selftest/cases/overlayfs.py | 82 +--
2 files changed, 57 insertions(+), 27 deletions(-)
create mode 100644 meta-selftest/recipes-test/base-files/base-files_%.bbappend
diff --git a/meta-selftest/recipes-test/base-files/base-files_%.bbappend
b/meta-selftest/recipes-test/base-files/base-files_%.bbappend
new file mode 100644
index 00..205720982c
--- /dev/null
+++ b/meta-selftest/recipes-test/base-files/base-files_%.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file
created by tests.
+include test_recipe.inc
diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py
b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 79d36fa93c..56ae48ce64 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -107,7 +107,7 @@ OVERLAYFS_MOUNT_POINT[usr-share-overlay] =
"/usr/share/overlay"
line = getline(res, "Missing required mount point for
OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration")
self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"),
msg=res.output)
-def test_correct_image(self):
+def _test_correct_image(self, recipe, data):
"""
Summary: Check that we can create an image when all parameters are
set correctly
@@ -124,31 +124,6 @@ VIRTUAL-RUNTIME_init_manager = "systemd"
# enable overlayfs in the kernel
KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
-"""
-
-systemd_machine_unit_append = """
-SYSTEMD_SERVICE:${PN} += " \
-mnt-overlay.mount \
-"
-
-do_install:append() {
-install -d ${D}${systemd_system_unitdir}
-cat < ${D}${systemd_system_unitdir}/mnt-overlay.mount
-[Unit]
-Description=Tmpfs directory
-DefaultDependencies=no
-
-[Mount]
-What=tmpfs
-Where=/mnt/overlay
-Type=tmpfs
-Options=mode=1777,strictatime,nosuid,nodev
-
-[Install]
-WantedBy=multi-user.target
-EOT
-}
-
"""
overlayfs_recipe_append = """
@@ -179,7 +154,7 @@ EOT
self.write_config(config)
self.add_overlay_conf_to_machine()
-self.write_recipeinc('systemd-machine-units',
systemd_machine_unit_append)
+self.write_recipeinc(recipe, data)
self.write_recipeinc('overlayfs-user', overlayfs_recipe_append)
bitbake('core-image-minimal')
@@ -210,6 +185,59 @@ EOT
line = getline_qemu(output,
"upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
self.assertTrue(line and line.startswith("overlay"), msg=output)
+def test_correct_image_fstab(self):
+"""
+Summary: Check that we can create an image when all parameters are
+ set correctly via fstab
+Expected: Image is created successfully
+Author:Stefan Herbrechtsmeier
+"""
+
+base_files_append = """
+do_install:append() {
+cat <> ${D}${sysconfdir}/fstab
+tmpfs/mnt/overlay tmpfs
mode=1777,strictatime,nosuid,nodev 0 0
+EOT
+}
+"""
+
+self._test_correct_image('base-files', base_files_append)
+
+def test_correct_image_unit(self):
+"""
+Summary: Check that we can create an image when all parameters are
+ set correctly via mount unit
+Expected: Image is created successfully
+Author:Vyacheslav Yurkov
+"""
+
+systemd_machine_unit_append = """
+SYSTEMD_SERVICE:${PN} += " \
+mnt-overlay.mount \
+"
+
+do_install:append() {
+install -d ${D}${systemd_system_unitdir}
+cat < ${D}${systemd_system_unitdir}/mnt-overlay.mount
+[Unit]
+Description=Tmpfs directory
+DefaultDependencies=no
+
+[Mount]
+What=tmpfs
+Where=/mnt/overlay
+Type=tmpfs
+Options=mode=1777,strictatime,nosuid,nodev
+
+[Install]
+WantedBy=multi-user.target
+EOT
+}
+
+"""
+
+self._test_correct_image('systemd-machine-units',
systemd_machine_unit_append)
+
class OverlayFSEtcRunTimeTests(OESelftestTestCase):
"""overlayfs-etc class tests"""
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162613):
https://lists.openembedded.org/g/openembedded-core/message/162613
Mute This Topic: https://lists.openembedded.org/mt/89502039/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3] systemd: move systemd shared library into its own package
From: Stefan Herbrechtsmeier
Move the systemd shared library (libsystemd-shared.so) into its own
package to prevent a runtime dependency from udev package to systemd
package and thereby to a second init manager.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v3:
- Add libdir to INSANE_SKIP for libsystemd-shared package
Changes in v2:
- Fix SUMMARY override
meta/recipes-core/systemd/systemd_250.3.bb | 6 ++
1 file changed, 6 insertions(+)
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 9121333f56..31c5c55c3f 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -371,6 +371,7 @@ PACKAGE_BEFORE_PN = "\
${PN}-journal-remote \
${PN}-extra-utils \
${PN}-udev-rules \
+libsystemd-shared \
udev \
udev-hwdb \
"
@@ -387,6 +388,8 @@ DESCRIPTION:${PN}-journal-upload = "systemd-journal-upload
uploads journal entri
SUMMARY:${PN}-journal-remote = "Receive journal messages over the network"
DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a command to
receive serialized journal events and store them to journal files."
+SUMMARY:libsystemd-shared = "Systemd shared library"
+
SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt',
'${PN}-binfmt', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-gatewayd', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-remote', '', d)} \
@@ -653,6 +656,9 @@ RRECOMMENDS:${PN} += "systemd-extra-utils \
INSANE_SKIP:${PN} += "dev-so libdir"
INSANE_SKIP:${PN}-dbg += "libdir"
INSANE_SKIP:${PN}-doc += " libdir"
+INSANE_SKIP:libsystemd-shared += "libdir"
+
+FILES:libsystemd-shared = "${rootlibexecdir}/systemd/libsystemd-shared*.so"
RPROVIDES:udev = "hotplug"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162406):
https://lists.openembedded.org/g/openembedded-core/message/162406
Mute This Topic: https://lists.openembedded.org/mt/89391009/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2] systemd: move systemd shared library into its own package
Am 24.02.2022 um 22:06 schrieb Richard Purdie: On Wed, 2022-02-23 at 17:17 +0100, Stefan Herbrechtsmeier wrote: From: Stefan Herbrechtsmeier Move the systemd shared library (libsystemd-shared.so) into its own package to prevent a runtime dependency from udev package to systemd package and thereby to a second init manager. Signed-off-by: Stefan Herbrechtsmeier --- Changes in v2: - Fix SUMMARY override This did trigger a warning on the autobuilder tests: https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/2657/steps/11/logs/warnings I will post a patch to add `INSANE_SKIP:libsystemd-shared += "libdir"` to the recipe. At the moment I have a problem to reproduce the warning. I don't see it on qemux86-64 and have to retest with qemux86. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162341): https://lists.openembedded.org/g/openembedded-core/message/162341 Mute This Topic: https://lists.openembedded.org/mt/89344344/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] files: overlayfs-create-dirs: split ExecStart into two commands
From: Stefan Herbrechtsmeier
Split ExecStart into two commands because systemd interpret an ExecStart
entry as a single executable with multiple parameters.
systemd[1]: Starting Overlayfs directories setup...
mkdir: cannot create directory '&&': Read-only file system
mkdir: cannot create directory 'mkdir': Read-only file system
Signed-off-by: Stefan Herbrechtsmeier
---
meta/files/overlayfs-create-dirs.service.in | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/files/overlayfs-create-dirs.service.in
b/meta/files/overlayfs-create-dirs.service.in
index 17204145f2..61b2b9321b 100644
--- a/meta/files/overlayfs-create-dirs.service.in
+++ b/meta/files/overlayfs-create-dirs.service.in
@@ -6,7 +6,8 @@ DefaultDependencies=no
[Service]
Type=oneshot
-ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p
{DATA_MOUNT_POINT}/upper{LOWERDIR}
+ExecStart=mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
+ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR}
RemainAfterExit=true
StandardOutput=journal
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162327):
https://lists.openembedded.org/g/openembedded-core/message/162327
Mute This Topic: https://lists.openembedded.org/mt/89369550/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] classes: rootfs-postcommands: avoid exception in overlayfs_qa_check
From: Stefan Herbrechtsmeier
The overlayfs_qa_check in the rootfs-postcommands class throws an
exception if the overlayfs distro feature is enabled but no global
OVERLAYFS_MOUNT_POINT is specifived.
File:
'/home/stefan/dalos-linutronix/poky/meta/classes/rootfs-postcommands.bbclass',
lineno: 389, function: overlayfs_qa_check
0385:searchpaths = [oe.path.join(imagepath, d.getVar("sysconfdir"),
"systemd", "system"),
0386: oe.path.join(imagepath,
d.getVar("systemd_system_unitdir"))]
0387:
0388:allUnitExist = True;
*** 0389:for mountPoint in overlayMountPoints:
0390:path = d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint)
0391:unit = mountUnitName(path)
0392:
0393:if not any(os.path.isfile(oe.path.join(dirpath, unit))
Exception: TypeError: 'NoneType' object is not iterable
Signed-off-by: Stefan Herbrechtsmeier
---
meta/classes/rootfs-postcommands.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index 74035c30b7..cd8986d8a9 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -380,7 +380,7 @@ python overlayfs_qa_check() {
from oe.overlayfs import mountUnitName
# this is a dumb check for unit existence, not its validity
-overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
+overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") or {}
imagepath = d.getVar("IMAGE_ROOTFS")
searchpaths = [oe.path.join(imagepath, d.getVar("sysconfdir"), "systemd",
"system"),
oe.path.join(imagepath, d.getVar("systemd_system_unitdir"))]
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162326):
https://lists.openembedded.org/g/openembedded-core/message/162326
Mute This Topic: https://lists.openembedded.org/mt/89369533/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] systemd: move systemd shared library into its own package
Am 23.02.2022 um 17:04 schrieb Konrad Weihmann:
On 23.02.22 16:59, Stefan Herbrechtsmeier wrote:
From: Stefan Herbrechtsmeier
Move the systemd shared library (libsystemd-shared.so) into its own
package to prevent a runtime dependency from udev package to systemd
package and thereby to a second init manager.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-core/systemd/systemd_250.3.bb | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 9121333f56..43ac66da62 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -371,6 +371,7 @@ PACKAGE_BEFORE_PN = "\
${PN}-journal-remote \
${PN}-extra-utils \
${PN}-udev-rules \
+ libsystemd-shared \
udev \
udev-hwdb \
"
@@ -387,6 +388,8 @@ DESCRIPTION:${PN}-journal-upload =
"systemd-journal-upload uploads journal entri
SUMMARY:${PN}-journal-remote = "Receive journal messages over the
network"
DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a
command to receive serialized journal events and store them to journal
files."
+SUMMARY:${PN}-container = "Systemd shared library"
That doesn't look right to me - shouldn't that be
SUMMARY:libsystemd-shared?
Thanks. I have fix the copy paste error and send a new version.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162271):
https://lists.openembedded.org/g/openembedded-core/message/162271
Mute This Topic: https://lists.openembedded.org/mt/89343939/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2] systemd: move systemd shared library into its own package
From: Stefan Herbrechtsmeier
Move the systemd shared library (libsystemd-shared.so) into its own
package to prevent a runtime dependency from udev package to systemd
package and thereby to a second init manager.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Fix SUMMARY override
meta/recipes-core/systemd/systemd_250.3.bb | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 9121333f56..04785827a2 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -371,6 +371,7 @@ PACKAGE_BEFORE_PN = "\
${PN}-journal-remote \
${PN}-extra-utils \
${PN}-udev-rules \
+libsystemd-shared \
udev \
udev-hwdb \
"
@@ -387,6 +388,8 @@ DESCRIPTION:${PN}-journal-upload = "systemd-journal-upload
uploads journal entri
SUMMARY:${PN}-journal-remote = "Receive journal messages over the network"
DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a command to
receive serialized journal events and store them to journal files."
+SUMMARY:libsystemd-shared = "Systemd shared library"
+
SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt',
'${PN}-binfmt', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-gatewayd', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-remote', '', d)} \
@@ -654,6 +657,8 @@ INSANE_SKIP:${PN} += "dev-so libdir"
INSANE_SKIP:${PN}-dbg += "libdir"
INSANE_SKIP:${PN}-doc += " libdir"
+FILES:libsystemd-shared = "${rootlibexecdir}/systemd/libsystemd-shared*.so"
+
RPROVIDES:udev = "hotplug"
RDEPENDS:udev-hwdb += "udev"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162270):
https://lists.openembedded.org/g/openembedded-core/message/162270
Mute This Topic: https://lists.openembedded.org/mt/89344344/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] systemd: move systemd shared library into its own package
From: Stefan Herbrechtsmeier
Move the systemd shared library (libsystemd-shared.so) into its own
package to prevent a runtime dependency from udev package to systemd
package and thereby to a second init manager.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-core/systemd/systemd_250.3.bb | 5 +
1 file changed, 5 insertions(+)
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 9121333f56..43ac66da62 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -371,6 +371,7 @@ PACKAGE_BEFORE_PN = "\
${PN}-journal-remote \
${PN}-extra-utils \
${PN}-udev-rules \
+libsystemd-shared \
udev \
udev-hwdb \
"
@@ -387,6 +388,8 @@ DESCRIPTION:${PN}-journal-upload = "systemd-journal-upload
uploads journal entri
SUMMARY:${PN}-journal-remote = "Receive journal messages over the network"
DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a command to
receive serialized journal events and store them to journal files."
+SUMMARY:${PN}-container = "Systemd shared library"
+
SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt',
'${PN}-binfmt', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-gatewayd', '', d)} \
${@bb.utils.contains('PACKAGECONFIG', 'microhttpd',
'${PN}-journal-remote', '', d)} \
@@ -654,6 +657,8 @@ INSANE_SKIP:${PN} += "dev-so libdir"
INSANE_SKIP:${PN}-dbg += "libdir"
INSANE_SKIP:${PN}-doc += " libdir"
+FILES:libsystemd-shared = "${rootlibexecdir}/systemd/libsystemd-shared*.so"
+
RPROVIDES:udev = "hotplug"
RDEPENDS:udev-hwdb += "udev"
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162267):
https://lists.openembedded.org/g/openembedded-core/message/162267
Mute This Topic: https://lists.openembedded.org/mt/89343939/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Am 23.02.2022 um 13:39 schrieb Richard Purdie:
On Wed, 2022-02-23 at 09:13 +0100, Stefan Herbrechtsmeier wrote:
Am 23.02.2022 um 02:49 schrieb Yu, Mingli:
On 2/22/22 9:26 PM, Stefan Herbrechtsmeier wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
Am 22.02.2022 um 10:20 schrieb Yu, Mingli:
On 2/2/22 3:35 PM, Stefan Herbrechtsmeier wrote:
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 739f3aae86..431781e6c7 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -66,6 +66,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig
pam selinux smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '',
d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '',
d)} \
+ ${@bb.utils.contains('DISTRO_FEATURES', 'systemd',
'link-udev-shared', '', d)} \
After this change, the iso image failed to boot with below error:
Cannot find rootfs.img file in /run/media/*, dropping to a shell
And more detailed about the error as the attached.
Steps to reproduce:
1, update conf/local.conf to use systemd as the default init manager
DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"
KERNEL_FEATURES:append = " cfg/systemd.scc"
You have to set the INIT_MANAGER variable instead or set the
VIRTUAL-RUNTIME_dev_manager variable to systemd.
2, bitbake core-image-minimal
3, runqemu qemux86-64 core-image-minimal iso
A revert of the commit solve your problem?
Yes, the issue is gone after revert the commit.
It looks like a loot of packages have a direct runtime dependency to
udev and after this patch udev have a runtime dependency to systemd. We
could revert this patch or move libsystemd-shared.so to its own package
to remove the dependency to systemd itself.
A separate package for it would seem like a good option and could smplify the
original issues we saw enabling this too?
I think so but it will increase the size of the udev package because of
the size and dependencies of libsystemd-shared if you use udev without
systemd.
If we create a extra package udev, systemctl, networkd, timesyncd and
boot could be used without systemd package.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162264):
https://lists.openembedded.org/g/openembedded-core/message/162264
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Hi,
Am 23.02.2022 um 02:49 schrieb Yu, Mingli:
On 2/22/22 9:26 PM, Stefan Herbrechtsmeier wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
Hi,
Am 22.02.2022 um 10:20 schrieb Yu, Mingli:
Hi,
On 2/2/22 3:35 PM, Stefan Herbrechtsmeier wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 739f3aae86..431781e6c7 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -66,6 +66,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig
pam selinux smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '',
d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '',
d)} \
+ ${@bb.utils.contains('DISTRO_FEATURES', 'systemd',
'link-udev-shared', '', d)} \
After this change, the iso image failed to boot with below error:
Cannot find rootfs.img file in /run/media/*, dropping to a shell
And more detailed about the error as the attached.
Steps to reproduce:
1, update conf/local.conf to use systemd as the default init manager
DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"
KERNEL_FEATURES:append = " cfg/systemd.scc"
You have to set the INIT_MANAGER variable instead or set the
VIRTUAL-RUNTIME_dev_manager variable to systemd.
2, bitbake core-image-minimal
3, runqemu qemux86-64 core-image-minimal iso
A revert of the commit solve your problem?
Yes, the issue is gone after revert the commit.
It looks like a loot of packages have a direct runtime dependency to
udev and after this patch udev have a runtime dependency to systemd. We
could revert this patch or move libsystemd-shared.so to its own package
to remove the dependency to systemd itself.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162246):
https://lists.openembedded.org/g/openembedded-core/message/162246
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Hi,
Am 22.02.2022 um 10:20 schrieb Yu, Mingli:
Hi,
On 2/2/22 3:35 PM, Stefan Herbrechtsmeier wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 739f3aae86..431781e6c7 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -66,6 +66,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig
pam selinux smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '',
d)} \
+ ${@bb.utils.contains('DISTRO_FEATURES', 'systemd',
'link-udev-shared', '', d)} \
After this change, the iso image failed to boot with below error:
Cannot find rootfs.img file in /run/media/*, dropping to a shell
And more detailed about the error as the attached.
Steps to reproduce:
1, update conf/local.conf to use systemd as the default init manager
DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"
KERNEL_FEATURES:append = " cfg/systemd.scc"
You have to set the INIT_MANAGER variable instead or set the
VIRTUAL-RUNTIME_dev_manager variable to systemd.
2, bitbake core-image-minimal
3, runqemu qemux86-64 core-image-minimal iso
A revert of the commit solve your problem?
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162132):
https://lists.openembedded.org/g/openembedded-core/message/162132
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 4/4] selftest: wic: Add rawcopy plugin unpack test
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/wic.py | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 96b3e1b6a5..a021f8d84b 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1065,13 +1065,14 @@ class Wic2(WicTestCase):
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
self.assertEqual(output, '2')
-def test_rawcopy_plugin(self):
+def _rawcopy_plugin(self, fstype):
"""Test rawcopy plugin"""
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
+params = ',unpack' if fstype.endswith('.gz') else ''
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
- % (img, machine))
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.%s%s"\n'\
+ % (img, machine, fstype, params))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
@@ -1079,6 +1080,17 @@ class Wic2(WicTestCase):
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+def test_rawcopy_plugin(self):
+self._rawcopy_plugin('ext4')
+
+def test_rawcopy_plugin_unpack(self):
+fstype = 'ext4.gz'
+config = 'IMAGE_FSTYPES = "%s"\n' % fstype
+self.append_config(config)
+self.assertEqual(0, bitbake('core-image-minimal').status)
+self.remove_config(config)
+self._rawcopy_plugin(fstype)
+
def test_empty_plugin(self):
"""Test empty plugin"""
config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "test_empty_plugin.wks"\n'
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161732):
https://lists.openembedded.org/g/openembedded-core/message/161732
Mute This Topic: https://lists.openembedded.org/mt/89142899/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 3/4] selftest: wic: Remove requirement of syslinux from test_rawcopy_plugin
From: Stefan Herbrechtsmeier
Remove bootimg-pcbios from wks to eliminate requirement of syslinux from
test_rawcopy_plugin to avoid the following error.
ERROR: Couldn't find correct bootimg_dir, exiting
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/wic.py | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 5fc8e65142..96b3e1b6a5 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1070,10 +1070,8 @@ class Wic2(WicTestCase):
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.writelines(['part /boot --active --source bootimg-pcbios\n',
-'part / --source rawcopy
--sourceparams="file=%s-%s.ext4" --use-uuid\n'\
- % (img, machine),
-'bootloader --timeout=0
--append="console=ttyS0,115200n8"\n'])
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
+ % (img, machine))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161731):
https://lists.openembedded.org/g/openembedded-core/message/161731
Mute This Topic: https://lists.openembedded.org/mt/89142897/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 2/4] wic: rawcopy: Add support for packed images
From: Stefan Herbrechtsmeier
Add support for packed images to wic rawcopy handler do minimize disk
usage in deploy directory and reuse of packed images between wic and
swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz
archives.
Example:
part / --source rawcopy
--sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack"
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v2)
Changes in v2:
- Reword WicError message and add compressor filename extension
scripts/lib/wic/plugins/source/rawcopy.py | 29 ++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py
b/scripts/lib/wic/plugins/source/rawcopy.py
index fa7b1eb8ac..7c90cd3cf8 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -4,6 +4,8 @@
import logging
import os
+import signal
+import subprocess
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -38,6 +40,25 @@ class RawCopyPlugin(SourcePlugin):
exec_cmd(cmd)
+@staticmethod
+def do_image_uncompression(src, dst, workdir):
+def subprocess_setup():
+# Python installs a SIGPIPE handler by default. This is usually
not what
+# non-Python subprocesses expect.
+# SIGPIPE errors are known issues with gzip/bash
+signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+extension = os.path.splitext(src)[1]
+decompressor = {
+".bz2": "bzip2",
+".gz": "gzip",
+".xz": "xz"
+}.get(extension)
+if not decompressor:
+raise WicError("Not supported compressor filename extension: %s" %
extension)
+cmd = "%s -dc %s > %s" % (decompressor, src, dst)
+subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True,
cwd=workdir)
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -56,7 +77,13 @@ class RawCopyPlugin(SourcePlugin):
if 'file' not in source_params:
raise WicError("No file specified")
-src = os.path.join(kernel_dir, source_params['file'])
+if 'unpack' in source_params:
+img = os.path.join(kernel_dir, source_params['file'])
+src = os.path.join(cr_workdir,
os.path.splitext(source_params['file'])[0])
+RawCopyPlugin.do_image_uncompression(img, src, cr_workdir)
+else:
+src = os.path.join(kernel_dir, source_params['file'])
+
dst = os.path.join(cr_workdir, "%s.%s" %
(os.path.basename(source_params['file']), part.lineno))
if not os.path.exists(os.path.dirname(dst)):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161730):
https://lists.openembedded.org/g/openembedded-core/message/161730
Mute This Topic: https://lists.openembedded.org/mt/89142895/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 1/4] wic: partition: Support valueless keys in sourceparams
From: Stefan Herbrechtsmeier
Accept valueless keys in sourceparams without equals sign (=) to match
the comment and support Boolean entries.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
scripts/lib/wic/partition.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index a25834048e..09e491dd49 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -171,7 +171,7 @@ class Partition():
# Split sourceparams string of the form key1=val1[,key2=val2,...]
# into a dict. Also accepts valueless keys i.e. without =
splitted = self.sourceparams.split(',')
-srcparams_dict = dict(par.split('=', 1) for par in splitted if par)
+srcparams_dict = dict((par.split('=', 1) + [None])[:2] for par in
splitted if par)
plugin = PluginMgr.get_plugins('source')[self.source]
plugin.do_configure_partition(self, srcparams_dict, creator,
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161729):
https://lists.openembedded.org/g/openembedded-core/message/161729
Mute This Topic: https://lists.openembedded.org/mt/89142894/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL
Hi Alex, Am 14.02.2022 um 17:28 schrieb Alexander Kanavin: This fixes one location where the problem can occur in selftests, but what about all the others? Other tests like efibootpartition, gcc, glibc or runqemu already set nographic. Generally, it is not selftest's job to ensure qemu can be started: Why selftest requires features which it doesn't need? you need to either tweak runqemu to detect WSL, configure qemu-system-native from your local.conf so that it doesn't enable sdl or gtk, or, better yet, fix the problem at the source - find out why graphical qemu doesn't work in WSL and fix that. The graphic support depends on the WSL version. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#161727): https://lists.openembedded.org/g/openembedded-core/message/161727 Mute This Topic: https://lists.openembedded.org/mt/89138943/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 2/5] wic: rawcopy: Add support for packed images
From: Stefan Herbrechtsmeier
Add support for packed images to wic rawcopy handler do minimize disk
usage in deploy directory and reuse of packed images between wic and
swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz
archives.
Example:
part / --source rawcopy
--sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack"
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Reword WicError message and add compressor filename extension
scripts/lib/wic/plugins/source/rawcopy.py | 29 ++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py
b/scripts/lib/wic/plugins/source/rawcopy.py
index fa7b1eb8ac..7c90cd3cf8 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -4,6 +4,8 @@
import logging
import os
+import signal
+import subprocess
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -38,6 +40,25 @@ class RawCopyPlugin(SourcePlugin):
exec_cmd(cmd)
+@staticmethod
+def do_image_uncompression(src, dst, workdir):
+def subprocess_setup():
+# Python installs a SIGPIPE handler by default. This is usually
not what
+# non-Python subprocesses expect.
+# SIGPIPE errors are known issues with gzip/bash
+signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+extension = os.path.splitext(src)[1]
+decompressor = {
+".bz2": "bzip2",
+".gz": "gzip",
+".xz": "xz"
+}.get(extension)
+if not decompressor:
+raise WicError("Not supported compressor filename extension: %s" %
extension)
+cmd = "%s -dc %s > %s" % (decompressor, src, dst)
+subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True,
cwd=workdir)
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -56,7 +77,13 @@ class RawCopyPlugin(SourcePlugin):
if 'file' not in source_params:
raise WicError("No file specified")
-src = os.path.join(kernel_dir, source_params['file'])
+if 'unpack' in source_params:
+img = os.path.join(kernel_dir, source_params['file'])
+src = os.path.join(cr_workdir,
os.path.splitext(source_params['file'])[0])
+RawCopyPlugin.do_image_uncompression(img, src, cr_workdir)
+else:
+src = os.path.join(kernel_dir, source_params['file'])
+
dst = os.path.join(cr_workdir, "%s.%s" %
(os.path.basename(source_params['file']), part.lineno))
if not os.path.exists(os.path.dirname(dst)):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161719):
https://lists.openembedded.org/g/openembedded-core/message/161719
Mute This Topic: https://lists.openembedded.org/mt/89138944/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL
From: Stefan Herbrechtsmeier
Disable graphic support of qemu to support qemu tests in WSL.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Add patch
meta/lib/oeqa/selftest/cases/wic.py | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index a021f8d84b..6f3dc27743 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -851,7 +851,7 @@ class Wic2(WicTestCase):
self.assertEqual(0, bitbake('wic-image-minimal').status)
self.remove_config(config)
-with runqemu('wic-image-minimal', ssh=False) as qemu:
+with runqemu('wic-image-minimal', ssh=False,
runqemuparams='nographic') as qemu:
cmd = "mount | grep '^/dev/' | cut -f1,3 -d ' ' | egrep -c -e
'/dev/sda1 /boot' " \
"-e '/dev/root /|/dev/sda2 /' -e '/dev/sda3 /media' -e
'/dev/sda4 /mnt'"
status, output = qemu.run_serial(cmd)
@@ -871,7 +871,7 @@ class Wic2(WicTestCase):
self.remove_config(config)
with runqemu('core-image-minimal', ssh=False,
- runqemuparams='ovmf', image_fstype='wic') as qemu:
+ runqemuparams='nographic ovmf', image_fstype='wic') as
qemu:
cmd = "grep sda. /proc/partitions |wc -l"
status, output = qemu.run_serial(cmd)
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
@@ -1059,7 +1059,8 @@ class Wic2(WicTestCase):
self.assertEqual(0, bitbake('core-image-minimal-mtdutils').status)
self.remove_config(config)
-with runqemu('core-image-minimal-mtdutils', ssh=False,
image_fstype='wic') as qemu:
+with runqemu('core-image-minimal-mtdutils', ssh=False,
+ runqemuparams='nographic', image_fstype='wic') as qemu:
cmd = "grep sda. /proc/partitions |wc -l"
status, output = qemu.run_serial(cmd)
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
@@ -1119,7 +1120,8 @@ class Wic2(WicTestCase):
self.assertEqual(0, bitbake('core-image-minimal').status)
self.remove_config(config)
-with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as
qemu:
+with runqemu('core-image-minimal', ssh=False,
+ runqemuparams='nographic', image_fstype='wic') as qemu:
# Check that we have ONLY two /dev/sda* partitions (/boot and /)
cmd = "grep sda. /proc/partitions | wc -l"
status, output = qemu.run_serial(cmd)
@@ -1180,7 +1182,7 @@ class Wic2(WicTestCase):
self.remove_config(config)
with runqemu('core-image-minimal', ssh=False,
- runqemuparams='ovmf', image_fstype='wic') as qemu:
+ runqemuparams='nographic ovmf', image_fstype='wic') as
qemu:
# Check that /boot has EFI bootx64.efi (required for EFI)
cmd = "ls /boot/EFI/BOOT/bootx64.efi | wc -l"
status, output = qemu.run_serial(cmd)
@@ -1418,7 +1420,7 @@ class Wic2(WicTestCase):
bb.utils.rename(new_image_path, image_path)
# Check if it boots in qemu
-with runqemu('core-image-minimal', ssh=False) as qemu:
+with runqemu('core-image-minimal', ssh=False,
runqemuparams='nographic') as qemu:
cmd = "ls /etc/"
status, output = qemu.run_serial('true')
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161718):
https://lists.openembedded.org/g/openembedded-core/message/161718
Mute This Topic: https://lists.openembedded.org/mt/89138943/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 4/5] selftest: wic: Add rawcopy plugin unpack test
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/wic.py | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 96b3e1b6a5..a021f8d84b 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1065,13 +1065,14 @@ class Wic2(WicTestCase):
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
self.assertEqual(output, '2')
-def test_rawcopy_plugin(self):
+def _rawcopy_plugin(self, fstype):
"""Test rawcopy plugin"""
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
+params = ',unpack' if fstype.endswith('.gz') else ''
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
- % (img, machine))
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.%s%s"\n'\
+ % (img, machine, fstype, params))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
@@ -1079,6 +1080,17 @@ class Wic2(WicTestCase):
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+def test_rawcopy_plugin(self):
+self._rawcopy_plugin('ext4')
+
+def test_rawcopy_plugin_unpack(self):
+fstype = 'ext4.gz'
+config = 'IMAGE_FSTYPES = "%s"\n' % fstype
+self.append_config(config)
+self.assertEqual(0, bitbake('core-image-minimal').status)
+self.remove_config(config)
+self._rawcopy_plugin(fstype)
+
def test_empty_plugin(self):
"""Test empty plugin"""
config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "test_empty_plugin.wks"\n'
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161717):
https://lists.openembedded.org/g/openembedded-core/message/161717
Mute This Topic: https://lists.openembedded.org/mt/89138942/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 3/5] selftest: wic: Remove requirement of syslinux from test_rawcopy_plugin
From: Stefan Herbrechtsmeier
Remove bootimg-pcbios from wks to eliminate requirement of syslinux from
test_rawcopy_plugin to avoid the following error.
ERROR: Couldn't find correct bootimg_dir, exiting
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/wic.py | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 5fc8e65142..96b3e1b6a5 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1070,10 +1070,8 @@ class Wic2(WicTestCase):
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.writelines(['part /boot --active --source bootimg-pcbios\n',
-'part / --source rawcopy
--sourceparams="file=%s-%s.ext4" --use-uuid\n'\
- % (img, machine),
-'bootloader --timeout=0
--append="console=ttyS0,115200n8"\n'])
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
+ % (img, machine))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161716):
https://lists.openembedded.org/g/openembedded-core/message/161716
Mute This Topic: https://lists.openembedded.org/mt/89138940/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 1/5] wic: partition: Support valueless keys in sourceparams
From: Stefan Herbrechtsmeier
Accept valueless keys in sourceparams without equals sign (=) to match
the comment and support Boolean entries.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
scripts/lib/wic/partition.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index a25834048e..09e491dd49 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -171,7 +171,7 @@ class Partition():
# Split sourceparams string of the form key1=val1[,key2=val2,...]
# into a dict. Also accepts valueless keys i.e. without =
splitted = self.sourceparams.split(',')
-srcparams_dict = dict(par.split('=', 1) for par in splitted if par)
+srcparams_dict = dict((par.split('=', 1) + [None])[:2] for par in
splitted if par)
plugin = PluginMgr.get_plugins('source')[self.source]
plugin.do_configure_partition(self, srcparams_dict, creator,
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161715):
https://lists.openembedded.org/g/openembedded-core/message/161715
Mute This Topic: https://lists.openembedded.org/mt/89138936/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/4] wic: rawcopy: Add support for packed images
Hi Ulrich,
Am 11.02.2022 um 17:17 schrieb Ulrich Ölmann:
On Thu, Feb 10 2022 at 17:17 +0100, "Stefan Herbrechtsmeier"
wrote:
From: Stefan Herbrechtsmeier
Add support for packed images to wic rawcopy handler do minimize disk
usage in deploy directory and reuse of packed images between wic and
swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz
archives.
Example:
part / --source rawcopy
--sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack"
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/wic/plugins/source/rawcopy.py | 28 ++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py
b/scripts/lib/wic/plugins/source/rawcopy.py
index fa7b1eb8ac..7816e00e49 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -4,6 +4,8 @@
import logging
import os
+import signal
+import subprocess
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -38,6 +40,24 @@ class RawCopyPlugin(SourcePlugin):
exec_cmd(cmd)
+@staticmethod
+def do_image_uncompression(src, dst, workdir):
+def subprocess_setup():
+# Python installs a SIGPIPE handler by default. This is usually
not what
+# non-Python subprocesses expect.
+# SIGPIPE errors are known issues with gzip/bash
+signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+decompressor = {
+".bz2": "bzip2",
+".gz": "gzip",
+".xz": "xz"
+}.get(os.path.splitext(src)[1])
+if not decompressor:
+raise WicError("Compression not support")
Here is a small typo: s/support/supported/ .
And what about substituting "Compression" by "Compressor" to express
that not compression in general but only this individual compressor is
currently unsupported? (Or even better: in addition explicitely name the
currently unsupported compressor in the error message?)
What about the following:
Not supported compressor extension: %s
Best regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161658):
https://lists.openembedded.org/g/openembedded-core/message/161658
Mute This Topic: https://lists.openembedded.org/mt/89049086/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/4] selftest: wic: Remove requirement of syslinux from test_rawcopy_plugin
From: Stefan Herbrechtsmeier
Remove bootimg-pcbios from wks to eliminate requirement of syslinux from
test_rawcopy_plugin to avoid the following error.
ERROR: Couldn't find correct bootimg_dir, exiting
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/wic.py | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 5fc8e65142..96b3e1b6a5 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1070,10 +1070,8 @@ class Wic2(WicTestCase):
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.writelines(['part /boot --active --source bootimg-pcbios\n',
-'part / --source rawcopy
--sourceparams="file=%s-%s.ext4" --use-uuid\n'\
- % (img, machine),
-'bootloader --timeout=0
--append="console=ttyS0,115200n8"\n'])
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
+ % (img, machine))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161619):
https://lists.openembedded.org/g/openembedded-core/message/161619
Mute This Topic: https://lists.openembedded.org/mt/89049088/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 4/4] selftest: wic: Add rawcopy plugin unpack test
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/wic.py | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 96b3e1b6a5..a021f8d84b 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1065,13 +1065,14 @@ class Wic2(WicTestCase):
self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
self.assertEqual(output, '2')
-def test_rawcopy_plugin(self):
+def _rawcopy_plugin(self, fstype):
"""Test rawcopy plugin"""
img = 'core-image-minimal'
machine = get_bb_var('MACHINE', img)
+params = ',unpack' if fstype.endswith('.gz') else ''
with NamedTemporaryFile("w", suffix=".wks") as wks:
-wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.ext4"\n'\
- % (img, machine))
+wks.write('part / --source rawcopy
--sourceparams="file=%s-%s.%s%s"\n'\
+ % (img, machine, fstype, params))
wks.flush()
cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
runCmd(cmd)
@@ -1079,6 +1080,17 @@ class Wic2(WicTestCase):
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+def test_rawcopy_plugin(self):
+self._rawcopy_plugin('ext4')
+
+def test_rawcopy_plugin_unpack(self):
+fstype = 'ext4.gz'
+config = 'IMAGE_FSTYPES = "%s"\n' % fstype
+self.append_config(config)
+self.assertEqual(0, bitbake('core-image-minimal').status)
+self.remove_config(config)
+self._rawcopy_plugin(fstype)
+
def test_empty_plugin(self):
"""Test empty plugin"""
config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "test_empty_plugin.wks"\n'
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161620):
https://lists.openembedded.org/g/openembedded-core/message/161620
Mute This Topic: https://lists.openembedded.org/mt/89049089/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/4] wic: rawcopy: Add support for packed images
From: Stefan Herbrechtsmeier
Add support for packed images to wic rawcopy handler do minimize disk
usage in deploy directory and reuse of packed images between wic and
swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz
archives.
Example:
part / --source rawcopy
--sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack"
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/wic/plugins/source/rawcopy.py | 28 ++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py
b/scripts/lib/wic/plugins/source/rawcopy.py
index fa7b1eb8ac..7816e00e49 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -4,6 +4,8 @@
import logging
import os
+import signal
+import subprocess
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -38,6 +40,24 @@ class RawCopyPlugin(SourcePlugin):
exec_cmd(cmd)
+@staticmethod
+def do_image_uncompression(src, dst, workdir):
+def subprocess_setup():
+# Python installs a SIGPIPE handler by default. This is usually
not what
+# non-Python subprocesses expect.
+# SIGPIPE errors are known issues with gzip/bash
+signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+decompressor = {
+".bz2": "bzip2",
+".gz": "gzip",
+".xz": "xz"
+}.get(os.path.splitext(src)[1])
+if not decompressor:
+raise WicError("Compression not support")
+cmd = "%s -dc %s > %s" % (decompressor, src, dst)
+subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True,
cwd=workdir)
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -56,7 +76,13 @@ class RawCopyPlugin(SourcePlugin):
if 'file' not in source_params:
raise WicError("No file specified")
-src = os.path.join(kernel_dir, source_params['file'])
+if 'unpack' in source_params:
+img = os.path.join(kernel_dir, source_params['file'])
+src = os.path.join(cr_workdir,
os.path.splitext(source_params['file'])[0])
+RawCopyPlugin.do_image_uncompression(img, src, cr_workdir)
+else:
+src = os.path.join(kernel_dir, source_params['file'])
+
dst = os.path.join(cr_workdir, "%s.%s" %
(os.path.basename(source_params['file']), part.lineno))
if not os.path.exists(os.path.dirname(dst)):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161618):
https://lists.openembedded.org/g/openembedded-core/message/161618
Mute This Topic: https://lists.openembedded.org/mt/89049086/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/4] wic: partition: Support valueless keys in sourceparams
From: Stefan Herbrechtsmeier
Accept valueless keys in sourceparams without equals sign (=) to match
the comment and support Boolean entries.
Signed-off-by: Stefan Herbrechtsmeier
---
scripts/lib/wic/partition.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index a25834048e..09e491dd49 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -171,7 +171,7 @@ class Partition():
# Split sourceparams string of the form key1=val1[,key2=val2,...]
# into a dict. Also accepts valueless keys i.e. without =
splitted = self.sourceparams.split(',')
-srcparams_dict = dict(par.split('=', 1) for par in splitted if par)
+srcparams_dict = dict((par.split('=', 1) + [None])[:2] for par in
splitted if par)
plugin = PluginMgr.get_plugins('source')[self.source]
plugin.do_configure_partition(self, srcparams_dict, creator,
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161617):
https://lists.openembedded.org/g/openembedded-core/message/161617
Mute This Topic: https://lists.openembedded.org/mt/89049084/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/2] gcc-target: fix glob to remove gcc- binary
From: Stefan Herbrechtsmeier
The glob to remove the gcc- binary expects a single-digit major
version which is no longer true.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-devtools/gcc/gcc-target.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/gcc/gcc-target.inc
b/meta/recipes-devtools/gcc/gcc-target.inc
index bf55e692e6..bcea75b2fa 100644
--- a/meta/recipes-devtools/gcc/gcc-target.inc
+++ b/meta/recipes-devtools/gcc/gcc-target.inc
@@ -193,7 +193,7 @@ do_install () {
rm -f *c++*
# We don't care about the gcc- ones for this
- rm -f *gcc-?.?*
+ rm -f *gcc-?*.?*
# Not sure why we end up with these but we don't want them...
rm -f ${TARGET_PREFIX}${TARGET_PREFIX}*
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161615):
https://lists.openembedded.org/g/openembedded-core/message/161615
Mute This Topic: https://lists.openembedded.org/mt/89048989/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/2] gcc-target: move cc1plus to g++ package
From: Stefan Herbrechtsmeier
Move cc1plus from gcc to g++ package. Therefor, remove the duplicate
FILES entry from gcc package and keep the entry in g++ package.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-devtools/gcc/gcc-target.inc | 1 -
1 file changed, 1 deletion(-)
diff --git a/meta/recipes-devtools/gcc/gcc-target.inc
b/meta/recipes-devtools/gcc/gcc-target.inc
index bcea75b2fa..66f737c9dc 100644
--- a/meta/recipes-devtools/gcc/gcc-target.inc
+++ b/meta/recipes-devtools/gcc/gcc-target.inc
@@ -44,7 +44,6 @@ FILES:${PN} = "\
${bindir}/${TARGET_PREFIX}gcc* \
${bindir}/${TARGET_PREFIX}lto* \
${libexecdir}/gcc/${TARGET_SYS}/${BINV}/collect2* \
-${libexecdir}/gcc/${TARGET_SYS}/${BINV}/cc1plus \
${libexecdir}/gcc/${TARGET_SYS}/${BINV}/g++-mapper-server \
${libexecdir}/gcc/${TARGET_SYS}/${BINV}/lto* \
${libexecdir}/gcc/${TARGET_SYS}/${BINV}/lib*${SOLIBS} \
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161616):
https://lists.openembedded.org/g/openembedded-core/message/161616
Mute This Topic: https://lists.openembedded.org/mt/89048993/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Am 06.02.2022 um 20:05 schrieb Mark Hatle:
On 2/6/22 11:35 AM, Richard Purdie wrote:
On Sun, 2022-02-06 at 18:31 +0100, Alexander Kanavin wrote:
On Sun, 6 Feb 2022 at 18:27, Mark Hatle
wrote:
It definitely works in Honister (I'm actively using it.)
We produce a single package set, then allow for multiple image
recipes to
select
which init manager to use. This simplifies the eSDK, sstate-cache and
binary
package feed -- while still giving the end user the ability to switch.
Selection of init manager is a distro setting, it can be used by
component
recipes to make decisions, and overriding it from images is neither
tested nor
supported.
There is a hybrid where you can enable sysvinit and systemd at the
same time and
use them in different situations. We do test that and it is that test
which
showed issues on the autobuilder with this patch.
It isn't as easy as pick and chose either in a given image but handy for
initramfs and some other uses.
I've got the following in my distro:
# Create packages that support both systemd and sysvinit
# but only on arm/arm64 systems, microblaze is sysvinit only
DISTRO_FEATURES_BACKFILL_CONSIDERED:remove:arm = "sysvinit systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED:remove:aarch64 = "sysvinit systemd"
DISTRO_FEATURES_BACKFILL:append:arm = " systemd"
DISTRO_FEATURES_BACKFILL:append:aarch64 = " systemd”
INIT_MANAGER_DEFAULT = "systemd"
INIT_MANAGER_DEFAULT:microblaze = "sysvinit"
INIT_MANAGER_DEFAULT:zynq = "sysvinit"
INIT_MANAGER ?= "${INIT_MANAGER_DEFAULT}"
Then a few configurations change the INIT_MANAGER, and (almost) all of
the sstate is re-usable and things don't need to be re-built.
The issue I have, microblaze if the filesystem is over about 80mb it
doesn't fit into memory. On Arm, some devices are memory limited the
same way microblaze is, while some have full access to flash storage and
thus can use systemd. And on aarch64 - most defined (but not all) have
flash storage.
This is all about sstate-cache re-use (time to build the image) and
ability to have a SINGLE binary distribution that supports the image
size requirements of the user.
After the change from Richard the optimization will only take place if
DISTRO_FEATURES doesn't contain sysvinit and you will get the
independent systemd and udev packages for all your platforms.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161459):
https://lists.openembedded.org/g/openembedded-core/message/161459
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Am 06.02.22 um 17:43 schrieb Stefan Herbrechtsmeier:
Am 06.02.22 um 10:53 schrieb Richard Purdie:
On Sun, 2022-02-06 at 08:10 +, Richard Purdie via
lists.openembedded.org
wrote:
On Wed, 2022-02-02 at 08:35 +0100, Stefan Herbrechtsmeier wrote:
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
This seems to break some of our tests:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/4708
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3121
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3085
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3089
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3127
Presumably it causes extra dependencies to be pulled in causing the
file overlap
where there was previously none.
Thinking a bit more, I think this needs to be:
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 2b6bfcc2c25..7851c4c16a7 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -69,7 +69,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig
pam selinux smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '',
d)} \
- ${@bb.utils.contains('DISTRO_FEATURES', 'systemd',
'link-udev-shared', '', d)} \
+ ${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', '',
'link-udev-shared', d)} \
backlight \
binfmt \
gshadow \
which I'll test...
Maybe the systemd naming for the option is sub optimal. The
link-udev-shared option create a dependency between udev and systemd
package. It links udev against the libsystemd-shared library. Without it
udev is linked statically against this library. The option should only
be enabled if udev is always used together with systemd.
Sorry, I should read the patch careful.
I we can assume that udev from systemd is only used with sysvinit or
systemd, this is okay.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161404):
https://lists.openembedded.org/g/openembedded-core/message/161404
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Am 06.02.22 um 10:53 schrieb Richard Purdie:
On Sun, 2022-02-06 at 08:10 +, Richard Purdie via lists.openembedded.org
wrote:
On Wed, 2022-02-02 at 08:35 +0100, Stefan Herbrechtsmeier wrote:
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
This seems to break some of our tests:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/4708
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3121
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3085
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3089
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3127
Presumably it causes extra dependencies to be pulled in causing the file overlap
where there was previously none.
Thinking a bit more, I think this needs to be:
diff --git a/meta/recipes-core/systemd/systemd_250.3.bb
b/meta/recipes-core/systemd/systemd_250.3.bb
index 2b6bfcc2c25..7851c4c16a7 100644
--- a/meta/recipes-core/systemd/systemd_250.3.bb
+++ b/meta/recipes-core/systemd/systemd_250.3.bb
@@ -69,7 +69,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig pam selinux
smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '', d)} \
-${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'link-udev-shared', '',
d)} \
+${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', '',
'link-udev-shared', d)} \
backlight \
binfmt \
gshadow \
which I'll test...
Maybe the systemd naming for the option is sub optimal. The
link-udev-shared option create a dependency between udev and systemd
package. It links udev against the libsystemd-shared library. Without it
udev is linked statically against this library. The option should only
be enabled if udev is always used together with systemd.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161403):
https://lists.openembedded.org/g/openembedded-core/message/161403
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
Am 06.02.22 um 09:10 schrieb Richard Purdie: On Wed, 2022-02-02 at 08:35 +0100, Stefan Herbrechtsmeier wrote: From: Stefan Herbrechtsmeier Link udev shared with systemd helper to minimize the udev package size if DISTRO_FEATURES contains systemd. It is only usefull to link udev static with systemd helper if udev should be installed without systemd. Signed-off-by: Stefan Herbrechtsmeier --- (no changes since v1) meta/recipes-core/systemd/systemd_249.7.bb | 1 + 1 file changed, 1 insertion(+) This seems to break some of our tests: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/4708 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3121 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3085 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3089 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3127 Presumably it causes extra dependencies to be pulled in causing the file overlap where there was previously none. Sorry, I was told sysvinit with systemd isn't a valid use case anymore. Would it be okay to enable it only if sysvinit isn't in distro? Otherwise reject the change because it is impossible to know if a dependency between udev and systemd is okay or not. Regads Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#161402): https://lists.openembedded.org/g/openembedded-core/message/161402 Mute This Topic: https://lists.openembedded.org/mt/88853822/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] recipetool/create: Scan for SDPX-License-Identifier
Am 04.02.2022 um 14:41 schrieb Richard Purdie:
On Fri, 2022-02-04 at 10:05 +0100, Stefan Herbrechtsmeier wrote:
Am 03.02.2022 um 22:24 schrieb Richard Purdie via lists.openembedded.org:
On Thu, 2022-02-03 at 09:07 -0800, Saul Wold wrote:
When a file can not be identified by checksum and they contain an SPDX
License-Identifier tag, use it as the found license.
[YOCTO #14529]
Tested with LICENSE files that contain 1 or more SPDX-License-Identifier tags
Signed-off-by: Saul Wold
---
scripts/lib/recipetool/create.py | 16 +++-
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 507a230511..9149c2d94f 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1221,14 +1221,20 @@ def guess_license(srctree, d):
for licfile in sorted(licfiles):
md5value = bb.utils.md5_file(licfile)
license = md5sums.get(md5value, None)
+license_list = []
if not license:
license, crunched_md5, lictext = crunch_license(licfile)
if lictext and not license:
-license = 'Unknown'
-logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
-"and replace `Unknown` with the license:\n" \
-"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
-if license:
+spdx_re = re.compile('SPDX-License-Identifier:\s+([-A-Za-z\d.
]+)[ |\n|\r\n]*?')
+license_list = re.findall(spdx_re, "\n".join(lictext))
+if not license_list:
+license_list.append('Unknown')
+logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
+"and replace `Unknown` with the license:\n" \
+"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
+else:
+license_list.append(license)
+for license in license_list:
licenses.append((license, os.path.relpath(licfile, srctree),
md5value))
# FIXME should we grab at least one source file with a license header and add that too?
I think to close this bug the code may need to go one step further and
effectively grep over the source tree.
Please keep in mind that we need a full license text and not only the
license name for license compliance. The current function only search
for license files with license text.
We'd probably want to list the value of any SPDX-License-Identifier: header
found in any of the source files for the user to then decide upon?
I think this is an other feature like a license checker because if you
have a SPDX-License-Identifier without a license text you have a license
violation.
This brings us to the problem that this code will interpret a file with
only a SPDX-License-Identifier as a license file with license text.
As I understand it the tool is there to help write a recipe so filling out
LICENSE and highlighting a missing full license text would be a valid approach
for the tool and helpful to the user?
Yes, but we should distinguish between license files which are guess via
hash of the content and SPDX-License-Identifier which labels the source
code’s license. In this case the SPDX-License-Identifier is non-material
text from a license file and should be filtered out inside
crunch_license function.
The collection of all used licenses via SPDX-License-Identifier is an
additional feature and we need a warning if a SPDX-License-Identifier
exists without license file.
It certainly isn't intended as full validation, just intended to assist the
creation of a recipe.
But this patch is an regress because it doesn't distinguish between a
license file with a known hash and a mostly empty file with a
SPDX-License-Identifier.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161366):
https://lists.openembedded.org/g/openembedded-core/message/161366
Mute This Topic: https://lists.openembedded.org/mt/7504/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] recipetool/create: Scan for SDPX-License-Identifier
Hi Richard,
Am 03.02.2022 um 22:24 schrieb Richard Purdie via lists.openembedded.org:
On Thu, 2022-02-03 at 09:07 -0800, Saul Wold wrote:
When a file can not be identified by checksum and they contain an SPDX
License-Identifier tag, use it as the found license.
[YOCTO #14529]
Tested with LICENSE files that contain 1 or more SPDX-License-Identifier tags
Signed-off-by: Saul Wold
---
scripts/lib/recipetool/create.py | 16 +++-
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 507a230511..9149c2d94f 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1221,14 +1221,20 @@ def guess_license(srctree, d):
for licfile in sorted(licfiles):
md5value = bb.utils.md5_file(licfile)
license = md5sums.get(md5value, None)
+license_list = []
if not license:
license, crunched_md5, lictext = crunch_license(licfile)
if lictext and not license:
-license = 'Unknown'
-logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
-"and replace `Unknown` with the license:\n" \
-"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
-if license:
+spdx_re = re.compile('SPDX-License-Identifier:\s+([-A-Za-z\d.
]+)[ |\n|\r\n]*?')
+license_list = re.findall(spdx_re, "\n".join(lictext))
+if not license_list:
+license_list.append('Unknown')
+logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
+"and replace `Unknown` with the license:\n" \
+"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
+else:
+license_list.append(license)
+for license in license_list:
licenses.append((license, os.path.relpath(licfile, srctree),
md5value))
# FIXME should we grab at least one source file with a license header and add that too?
I think to close this bug the code may need to go one step further and
effectively grep over the source tree.
Please keep in mind that we need a full license text and not only the
license name for license compliance. The current function only search
for license files with license text.
We'd probably want to list the value of any SPDX-License-Identifier: header
found in any of the source files for the user to then decide upon?
I think this is an other feature like a license checker because if you
have a SPDX-License-Identifier without a license text you have a license
violation.
This brings us to the problem that this code will interpret a file with
only a SPDX-License-Identifier as a license file with license text.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161342):
https://lists.openembedded.org/g/openembedded-core/message/161342
Mute This Topic: https://lists.openembedded.org/mt/7504/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] recipetool/create: Scan for SDPX-License-Identifier
Hi Saul,
Am 03.02.2022 um 18:07 schrieb Saul Wold via lists.openembedded.org:
When a file can not be identified by checksum and they contain an SPDX
License-Identifier tag, use it as the found license.
[YOCTO #14529]
Tested with LICENSE files that contain 1 or more SPDX-License-Identifier tags
Can you please give an example for an project with use a
SPDX-License-Identifier inside a license file.
Signed-off-by: Saul Wold
---
scripts/lib/recipetool/create.py | 16 +++-
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 507a230511..9149c2d94f 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1221,14 +1221,20 @@ def guess_license(srctree, d):
for licfile in sorted(licfiles):
md5value = bb.utils.md5_file(licfile)
license = md5sums.get(md5value, None)
+license_list = []
Could you please use an other name. We already have licenses and it is
hard to distinguish the difference between licenses and license_list.
if not license:
license, crunched_md5, lictext = crunch_license(licfile)
if lictext and not license:
-license = 'Unknown'
-logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
-"and replace `Unknown` with the license:\n" \
-"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
-if license:
+spdx_re = re.compile('SPDX-License-Identifier:\s+([-A-Za-z\d.
]+)[ |\n|\r\n]*?')
+license_list = re.findall(spdx_re, "\n".join(lictext))
+if not license_list:
+license_list.append('Unknown')
+logger.info("Please add the following line for '%s' to a
'lib/recipetool/licenses.csv' " \
+"and replace `Unknown` with the license:\n" \
+"%s,Unknown" % (os.path.relpath(licfile, srctree),
md5value))
+else:
+license_list.append(license)
+for license in license_list:
licenses.append((license, os.path.relpath(licfile, srctree),
md5value))
# FIXME should we grab at least one source file with a license header and add that too?
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161340):
https://lists.openembedded.org/g/openembedded-core/message/161340
Mute This Topic: https://lists.openembedded.org/mt/7504/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 739f3aae86..431781e6c7 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -66,6 +66,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig pam selinux
smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '', d)} \
+${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'link-udev-shared', '',
d)} \
backlight \
binfmt \
gshadow \
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161182):
https://lists.openembedded.org/g/openembedded-core/message/161182
Mute This Topic: https://lists.openembedded.org/mt/88853822/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 2/3] systemd: Add link-udev-shared PACKAGECONFIG
From: Stefan Herbrechtsmeier
Add a PACKAGECONFIG to link systemd-udev and its helpers to
libsystemd-shared.so. If enabled the udev package depends on the systemd
package.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/recipes-core/systemd/systemd_249.7.bb | 8 +++-
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 53e7f6738d..739f3aae86 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -156,6 +156,9 @@ PACKAGECONFIG[kmod] = "-Dkmod=true,-Dkmod=false,kmod"
PACKAGECONFIG[ldconfig] = "-Dldconfig=true,-Dldconfig=false,,ldconfig"
PACKAGECONFIG[libidn] = "-Dlibidn=true,-Dlibidn=false,libidn,,libidn"
PACKAGECONFIG[libidn2] = "-Dlibidn2=true,-Dlibidn2=false,libidn2,,libidn2"
+# Link udev shared with systemd helper library.
+# If enabled the udev package depends on the systemd package (which has the
needed shared library).
+PACKAGECONFIG[link-udev-shared] =
"-Dlink-udev-shared=true,-Dlink-udev-shared=false"
PACKAGECONFIG[localed] = "-Dlocaled=true,-Dlocaled=false"
PACKAGECONFIG[logind] = "-Dlogind=true,-Dlogind=false"
PACKAGECONFIG[lz4] = "-Dlz4=true,-Dlz4=false,lz4"
@@ -215,11 +218,6 @@ rootprefix ?= "${root_prefix}"
rootlibdir ?= "${base_libdir}"
rootlibexecdir = "${rootprefix}/lib"
-# This links udev statically with systemd helper library.
-# Otherwise udev package would depend on systemd package (which has the needed
shared library),
-# and always pull it into images.
-EXTRA_OEMESON += "-Dlink-udev-shared=false"
-
EXTRA_OEMESON += "-Dnobody-user=nobody \
-Dnobody-group=nobody \
-Drootlibdir=${rootlibdir} \
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161181):
https://lists.openembedded.org/g/openembedded-core/message/161181
Mute This Topic: https://lists.openembedded.org/mt/88853821/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 1/3] cve-check: create directory of CVE_CHECK_MANIFEST before copy
From: Stefan Herbrechtsmeier
Create directory of the CVE_CHECK_MANIFEST variable before copy to it,
so that the variable can use an arbitrary directory name.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Reword commit
meta/classes/cve-check.bbclass | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 70d1988a70..a4d735c323 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -143,6 +143,7 @@ python cve_check_write_rootfs_manifest () {
manifest_name = d.getVar("CVE_CHECK_MANIFEST")
cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE")
+bb.utils.mkdirhier(os.path.dirname(manifest_name))
shutil.copyfile(cve_tmp_file, manifest_name)
if manifest_name and os.path.exists(manifest_name):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161180):
https://lists.openembedded.org/g/openembedded-core/message/161180
Mute This Topic: https://lists.openembedded.org/mt/88853820/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/3] cve-check: create directory before copy
Am 01.02.2022 um 18:00 schrieb Alexander Kanavin:
Can you please describe the issue this is fixing in the commit message?
Why is this not seen by others?
Isn't it best practice to create a directory before copy something into
the directory? The patch shows that the arbitrary directory of the
CVE_CHECK_MANIFEST variable isn't created.
On Tue, 1 Feb 2022 at 17:17, Stefan Herbrechtsmeier
<mailto:stefan.herbrechtsmeier-...@weidmueller.com>> wrote:
From: Stefan Herbrechtsmeier mailto:stefan.herbrechtsme...@weidmueller.com>>
Signed-off-by: Stefan Herbrechtsmeier
mailto:stefan.herbrechtsme...@weidmueller.com>>
---
meta/classes/cve-check.bbclass | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/classes/cve-check.bbclass
b/meta/classes/cve-check.bbclass
index 70d1988a70..a4d735c323 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -143,6 +143,7 @@ python cve_check_write_rootfs_manifest () {
manifest_name = d.getVar("CVE_CHECK_MANIFEST")
cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE")
+ bb.utils.mkdirhier(os.path.dirname(manifest_name))
shutil.copyfile(cve_tmp_file, manifest_name)
if manifest_name and os.path.exists(manifest_name):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161166):
https://lists.openembedded.org/g/openembedded-core/message/161166
Mute This Topic: https://lists.openembedded.org/mt/88836328/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 3/3] systemd: Minimize udev package size if DISTRO_FEATURES contains systemd
From: Stefan Herbrechtsmeier
Link udev shared with systemd helper to minimize the udev package size
if DISTRO_FEATURES contains systemd.
It is only usefull to link udev static with systemd helper if udev
should be installed without systemd.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-core/systemd/systemd_249.7.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 739f3aae86..431781e6c7 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -66,6 +66,7 @@ PACKAGECONFIG ??= " \
${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi ldconfig pam selinux
smack usrmerge polkit seccomp', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '', d)} \
+${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'link-udev-shared', '',
d)} \
backlight \
binfmt \
gshadow \
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161161):
https://lists.openembedded.org/g/openembedded-core/message/161161
Mute This Topic: https://lists.openembedded.org/mt/88836333/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 1/3] cve-check: create directory before copy
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
meta/classes/cve-check.bbclass | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 70d1988a70..a4d735c323 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -143,6 +143,7 @@ python cve_check_write_rootfs_manifest () {
manifest_name = d.getVar("CVE_CHECK_MANIFEST")
cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE")
+bb.utils.mkdirhier(os.path.dirname(manifest_name))
shutil.copyfile(cve_tmp_file, manifest_name)
if manifest_name and os.path.exists(manifest_name):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161159):
https://lists.openembedded.org/g/openembedded-core/message/161159
Mute This Topic: https://lists.openembedded.org/mt/88836328/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 2/3] systemd: Add link-udev-shared PACKAGECONFIG
From: Stefan Herbrechtsmeier
Add a PACKAGECONFIG to link systemd-udev and its helpers to
libsystemd-shared.so. If enabled the udev package depends on the systemd
package.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/recipes-core/systemd/systemd_249.7.bb | 8 +++-
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/meta/recipes-core/systemd/systemd_249.7.bb
b/meta/recipes-core/systemd/systemd_249.7.bb
index 53e7f6738d..739f3aae86 100644
--- a/meta/recipes-core/systemd/systemd_249.7.bb
+++ b/meta/recipes-core/systemd/systemd_249.7.bb
@@ -156,6 +156,9 @@ PACKAGECONFIG[kmod] = "-Dkmod=true,-Dkmod=false,kmod"
PACKAGECONFIG[ldconfig] = "-Dldconfig=true,-Dldconfig=false,,ldconfig"
PACKAGECONFIG[libidn] = "-Dlibidn=true,-Dlibidn=false,libidn,,libidn"
PACKAGECONFIG[libidn2] = "-Dlibidn2=true,-Dlibidn2=false,libidn2,,libidn2"
+# Link udev shared with systemd helper library.
+# If enabled the udev package depends on the systemd package (which has the
needed shared library).
+PACKAGECONFIG[link-udev-shared] =
"-Dlink-udev-shared=true,-Dlink-udev-shared=false"
PACKAGECONFIG[localed] = "-Dlocaled=true,-Dlocaled=false"
PACKAGECONFIG[logind] = "-Dlogind=true,-Dlogind=false"
PACKAGECONFIG[lz4] = "-Dlz4=true,-Dlz4=false,lz4"
@@ -215,11 +218,6 @@ rootprefix ?= "${root_prefix}"
rootlibdir ?= "${base_libdir}"
rootlibexecdir = "${rootprefix}/lib"
-# This links udev statically with systemd helper library.
-# Otherwise udev package would depend on systemd package (which has the needed
shared library),
-# and always pull it into images.
-EXTRA_OEMESON += "-Dlink-udev-shared=false"
-
EXTRA_OEMESON += "-Dnobody-user=nobody \
-Dnobody-group=nobody \
-Drootlibdir=${rootlibdir} \
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161160):
https://lists.openembedded.org/g/openembedded-core/message/161160
Mute This Topic: https://lists.openembedded.org/mt/88836330/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Am 27.12.21 um 15:54 schrieb Eero Aaltonen: On Mon, 2021-12-27 at 14:38 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote: Hi Alex, Am 25.12.21 um 20:41 schrieb Alexander Kanavin: On Sat, 25 Dec 2021 at 20:32, Stefan Herbrechtsmeier mailto:ste...@herbrechtsmeier.net>> wrote: > I'm not sure how to deal with that, so there aren't that many options here. This is a common problem for all language specific package managers (python / pip, Node.js / npm, Rust / Carge, go) and we need a common solution. I tend to think that the best (and the hardest) option is to improve these tools so that they're usable inside do_fetch (e.g. fulfil the caching/reproducibility criteria for a bitbake fetcher), and the needed changes are acceptable to upstreams. The real problem is the different philosophy between OE and the package manager. The package manager doesn't care about duplicate versions, maintenance versions, version updates of indirect dependencies, license compliance, CVE checks or dead code (examples, documentations, test, ...) and if they care every package manager have its own solution. The Java OSS ecosystem has had fairly well defined build time and runtime dependency metadata commonly available for many years now. The more widely used build tools are capable of using this metadata to calculate a solution Set(lib, version) that satisfies the version requirements of the dependency graph - or report that no solution is available. Cryptographic hashes are however by default provided by the artifact server(s). For Python distribution packages, the 2020 pip resolver is advertised to be better at calculating solutions to required package dependencies. Cryptographic hashes are specified for pip, use is optional. The PyPA Core metadata includes some potentially useful fields, such as the license and what the project requires. The language specific package managers could potentially be used to export dependency metadata, and also potentially to check "with (foo i.j.k) and (bar x.y.z), is there a set of compatible package versions available for baz and it's dependencies?" This is possible and already done in recipetool. I see two possible solution to support language specific package managers: a) Use the package manager manifest direct (go.sum, npm-shrinkwrap.json, ...) to fetch the dependencies. b) Create recipes from the package manager manifest. Either we give away the control over the dependencies and use the language specific way of manage dependencies or we have to extract the information as suggestion. Everything between means that we lose the benefits from the language specific package manager and OE together. Regard Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160032): https://lists.openembedded.org/g/openembedded-core/message/160032 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Hi Alex, Am 25.12.21 um 20:41 schrieb Alexander Kanavin: On Sat, 25 Dec 2021 at 20:32, Stefan Herbrechtsmeier mailto:ste...@herbrechtsmeier.net>> wrote: > I'm not sure how to deal with that, so there aren't that many options here. This is a common problem for all language specific package managers (python / pip, Node.js / npm, Rust / Carge, go) and we need a common solution. I tend to think that the best (and the hardest) option is to improve these tools so that they're usable inside do_fetch (e.g. fulfil the caching/reproducibility criteria for a bitbake fetcher), and the needed changes are acceptable to upstreams. Is the fetcher really the problem? In all cases the input and output of the package manager fetch task is well defined. In the npm case the bitbake npmsw fetcher and my recipetool approach translate this configuration into bitbake fetch and unpack commands. The real problem is the different philosophy between OE and the package manager. The package manager doesn't care about duplicate versions, maintenance versions, version updates of indirect dependencies, license compliance, CVE checks or dead code (examples, documentations, test, ...) and if they care every package manager have its own solution. Why C/C++ and Python doesn't fetch all its dependencies inside a single recipe and why do we try to replace embedded dependencies? I think we have good reasons for it and we shouldn't discard it for other languages. Independent of the language an update of a dependency need a test inside a user and with Node-RED as an example I show that this is possible for npm modules. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160024): https://lists.openembedded.org/g/openembedded-core/message/160024 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Am 25.12.21 um 21:58 schrieb Konrad Weihmann: On 25.12.21 20:32, Stefan Herbrechtsmeier wrote: Isn't the circular dependencies problem mainly a problem of the native packages? In many cases the dependency only exists at compile time of the main project and not at compile time of the dependency itself. In case of a native package all dependency sysroots need to be finished before the recipe sysroot itself. Unfortunately it is not - it's happens that the go.mod already creates this circular dependency. The classic example is that golang.org/x/tools depends on parts of golang.org/x/text (and guess what) golang.org/x/text depends solely on golang.org/x/tools :facepalm: Does this circular dependency exists in the executables (main) only? For go itself this isn't a big issue, as the compiler just pulls all the needed code into the compile workspace and build a binary out of it, which can be reused later on. But for bitbake it's a big issue, as either you pull in golang.org/x/text + golang.org/x/tools sources into the golang.org/x/text workspace, stripping you off the possibility to properly watch the revision of golang.org/x/tools (in terms of devtools and co) - or you simply end up with a circular dependency. I scratched my head on this one for weeks, till I found a way to actually use the latest greatest of each go module each coming as a recipe of its own, while avoiding above mentioned situations -- and it's kind of a huge break to how things are done right now. I'm pretty sure the same will happen in rust sooner or later. And I personally don't see how to break it as I don't know any way go distinguishes between compile and runtime dependency - afaik they all need to be present at compile time Is it possible to create a recipe for the source and a recipe for the binaries which depends on the source recipe? In this case the DEPENDS is always the source package and the RDEPENDS the binary package. A BBCLASSEXTEND could be used to create the source recipe automatically. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160020): https://lists.openembedded.org/g/openembedded-core/message/160020 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Hi Konrad, Am 25.12.21 um 21:43 schrieb Konrad Weihmann: What I so far don't really get is why increase in parsing time is such a big deal. I admit when we're talking about npm it's some kind of a drastic increase in recipes one would have to maintain, just because some random project decides to use a trillion dependencies instead of writing two or three lines of code more. Still I come to think this might be actually beneficial, as it shows how broken npm is from a distribution perspective - as it may be that some users actually start to access the situation when they are actually aware what monstrosity of a dep tree they are inheriting by just a single npm module. So this is mind - and I don't want to sound radical - I would rather abandon npm and go support in core than to sacrifice closing the one main loophole in core that is preventing true accessibility at the moment. Which is uncontrolled fetching outside of the fetching task, as this invalidates everything you will get in the end in terms of licensing, quality reports that haven't been done as part of the build and even CVE checking becomes pretty much worthless if one is allowed to inject random code on the fly into the build - and in the end everything that can't be assessed outside of the build is pretty much non-existing for most of the assessments I had to work with. In the past I showed that npm and go can be made to work with these principles, even if they would introduce a different way of working and maybe the need for better tooling - *but* they can work with how oe-core works at the moment - to the expense of parsing time - and that's pretty much it from my point of view. Still the gains outweigh it by far in my opinion, as it would make all of that accessible by common tooling already in place - including true reproducibility and builtin quality reports. BTW I don't think rust and therefore cargo is heavily affected by it, as the cargo to bitbake scripting works kind in the way I would imagine for npm and go too - so far just npm and go are really really bad to handle in this case. I'm happy to help on such scripting and tooling, but I don't see much worth in either accepting the fact that "modern languages" have to include some not validatable "magic" (which then would mean allowing uncontrolled network access) or working around the fact that a trillion dependencies is still a trillion dependencies no matter how you put it ( :) ) I fully agree with you and would be happy to add a 'recursive' option to devtool / recipetool to create multiple recipes on the fly instead of one big recipe with more than 100 different projects inclusive duplicate and different compatible versions. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160019): https://lists.openembedded.org/g/openembedded-core/message/160019 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Hi Konrad, Am 24.12.21 um 11:36 schrieb Konrad Weihmann: I had a look at the failures and most of them are fairly easy to fix - all but the go recipes like influxdb. The go class doesn't work without network by default if the project doesn't provide a vendor folder. I'm not sure how to deal with that, so there aren't that many options here. This is a common problem for all language specific package managers (python / pip, Node.js / npm, Rust / Carge, go) and we need a common solution. Either - inject all the needed with a fixed revision, but that would prevent devtool from catching updates to them properly This is possible if we extend recipetool to add the dependencies to the recipe. Rust and npm are using this option and I have a WIP to improve the npm solution. But in any case we loose a lot of advantages of OE. - create recipes for all of them and pray that there won't be a circular dependency in any of them The circular dependency is only the least problem and could be fixed by splitting the dependency and code into separate recipes. The main problem is the huge increase of recipe count and parse time. Furthermore we need the possibility to test recipe updates of dependencies. Python is using this option. - allow go to work with network (the least favorable option IMO) This is how the current go and especially go-mod class work. in the case of influxdb for instance we are talking about github.com/peterh/liner golang.org/x/crypto github.com/influxdata/influxql github.com/influxdata/flux github.com/BurntSushi/toml github.com/influxdata/usage-client golang.org/x/text go.uber.org/zap github.com/klauspost/pgzip github.com/prometheus/client_golang github.com/jsternberg/zap-logfmt github.com/mattn/go-isatty collectd.org github.com/bmizerany/pat github.com/dgrijalva/jwt-go/v4 github.com/gogo/protobuf github.com/golang/snappy github.com/tinylib/msgp github.com/opentracing/opentracing-go golang.org/x/sync github.com/apache/arrow/go/arrow github.com/pkg/errors google.golang.org/grpc github.com/kraj/xxhash github.com/influxdata/roaring github.com/xlab/treeprint golang.org/x/time golang.org/x/sys github.com/jwilder/encoding github.com/dgryski/go-bitstream where especially the golang.org/x modules are known to have circular dependencies like x/a relies on x/b, while x/b requires x/c and x/c needs x/a for building. Isn't the circular dependencies problem mainly a problem of the native packages? In many cases the dependency only exists at compile time of the main project and not at compile time of the dependency itself. In case of a native package all dependency sysroots need to be finished before the recipe sysroot itself. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160005): https://lists.openembedded.org/g/openembedded-core/message/160005 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
Hi Richard, Am 24.12.21 um 09:30 schrieb Richard Purdie: On Thu, 2021-12-23 at 22:00 -0800, Khem Raj wrote: is this list of failures due to this patch ? https://errors.yoctoproject.org/Errors/Build/137579/ Looks likely, yes. Recipes shouldn't be attempting downloads in do_compile :( In this case the go and especially go-mod class is broken by design because it downloads all its dependencies after do_fetch. Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#160004): https://lists.openembedded.org/g/openembedded-core/message/160004 Mute This Topic: https://lists.openembedded.org/mt/87909311/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 3/4] selftest: recipetool: Add test for split_pkg_licenses function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 64 ++
1 file changed, 64 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 95e4753976..1c73b2c5e0 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -541,9 +541,13 @@ class RecipetoolTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
+import sys
+
super(RecipetoolTests, cls).setUpClass()
bb_vars = get_bb_vars(['BBPATH'])
cls.bbpath = bb_vars['BBPATH']
+libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib',
'recipetool')
+sys.path.insert(0, libpath)
def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
dstdir = basedstdir
@@ -588,6 +592,66 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_split_pkg_licenses(self):
+from create import split_pkg_licenses
+licvalues = [
+# Duplicate licenses
+('BSD-2-Clause', 'x/COPYING', None),
+('BSD-2-Clause', 'x/LICENSE', None),
+# Multiple licenses
+('MIT', 'x/a/LICENSE.MIT', None),
+('ISC', 'x/a/LICENSE.ISC', None),
+# Alternative licenses
+('(MIT | ISC)', 'x/b/LICENSE', None),
+# Alternative licenses without brackets
+('MIT | BSD-2-Clause', 'x/c/LICENSE', None),
+# Multi licenses with alternatives
+('MIT', 'x/d/COPYING', None),
+('MIT | BSD-2-Clause', 'x/d/LICENSE', None),
+# Multi licenses with alternatives and brackets
+('Apache-2.0 & ((MIT | ISC) & BSD-3-Clause)', 'x/e/LICENSE', None)
+]
+packages = {
+'${PN}': '',
+'a': 'x/a',
+'b': 'x/b',
+'c': 'x/c',
+'d': 'x/d',
+'e': 'x/e',
+'f': 'x/f',
+'g': 'x/g',
+}
+fallback_licenses = {
+# Ignored
+'a': 'BSD-3-Clause',
+# Used
+'f': 'BSD-3-Clause'
+}
+outlines = []
+outlicenses = split_pkg_licenses(licvalues, packages, outlines,
fallback_licenses)
+expected_outlicenses = {
+'${PN}': ['BSD-2-Clause'],
+'a': ['ISC', 'MIT'],
+'b': ['(ISC | MIT)'],
+'c': ['(BSD-2-Clause | MIT)'],
+'d': ['(BSD-2-Clause | MIT)', 'MIT'],
+'e': ['(ISC | MIT)', 'Apache-2.0', 'BSD-3-Clause'],
+'f': ['BSD-3-Clause'],
+'g': ['Unknown']
+}
+self.assertEqual(outlicenses, expected_outlicenses)
+expected_outlines = [
+'LICENSE:${PN} = "BSD-2-Clause"',
+'LICENSE:a = "ISC & MIT"',
+'LICENSE:b = "(ISC | MIT)"',
+'LICENSE:c = "(BSD-2-Clause | MIT)"',
+'LICENSE:d = "(BSD-2-Clause | MIT) & MIT"',
+'LICENSE:e = "(ISC | MIT) & Apache-2.0 & BSD-3-Clause"',
+'LICENSE:f = "BSD-3-Clause"',
+'LICENSE:g = "Unknown"'
+]
+self.assertEqual(outlines, expected_outlines)
+
class RecipetoolAppendsrcBase(RecipetoolBase):
def _try_recipetool_appendsrcfile(self, testrecipe, newfile, destfile,
options, expectedlines, expectedfiles):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159734):
https://lists.openembedded.org/g/openembedded-core/message/159734
Mute This Topic: https://lists.openembedded.org/mt/87746715/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 4/4] selftest: recipetool: Add test for handle_license_vars function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v3:
- Sort LIC_FILES_CHKSUM values
meta/lib/oeqa/selftest/cases/recipetool.py | 62 ++
1 file changed, 62 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1c73b2c5e0..9db1ddb532 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -592,6 +592,68 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_handle_license_vars(self):
+from create import handle_license_vars
+from unittest.mock import Mock
+
+commonlicdir = get_bb_var('COMMON_LICENSE_DIR')
+
+d = bb.tinfoil.TinfoilDataStoreConnector
+d.getVar = Mock(return_value=commonlicdir)
+
+srctree = tempfile.mkdtemp(prefix='recipetoolqa')
+self.track_for_cleanup(srctree)
+
+# Multiple licenses
+licenses = ['MIT', 'ISC', 'BSD-3-Clause', 'Apache-2.0']
+for licence in licenses:
+shutil.copy(os.path.join(commonlicdir, licence),
os.path.join(srctree, 'LICENSE.' + licence))
+# Duplicate license
+shutil.copy(os.path.join(commonlicdir, 'MIT'), os.path.join(srctree,
'LICENSE'))
+
+extravalues = {
+# Duplicate and missing licenses
+'LICENSE': 'Zlib & BSD-2-Clause & Zlib',
+'LIC_FILES_CHKSUM': [
+'file://README.md;md5=0123456789abcdef0123456789abcd'
+]
+}
+lines_before = []
+handled = []
+licvalues = handle_license_vars(srctree, lines_before, handled,
extravalues, d)
+expected_lines_before = [
+'# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are
best guesses - it is',
+'# your responsibility to verify that the values are complete and
correct.',
+'# NOTE: Original package / source metadata indicates license is:
BSD-2-Clause & Zlib',
+'#',
+'# NOTE: multiple licenses have been detected; they have been
separated with &',
+'# in the LICENSE value for now since it is a reasonable
assumption that all',
+'# of the licenses apply. If instead there is a choice between the
multiple',
+'# licenses then you should change the value to separate the
licenses with |',
+'# instead of &. If there is any doubt, check the accompanying
documentation',
+'# to determine which situation is applicable.',
+'LICENSE = "Apache-2.0 & BSD-2-Clause & BSD-3-Clause & ISC & MIT &
Zlib"',
+'LIC_FILES_CHKSUM =
"file://LICENSE;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.Apache-2.0;md5=89aea4e17d99a7cacdbeed46a0096b10 \\\n'
+'
file://LICENSE.BSD-3-Clause;md5=550794465ba0ec5312d6919e203a55f9 \\\n'
+'
file://LICENSE.ISC;md5=f3b90e78ea0cffb20bf5cca7947a896d \\\n'
+'
file://LICENSE.MIT;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://README.md;md5=0123456789abcdef0123456789abcd"',
+''
+]
+self.assertEqual(lines_before, expected_lines_before)
+expected_licvalues = [
+('MIT', 'LICENSE', '0835ade698e0bcf8506ecda2f7b4f302'),
+('Apache-2.0', 'LICENSE.Apache-2.0',
'89aea4e17d99a7cacdbeed46a0096b10'),
+('BSD-3-Clause', 'LICENSE.BSD-3-Clause',
'550794465ba0ec5312d6919e203a55f9'),
+('ISC', 'LICENSE.ISC', 'f3b90e78ea0cffb20bf5cca7947a896d'),
+('MIT', 'LICENSE.MIT', '0835ade698e0bcf8506ecda2f7b4f302')
+]
+self.assertEqual(handled, [('license', expected_licvalues)])
+self.assertEqual(extravalues, {})
+self.assertEqual(licvalues, expected_licvalues)
+
+
def test_recipetool_split_pkg_licenses(self):
from create import split_pkg_licenses
licvalues = [
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159733):
https://lists.openembedded.org/g/openembedded-core/message/159733
Mute This Topic: https://lists.openembedded.org/mt/87746714/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 1/4] recipetool: Sort output of guess_license function to be deterministic
From: Stefan Herbrechtsmeier Sort the output of guess_license function by license file to be deterministic and to support self-testing. Signed-off-by: Stefan Herbrechtsmeier --- Changes in v3: - Add commit to fix selftest failure of autobuilder scripts/lib/recipetool/create.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 406c97f1c5..28224dbc24 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1201,7 +1201,7 @@ def guess_license(srctree, d): fullpath = os.path.join(root, fn) if not fullpath in licfiles: licfiles.append(fullpath) -for licfile in licfiles: +for licfile in sorted(licfiles): md5value = bb.utils.md5_file(licfile) license = md5sums.get(md5value, None) if not license: -- 2.30.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#159731): https://lists.openembedded.org/g/openembedded-core/message/159731 Mute This Topic: https://lists.openembedded.org/mt/87746711/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 2/4] recipetool: Separate licenses with & operator
From: Stefan Herbrechtsmeier
Separate licenses with & operator since it should be satisfied most use
cases and it is a reasonable assumption that all the licenses apply.
Furthermore flat, split and sort the licenses to minimize license string
changes.
Separate package licenses with & operator:
-LICENSE:${PN} = "MIT ISC"
+LICENSE:${PN} = "ISC & MIT"
Respect | and brackets in LICENSE:
-LICENSE = "BSD-3-Clause & (ISC & | & MIT)"
+LICENSE = "BSD-3-Clause & (ISC | MIT)"
Sort licenses:
-LICENSE = "MIT & BSD-3-Clause & ISC"
+LICENSE = "BSD-3-Clause & ISC & MIT"
Remove duplicates:
-LICENSE = "MIT & ISC & MIT"
+LICENSE = "ISC & MIT"
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 4 +--
scripts/lib/recipetool/create.py | 39 +-
2 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 439e41597c..95e4753976 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -426,7 +426,7 @@ class RecipetoolCreateTests(RecipetoolBase):
checkvars = {}
checkvars['SUMMARY'] = 'Node Server Example'
checkvars['HOMEPAGE'] =
'https://github.com/savoirfairelinux/node-server-example#readme'
-checkvars['LICENSE'] = set(['MIT', 'ISC', 'Unknown'])
+checkvars['LICENSE'] = 'BSD-3-Clause & ISC & MIT & Unknown'
urls = []
urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
@@ -483,7 +483,7 @@ class RecipetoolCreateTests(RecipetoolBase):
result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
self.assertTrue(os.path.isfile(recipefile))
checkvars = {}
-checkvars['LICENSE'] = set(['PSF', '&', 'BSD-3-Clause', 'GPL'])
+checkvars['LICENSE'] = 'BSD-3-Clause & GPL & PSF'
checkvars['LIC_FILES_CHKSUM'] =
'file://COPYING.txt;md5=35a23d42b615470583563132872c97d6'
checkvars['SRC_URI'] =
'https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-${PV}.tar.gz'
checkvars['SRC_URI[md5sum]'] = 'c53768d63db3873b7d452833553469de'
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 28224dbc24..507a230511 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -919,6 +919,22 @@ def split_value(value):
else:
return value
+def fixup_license(value):
+# Ensure licenses with OR starts and ends with brackets
+if '|' in value:
+return '(' + value + ')'
+return value
+
+def tidy_licenses(value):
+"""Flat, split and sort licenses"""
+from oe.license import flattened_licenses
+def _choose(a, b):
+str_a, str_b = sorted((" & ".join(a), " & ".join(b)),
key=str.casefold)
+return ["(%s | %s)" % (str_a, str_b)]
+if not isinstance(value, str):
+value = " & ".join(value)
+return sorted(list(set(flattened_licenses(value, _choose))),
key=str.casefold)
+
def handle_license_vars(srctree, lines_before, handled, extravalues, d):
lichandled = [x for x in handled if x[0] == 'license']
if lichandled:
@@ -932,10 +948,13 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
lines = []
if licvalues:
for licvalue in licvalues:
-if not licvalue[0] in licenses:
-licenses.append(licvalue[0])
+license = licvalue[0]
+lics = tidy_licenses(fixup_license(license))
+lics = [lic for lic in lics if lic not in licenses]
+if len(lics):
+licenses.extend(lics)
lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1],
licvalue[2]))
-if licvalue[0] == 'Unknown':
+if license == 'Unknown':
lic_unknown.append(licvalue[1])
if lic_unknown:
lines.append('#')
@@ -944,9 +963,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
for licfile in lic_unknown:
lines.append('# %s' % licfile)
-extra_license = split_value(extravalues.pop('LICENSE', []))
-if '&' in extra_license:
-extra_license.remove('&')
+extra_license = tidy_licenses(extravalues.pop('LICENSE', ''))
if extra_license:
if licenses == ['Unknown']:
licenses = extra_license
@@ -987,7 +1004,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
l
Re: [OE-core] [RFC PATCH v2 3/3] selftest: recipetool: Add test for handle_license_vars function
Hi Richard,
Am 14.12.2021 um 14:01 schrieb Richard Purdie:
On Mon, 2021-12-13 at 16:05 +0100, Stefan Herbrechtsmeier wrote:
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 62 ++
1 file changed, 62 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1c73b2c5e0..5330d73cbf 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -592,6 +592,68 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_handle_license_vars(self):
+from create import handle_license_vars
+from unittest.mock import Mock
+
+commonlicdir = get_bb_var('COMMON_LICENSE_DIR')
+
+d = bb.tinfoil.TinfoilDataStoreConnector
+d.getVar = Mock(return_value=commonlicdir)
+
+srctree = tempfile.mkdtemp(prefix='recipetoolqa')
+self.track_for_cleanup(srctree)
+
+# Multiple licenses
+licenses = ['MIT', 'ISC', 'BSD-3-Clause', 'Apache-2.0']
+for licence in licenses:
+shutil.copy(os.path.join(commonlicdir, licence),
os.path.join(srctree, 'LICENSE.' + licence))
+# Duplicate license
+shutil.copy(os.path.join(commonlicdir, 'MIT'), os.path.join(srctree,
'LICENSE'))
+
+extravalues = {
+# Duplicate and missing licenses
+'LICENSE': 'Zlib & BSD-2-Clause & Zlib',
+'LIC_FILES_CHKSUM': [
+'file://README.md;md5=0123456789abcdef0123456789abcd'
+]
+}
+lines_before = []
+handled = []
+licvalues = handle_license_vars(srctree, lines_before, handled,
extravalues, d)
+expected_lines_before = [
+'# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are
best guesses - it is',
+'# your responsibility to verify that the values are complete and
correct.',
+'# NOTE: Original package / source metadata indicates license is:
BSD-2-Clause & Zlib',
+'#',
+'# NOTE: multiple licenses have been detected; they have been
separated with &',
+'# in the LICENSE value for now since it is a reasonable
assumption that all',
+'# of the licenses apply. If instead there is a choice between the
multiple',
+'# licenses then you should change the value to separate the
licenses with |',
+'# instead of &. If there is any doubt, check the accompanying
documentation',
+'# to determine which situation is applicable.',
+'LICENSE = "Apache-2.0 & BSD-2-Clause & BSD-3-Clause & ISC & MIT &
Zlib"',
+'LIC_FILES_CHKSUM =
"file://LICENSE;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.MIT;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.ISC;md5=f3b90e78ea0cffb20bf5cca7947a896d \\\n'
+'
file://LICENSE.Apache-2.0;md5=89aea4e17d99a7cacdbeed46a0096b10 \\\n'
+'
file://LICENSE.BSD-3-Clause;md5=550794465ba0ec5312d6919e203a55f9 \\\n'
+'
file://README.md;md5=0123456789abcdef0123456789abcd"',
+''
+]
+self.assertEqual(lines_before, expected_lines_before)
+expected_licvalues = [
+('MIT', 'LICENSE', '0835ade698e0bcf8506ecda2f7b4f302'),
+('MIT', 'LICENSE.MIT', '0835ade698e0bcf8506ecda2f7b4f302'),
+('ISC', 'LICENSE.ISC', 'f3b90e78ea0cffb20bf5cca7947a896d'),
+('Apache-2.0', 'LICENSE.Apache-2.0',
'89aea4e17d99a7cacdbeed46a0096b10'),
+('BSD-3-Clause', 'LICENSE.BSD-3-Clause',
'550794465ba0ec5312d6919e203a55f9')
+]
+self.assertEqual(handled, [('license', expected_licvalues)])
+self.assertEqual(extravalues, {})
+self.assertEqual(licvalues, expected_licvalues)
+
+
def test_recipetool_split_pkg_licenses(self):
from create import split_pkg_licenses
licvalues = [
The test failed in testing:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2956/steps/14/logs/stdio
I didn't know that os.walk() is not deterministic.
I will fix it.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159692):
https://lists.openembedded.org/g/openembedded-core/message/159692
Mute This Topic: https://lists.openembedded.org/mt/87699203/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH v2 1/3] recipetool: Separate licenses with & operator
From: Stefan Herbrechtsmeier
Separate licenses with & operator since it should be satisfied most use
cases and it is a reasonable assumption that all the licenses apply.
Furthermore flat, split and sort the licenses to minimize license string
changes.
Separate package licenses with & operator:
-LICENSE:${PN} = "MIT ISC"
+LICENSE:${PN} = "ISC & MIT"
Respect | and brackets in LICENSE:
-LICENSE = "BSD-3-Clause & (ISC & | & MIT)"
+LICENSE = "BSD-3-Clause & (ISC | MIT)"
Sort licenses:
-LICENSE = "MIT & BSD-3-Clause & ISC"
+LICENSE = "BSD-3-Clause & ISC & MIT"
Remove duplicates:
-LICENSE = "MIT & ISC & MIT"
+LICENSE = "ISC & MIT"
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 4 +--
scripts/lib/recipetool/create.py | 39 +-
2 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 439e41597c..95e4753976 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -426,7 +426,7 @@ class RecipetoolCreateTests(RecipetoolBase):
checkvars = {}
checkvars['SUMMARY'] = 'Node Server Example'
checkvars['HOMEPAGE'] =
'https://github.com/savoirfairelinux/node-server-example#readme'
-checkvars['LICENSE'] = set(['MIT', 'ISC', 'Unknown'])
+checkvars['LICENSE'] = 'BSD-3-Clause & ISC & MIT & Unknown'
urls = []
urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
@@ -483,7 +483,7 @@ class RecipetoolCreateTests(RecipetoolBase):
result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
self.assertTrue(os.path.isfile(recipefile))
checkvars = {}
-checkvars['LICENSE'] = set(['PSF', '&', 'BSD-3-Clause', 'GPL'])
+checkvars['LICENSE'] = 'BSD-3-Clause & GPL & PSF'
checkvars['LIC_FILES_CHKSUM'] =
'file://COPYING.txt;md5=35a23d42b615470583563132872c97d6'
checkvars['SRC_URI'] =
'https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-${PV}.tar.gz'
checkvars['SRC_URI[md5sum]'] = 'c53768d63db3873b7d452833553469de'
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 406c97f1c5..8e8a621b4f 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -919,6 +919,22 @@ def split_value(value):
else:
return value
+def fixup_license(value):
+# Ensure licenses with OR starts and ends with brackets
+if '|' in value:
+return '(' + value + ')'
+return value
+
+def tidy_licenses(value):
+"""Flat, split and sort licenses"""
+from oe.license import flattened_licenses
+def _choose(a, b):
+str_a, str_b = sorted((" & ".join(a), " & ".join(b)),
key=str.casefold)
+return ["(%s | %s)" % (str_a, str_b)]
+if not isinstance(value, str):
+value = " & ".join(value)
+return sorted(list(set(flattened_licenses(value, _choose))),
key=str.casefold)
+
def handle_license_vars(srctree, lines_before, handled, extravalues, d):
lichandled = [x for x in handled if x[0] == 'license']
if lichandled:
@@ -932,10 +948,13 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
lines = []
if licvalues:
for licvalue in licvalues:
-if not licvalue[0] in licenses:
-licenses.append(licvalue[0])
+license = licvalue[0]
+lics = tidy_licenses(fixup_license(license))
+lics = [lic for lic in lics if lic not in licenses]
+if len(lics):
+licenses.extend(lics)
lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1],
licvalue[2]))
-if licvalue[0] == 'Unknown':
+if license == 'Unknown':
lic_unknown.append(licvalue[1])
if lic_unknown:
lines.append('#')
@@ -944,9 +963,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
for licfile in lic_unknown:
lines.append('# %s' % licfile)
-extra_license = split_value(extravalues.pop('LICENSE', []))
-if '&' in extra_license:
-extra_license.remove('&')
+extra_license = tidy_licenses(extravalues.pop('LICENSE', ''))
if extra_license:
if licenses == ['Unknown']:
licenses = extra_license
@@ -987,7 +1004,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
l
[OE-core] [RFC PATCH v2 3/3] selftest: recipetool: Add test for handle_license_vars function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 62 ++
1 file changed, 62 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1c73b2c5e0..5330d73cbf 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -592,6 +592,68 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_handle_license_vars(self):
+from create import handle_license_vars
+from unittest.mock import Mock
+
+commonlicdir = get_bb_var('COMMON_LICENSE_DIR')
+
+d = bb.tinfoil.TinfoilDataStoreConnector
+d.getVar = Mock(return_value=commonlicdir)
+
+srctree = tempfile.mkdtemp(prefix='recipetoolqa')
+self.track_for_cleanup(srctree)
+
+# Multiple licenses
+licenses = ['MIT', 'ISC', 'BSD-3-Clause', 'Apache-2.0']
+for licence in licenses:
+shutil.copy(os.path.join(commonlicdir, licence),
os.path.join(srctree, 'LICENSE.' + licence))
+# Duplicate license
+shutil.copy(os.path.join(commonlicdir, 'MIT'), os.path.join(srctree,
'LICENSE'))
+
+extravalues = {
+# Duplicate and missing licenses
+'LICENSE': 'Zlib & BSD-2-Clause & Zlib',
+'LIC_FILES_CHKSUM': [
+'file://README.md;md5=0123456789abcdef0123456789abcd'
+]
+}
+lines_before = []
+handled = []
+licvalues = handle_license_vars(srctree, lines_before, handled,
extravalues, d)
+expected_lines_before = [
+'# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are
best guesses - it is',
+'# your responsibility to verify that the values are complete and
correct.',
+'# NOTE: Original package / source metadata indicates license is:
BSD-2-Clause & Zlib',
+'#',
+'# NOTE: multiple licenses have been detected; they have been
separated with &',
+'# in the LICENSE value for now since it is a reasonable
assumption that all',
+'# of the licenses apply. If instead there is a choice between the
multiple',
+'# licenses then you should change the value to separate the
licenses with |',
+'# instead of &. If there is any doubt, check the accompanying
documentation',
+'# to determine which situation is applicable.',
+'LICENSE = "Apache-2.0 & BSD-2-Clause & BSD-3-Clause & ISC & MIT &
Zlib"',
+'LIC_FILES_CHKSUM =
"file://LICENSE;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.MIT;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.ISC;md5=f3b90e78ea0cffb20bf5cca7947a896d \\\n'
+'
file://LICENSE.Apache-2.0;md5=89aea4e17d99a7cacdbeed46a0096b10 \\\n'
+'
file://LICENSE.BSD-3-Clause;md5=550794465ba0ec5312d6919e203a55f9 \\\n'
+'
file://README.md;md5=0123456789abcdef0123456789abcd"',
+''
+]
+self.assertEqual(lines_before, expected_lines_before)
+expected_licvalues = [
+('MIT', 'LICENSE', '0835ade698e0bcf8506ecda2f7b4f302'),
+('MIT', 'LICENSE.MIT', '0835ade698e0bcf8506ecda2f7b4f302'),
+('ISC', 'LICENSE.ISC', 'f3b90e78ea0cffb20bf5cca7947a896d'),
+('Apache-2.0', 'LICENSE.Apache-2.0',
'89aea4e17d99a7cacdbeed46a0096b10'),
+('BSD-3-Clause', 'LICENSE.BSD-3-Clause',
'550794465ba0ec5312d6919e203a55f9')
+]
+self.assertEqual(handled, [('license', expected_licvalues)])
+self.assertEqual(extravalues, {})
+self.assertEqual(licvalues, expected_licvalues)
+
+
def test_recipetool_split_pkg_licenses(self):
from create import split_pkg_licenses
licvalues = [
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159642):
https://lists.openembedded.org/g/openembedded-core/message/159642
Mute This Topic: https://lists.openembedded.org/mt/87699203/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH v2 2/3] selftest: recipetool: Add test for split_pkg_licenses function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 64 ++
1 file changed, 64 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 95e4753976..1c73b2c5e0 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -541,9 +541,13 @@ class RecipetoolTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
+import sys
+
super(RecipetoolTests, cls).setUpClass()
bb_vars = get_bb_vars(['BBPATH'])
cls.bbpath = bb_vars['BBPATH']
+libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib',
'recipetool')
+sys.path.insert(0, libpath)
def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
dstdir = basedstdir
@@ -588,6 +592,66 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_split_pkg_licenses(self):
+from create import split_pkg_licenses
+licvalues = [
+# Duplicate licenses
+('BSD-2-Clause', 'x/COPYING', None),
+('BSD-2-Clause', 'x/LICENSE', None),
+# Multiple licenses
+('MIT', 'x/a/LICENSE.MIT', None),
+('ISC', 'x/a/LICENSE.ISC', None),
+# Alternative licenses
+('(MIT | ISC)', 'x/b/LICENSE', None),
+# Alternative licenses without brackets
+('MIT | BSD-2-Clause', 'x/c/LICENSE', None),
+# Multi licenses with alternatives
+('MIT', 'x/d/COPYING', None),
+('MIT | BSD-2-Clause', 'x/d/LICENSE', None),
+# Multi licenses with alternatives and brackets
+('Apache-2.0 & ((MIT | ISC) & BSD-3-Clause)', 'x/e/LICENSE', None)
+]
+packages = {
+'${PN}': '',
+'a': 'x/a',
+'b': 'x/b',
+'c': 'x/c',
+'d': 'x/d',
+'e': 'x/e',
+'f': 'x/f',
+'g': 'x/g',
+}
+fallback_licenses = {
+# Ignored
+'a': 'BSD-3-Clause',
+# Used
+'f': 'BSD-3-Clause'
+}
+outlines = []
+outlicenses = split_pkg_licenses(licvalues, packages, outlines,
fallback_licenses)
+expected_outlicenses = {
+'${PN}': ['BSD-2-Clause'],
+'a': ['ISC', 'MIT'],
+'b': ['(ISC | MIT)'],
+'c': ['(BSD-2-Clause | MIT)'],
+'d': ['(BSD-2-Clause | MIT)', 'MIT'],
+'e': ['(ISC | MIT)', 'Apache-2.0', 'BSD-3-Clause'],
+'f': ['BSD-3-Clause'],
+'g': ['Unknown']
+}
+self.assertEqual(outlicenses, expected_outlicenses)
+expected_outlines = [
+'LICENSE:${PN} = "BSD-2-Clause"',
+'LICENSE:a = "ISC & MIT"',
+'LICENSE:b = "(ISC | MIT)"',
+'LICENSE:c = "(BSD-2-Clause | MIT)"',
+'LICENSE:d = "(BSD-2-Clause | MIT) & MIT"',
+'LICENSE:e = "(ISC | MIT) & Apache-2.0 & BSD-3-Clause"',
+'LICENSE:f = "BSD-3-Clause"',
+'LICENSE:g = "Unknown"'
+]
+self.assertEqual(outlines, expected_outlines)
+
class RecipetoolAppendsrcBase(RecipetoolBase):
def _try_recipetool_appendsrcfile(self, testrecipe, newfile, destfile,
options, expectedlines, expectedfiles):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159640):
https://lists.openembedded.org/g/openembedded-core/message/159640
Mute This Topic: https://lists.openembedded.org/mt/87699201/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 1/3] recipetool: Separate licenses with & operator
Am 13.12.2021 um 14:00 schrieb Stefan Herbrechtsmeier via lists.openembedded.org: Am 13.12.2021 um 13:57 schrieb Alexander Kanavin: On Mon, 13 Dec 2021 at 13:38, Stefan Herbrechtsmeier <mailto:stefan.herbrechtsmeier-...@weidmueller.com>> wrote: From: Stefan Herbrechtsmeier mailto:stefan.herbrechtsme...@weidmueller.com>> Separate licenses with & operator since it should be satisfied most use cases and it is a reasonable assumption that all the licenses apply. Furthermore flat, split and sort the licenses to minimize license string changes. I think this is better explained with an example. What did not work before, and what works with this change? Do you mean something like: -LICENSE = "MIT ISC Unknown" +LICENSE = "BSD-3-Clause & ISC & MIT & Unknown" Sorry I mean: -LICENSE = "MIT ISC Unknown" +LICENSE = "ISC & MIT & Unknown" -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#159624): https://lists.openembedded.org/g/openembedded-core/message/159624 Mute This Topic: https://lists.openembedded.org/mt/87696347/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [RFC PATCH 1/3] recipetool: Separate licenses with & operator
Am 13.12.2021 um 13:57 schrieb Alexander Kanavin:
On Mon, 13 Dec 2021 at 13:38, Stefan Herbrechtsmeier
<mailto:stefan.herbrechtsmeier-...@weidmueller.com>> wrote:
From: Stefan Herbrechtsmeier mailto:stefan.herbrechtsme...@weidmueller.com>>
Separate licenses with & operator since it should be satisfied most use
cases and it is a reasonable assumption that all the licenses apply.
Furthermore flat, split and sort the licenses to minimize license string
changes.
I think this is better explained with an example. What did not work
before, and what works with this change?
Do you mean something like:
-LICENSE = "MIT ISC Unknown"
+LICENSE = "BSD-3-Clause & ISC & MIT & Unknown"
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159623):
https://lists.openembedded.org/g/openembedded-core/message/159623
Mute This Topic: https://lists.openembedded.org/mt/87696347/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 3/3] selftest: recipetool: Add test for handle_license_vars function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/recipetool.py | 62 ++
1 file changed, 62 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1c73b2c5e0..5330d73cbf 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -592,6 +592,68 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_handle_license_vars(self):
+from create import handle_license_vars
+from unittest.mock import Mock
+
+commonlicdir = get_bb_var('COMMON_LICENSE_DIR')
+
+d = bb.tinfoil.TinfoilDataStoreConnector
+d.getVar = Mock(return_value=commonlicdir)
+
+srctree = tempfile.mkdtemp(prefix='recipetoolqa')
+self.track_for_cleanup(srctree)
+
+# Multiple licenses
+licenses = ['MIT', 'ISC', 'BSD-3-Clause', 'Apache-2.0']
+for licence in licenses:
+shutil.copy(os.path.join(commonlicdir, licence),
os.path.join(srctree, 'LICENSE.' + licence))
+# Duplicate license
+shutil.copy(os.path.join(commonlicdir, 'MIT'), os.path.join(srctree,
'LICENSE'))
+
+extravalues = {
+# Duplicate and missing licenses
+'LICENSE': 'Zlib & BSD-2-Clause & Zlib',
+'LIC_FILES_CHKSUM': [
+'file://README.md;md5=0123456789abcdef0123456789abcd'
+]
+}
+lines_before = []
+handled = []
+licvalues = handle_license_vars(srctree, lines_before, handled,
extravalues, d)
+expected_lines_before = [
+'# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are
best guesses - it is',
+'# your responsibility to verify that the values are complete and
correct.',
+'# NOTE: Original package / source metadata indicates license is:
BSD-2-Clause & Zlib',
+'#',
+'# NOTE: multiple licenses have been detected; they have been
separated with &',
+'# in the LICENSE value for now since it is a reasonable
assumption that all',
+'# of the licenses apply. If instead there is a choice between the
multiple',
+'# licenses then you should change the value to separate the
licenses with |',
+'# instead of &. If there is any doubt, check the accompanying
documentation',
+'# to determine which situation is applicable.',
+'LICENSE = "Apache-2.0 & BSD-2-Clause & BSD-3-Clause & ISC & MIT &
Zlib"',
+'LIC_FILES_CHKSUM =
"file://LICENSE;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.MIT;md5=0835ade698e0bcf8506ecda2f7b4f302 \\\n'
+'
file://LICENSE.ISC;md5=f3b90e78ea0cffb20bf5cca7947a896d \\\n'
+'
file://LICENSE.Apache-2.0;md5=89aea4e17d99a7cacdbeed46a0096b10 \\\n'
+'
file://LICENSE.BSD-3-Clause;md5=550794465ba0ec5312d6919e203a55f9 \\\n'
+'
file://README.md;md5=0123456789abcdef0123456789abcd"',
+''
+]
+self.assertEqual(lines_before, expected_lines_before)
+expected_licvalues = [
+('MIT', 'LICENSE', '0835ade698e0bcf8506ecda2f7b4f302'),
+('MIT', 'LICENSE.MIT', '0835ade698e0bcf8506ecda2f7b4f302'),
+('ISC', 'LICENSE.ISC', 'f3b90e78ea0cffb20bf5cca7947a896d'),
+('Apache-2.0', 'LICENSE.Apache-2.0',
'89aea4e17d99a7cacdbeed46a0096b10'),
+('BSD-3-Clause', 'LICENSE.BSD-3-Clause',
'550794465ba0ec5312d6919e203a55f9')
+]
+self.assertEqual(handled, [('license', expected_licvalues)])
+self.assertEqual(extravalues, {})
+self.assertEqual(licvalues, expected_licvalues)
+
+
def test_recipetool_split_pkg_licenses(self):
from create import split_pkg_licenses
licvalues = [
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159621):
https://lists.openembedded.org/g/openembedded-core/message/159621
Mute This Topic: https://lists.openembedded.org/mt/87696349/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 2/3] selftest: recipetool: Add test for split_pkg_licenses function
From: Stefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/recipetool.py | 64 ++
1 file changed, 64 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 95e4753976..1c73b2c5e0 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -541,9 +541,13 @@ class RecipetoolTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
+import sys
+
super(RecipetoolTests, cls).setUpClass()
bb_vars = get_bb_vars(['BBPATH'])
cls.bbpath = bb_vars['BBPATH']
+libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib',
'recipetool')
+sys.path.insert(0, libpath)
def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
dstdir = basedstdir
@@ -588,6 +592,66 @@ class RecipetoolTests(RecipetoolBase):
with open(srcfile, 'w') as fh:
fh.writelines(plugincontent)
+def test_recipetool_split_pkg_licenses(self):
+from create import split_pkg_licenses
+licvalues = [
+# Duplicate licenses
+('BSD-2-Clause', 'x/COPYING', None),
+('BSD-2-Clause', 'x/LICENSE', None),
+# Multiple licenses
+('MIT', 'x/a/LICENSE.MIT', None),
+('ISC', 'x/a/LICENSE.ISC', None),
+# Alternative licenses
+('(MIT | ISC)', 'x/b/LICENSE', None),
+# Alternative licenses without brackets
+('MIT | BSD-2-Clause', 'x/c/LICENSE', None),
+# Multi licenses with alternatives
+('MIT', 'x/d/COPYING', None),
+('MIT | BSD-2-Clause', 'x/d/LICENSE', None),
+# Multi licenses with alternatives and brackets
+('Apache-2.0 & ((MIT | ISC) & BSD-3-Clause)', 'x/e/LICENSE', None)
+]
+packages = {
+'${PN}': '',
+'a': 'x/a',
+'b': 'x/b',
+'c': 'x/c',
+'d': 'x/d',
+'e': 'x/e',
+'f': 'x/f',
+'g': 'x/g',
+}
+fallback_licenses = {
+# Ignored
+'a': 'BSD-3-Clause',
+# Used
+'f': 'BSD-3-Clause'
+}
+outlines = []
+outlicenses = split_pkg_licenses(licvalues, packages, outlines,
fallback_licenses)
+expected_outlicenses = {
+'${PN}': ['BSD-2-Clause'],
+'a': ['ISC', 'MIT'],
+'b': ['(ISC | MIT)'],
+'c': ['(BSD-2-Clause | MIT)'],
+'d': ['(BSD-2-Clause | MIT)', 'MIT'],
+'e': ['(ISC | MIT)', 'Apache-2.0', 'BSD-3-Clause'],
+'f': ['BSD-3-Clause'],
+'g': ['Unknown']
+}
+self.assertEqual(outlicenses, expected_outlicenses)
+expected_outlines = [
+'LICENSE:${PN} = "BSD-2-Clause"',
+'LICENSE:a = "ISC & MIT"',
+'LICENSE:b = "(ISC | MIT)"',
+'LICENSE:c = "(BSD-2-Clause | MIT)"',
+'LICENSE:d = "(BSD-2-Clause | MIT) & MIT"',
+'LICENSE:e = "(ISC | MIT) & Apache-2.0 & BSD-3-Clause"',
+'LICENSE:f = "BSD-3-Clause"',
+'LICENSE:g = "Unknown"'
+]
+self.assertEqual(outlines, expected_outlines)
+
class RecipetoolAppendsrcBase(RecipetoolBase):
def _try_recipetool_appendsrcfile(self, testrecipe, newfile, destfile,
options, expectedlines, expectedfiles):
--
2.30.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159620):
https://lists.openembedded.org/g/openembedded-core/message/159620
Mute This Topic: https://lists.openembedded.org/mt/87696348/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [RFC PATCH 1/3] recipetool: Separate licenses with & operator
From: Stefan Herbrechtsmeier
Separate licenses with & operator since it should be satisfied most use
cases and it is a reasonable assumption that all the licenses apply.
Furthermore flat, split and sort the licenses to minimize license string
changes.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/recipetool.py | 4 +--
scripts/lib/recipetool/create.py | 39 +-
2 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 439e41597c..95e4753976 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -426,7 +426,7 @@ class RecipetoolCreateTests(RecipetoolBase):
checkvars = {}
checkvars['SUMMARY'] = 'Node Server Example'
checkvars['HOMEPAGE'] =
'https://github.com/savoirfairelinux/node-server-example#readme'
-checkvars['LICENSE'] = set(['MIT', 'ISC', 'Unknown'])
+checkvars['LICENSE'] = 'BSD-3-Clause & ISC & MIT & Unknown'
urls = []
urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
@@ -483,7 +483,7 @@ class RecipetoolCreateTests(RecipetoolBase):
result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
self.assertTrue(os.path.isfile(recipefile))
checkvars = {}
-checkvars['LICENSE'] = set(['PSF', '&', 'BSD-3-Clause', 'GPL'])
+checkvars['LICENSE'] = 'BSD-3-Clause & GPL & PSF'
checkvars['LIC_FILES_CHKSUM'] =
'file://COPYING.txt;md5=35a23d42b615470583563132872c97d6'
checkvars['SRC_URI'] =
'https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-${PV}.tar.gz'
checkvars['SRC_URI[md5sum]'] = 'c53768d63db3873b7d452833553469de'
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 406c97f1c5..8e8a621b4f 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -919,6 +919,22 @@ def split_value(value):
else:
return value
+def fixup_license(value):
+# Ensure licenses with OR starts and ends with brackets
+if '|' in value:
+return '(' + value + ')'
+return value
+
+def tidy_licenses(value):
+"""Flat, split and sort licenses"""
+from oe.license import flattened_licenses
+def _choose(a, b):
+str_a, str_b = sorted((" & ".join(a), " & ".join(b)),
key=str.casefold)
+return ["(%s | %s)" % (str_a, str_b)]
+if not isinstance(value, str):
+value = " & ".join(value)
+return sorted(list(set(flattened_licenses(value, _choose))),
key=str.casefold)
+
def handle_license_vars(srctree, lines_before, handled, extravalues, d):
lichandled = [x for x in handled if x[0] == 'license']
if lichandled:
@@ -932,10 +948,13 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
lines = []
if licvalues:
for licvalue in licvalues:
-if not licvalue[0] in licenses:
-licenses.append(licvalue[0])
+license = licvalue[0]
+lics = tidy_licenses(fixup_license(license))
+lics = [lic for lic in lics if lic not in licenses]
+if len(lics):
+licenses.extend(lics)
lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1],
licvalue[2]))
-if licvalue[0] == 'Unknown':
+if license == 'Unknown':
lic_unknown.append(licvalue[1])
if lic_unknown:
lines.append('#')
@@ -944,9 +963,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
for licfile in lic_unknown:
lines.append('# %s' % licfile)
-extra_license = split_value(extravalues.pop('LICENSE', []))
-if '&' in extra_license:
-extra_license.remove('&')
+extra_license = tidy_licenses(extravalues.pop('LICENSE', ''))
if extra_license:
if licenses == ['Unknown']:
licenses = extra_license
@@ -987,7 +1004,7 @@ def handle_license_vars(srctree, lines_before, handled,
extravalues, d):
lines.append('# instead of &. If there is any doubt, check the
accompanying documentation')
lines.append('# to determine which situation is applicable.')
-lines.append('LICENSE = "%s"' % ' & '.join(licenses))
+lines.append('LICENSE = "%s"' % ' & '.join(sorted(licenses,
key=str.casefold)))
lines.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n
'.join(lic_files_chksum))
lines.append('')
@@ -1226,6 +1243,7 @@ def split_pkg_licenses(licvalues, packages
[OE-core] [PATCH v2 2/5] selftest/devtool: Check branch in git fetch
From: Stefan Herbrechtsmeier
The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
forget the url branch= parameter in the devtool git fetch test.
Signed-off-by: Stefan Herbrechtsmeier
---
Changes in v2:
- Add commit to check branch in selftest/devtool
meta/lib/oeqa/selftest/cases/devtool.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index d2b31af80e..a2b4d7f7d1 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -442,6 +442,7 @@ class DevtoolAddTests(DevtoolBase):
tempdir = tempfile.mkdtemp(prefix='devtoolqa')
self.track_for_cleanup(tempdir)
url = 'gitsm://git.yoctoproject.org/mraa'
+url_branch = '%s;branch=master' % url
checkrev = 'ae127b19a50aa54255e4330ccfdd9a5d058e581d'
testrecipe = 'mraa'
srcdir = os.path.join(tempdir, testrecipe)
@@ -462,7 +463,7 @@ class DevtoolAddTests(DevtoolBase):
checkvars = {}
checkvars['S'] = '${WORKDIR}/git'
checkvars['PV'] = '1.0+git${SRCPV}'
-checkvars['SRC_URI'] = url
+checkvars['SRC_URI'] = url_branch
checkvars['SRCREV'] = '${AUTOREV}'
self._test_recipe_contents(recipefile, checkvars, [])
# Try with revision and version specified
@@ -481,7 +482,7 @@ class DevtoolAddTests(DevtoolBase):
checkvars = {}
checkvars['S'] = '${WORKDIR}/git'
checkvars['PV'] = '1.5+git${SRCPV}'
-checkvars['SRC_URI'] = url
+checkvars['SRC_URI'] = url_branch
checkvars['SRCREV'] = checkrev
self._test_recipe_contents(recipefile, checkvars, [])
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159356):
https://lists.openembedded.org/g/openembedded-core/message/159356
Mute This Topic: https://lists.openembedded.org/mt/87585450/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 3/5] selftest/recipetool: Split tests into separate test classes
From: Stefan Herbrechtsmeier
Split tests into separate test classes to speed up individual test runs
by reducing the test setup to a minimum.
The pkgdata generation is only needed for the append tests and slow down
the other tests.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 17 ++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 8dc00ac10e..1aedc02b99 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -68,7 +68,7 @@ class RecipetoolBase(devtool.DevtoolBase):
return bbappendfile, result.output
-class RecipetoolTests(RecipetoolBase):
+class RecipetoolAppendTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
@@ -76,9 +76,8 @@ class RecipetoolTests(RecipetoolBase):
# Ensure we have the right data in shlibs/pkgdata
cls.logger.info('Running bitbake to generate pkgdata')
bitbake('-c packagedata base-files coreutils busybox
selftest-recipetool-appendfile')
-bb_vars = get_bb_vars(['COREBASE', 'BBPATH'])
+bb_vars = get_bb_vars(['COREBASE'])
cls.corebase = bb_vars['COREBASE']
-cls.bbpath = bb_vars['BBPATH']
def _try_recipetool_appendfile(self, testrecipe, destfile, newfile,
options, expectedlines, expectedfiles):
cmd = 'recipetool appendfile %s %s %s %s' % (self.templayerdir,
destfile, newfile, options)
@@ -332,6 +331,9 @@ class RecipetoolTests(RecipetoolBase):
filename = try_appendfile_wc('-w')
self.assertEqual(filename, recipefn.split('_')[0] + '_%.bbappend')
+
+class RecipetoolCreateTests(RecipetoolBase):
+
def test_recipetool_create(self):
# Try adding a recipe
tempsrc = os.path.join(self.tempdir, 'srctree')
@@ -518,6 +520,15 @@ class RecipetoolTests(RecipetoolBase):
inherits = ['pkgconfig', 'autotools']
self._test_recipe_contents(recipefile, checkvars, inherits)
+
+class RecipetoolTests(RecipetoolBase):
+
+@classmethod
+def setUpClass(cls):
+super(RecipetoolTests, cls).setUpClass()
+bb_vars = get_bb_vars(['BBPATH'])
+cls.bbpath = bb_vars['BBPATH']
+
def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
dstdir = basedstdir
self.assertTrue(os.path.exists(dstdir))
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159358):
https://lists.openembedded.org/g/openembedded-core/message/159358
Mute This Topic: https://lists.openembedded.org/mt/87585452/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 1/5] recipetool: Set master branch only as fallback
From: Stefan Herbrechtsmeier
The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
sets the branch= parameter too early to master and thereby breaks the
-B/--srcbranch option.
ERROR: branch= parameter and -B/--srcbranch option cannot both be specified -
use one or the other
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
scripts/lib/recipetool/create.py | 15 +++
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4f6e01c639..406c97f1c5 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -389,9 +389,6 @@ def reformat_git_uri(uri):
parms.update({('protocol', 'ssh')})
elif (scheme == "http" or scheme == 'https' or scheme == 'ssh') and
not ('protocol' in parms):
parms.update({('protocol', scheme)})
-# We assume 'master' branch if not set
-if not 'branch' in parms:
-parms.update({('branch', 'master')})
# Always append 'git://'
fUrl = bb.fetch2.encodeurl(('git', host, path, user, pswd, parms))
return fUrl
@@ -481,6 +478,9 @@ def create_recipe(args):
storeTagName = params['tag']
params['nobranch'] = '1'
del params['tag']
+# Assume 'master' branch if not set
+if scheme in ['git', 'gitsm'] and 'branch' not in params and
'nobranch' not in params:
+params['branch'] = 'master'
fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd,
params))
tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
@@ -530,10 +530,9 @@ def create_recipe(args):
# Remove HEAD reference point and drop remote prefix
get_branch = [x.split('/', 1)[1] for x in get_branch if not
x.startswith('origin/HEAD')]
if 'master' in get_branch:
-# If it is master, we do not need to append 'branch=master' as
this is default.
# Even with the case where get_branch has multiple objects, if
'master' is one
# of them, we should default take from 'master'
-srcbranch = ''
+srcbranch = 'master'
elif len(get_branch) == 1:
# If 'master' isn't in get_branch and get_branch contains only
ONE object, then store result into 'srcbranch'
srcbranch = get_branch[0]
@@ -546,8 +545,8 @@ def create_recipe(args):
# Since we might have a value in srcbranch, we need to
# recontruct the srcuri to include 'branch' in params.
scheme, network, path, user, passwd, params =
bb.fetch2.decodeurl(srcuri)
-if srcbranch:
-params['branch'] = srcbranch
+if scheme in ['git', 'gitsm']:
+params['branch'] = srcbranch or 'master'
if storeTagName and scheme in ['git', 'gitsm']:
# Check srcrev using tag and check validity of the tag
@@ -606,7 +605,7 @@ def create_recipe(args):
splitline = line.split()
if len(splitline) > 1:
if splitline[0] == 'origin' and
scriptutils.is_src_url(splitline[1]):
-srcuri = reformat_git_uri(splitline[1])
+srcuri = reformat_git_uri(splitline[1]) +
';branch=master'
srcsubdir = 'git'
break
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159355):
https://lists.openembedded.org/g/openembedded-core/message/159355
Mute This Topic: https://lists.openembedded.org/mt/87585448/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 4/5] selftest/recipetool: Add tests for branch parameter and srcbranch option
From: Stefan Herbrechtsmeier
The recipetool support two ways to pass a branch and fallback to master
if no branch is defined. Add tests for default branch, branch parameter
and srcbranch option.
Signed-off-by: Stefan Herbrechtsmeier
---
This commit changes the test repository from matchbox-terminal to
matchbox-keyboard because matchbox-terminal has only one master branch.
Maybe the test should use a special test repository.
(no changes since v1)
meta/lib/oeqa/selftest/cases/recipetool.py | 38 +++---
1 file changed, 27 insertions(+), 11 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1aedc02b99..b77a2712f5 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -350,7 +350,7 @@ class RecipetoolCreateTests(RecipetoolBase):
checkvars['SRC_URI[sha256sum]'] =
'2e6a401cac9024db2288297e3be1a8ab60e7401ba8e91225218aaf4a27e82a07'
self._test_recipe_contents(recipefile, checkvars, [])
-def test_recipetool_create_git(self):
+def test_recipetool_create_autotools(self):
if 'x11' not in get_bb_var('DISTRO_FEATURES'):
self.skipTest('Test requires x11 as distro feature')
# Ensure we have the right data in shlibs/pkgdata
@@ -359,7 +359,7 @@ class RecipetoolCreateTests(RecipetoolBase):
tempsrc = os.path.join(self.tempdir, 'srctree')
os.makedirs(tempsrc)
recipefile = os.path.join(self.tempdir, 'libmatchbox.bb')
-srcuri = 'git://git.yoctoproject.org/libmatchbox;branch=master'
+srcuri = 'git://git.yoctoproject.org/libmatchbox'
result = runCmd(['recipetool', 'create', '-o', recipefile, srcuri +
";rev=9f7cf8895ae2d39c465c04cc78e918c157420269", '-x', tempsrc])
self.assertTrue(os.path.isfile(recipefile), 'recipetool did not create
recipe file; output:\n%s' % result.output)
checkvars = {}
@@ -367,7 +367,7 @@ class RecipetoolCreateTests(RecipetoolBase):
checkvars['LIC_FILES_CHKSUM'] =
'file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34'
checkvars['S'] = '${WORKDIR}/git'
checkvars['PV'] = '1.11+git${SRCPV}'
-checkvars['SRC_URI'] = srcuri
+checkvars['SRC_URI'] = srcuri + ';branch=master'
checkvars['DEPENDS'] = set(['libcheck', 'libjpeg-turbo', 'libpng',
'libx11', 'libxext', 'pango'])
inherits = ['autotools', 'pkgconfig']
self._test_recipe_contents(recipefile, checkvars, inherits)
@@ -506,19 +506,35 @@ class RecipetoolCreateTests(RecipetoolBase):
inherits = ['setuptools3']
self._test_recipe_contents(recipefile, checkvars, inherits)
-def test_recipetool_create_git_http(self):
+def _test_recipetool_create_git(self, srcuri, branch=None):
# Basic test to check http git URL mangling works
temprecipe = os.path.join(self.tempdir, 'recipe')
os.makedirs(temprecipe)
-recipefile = os.path.join(temprecipe, 'matchbox-terminal_git.bb')
-srcuri = 'http://git.yoctoproject.org/git/matchbox-terminal'
-result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+name = srcuri.split(';')[0].split('/')[-1]
+recipefile = os.path.join(temprecipe, name + '_git.bb')
+options = ' -B %s' % branch if branch else ''
+result = runCmd('recipetool create -o %s%s "%s"' % (temprecipe,
options, srcuri))
self.assertTrue(os.path.isfile(recipefile))
checkvars = {}
-checkvars['LICENSE'] = set(['GPLv2'])
-checkvars['SRC_URI'] =
'git://git.yoctoproject.org/git/matchbox-terminal;protocol=http;branch=master'
-inherits = ['pkgconfig', 'autotools']
-self._test_recipe_contents(recipefile, checkvars, inherits)
+checkvars['SRC_URI'] = srcuri
+for scheme in ['http', 'https']:
+if srcuri.startswith(scheme + ":"):
+checkvars['SRC_URI'] = 'git%s;protocol=%s' %
(srcuri[len(scheme):], scheme)
+if ';branch=' not in srcuri:
+checkvars['SRC_URI'] += ';branch=' + (branch or 'master')
+self._test_recipe_contents(recipefile, checkvars, [])
+
+def test_recipetool_create_git_http(self):
+
self._test_recipetool_create_git('http://git.yoctoproject.org/git/matchbox-keyboard')
+
+def test_recipetool_create_git_srcuri_master(self):
+
self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard;branch=master')
+
+def test_recipetool_create_git_srcuri_branch(self):
+
self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard;branch=matchbox-keyboard-0-1')
+
+def test_recipetool_create_git_srcbranch(self):
+
self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard',
'matchbox-keyboard-0-1')
class RecipetoolTests(RecipetoolBase):
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links:
[OE-core] [PATCH v2 5/5] selftest: devtool: Separate common functions and devtool sstate setup into two classes
From: Stefan Herbrechtsmeier
The selftest recipetool base class reuse the selftest devtool base
class. Thereby the selftest devtool base class setup its own devtool
sstate and the selftest recipetool classes trigger the build of recipes.
This leads to the problem that the build artifacts doesn't reach the
persistent sstate cache and rebuild on every selftest run.
Move the common selftest devtool functions into its own class and use
the sstate cache in the recipetool tests.
Signed-off-by: Stefan Herbrechtsmeier
---
(no changes since v1)
meta/lib/oeqa/selftest/cases/devtool.py| 45 +-
meta/lib/oeqa/selftest/cases/recipetool.py | 4 +-
2 files changed, 28 insertions(+), 21 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index a2b4d7f7d1..23d55903fb 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -80,32 +80,15 @@ def tearDownModule():
bb.utils.edit_bblayers_conf(bblayers_conf, None, None,
bblayers_edit_cb)
shutil.rmtree(templayerdir)
-class DevtoolBase(OESelftestTestCase):
-
-@classmethod
-def setUpClass(cls):
-super(DevtoolBase, cls).setUpClass()
-bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
-cls.original_sstate = bb_vars['SSTATE_DIR']
-cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
-cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
-cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
-% cls.original_sstate)
-
-@classmethod
-def tearDownClass(cls):
-cls.logger.debug('Deleting devtool sstate cache on %s' %
cls.devtool_sstate)
-runCmd('rm -rf %s' % cls.devtool_sstate)
-super(DevtoolBase, cls).tearDownClass()
+class DevtoolTestCase(OESelftestTestCase):
def setUp(self):
"""Test case setup function"""
-super(DevtoolBase, self).setUp()
+super(DevtoolTestCase, self).setUp()
self.workspacedir = os.path.join(self.builddir, 'workspace')
self.assertTrue(not os.path.exists(self.workspacedir),
'This test cannot be run with a workspace directory '
'under the build directory')
-self.append_config(self.sstate_conf)
def _check_src_repo(self, repo_dir):
"""Check srctree git repository"""
@@ -236,6 +219,30 @@ class DevtoolBase(OESelftestTestCase):
return filelist
+class DevtoolBase(DevtoolTestCase):
+
+@classmethod
+def setUpClass(cls):
+super(DevtoolBase, cls).setUpClass()
+bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
+cls.original_sstate = bb_vars['SSTATE_DIR']
+cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
+cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
+cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
+% cls.original_sstate)
+
+@classmethod
+def tearDownClass(cls):
+cls.logger.debug('Deleting devtool sstate cache on %s' %
cls.devtool_sstate)
+runCmd('rm -rf %s' % cls.devtool_sstate)
+super(DevtoolBase, cls).tearDownClass()
+
+def setUp(self):
+"""Test case setup function"""
+super(DevtoolBase, self).setUp()
+self.append_config(self.sstate_conf)
+
+
class DevtoolTests(DevtoolBase):
def test_create_workspace(self):
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index b77a2712f5..439e41597c 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -25,7 +25,7 @@ def tearDownModule():
runCmd('rm -rf %s' % templayerdir)
-class RecipetoolBase(devtool.DevtoolBase):
+class RecipetoolBase(devtool.DevtoolTestCase):
def setUpLocal(self):
super(RecipetoolBase, self).setUpLocal()
@@ -72,7 +72,7 @@ class RecipetoolAppendTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
-super(RecipetoolTests, cls).setUpClass()
+super(RecipetoolAppendTests, cls).setUpClass()
# Ensure we have the right data in shlibs/pkgdata
cls.logger.info('Running bitbake to generate pkgdata')
bitbake('-c packagedata base-files coreutils busybox
selftest-recipetool-appendfile')
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159357):
https://lists.openembedded.org/g/openembedded-core/message/159357
Mute This Topic: https://lists.openembedded.org/mt/87585451/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback
Am 08.12.2021 um 09:32 schrieb Alexander Kanavin via lists.openembedded.org:
I haven't looked at this, but probably the test could be adjusted to
first build the needed parts with only the original cache?
This needs a rework of the tests because the private sstate cache is set
in the test setup and we need to know the dependencies of the test to
build those manual.
On Wed, 8 Dec 2021 at 09:27, Stefan Herbrechtsmeier
<mailto:stefan.herbrechtsmeier-...@weidmueller.com>> wrote:
Am 08.12.2021 um 09:10 schrieb Alexander Kanavin:
> On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier
> mailto:stefan.herbrechtsmeier-...@weidmueller.com>
> <mailto:stefan.herbrechtsmeier-...@weidmueller.com
<mailto:stefan.herbrechtsmeier-...@weidmueller.com>>> wrote:
>
> I will update my patch series but need some time to test it
locally.
> The
> devtool test build a lot of recipe on every run because it use a
> private
> sstate cache. How does the autobuilder overcome this problem?
>
>
> If you don't rebase the branch used for testing, and set the
cache to a
> directory outside of build directory (e.g. /home/stefan/sstate )
it will
> reuse that cache between selftest invocations.
The devtool test use its own PRIVATE sstate cache:
def setUpClass(cls):
...
cls.original_sstate = bb_vars['SSTATE_DIR']
cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
% cls.original_sstate)
And remove it after the test run:
def tearDownClass(cls):
...
runCmd('rm -rf %s' % cls.devtool_sstate)
Even a keep builddir doesn't help to keep the private sstate cache.
> The autobuilder uses a single writeable cache for all builds and
> builders, mounted as read-write NFS partition.
This doesn't help in this case. The devtool test clean parts of the
sstate cache and therefore use its private sstate cache and the
original
sstate cache as mirror. This is a problem if a test triggers a build of
dependencies.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159347):
https://lists.openembedded.org/g/openembedded-core/message/159347
Mute This Topic: https://lists.openembedded.org/mt/87536000/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback
Am 08.12.2021 um 09:10 schrieb Alexander Kanavin:
On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier
<mailto:stefan.herbrechtsmeier-...@weidmueller.com>> wrote:
I will update my patch series but need some time to test it locally.
The
devtool test build a lot of recipe on every run because it use a
private
sstate cache. How does the autobuilder overcome this problem?
If you don't rebase the branch used for testing, and set the cache to a
directory outside of build directory (e.g. /home/stefan/sstate ) it will
reuse that cache between selftest invocations.
The devtool test use its own PRIVATE sstate cache:
def setUpClass(cls):
...
cls.original_sstate = bb_vars['SSTATE_DIR']
cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
% cls.original_sstate)
And remove it after the test run:
def tearDownClass(cls):
...
runCmd('rm -rf %s' % cls.devtool_sstate)
Even a keep builddir doesn't help to keep the private sstate cache.
The autobuilder uses a single writeable cache for all builds and
builders, mounted as read-write NFS partition.
This doesn't help in this case. The devtool test clean parts of the
sstate cache and therefore use its private sstate cache and the original
sstate cache as mirror. This is a problem if a test triggers a build of
dependencies.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159344):
https://lists.openembedded.org/g/openembedded-core/message/159344
Mute This Topic: https://lists.openembedded.org/mt/87536000/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback
Hi Richard, Am 07.12.2021 um 16:45 schrieb Richard Purdie: On Mon, 2021-12-06 at 09:04 +0100, Stefan Herbrechtsmeier wrote: From: Stefan Herbrechtsmeier The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)' sets the branch= parameter too early to master and thereby breaks the -B/--srcbranch option. ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other Signed-off-by: Stefan Herbrechtsmeier --- scripts/lib/recipetool/create.py | 15 +++ 1 file changed, 7 insertions(+), 8 deletions(-) I think something in this series is causing: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2926/steps/14/logs/stdio Sorry I missed the adaption of the test test_devtool_add_fetch_git. It doesn't expect a "branch=master" which is wrong. - gitsm://git.yoctoproject.org/mraa;branch=master ? -- + gitsm://git.yoctoproject.org/mraa I will update my patch series but need some time to test it locally. The devtool test build a lot of recipe on every run because it use a private sstate cache. How does the autobuilder overcome this problem? (the other selftest builds also failed) What do you mean by this? Regards Stefan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#159341): https://lists.openembedded.org/g/openembedded-core/message/159341 Mute This Topic: https://lists.openembedded.org/mt/87536000/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 4/4] selftest: devtool: Separate common functions and devtool sstate setup into two classes
From: Stefan Herbrechtsmeier
The selftest recipetool base class reuse the selftest devtool base
class. Thereby the selftest devtool base class setup its own devtool
sstate and the selftest recipetool classes trigger the build of recipes.
This leads to the problem that the build artifacts doesn't reach the
persistent sstate cache and rebuild on every selftest run.
Move the common selftest devtool functions into its own class and use
the sstate cache in the recipetool tests.
Signed-off-by: Stefan Herbrechtsmeier
---
meta/lib/oeqa/selftest/cases/devtool.py| 45 +-
meta/lib/oeqa/selftest/cases/recipetool.py | 4 +-
2 files changed, 28 insertions(+), 21 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py
b/meta/lib/oeqa/selftest/cases/devtool.py
index d2b31af80e..96f40ac9a0 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -80,32 +80,15 @@ def tearDownModule():
bb.utils.edit_bblayers_conf(bblayers_conf, None, None,
bblayers_edit_cb)
shutil.rmtree(templayerdir)
-class DevtoolBase(OESelftestTestCase):
-
-@classmethod
-def setUpClass(cls):
-super(DevtoolBase, cls).setUpClass()
-bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
-cls.original_sstate = bb_vars['SSTATE_DIR']
-cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
-cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
-cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
-% cls.original_sstate)
-
-@classmethod
-def tearDownClass(cls):
-cls.logger.debug('Deleting devtool sstate cache on %s' %
cls.devtool_sstate)
-runCmd('rm -rf %s' % cls.devtool_sstate)
-super(DevtoolBase, cls).tearDownClass()
+class DevtoolTestCase(OESelftestTestCase):
def setUp(self):
"""Test case setup function"""
-super(DevtoolBase, self).setUp()
+super(DevtoolTestCase, self).setUp()
self.workspacedir = os.path.join(self.builddir, 'workspace')
self.assertTrue(not os.path.exists(self.workspacedir),
'This test cannot be run with a workspace directory '
'under the build directory')
-self.append_config(self.sstate_conf)
def _check_src_repo(self, repo_dir):
"""Check srctree git repository"""
@@ -236,6 +219,30 @@ class DevtoolBase(OESelftestTestCase):
return filelist
+class DevtoolBase(DevtoolTestCase):
+
+@classmethod
+def setUpClass(cls):
+super(DevtoolBase, cls).setUpClass()
+bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
+cls.original_sstate = bb_vars['SSTATE_DIR']
+cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
+cls.sstate_conf = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
+cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
+% cls.original_sstate)
+
+@classmethod
+def tearDownClass(cls):
+cls.logger.debug('Deleting devtool sstate cache on %s' %
cls.devtool_sstate)
+runCmd('rm -rf %s' % cls.devtool_sstate)
+super(DevtoolBase, cls).tearDownClass()
+
+def setUp(self):
+"""Test case setup function"""
+super(DevtoolBase, self).setUp()
+self.append_config(self.sstate_conf)
+
+
class DevtoolTests(DevtoolBase):
def test_create_workspace(self):
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py
b/meta/lib/oeqa/selftest/cases/recipetool.py
index b77a2712f5..439e41597c 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -25,7 +25,7 @@ def tearDownModule():
runCmd('rm -rf %s' % templayerdir)
-class RecipetoolBase(devtool.DevtoolBase):
+class RecipetoolBase(devtool.DevtoolTestCase):
def setUpLocal(self):
super(RecipetoolBase, self).setUpLocal()
@@ -72,7 +72,7 @@ class RecipetoolAppendTests(RecipetoolBase):
@classmethod
def setUpClass(cls):
-super(RecipetoolTests, cls).setUpClass()
+super(RecipetoolAppendTests, cls).setUpClass()
# Ensure we have the right data in shlibs/pkgdata
cls.logger.info('Running bitbake to generate pkgdata')
bitbake('-c packagedata base-files coreutils busybox
selftest-recipetool-appendfile')
--
2.20.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159211):
https://lists.openembedded.org/g/openembedded-core/message/159211
Mute This Topic: https://lists.openembedded.org/mt/87536004/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
