[OE-core][kirkstone][PATCH] go : CVE-2022-27664

[Teoh, Jay Shen]

From: Teoh Jay Shen 

Upstream-Status: Backport 
[https://github.com/golang/go/commit/5bc9106458fc07851ac324a4157132a91b1f3479]
Signed-off-by: Teoh Jay Shen 
---
 meta/recipes-devtools/go/go-1.17.13.inc   |   1 +
 .../go/go-1.18/CVE-2022-27664.patch   | 102 ++
 2 files changed, 103 insertions(+)
 create mode 100644 meta/recipes-devtools/go/go-1.18/CVE-2022-27664.patch

diff --git a/meta/recipes-devtools/go/go-1.17.13.inc 
b/meta/recipes-devtools/go/go-1.17.13.inc
index 95d0fb7e98..b18de66f42 100644
--- a/meta/recipes-devtools/go/go-1.17.13.inc
+++ b/meta/recipes-devtools/go/go-1.17.13.inc
@@ -16,6 +16,7 @@ SRC_URI += "\
 file://0009-Revert-cmd-go-make-sure-CC-and-CXX-are-absolute.patch \
 file://0001-exec.go-do-not-write-linker-flags-into-buildids.patch \
 file://0001-src-cmd-dist-buildgo.go-do-not-hardcode-host-compile.patch \
+file://CVE-2022-27664.patch \
 "
 SRC_URI[main.sha256sum] = 
"a1a48b23afb206f95e7bbaa9b898d965f90826f6f1d1fc0c1d784ada0cd300fd"
 
diff --git a/meta/recipes-devtools/go/go-1.18/CVE-2022-27664.patch 
b/meta/recipes-devtools/go/go-1.18/CVE-2022-27664.patch
new file mode 100644
index 00..fba4f054ee
--- /dev/null
+++ b/meta/recipes-devtools/go/go-1.18/CVE-2022-27664.patch
@@ -0,0 +1,102 @@
+From 5bc9106458fc07851ac324a4157132a91b1f3479 Mon Sep 17 00:00:00 2001
+From: Damien Neil 
+Date: Mon, 22 Aug 2022 16:21:02 -0700
+Subject: [PATCH] [release-branch.go1.18] net/http: update bundled
+ golang.org/x/net/http2
+
+Disable cmd/internal/moddeps test, since this update includes PRIVATE
+track fixes.
+
+Fixes CVE-2022-27664
+Fixes #53977
+For #54658.
+
+Change-Id: I84b0b8f61e49e15ef55ef8d738730107a3cf849b
+Reviewed-on: 
https://team-review.git.corp.google.com/c/golang/go-private/+/1554415
+Reviewed-by: Roland Shoemaker 
+Reviewed-by: Tatiana Bradley 
+Reviewed-on: https://go-review.googlesource.com/c/go/+/428635
+Reviewed-by: Tatiana Bradley 
+Run-TryBot: Michael Knyszek 
+TryBot-Result: Gopher Robot 
+Reviewed-by: Carlos Amedee 
+
+Upstream-Status: Backport
+CVE: CVE-2022-27664 
+
+Reference to upstream patch: 
https://github.com/golang/go/commit/5bc9106458fc07851ac324a4157132a91b1f3479
+Signed-off-by: Teoh Jay Shen 
+---
+ src/cmd/internal/moddeps/moddeps_test.go |  2 ++
+ src/net/http/h2_bundle.go| 21 +
+ 2 files changed, 15 insertions(+), 8 deletions(-)
+
+diff --git a/src/cmd/internal/moddeps/moddeps_test.go 
b/src/cmd/internal/moddeps/moddeps_test.go
+index 56c3b2585c..3306e29431 100644
+--- a/src/cmd/internal/moddeps/moddeps_test.go
 b/src/cmd/internal/moddeps/moddeps_test.go
+@@ -34,6 +34,8 @@ import (
+ // See issues 36852, 41409, and 43687.
+ // (Also see golang.org/issue/27348.)
+ func TestAllDependencies(t *testing.T) {
++  t.Skip("TODO(#53977): 1.18.5 contains unreleased changes from vendored 
modules")
++
+   goBin := testenv.GoToolPath(t)
+ 
+   // Ensure that all packages imported within GOROOT
+diff --git a/src/net/http/h2_bundle.go b/src/net/http/h2_bundle.go
+index bb82f24585..1e78f6cdb9 100644
+--- a/src/net/http/h2_bundle.go
 b/src/net/http/h2_bundle.go
+@@ -3384,10 +3384,11 @@ func (s http2SettingID) String() string {
+ // name (key). See httpguts.ValidHeaderName for the base rules.
+ //
+ // Further, http2 says:
+-//   "Just as in HTTP/1.x, header field names are strings of ASCII
+-//   characters that are compared in a case-insensitive
+-//   fashion. However, header field names MUST be converted to
+-//   lowercase prior to their encoding in HTTP/2. "
++//
++//"Just as in HTTP/1.x, header field names are strings of ASCII
++//characters that are compared in a case-insensitive
++//fashion. However, header field names MUST be converted to
++//lowercase prior to their encoding in HTTP/2. "
+ func http2validWireHeaderFieldName(v string) bool {
+   if len(v) == 0 {
+   return false
+@@ -3578,8 +3579,8 @@ func (s *http2sorter) SortStrings(ss []string) {
+ // validPseudoPath reports whether v is a valid :path pseudo-header
+ // value. It must be either:
+ //
+-// *) a non-empty string starting with '/'
+-// *) the string '*', for OPTIONS requests.
++//*) a non-empty string starting with '/'
++//*) the string '*', for OPTIONS requests.
+ //
+ // For now this is only used a quick check for deciding when to clean
+ // up Opaque URLs before sending requests from the Transport.
+@@ -5053,6 +5054,9 @@ func (sc *http2serverConn) 
startGracefulShutdownInternal() {
+ func (sc *http2serverConn) goAway(code http2ErrCode) {
+   sc.serveG.check()
+   if sc.inGoAway {
++  if sc.goAwayCode == http2ErrCodeNo {
++  sc.goAwayCode = code
++  }
+   return
+   }
+   sc.inGoAway = true
+@@ -6265,8 +6269,9 @@ func (rws *http2responseWriterState) writeChunk(p 
[]byte) (n int, err error) {
+ // prior to the headers being written. If the set of trailers is fixed
+ // or known 

Re: [OE-core] [Openembedded-architecture] Adding more information to the SBOM

[Mark Hatle]

On 9/14/22 9:56 AM, Joshua Watt wrote:

On Wed, Sep 14, 2022 at 9:16 AM Marta Rybczynska  wrote:


Dear all,
(cross-posting to oe-core and *-architecture)
In the last months, we have worked in Oniro on using the create-spdx
class for both IP compliance and security.

During this work, Alberto Pianon has found that some information is
missing from the SBOM and it does not contain enough for Software
Composition Analysis. The main missing point is the relation between
the actual upstream sources and the final binaries (create-spdx uses
composite sources).


I believe we map the binaries to the source code from the -dbg
packages; is the premise that this is insufficient? Can you elaborate
more on why that is, I don't quite understand. The debug sources are
(basically) what we actually compiled (e.g. post-do_patch) to produce
the binary, and you can in turn follow these back to the upstream
sources with the downloadLocation property.


When I last looked at this, it was critical that the analysis be:

binary -> patched & configured source (dbg package) -> how the sources were 
constructed.


As Joshua said above.  I believe all of the information is present for this as 
you can tie the binary (through debug symbols) back to the debug package.. and 
the source of the debug package back to the sources that constructed it via 
heuristics.  (If you enable the git patch mechanism.  It should even be possible 
to use git blame to find exactly what upstreams constructed the patched sources.


For generated content, it's more difficult -- but for those items usually there 
is a header which indicates what generated the content so other heuristics can 
be used.




Alberto has worked on how to obtain the missing data and now has a
POC. This POC provides full source-to-binary tracking of Yocto builds
through a couple of scripts (intended to be transformed into a new
bbclass at a later stage). The goal is to add the missing pieces of
information in order to get a "real" SBOM from Yocto, which should, at
a minimum:


Please be a little careful with the wording; SBoMs have a lot of uses,
and many of them we can satisfy with what we currently generate; it
may not do the exact use case you are looking for, but that doesn't
mean it's not a "real" SBoM :)



- carefully describe what is found in a final image (i.e. binary files
and their dependencies), since that is what is actually distributed
and goes into the final product;
- describe how such binary files have been generated and where they
come from (i.e. upstream sources, including patches and other stuff
added from meta-layers); provenance is important for a number of
reasons related to IP Compliance and security.


Full compliance will require binaries mapped to patched source to upstream 
sources _AND_ the instructions (layer/recipe/configuration) used to build them. 
 But it's up to the local legal determination to figure out 'how far you really 
need to go', vs just "here are the layers I used to build my project".)



The aim is to become able to:

- map binaries to their corresponding upstream source packages (and
not to the "internal" source packages created by recipes by combining
multiple upstream sources and patches)
- map binaries to the source files that have been actually used to
build them - which usually are a small subset of the whole source
package

With respect to IP compliance, this would allow to, among other things:

- get the real license text for each binary file, by getting the
license of the specific source files it has been generated from
(provided by Fossology, for instance), - and not the main license
stated in the corresponding recipe (which may be as confusing as
GPL-2.0-or-later & LGPL-2.1-or-later & BSD-3-Clause & BSD-4-Clause, or
even worse)


IIUC this is the difference between the "Declared" license and the
"Concluded" license. You can report both, and I think
create-spdx.bbclass can currently do this with its rudimentary source
license scanning. You really do want both and it's a great way to make
sure that the "Declared" license (that is the license in the recipe)
reflects the reality of the source code.


And the thing to keep in mind is that in a given package the "Declared" is 
usually what a LICENSE file or header says.  But the "Concluded" has levels of 
quality behind them.  The first level of quality is "Declared".  The next level 
is automation (something like fossology), the next level is human reviewed, and 
the highest level is "lawyer reviewed".


So being able to inject SPDX information with Concluded values for evaluation 
and track the 'quality level' has always been something I wanted to do, but 
never had time.


At the time, my idea was a database (and/or bbappend) for each component that 
would included pre-processed SPDX data for each recipe.  This data would run 
through a validation step to show it actually matches the patched sources.  (If 
any file checksums do NOT match, then they would be flagged for 

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-core@lists.openembedded.org  c...@lists.openembedded.org> On Behalf Of Alexander Kanavin
> Sent: den 14 september 2022 09:42
> To: ChenQi 
> Cc: OE-core ; Alexander Kanavin
> 
> Subject: Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check
> that TEMPLATECONF is valid
> 
> On Wed, 14 Sept 2022 at 10:17, ChenQi  wrote:
> > The codes are in: https://github.com/WindRiver-Labs/wrlinux-x
> >
> > e.g.
> >
> > ./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)
> >
> > ./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all
> > supported layers, together with their needed tarballs/git repos)
> >
> > ./wrlinux-x/setup.sh (this pulls down the default list of layers)
> >
> > ./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list
> > of layers + meta-xxx and its dependencies)
> >
> > And the final list of layers are in
> > /config/bblayers.conf.sample, which will be updated if
> > needed when re-running the setup.sh script.
> 
> May I suggest that the tool skips the bblayers.conf.sample step
> altogether, and simply writes the desired layers directly into
> bblayers.conf? It can do this with 'bitbake-layers add-layer'. Is
> there a need for doing the intermediate step of generating the
> template?

I know you have suggested this to me too as we use a similar setup to 
generate the bblayers.conf.sample file based on the layers that repo 
has checked out in our environment, and now I realized why this is not 
as simple as it seems. The problem (at least in our case) is that we 
run as a wrapper around oe-init-build-env. This means that before the 
wrapper sources oe-init-build-env, the build directory and thus the 
conf directory inside it (where the bblayers.conf file is supposed to 
go) may not exist. And after it has sourced oe-init-build-env it is 
too late as the bblayers.conf file needs to exists before. So to create 
the bblayers.conf file, the wrapper would have to duplicate all the 
non-trivial code in oe-setup-builddir that determines what the build 
directory should be called. And while this of course is possible to 
do, it means duplicating code that is internal to bitbake, and risking 
missing important changes to the upstream code in the future. Thus it 
is much easier and safer to generate the template files and let 
oe-init-build-env do its job.

> 
> Just to be clear where I am coming from: anything in TEMPLATECONF must
> not be auto-generated during build setup, and must be under version
> control. We are trying to standardize layer setup and configuration
> handling, and this would be a necessary step towards that.

Where I (and apparently WindRiver) are coming from, the decision on what 
layers are part of the configuration is not something a specific layer 
is responsible for. I.e., in our setup we fetch the layers we want for 
a build, write the list of layers to bblayers.conf.sample and source 
oe-init-build-env. With your setup, if I want a project with the two 
layers meta-a and meta-b, I would also have to create a layer meta-x 
where I create a static bblayers.conf.sample file that lists these 
three layers. Then, if another project wants to use meta-a and meta-c, 
they would have to add a second static bblayers.conf.sample file in 
meta-x that lists those three layers. Basically every time there is a 
new combination of layers fetched, there would also have to be a 
duplicate of the information in a static bblayers.conf.sample file 
somewhere. I am probably missing something here, but to me this seems 
suboptimal. If the list of layers in the static file had been used to 
fetch the layers, I would better understand it, but then the problem 
becomes that the list is now in one of the arbitrary layers that are 
supposed to be fetched.

For reference, today when one of our teams start a new development 
project, what they do is they create a specific layer (e.g., meta-foo) 
for that project where they can do all their development work and 
experimentation until it is time for them to integrate to our main 
platform (referred to as cfp/cfp.xml below). Then they create a repo 
manifest that looks like: 



  

  


in our manifest repository. This simple manifest is all the project 
team needs to create to build with their project specific layer. It 
also means that the only input to our Jenkins jobs to build for this 
project (instead of the main platform) is the name of the project's 
repo manifest. It also means that if we make any changes to the main 
platform, e.g., add a new layer (which is a one line change to the 
cfp.xml manifest), all projects will automatically get and use that 
new layer without having to make any changes to all the project 
manifests.

> 
> Alex

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170677): 
https://lists.openembedded.org/g/openembedded-core/message/170677
Mute This Topic: 

[OE-core] [PATCH] curl: Add missing perl modules in ptest rdepends

[Khem Raj]

These modules outside of perl-modules are necessary to run curl-ptests

Signed-off-by: Khem Raj 
---
 meta/recipes-support/curl/curl_7.85.0.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-support/curl/curl_7.85.0.bb 
b/meta/recipes-support/curl/curl_7.85.0.bb
index 3b55830415..ad6a5175bc 100644
--- a/meta/recipes-support/curl/curl_7.85.0.bb
+++ b/meta/recipes-support/curl/curl_7.85.0.bb
@@ -100,7 +100,8 @@ do_install_ptest() {
cp -rf ${D}${bindir}/curl-config ${D}${PTEST_PATH}
 }
 
-RDEPENDS:${PN}-ptest += "bash perl-modules"
+RDEPENDS:${PN}-ptest += "bash perl-modules perl-module-time-hires 
perl-module-digest-md5 \
+ perl-module-digest perl-module-ipc-open2"
 
 PACKAGES =+ "lib${BPN}"
 
-- 
2.37.3


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170676): 
https://lists.openembedded.org/g/openembedded-core/message/170676
Mute This Topic: https://lists.openembedded.org/mt/93690031/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v3 2/2] nfs-ganesha: Replace unfs3 with nfs-ganesha

[Adrian Freihofer]

Hi Paulo

Is it possible to do the transition from unfsd to nfs-ganesha step by
step?

   1. Add nfs-ganesha
   2. Refactor runqemu-export-rootfs to work with ganesha.nfsd instead
  of unfsd.
   3. Remove unfs3

My conclusion so far is that just removing the unfs3 recipe would break
runqemu-export-rootfs and some features of runqemu. Therefore I would
suggest that you send a V4 which only adds nfs-ganesha as a first step.
Note: There is also a typo: The "-D" from USE_ACL_MAPPING=OFF is
missing.

Regards,
Adrian




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170675): 
https://lists.openembedded.org/g/openembedded-core/message/170675
Mute This Topic: https://lists.openembedded.org/mt/93311086/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Adding more information to the SBOM

[Joshua Watt]

On Wed, Sep 14, 2022 at 12:10 PM Alberto Pianon  wrote:
>
> Hi Joshua,
>
> nice to meet you!
>
> I'm new to this list, and I've always approached Yocto just from the
> "IP compliance side", so I may miss important pieces of information.
> That
> is why Marta encouraged me and is helping me to ask community feedback.
>
> Il 2022-09-14 16:56 Joshua Watt ha scritto:
> > On Wed, Sep 14, 2022 at 9:16 AM Marta Rybczynska 
> > wrote:
> >>
> >> Dear all,
> >> (cross-posting to oe-core and *-architecture)
> >> In the last months, we have worked in Oniro on using the create-spdx
> >> class for both IP compliance and security.
> >>
> >> During this work, Alberto Pianon has found that some information is
> >> missing from the SBOM and it does not contain enough for Software
> >> Composition Analysis. The main missing point is the relation between
> >> the actual upstream sources and the final binaries (create-spdx uses
> >> composite sources).
> >
> > I believe we map the binaries to the source code from the -dbg
> > packages; is the premise that this is insufficient? Can you elaborate
> > more on why that is, I don't quite understand. The debug sources are
> > (basically) what we actually compiled (e.g. post-do_patch) to produce
> > the binary, and you can in turn follow these back to the upstream
> > sources with the downloadLocation property.
>
> This was also my assumption at the beginning. But then I found that
> there
> are recipes with multiple upstream sources, which may be combined/mixed
> together in recipes' WORKDIR. For instance this one:
>
> https://git.yoctoproject.org/meta-virtualization/tree/recipes-networking/cni/cni_git.bb
>
> SRC_URI = "\
> 
> git://github.com/containernetworking/cni.git;branch=main;name=cni;protocol=https
> \
>
> git://github.com/containernetworking/plugins.git;branch=release-1.1;destsuffix=${S}/src/github.com/containernetworking/plugins;name=plugins;protocol=https
> \
>
> git://github.com/flannel-io/cni-plugin;branch=main;name=flannel_plugin;protocol=https;destsuffix=${S}/src/github.com/containernetworking/plugins/plugins/meta/flannel
> \
> "
>
> (The third source is unpacked in a subdir of the second one)
>
>  From here I discovered that we can't assume that the first non-local URI
> is the downloadLocation for all source files, because it is not always
> the case.

This is true, but I think that's more of a problem with the inability
to express multiple download locations in the SPDX, not that we don't
have all the source when we generate the SPDX, correct? I _beleive_
the -dbg package still contains all the source code from all three
URLs?

>
> Moreover, in the context of our project we also needed to find the
> upstream
> sources also for local patches, scripts, etc. added by recipes (i.e. the
> corresponding layers' repos).

Ok, so this makes me wonder: If we implement the better source
extraction in OE core, does that help this problem? Is the primary
problem that you want the unpatched upstream source code files instead
of the patched ones, or is it some other problem?

AFAIK, the -dbg package contains the source code we actually
compiled. so I have a hard time understanding what's "incorrect"
(or not ideal) about referencing it; but I think I'm missing something
important :)

>
> >
> >>
> >> Alberto has worked on how to obtain the missing data and now has a
> >> POC. This POC provides full source-to-binary tracking of Yocto builds
> >> through a couple of scripts (intended to be transformed into a new
> >> bbclass at a later stage). The goal is to add the missing pieces of
> >> information in order to get a "real" SBOM from Yocto, which should, at
> >> a minimum:
> >
> > Please be a little careful with the wording; SBoMs have a lot of uses,
> > and many of them we can satisfy with what we currently generate; it
> > may not do the exact use case you are looking for, but that doesn't
> > mean it's not a "real" SBoM :)
>
> You are right, sorry! "real" is meant in the context of our project,
> where we need to make our Fossology Audit Team work on "original"
> upstream source packages/repos, for a number of reasons (the main being
> that in Oniro project we have a complex build matrix with a lot of
> available target machines and quite a number of different overrides
> depending on the machine, so when it comes to IP compliance we need to
> aggregate and simplify, otherwise our IP auditors would die :) )
>
> But since our Audit Team, differently from a commercial project,
> is working fully in the open, also other projects may benefit
> from this approach: having fully reviewed file-level license
> data publicly available for quite a number of upstream sources and
> Yocto layers, a complete source-to-binary tracking system would
> enable any Yocto projects to get very detailed license information
> for their images, to automatically detect license incompatibilities
> between linked binary files, etc.

Ok, so let me see if I can follow what you want here:
 

[OE-core][kirkstone 5/5] vim: Upgrade 9.0.0341 -> 9.0.0453

[Steve Sakoman]

From: Richard Purdie 

Includes fixes for CVE-2022-3099 and CVE-2022-3134.

Signed-off-by: Richard Purdie 
(cherry picked from commit d042923262130b6b96f703b5cd4184f659caeb92)
Signed-off-by: Steve Sakoman 
---
 meta/recipes-support/vim/vim.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-support/vim/vim.inc b/meta/recipes-support/vim/vim.inc
index 33a8299243..70dc2dfecf 100644
--- a/meta/recipes-support/vim/vim.inc
+++ b/meta/recipes-support/vim/vim.inc
@@ -20,8 +20,8 @@ SRC_URI = 
"git://github.com/vim/vim.git;branch=master;protocol=https \
file://no-path-adjust.patch \
"
 
-PV .= ".0341"
-SRCREV = "92a3d20682d46359bb50a452b4f831659e799155"
+PV .= ".0453"
+SRCREV = "83a19c5fda0556330860899bfb484addf9178cd0"
 
 # Remove when 8.3 is out
 UPSTREAM_VERSION_UNKNOWN = "1"
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170673): 
https://lists.openembedded.org/g/openembedded-core/message/170673
Mute This Topic: https://lists.openembedded.org/mt/93685088/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core][kirkstone 4/5] lighttpd: upgrade 1.4.65 -> 1.4.66

[Steve Sakoman]

From: Alexander Kanavin 

Signed-off-by: Alexander Kanavin 
Signed-off-by: Alexandre Belloni 
Signed-off-by: Richard Purdie 
(cherry picked from commit 3163134b0f58c58aaabe4e957c30109e63b2d60f)
Signed-off-by: Sakib Sajal 
Signed-off-by: Steve Sakoman 
---
 .../lighttpd/{lighttpd_1.4.65.bb => lighttpd_1.4.66.bb} | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 rename meta/recipes-extended/lighttpd/{lighttpd_1.4.65.bb => 
lighttpd_1.4.66.bb} (97%)

diff --git a/meta/recipes-extended/lighttpd/lighttpd_1.4.65.bb 
b/meta/recipes-extended/lighttpd/lighttpd_1.4.66.bb
similarity index 97%
rename from meta/recipes-extended/lighttpd/lighttpd_1.4.65.bb
rename to meta/recipes-extended/lighttpd/lighttpd_1.4.66.bb
index 10aa27f072..801162867c 100644
--- a/meta/recipes-extended/lighttpd/lighttpd_1.4.65.bb
+++ b/meta/recipes-extended/lighttpd/lighttpd_1.4.66.bb
@@ -19,7 +19,7 @@ SRC_URI = 
"http://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-${PV}.t
file://lighttpd \
"
 
-SRC_URI[sha256sum] = 
"bf0fa68a629fbc404023a912b377e70049331d6797bcbb4b3e8df4c3b42328be"
+SRC_URI[sha256sum] = 
"47ac6e60271aa0196e65472d02d019556dc7c6d09df3b65df2c1ab6866348e3b"
 
 DEPENDS = "virtual/crypt"
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170672): 
https://lists.openembedded.org/g/openembedded-core/message/170672
Mute This Topic: https://lists.openembedded.org/mt/93685085/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core][kirkstone 3/5] lighttpd: upgrade 1.4.64 -> 1.4.65

[Steve Sakoman]

From: wangmy 

Changelog:
==
  * [build] meson: fix typo in variable name
  * [build] autoconf: report if building with zstd
  * [build] meson -Dlua_version=... to specify lua ver
  * [core] avoid CCRandomGenerateBytes on MacOS <10.12 (fixes #3140)
  * [core] use diff var name w/ CCRandomGenerateBytes (fixes #3141)
  * [core] parse conf cmds with SHELL or /bin/sh
  * [core] fix HMAC with openssl 3.0
  * [mod_webdav] no COPYFILE_CLONE_FORCE on OSX <10.12 (fixes #3142)
  * [mod_deflate] fix to return 304 with If-None-Match (fixes #3143)
  * [core] Illumos epoll incompatible w/ lighttpd impl
  * [core] feature flag to allow Range w/ HTTP/1.0
  * [mod_mbedtls] set usekeysize for mbedtls 3.2.0+
  * [mod_deflate] collect mmap code
  * [mod_deflate] prototype using libdeflate w/ mmap
  * [mod_deflate] --with-libdeflate to use libdeflate
  * [mod_deflate] mark input bytes const
  * [core] sys-setjmp.[ch]
  * [mod_magnet] check lighty.result.content b4 setjmp
  * [core] include guard consistency in sys-time.h
  * [core] network_write_file_chunk_remap separate fn
  * [multiple] use new sys_setjmp_eval3() interface
  * [multiple] pedantic chunk.c checks for 0-len chunk
  * [multiple] shared code for struct chunk and mmap
  * [mod_deflate] use pread if available
  * [mod_deflate] improve loop compressing file chunk
  * [core] prep server_tag at startup for h2 resp hdr
  * [mod_magnet] defer req_env init unless needed
  * [mod_magnet] reset after error attaching content
  * [mod_magnet] lua_tointegerx() avoids raising error
  * [mod_mbedtls] use newer mbedtls 3.2.0+ interfaces
  * [mod_magnet] adjust hot path for more inlining
  * [mod_magnet] collect chk for magnet lua_State init
  * [mod_magnet] use type returned from lua_getfield()
  * [core] chunk_file_pread() to wrap pread()
  * [core] disable keep-alive if forcing HTTP/1.0 resp
  * [mod_magnet] use lua_getextraspace() to store r
  * [core] fall back to getauxval(AT_RANDOM), if avail
  * [mod_magnet] keep message handler on stack
  * [doc] update external links
  * [mod_magnet] pass lighty table index, defer pops
  * [mod_magnet] clear and reuse script-env table
  * [mod_magnet] clear stack when reloading script
  * [mod_magnet] use lua_isnoneornil() in interfaces
  * [mod_magnet] fix lighty.c.cookie_tokens()
  * [mod_magnet] fix lighty.c.urldec_query()
  * [mod_magnet] remove duplicated NULL checks
  * [mod_magnet] adjust magnet_lighty_result_get()
  * [mod_magnet] magnet_tmpbuf_acquire(),release()
  * [mod_magnet] lighty.c.quotedenc(),dec() funcs
  * [mod_magnet] fix header,content legacy table clear
  * [mod_cgi] cgi.local-redir request_reset thru fnptr
  * [core] isolate plugins_*() funcs to main server
  * [mod_wolfssl] wolfssl v5.0.0 defines DH_set0_pqg()
  * [mod_auth] save letter-case diff in require config
  * [mod_magnet] magnet_push_quoted_string shared code
  * [mod_magnet] lighty.c.header_tokens convenience fn
  * [core] fill in un.sun_path after accept() (fixes #3147)
  * [mod_extforward] adjust trust check for HTTP/2
  * [mod_proxy] adjust handling of legacy X-* headers
  * [core] permit env w/ blank value (fix regression)
  * [TLS] consistent debug.log-ssl-noise config type
  * [mod_magnet] allow removal of req_env elt via nil
  * [core] compiler workarounds for very old gcc,glibc
  * [mod_mbedtls] use newer mbedtls 3.2.0+ interfaces
  * [mod_ssi] check http_chunk_transfer_cqlen for err
  * [core] chunkqueue_steal() handle unexpected 0 len
  * [core] discard DATA from REFUSED_STREAM at h2 init
  * [multiple] WebSockets over HTTP/2 (fixes #3151)
  * [multiple] immed connect to backend for streaming
  * [core] ensure socket ready before checking connect
  * [core] reduce trace on Upgrade backend connection
  * [core] adjust when TCP_CORK used on TLS connection
  * [mod_cgi] disable input optim if might Upgrade
  * [mod_cgi] immed start CGI if Upgrade
  * [mod_wolfssl] wolfssl v5.0.0 adds ASN1_TIME_diff()
  * [mod_openssl] libressl v3.5.0 adds ASN1_TIME_diff
  * [TLS] warn if leaf cert read is inactive/expired
  * [core] stricter conformance w/ upcoming HTTP/2 rev
  * [build] -D_DEFAULT_SOURCE consistency in builds
  * [mod_extforward] support addtl IPv6 syntax w/ "[]"
  * [core] build fix for cygwin and lmingw
  * [core] short-circuit earlier parsing h2 trailers
  * [core] reformat h2.h for cleaner enum additions
  * [core] consolidate trace for log-state-handling
  * [core] request_config bitmasks for smaller struct
  * [core] prefix (=^), suffix (=$) config conditions (fixes #3153)
  * [core] tighten config parsing loop
  * [core] convert simple config cond regex to pre/sfx
  * [tests] able to run tests when built w/o pcre
  * [core] allow redirect,rewrite ext subst w/o pcre
  * [mod_sockproxy] reset http vers, avoid rare crash (fixes #3152)
  * [core] HTTP/2 PRIORITY_UPDATE frame (experimental)
  * [core] send HTTP/2 SETTINGS_NO_RFC7540_PRIORITIES
  * [core] stricter check of HTTP/2 GOAWAY frame size
  * [mod_mbedtls] use newer mbedtls 

[OE-core][kirkstone 2/5] systemd: Add 'no-dns-fallback' PACKAGECONFIG option

[Steve Sakoman]

From: "niko.ma...@vaisala.com" 

systemd defines a default set of fallback DNS servers in
https://github.com/systemd/systemd/blob/v251/meson_options.txt#L328-L330

By adding a PACKAGECONFIG knob providing a convenient way to opt out,
and then adding that value to systemd's PACKAGECONFIG, the output from
runtime 'resolvectl status' command no longer contains the following
line:

  Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 
1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 
2606:4700:4700::#cloudflare-dns.com 2001:4860:4860::#dns.google 
2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

(From OE-Core rev: 2b300d6b9ec6288a99d9dacb24a86949caf99e55)

Signed-off-by: Niko Mauno 
Signed-off-by: Richard Purdie 
Signed-off-by: Steve Sakoman 
---
 meta/recipes-core/systemd/systemd_250.5.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/systemd/systemd_250.5.bb 
b/meta/recipes-core/systemd/systemd_250.5.bb
index 9923312830..5d568f639e 100644
--- a/meta/recipes-core/systemd/systemd_250.5.bb
+++ b/meta/recipes-core/systemd/systemd_250.5.bb
@@ -165,6 +165,7 @@ PACKAGECONFIG[manpages] = 
"-Dman=true,-Dman=false,libxslt-native xmlto-native do
 PACKAGECONFIG[microhttpd] = 
"-Dmicrohttpd=true,-Dmicrohttpd=false,libmicrohttpd"
 PACKAGECONFIG[myhostname] = 
"-Dnss-myhostname=true,-Dnss-myhostname=false,,libnss-myhostname"
 PACKAGECONFIG[networkd] = "-Dnetworkd=true,-Dnetworkd=false"
+PACKAGECONFIG[no-dns-fallback] = "-Ddns-servers="
 PACKAGECONFIG[nss] = "-Dnss-systemd=true,-Dnss-systemd=false"
 PACKAGECONFIG[nss-mymachines] = "-Dnss-mymachines=true,-Dnss-mymachines=false"
 PACKAGECONFIG[nss-resolve] = "-Dnss-resolve=true,-Dnss-resolve=false"
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170670): 
https://lists.openembedded.org/g/openembedded-core/message/170670
Mute This Topic: https://lists.openembedded.org/mt/93685080/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core][kirkstone 1/5] systemd: Fix unwritable /var/lock when no sysvinit handling

[Steve Sakoman]

From: "niko.ma...@vaisala.com" 

Commit 8089cefed8e83c0348037768c292058f1bcbbbe5 ("systemd: Add
PACKAGECONFIG for sysvinit") decoupled enabling of systemd's sysvinit
handling behavior behind a distinct PACKAGECONFIG feature.

This new option affects among other things the installing of
tmpfiles.d/legacy.conf, which is responsible for creating /run/lock
directory, which is pointed to by /var/lock symlink provided by
base-files package.

In case the option is not enabled, then base-files provided /var/lock
is a dangling symlink on resulting rootfs, causing problems with
certain Linux userspace components that rely on existence of writable
/var/lock directory. As an example:

  # fw_printenv
  Error opening lock file /var/lock/fw_printenv.lock

Since Filesystem Hierarchy Standard Version 3.0 states in
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s09.html that

  Lock files should be stored within the /var/lock directory structure.

Ensure the /run/lock directory is always created, so that lock files
can be stored under /var/lock also when 'sysvinit' handling is
disabled.

(From OE-Core rev: 85e5ee2c35cf5778c3aefda45f526e8f6a511131)

Signed-off-by: Niko Mauno 
Signed-off-by: Alexandre Belloni 
Signed-off-by: Richard Purdie 
Signed-off-by: Steve Sakoman 
---
 meta/recipes-core/systemd/systemd/00-create-volatile.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/systemd/systemd/00-create-volatile.conf 
b/meta/recipes-core/systemd/systemd/00-create-volatile.conf
index 87cbe1e7d3..c4277221a2 100644
--- a/meta/recipes-core/systemd/systemd/00-create-volatile.conf
+++ b/meta/recipes-core/systemd/systemd/00-create-volatile.conf
@@ -3,5 +3,6 @@
 # inside /var/log.
 
 
+d  /run/lock   1777-   -   -
 d  /var/volatile/log   -   -   -   -
 d  /var/volatile/tmp   1777-   -
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170669): 
https://lists.openembedded.org/g/openembedded-core/message/170669
Mute This Topic: https://lists.openembedded.org/mt/93685079/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core][kirkstone 0/5] Patch review

[Steve Sakoman]

Please review this set of patches for kirkstone and have comments back by end
of day Friday.  This should be the final set of patches for the 4.0.4 release.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4225

The following changes since commit 08406e03abddc7290c0c2296aa179725a58155d3:

  runqemu: display host uptime when starting (2022-09-12 04:45:14 -1000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib stable/kirkstone-nut
  
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/kirkstone-nut

Alexander Kanavin (1):
  lighttpd: upgrade 1.4.65 -> 1.4.66

Richard Purdie (1):
  vim: Upgrade 9.0.0341 -> 9.0.0453

niko.ma...@vaisala.com (2):
  systemd: Fix unwritable /var/lock when no sysvinit handling
  systemd: Add 'no-dns-fallback' PACKAGECONFIG option

wangmy (1):
  lighttpd: upgrade 1.4.64 -> 1.4.65

 meta/recipes-core/systemd/systemd/00-create-volatile.conf | 1 +
 meta/recipes-core/systemd/systemd_250.5.bb| 1 +
 .../lighttpd/{lighttpd_1.4.64.bb => lighttpd_1.4.66.bb}   | 2 +-
 meta/recipes-support/vim/vim.inc  | 4 ++--
 4 files changed, 5 insertions(+), 3 deletions(-)
 rename meta/recipes-extended/lighttpd/{lighttpd_1.4.64.bb => 
lighttpd_1.4.66.bb} (97%)

-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170668): 
https://lists.openembedded.org/g/openembedded-core/message/170668
Mute This Topic: https://lists.openembedded.org/mt/93685078/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] cockpit recipe

[razriel]

I am trying to integrate cockpit recipe in my image.
cockpit is a web based linux managment tool.
cockpit is supported mosly in well know distors.
the recipe i am using is based on tag 220.
main problems I am trying to overcome:
1. in first login after image was flashed to emmc. the user is requiered to 
change password. will not display the change password screen.
2. account managment is not working.
3. work with user different from root.
I have tried to understand the origin of the problem. it seems the 
cockpit.spawn command in the javascript returnes with error when trying to set 
password for user.
anyone can guide me throuth this issue ?

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170667): 
https://lists.openembedded.org/g/openembedded-core/message/170667
Mute This Topic: https://lists.openembedded.org/mt/93683211/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Adding more information to the SBOM

[Alberto Pianon]

Hi Joshua,

nice to meet you!

I'm new to this list, and I've always approached Yocto just from the
"IP compliance side", so I may miss important pieces of information. 
That

is why Marta encouraged me and is helping me to ask community feedback.

Il 2022-09-14 16:56 Joshua Watt ha scritto:
On Wed, Sep 14, 2022 at 9:16 AM Marta Rybczynska  
wrote:


Dear all,
(cross-posting to oe-core and *-architecture)
In the last months, we have worked in Oniro on using the create-spdx
class for both IP compliance and security.

During this work, Alberto Pianon has found that some information is
missing from the SBOM and it does not contain enough for Software
Composition Analysis. The main missing point is the relation between
the actual upstream sources and the final binaries (create-spdx uses
composite sources).


I believe we map the binaries to the source code from the -dbg
packages; is the premise that this is insufficient? Can you elaborate
more on why that is, I don't quite understand. The debug sources are
(basically) what we actually compiled (e.g. post-do_patch) to produce
the binary, and you can in turn follow these back to the upstream
sources with the downloadLocation property.


This was also my assumption at the beginning. But then I found that 
there

are recipes with multiple upstream sources, which may be combined/mixed
together in recipes' WORKDIR. For instance this one:

https://git.yoctoproject.org/meta-virtualization/tree/recipes-networking/cni/cni_git.bb

SRC_URI = "\
	git://github.com/containernetworking/cni.git;branch=main;name=cni;protocol=https 
\

git://github.com/containernetworking/plugins.git;branch=release-1.1;destsuffix=${S}/src/github.com/containernetworking/plugins;name=plugins;protocol=https 
\

git://github.com/flannel-io/cni-plugin;branch=main;name=flannel_plugin;protocol=https;destsuffix=${S}/src/github.com/containernetworking/plugins/plugins/meta/flannel 
\

"

(The third source is unpacked in a subdir of the second one)

From here I discovered that we can't assume that the first non-local URI
is the downloadLocation for all source files, because it is not always
the case.

Moreover, in the context of our project we also needed to find the 
upstream

sources also for local patches, scripts, etc. added by recipes (i.e. the
corresponding layers' repos).





Alberto has worked on how to obtain the missing data and now has a
POC. This POC provides full source-to-binary tracking of Yocto builds
through a couple of scripts (intended to be transformed into a new
bbclass at a later stage). The goal is to add the missing pieces of
information in order to get a "real" SBOM from Yocto, which should, at
a minimum:


Please be a little careful with the wording; SBoMs have a lot of uses,
and many of them we can satisfy with what we currently generate; it
may not do the exact use case you are looking for, but that doesn't
mean it's not a "real" SBoM :)


You are right, sorry! "real" is meant in the context of our project,
where we need to make our Fossology Audit Team work on "original"
upstream source packages/repos, for a number of reasons (the main being
that in Oniro project we have a complex build matrix with a lot of
available target machines and quite a number of different overrides
depending on the machine, so when it comes to IP compliance we need to
aggregate and simplify, otherwise our IP auditors would die :) )

But since our Audit Team, differently from a commercial project,
is working fully in the open, also other projects may benefit
from this approach: having fully reviewed file-level license
data publicly available for quite a number of upstream sources and
Yocto layers, a complete source-to-binary tracking system would
enable any Yocto projects to get very detailed license information
for their images, to automatically detect license incompatibilities
between linked binary files, etc.





- carefully describe what is found in a final image (i.e. binary files
and their dependencies), since that is what is actually distributed
and goes into the final product;
- describe how such binary files have been generated and where they
come from (i.e. upstream sources, including patches and other stuff
added from meta-layers); provenance is important for a number of
reasons related to IP Compliance and security.

The aim is to become able to:

- map binaries to their corresponding upstream source packages (and
not to the "internal" source packages created by recipes by combining
multiple upstream sources and patches)
- map binaries to the source files that have been actually used to
build them - which usually are a small subset of the whole source
package

With respect to IP compliance, this would allow to, among other 
things:


- get the real license text for each binary file, by getting the
license of the specific source files it has been generated from
(provided by Fossology, for instance), - and not the main license
stated in the corresponding recipe 

Re: [OE-core] [PATCH] opkg: enable zstd support

[Khem Raj]

On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart  wrote:
>
> Thanks for checking.
>
> I'd be interested to know if setting a higher compression level for zstd
> can get us to a similar compression ratio to xz. If so, then I think it
> could be some real value to distro maintainers to be able to *tune*
> their compression.

yeah it will be interesting to say try something like level 9 but I think times
might regress with that but it might be good to know the balance and perhaps
suggest size mode and performance mode of zstd instead of xz

>
> That's not blocking for your new PR though.
>
>
> On 9/14/22 05:08, Etienne Cordonnier wrote:
> > Also note that zstd's default compression level is 3 per default (from
> > a 1 to 19 scale). I did not test other compression levels.
> >
> > On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
> >  wrote:
> >
> > I ran a build of core-image-full-cmdline using xz and zstd, using
> > pre-populated downloads and sstate-cache directories but with
> > empty tmp directory. Here are the numbers:
> > zstd:
> > bitbake core-image-full-cmdline took 2m52.768s (real), the
> > resulting directory tmp/deploy/ipk is 1.6GB big.
> > xz:
> > bitbake core-image-full-cmdline took 4m14.214s (real), the
> > resulting directory tmp/deploy/ipk/ is 996M big
> >
> > So the build with xz is 47% slower (254/172) than with zstd for
> > this "incremental build" use-case.
> >
> > See the 5 biggest packages, the difference in compression-ratio
> > increases with big files:
> > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> > total 1.6G
> > -rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
> > gcc-dbg_12.2.0-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
> > openssl-dbg_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
> > openssl-ptest_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
> > binutils-dbg_2.39-r0_core2-64.ipk
> > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> > total 963M
> > -rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
> > gcc-dbg_12.2.0-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
> > openssl-dbg_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
> > openssl-ptest_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
> > binutils-dbg_2.39-r0_core2-64.ipk
> > ...
> >
> > I think for use-cases where the size of the ipk packages matters,
> > xz is better. For use-cases where it does not matter (ipk packages
> > not deployed), build-time matters more than compression-ratio and
> > zstd is better.
> >
> > Regarding the enablement of zstd in opkg per default, I'll send a
> > new version of the patch without this line.
> > My thinking for enabling zstd per default in opkg was that zstd is
> > already enabled per default in libarchive's PACKAGECONFIG, and
> > disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
> > code from opkg (opkg uses libarchive for doing the actual
> > compression/decompression).
> >
> > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
> >  wrote:
> >
> >
> >
> > On 9/13/22 15:20, Alex Feinman wrote:
> > > I do have some numbers. When I was selling this change
> > internally, I
> > > did a comparison on our internal build.
> > >Combined write IPK times (Σ t do_package_write_ipk)
> > > xz 162m 35s
> > > gz 52m 13s
> > > zstd   33m 49s
> > > Compression rate for zstd was closer to xz than to gz but
> > not as good
> > > as xz. For systems that have to cache packages on the device
> > with
> > > limited storage xz might be a better option, but for the
> > bulk of
> > > projects zstd is the best choice
> > > Additionally, zstd offers much faster decompression than xz
> > so the
> > > rootfs build step that includes unpacking all of the ipks,
> > takes 3m
> > > 58s with xz and 2m 44s with zstd.
> > > One other thing of note - if your build includes debug
> > packages, some
> > > may be quite large. E.g. one of our components produces a
> > 2.2 GB debug
> > > package (uncompressed). On large files xz requires a
> > disproportionally
> > > large amount of time resulting in 15 minutes needed to
> > simply write
> > > ipk for the abovementioned packages, whereas zstd took about
> > 45 sec.
> > > For frequent tasks like bitbaking a single package this
> > translates in
> > > a lot of saved time.
> >
> > Those are certainly compelling performance 

Re: [OE-core] [PATCH] opkg: enable zstd support

[Alex Stewart]

Thanks for checking.

I'd be interested to know if setting a higher compression level for zstd 
can get us to a similar compression ratio to xz. If so, then I think it 
could be some real value to distro maintainers to be able to *tune* 
their compression.


That's not blocking for your new PR though.


On 9/14/22 05:08, Etienne Cordonnier wrote:
Also note that zstd's default compression level is 3 per default (from 
a 1 to 19 scale). I did not test other compression levels.


On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier 
 wrote:


I ran a build of core-image-full-cmdline using xz and zstd, using
pre-populated downloads and sstate-cache directories but with
empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the
resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the
resulting directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for
this "incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio
increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters,
xz is better. For use-cases where it does not matter (ipk packages
not deployed), build-time matters more than compression-ratio and
zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a
new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is
already enabled per default in libarchive's PACKAGECONFIG, and
disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
code from opkg (opkg uses libarchive for doing the actual
compression/decompression).

On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
 wrote:



On 9/13/22 15:20, Alex Feinman wrote:
> I do have some numbers. When I was selling this change
internally, I
> did a comparison on our internal build.
>            Combined write IPK times (Σ t do_package_write_ipk)
> xz         162m 35s
> gz         52m 13s
> zstd       33m 49s
> Compression rate for zstd was closer to xz than to gz but
not as good
> as xz. For systems that have to cache packages on the device
with
> limited storage xz might be a better option, but for the
bulk of
> projects zstd is the best choice
> Additionally, zstd offers much faster decompression than xz
so the
> rootfs build step that includes unpacking all of the ipks,
takes 3m
> 58s with xz and 2m 44s with zstd.
> One other thing of note - if your build includes debug
packages, some
> may be quite large. E.g. one of our components produces a
2.2 GB debug
> package (uncompressed). On large files xz requires a
disproportionally
> large amount of time resulting in 15 minutes needed to
simply write
> ipk for the abovementioned packages, whereas zstd took about
45 sec.
> For frequent tasks like bitbaking a single package this
translates in
> a lot of saved time.

Those are certainly compelling performance improvements.
Assuming that
the final data-segment size is within 5%-ish of xz, then I
would agree
with the rest of the thread that it should probably be the
contemporary
default.

And if we make it the default compressor for OE IPKs, then
obviously my
criticism in the original PR is satisfied.

> Bottom line - I think making xz a default package compressor
was not
> entirely thought through. gzip or zstd is what the default
should be.

ZStandard support was only added to opkg last September [1].
Before
that, xz was the new hotness 

Re: [OE-core] [PATCH v2] opkg: add option for zstd support

[Alex Stewart]

ACK from me.

On 9/14/22 04:59, Etienne Cordonnier via lists.openembedded.org wrote:

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

Signed-off-by: Alex Feinman 
Signed-off-by: Etienne Cordonnier 
---
  meta/recipes-devtools/opkg/opkg_0.6.0.bb | 1 +
  1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb 
b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
index 7b351e8123..4cd589cd29 100644
--- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
+++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
  PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
  PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
  PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
  PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
  
  EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"






--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.stew...@ni.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170663): 
https://lists.openembedded.org/g/openembedded-core/message/170663
Mute This Topic: https://lists.openembedded.org/mt/93674358/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH][dunfell] go: fix and ignore several CVEs

[Lee Chee Yang]

From: Chee Yang Lee 

backport fixes:
CVE-2021-27918
CVE-2021-36221
CVE-2021-39293
CVE-2021-41771

ignore:
CVE-2022-29526
CVE-2022-30634

Signed-off-by: Chee Yang Lee 
---
 meta/recipes-devtools/go/go-1.14.inc  |  10 +
 .../go/go-1.14/CVE-2021-27918.patch   | 191 ++
 .../go/go-1.14/CVE-2021-36221.patch   | 101 +
 .../go/go-1.14/CVE-2021-39293.patch   |  79 
 .../go/go-1.14/CVE-2021-41771.patch   |  86 
 5 files changed, 467 insertions(+)
 create mode 100644 meta/recipes-devtools/go/go-1.14/CVE-2021-27918.patch
 create mode 100644 meta/recipes-devtools/go/go-1.14/CVE-2021-36221.patch
 create mode 100644 meta/recipes-devtools/go/go-1.14/CVE-2021-39293.patch
 create mode 100644 meta/recipes-devtools/go/go-1.14/CVE-2021-41771.patch

diff --git a/meta/recipes-devtools/go/go-1.14.inc 
b/meta/recipes-devtools/go/go-1.14.inc
index 1458a11b3f..af6345205e 100644
--- a/meta/recipes-devtools/go/go-1.14.inc
+++ b/meta/recipes-devtools/go/go-1.14.inc
@@ -32,6 +32,10 @@ SRC_URI += "\
 file://CVE-2022-30635.patch \
 file://CVE-2022-32148.patch \
 file://CVE-2022-32189.patch \
+file://CVE-2021-27918.patch \
+file://CVE-2021-36221.patch \
+file://CVE-2021-39293.patch \
+file://CVE-2021-41771.patch \
 "
 
 SRC_URI_append_libc-musl = " 
file://0009-ld-replace-glibc-dynamic-linker-with-musl.patch"
@@ -42,3 +46,9 @@ SRC_URI[main.sha256sum] = 
"7ed13b2209e54a451835997f78035530b331c5b6943cdcd68a3d8
 # https://github.com/golang/go/issues/30999#issuecomment-910470358
 CVE_CHECK_WHITELIST += "CVE-2021-29923"
 
+# this issue affected go1.15 onwards
+# https://security-tracker.debian.org/tracker/CVE-2022-29526
+CVE_CHECK_WHITELIST += "CVE-2022-29526"
+
+# Issue only on windows
+CVE_CHECK_WHITELIST += "CVE-2022-30634"
diff --git a/meta/recipes-devtools/go/go-1.14/CVE-2021-27918.patch 
b/meta/recipes-devtools/go/go-1.14/CVE-2021-27918.patch
new file mode 100644
index 00..faa3f7f641
--- /dev/null
+++ b/meta/recipes-devtools/go/go-1.14/CVE-2021-27918.patch
@@ -0,0 +1,191 @@
+From d0b79e3513a29628f3599dc8860666b6eed75372 Mon Sep 17 00:00:00 2001
+From: Katie Hockman 
+Date: Mon, 1 Mar 2021 09:54:00 -0500
+Subject: [PATCH] encoding/xml: prevent infinite loop while decoding
+
+This change properly handles a TokenReader which
+returns an EOF in the middle of an open XML
+element.
+
+Thanks to Sam Whited for reporting this.
+
+Fixes CVE-2021-27918
+Fixes #44913
+
+Change-Id: Id02a3f3def4a1b415fa2d9a8e3b373eb6cb0f433
+Reviewed-on: 
https://team-review.git.corp.google.com/c/golang/go-private/+/1004594
+Reviewed-by: Russ Cox 
+Reviewed-by: Roland Shoemaker 
+Reviewed-by: Filippo Valsorda 
+Reviewed-on: https://go-review.googlesource.com/c/go/+/300391
+Trust: Katie Hockman 
+Run-TryBot: Katie Hockman 
+TryBot-Result: Go Bot 
+Reviewed-by: Alexander Rakoczy 
+Reviewed-by: Filippo Valsorda 
+
+https://github.com/golang/go/commit/d0b79e3513a29628f3599dc8860666b6eed75372
+CVE: CVE-2021-27918
+Upstream-Status: Backport
+Signed-off-by: Chee Yang Lee 
+---
+ src/encoding/xml/xml.go  |  19 ---
+ src/encoding/xml/xml_test.go | 104 +++
+ 2 files changed, 92 insertions(+), 31 deletions(-)
+
+diff --git a/src/encoding/xml/xml.go b/src/encoding/xml/xml.go
+index adaf4daf198b9..6f9594d7ba7a3 100644
+--- a/src/encoding/xml/xml.go
 b/src/encoding/xml/xml.go
+@@ -271,7 +271,7 @@ func NewTokenDecoder(t TokenReader) *Decoder {
+ // it will return an error.
+ //
+ // Token implements XML name spaces as described by
+-// https://www.w3.org/TR/REC-xml-names/.  Each of the
++// https://www.w3.org/TR/REC-xml-names/. Each of the
+ // Name structures contained in the Token has the Space
+ // set to the URL identifying its name space when known.
+ // If Token encounters an unrecognized name space prefix,
+@@ -285,16 +285,17 @@ func (d *Decoder) Token() (Token, error) {
+   if d.nextToken != nil {
+   t = d.nextToken
+   d.nextToken = nil
+-  } else if t, err = d.rawToken(); err != nil {
+-  switch {
+-  case err == io.EOF && d.t != nil:
+-  err = nil
+-  case err == io.EOF && d.stk != nil && d.stk.kind != stkEOF:
+-  err = d.syntaxError("unexpected EOF")
++  } else {
++  if t, err = d.rawToken(); t == nil && err != nil {
++  if err == io.EOF && d.stk != nil && d.stk.kind != 
stkEOF {
++  err = d.syntaxError("unexpected EOF")
++  }
++  return nil, err
+   }
+-  return t, err
++  // We still have a token to process, so clear any
++  // errors (e.g. EOF) and proceed.
++  err = nil
+   }
+-
+   if !d.Strict {
+   if t1, ok := d.autoClose(t); ok {
+   d.nextToken = t
+diff --git a/src/encoding/xml/xml_test.go 

Re: [OE-core] [kirkstone][PATCH v2] busybox: add devmem 128-bit support

[Steve Sakoman]

On Wed, Sep 14, 2022 at 4:08 AM Randy MacLeod
 wrote:
>
> On 2022-09-14 03:20, Yu, Mingli wrote:
>
> From: Mingli Yu 
>
> Add devmem 128-bit support [1].
>
> We should merge to master as well but I don't think we need to enable devmem, 
> especially in kirkstone.
> If a user wants to do that, that's fine.

IIf this is also suitable for master, then the patch should be
resubmitted for master and I can then cherry-pick to kirkstone.

Steve

>
> My notes:
>
> I think that back-porting this commit is low risk since it only adds a new, 
> non-default ability in a compatible manner and
> more importantly 'devmem' is NOT built by default. Therefore, other users are 
> not affected by including this patch.
>
> devmem is a small program that reads and writes from physical memory using 
> /dev/mem.
>
> Usage is:
>
> $ busybox devmem --help
> ...
> Usage: devmem ADDRESS [WIDTH [VALUE]]
> Read/write from physical address
> ADDRESSAddress to act upon
> WIDTHWidth (8/16/...)
> VALUEData to be written
>
>
>
> This commit adds 128 bit as a supported WIDTH.
>
> The patch cherry-picks back to 1.35.0 cleanly and doesn't require pulling in 
> any other commits.
> It does add new functionality but the patch is written such that if 
> _SIZEOF_INT128_ isn't define, the code is practically unchanged.
>
>
> There hasn't been a release of busybox since Dec 2021 so we should backport 
> this patch to oe-core/master and oe-core/kirkstone.
>
>
>
> [1] 
> https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1
>
> Signed-off-by: Mingli Yu 
> ---
>  .../0001-devmem-add-128-bit-width.patch   | 128 ++
>  meta/recipes-core/busybox/busybox/defconfig   |   2 +-
>  meta/recipes-core/busybox/busybox_1.35.0.bb   |   1 +
>  3 files changed, 130 insertions(+), 1 deletion(-)
>  create mode 100644 
> meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
>
> diff --git 
> a/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch 
> b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
> new file mode 100644
> index 00..985e2bf1d9
> --- /dev/null
> +++ b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
> @@ -0,0 +1,128 @@
> +From d432049f288c9acdc4a7caa729c68ceba3c5dca1 Mon Sep 17 00:00:00 2001
> +From: Aaro Koskinen 
> +Date: Thu, 25 Aug 2022 18:47:02 +0300
> +Subject: [PATCH] devmem: add 128-bit width
> +
> +Add 128-bit width if the compiler provides the needed type.
> +
> +function old new   delta
> +devmem_main  405 464 +59
> +.rodata   109025  109043 +18
> +--
> +(add/remove: 0/0 grow/shrink: 2/0 up/down: 77/0)   Total: 77 
> bytes
> +
> +Upstream-Status: Backport 
> [https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1]
> +
> +Signed-off-by: Aaro Koskinen 
> +Signed-off-by: Aaro Koskinen 
> +Signed-off-by: Denys Vlasenko 
> +Signed-off-by: Mingli Yu 
> +---
> + miscutils/devmem.c | 68 ++
> + 1 file changed, 44 insertions(+), 24 deletions(-)
> +
> +diff --git a/miscutils/devmem.c b/miscutils/devmem.c
> +index f9f0276bc..f21621bd6 100644
> +--- a/miscutils/devmem.c
>  b/miscutils/devmem.c
> +@@ -29,7 +29,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
> + {
> + void *map_base, *virt_addr;
> + uint64_t read_result;
> +- uint64_t writeval = writeval; /* for compiler */
> + off_t target;
> + unsigned page_size, mapped_size, offset_in_page;
> + int fd;
> +@@ -64,9 +63,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
> + width = strchrnul(bhwl, (argv[2][0] | 0x20)) - bhwl;
> + width = sizes[width];
> + }
> +- /* VALUE */
> +- if (argv[3])
> +- writeval = bb_strtoull(argv[3], NULL, 0);
> + } else { /* argv[2] == NULL */
> + /* make argv[3] to be a valid thing to fetch */
> + argv--;
> +@@ -96,28 +92,46 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
> + virt_addr = (char*)map_base + offset_in_page;
> +
> + if (!argv[3]) {
> +- switch (width) {
> +- case 8:
> +- read_result = *(volatile uint8_t*)virt_addr;
> +- break;
> +- case 16:
> +- read_result = *(volatile uint16_t*)virt_addr;
> +- break;
> +- case 32:
> +- read_result = *(volatile uint32_t*)virt_addr;
> +- break;
> +- case 64:
> +- read_result = *(volatile uint64_t*)virt_addr;
> +- break;
> +- default:
> +- bb_simple_error_msg_and_die("bad width");
> ++#ifdef __SIZEOF_INT128__
> ++ if (width == 128) {
> ++ unsigned __int128 rd =
> ++ *(volatile unsigned __int128 *)virt_addr;
> ++ printf("0x%016llX%016llX\n",
> ++ (unsigned long long)(uint64_t)(rd >> 64),
> ++ (unsigned long long)(uint64_t)rd
> ++ );
> ++ } else
> ++#endif
> ++ {
> ++ switch (width) {
> ++ case 8:
> ++ read_result = *(volatile uint8_t*)virt_addr;
> ++ break;
> 

Re: [OE-core] Adding more information to the SBOM

[Joshua Watt]

On Wed, Sep 14, 2022 at 9:16 AM Marta Rybczynska  wrote:
>
> Dear all,
> (cross-posting to oe-core and *-architecture)
> In the last months, we have worked in Oniro on using the create-spdx
> class for both IP compliance and security.
>
> During this work, Alberto Pianon has found that some information is
> missing from the SBOM and it does not contain enough for Software
> Composition Analysis. The main missing point is the relation between
> the actual upstream sources and the final binaries (create-spdx uses
> composite sources).

I believe we map the binaries to the source code from the -dbg
packages; is the premise that this is insufficient? Can you elaborate
more on why that is, I don't quite understand. The debug sources are
(basically) what we actually compiled (e.g. post-do_patch) to produce
the binary, and you can in turn follow these back to the upstream
sources with the downloadLocation property.

>
> Alberto has worked on how to obtain the missing data and now has a
> POC. This POC provides full source-to-binary tracking of Yocto builds
> through a couple of scripts (intended to be transformed into a new
> bbclass at a later stage). The goal is to add the missing pieces of
> information in order to get a "real" SBOM from Yocto, which should, at
> a minimum:

Please be a little careful with the wording; SBoMs have a lot of uses,
and many of them we can satisfy with what we currently generate; it
may not do the exact use case you are looking for, but that doesn't
mean it's not a "real" SBoM :)

>
> - carefully describe what is found in a final image (i.e. binary files
> and their dependencies), since that is what is actually distributed
> and goes into the final product;
> - describe how such binary files have been generated and where they
> come from (i.e. upstream sources, including patches and other stuff
> added from meta-layers); provenance is important for a number of
> reasons related to IP Compliance and security.
>
> The aim is to become able to:
>
> - map binaries to their corresponding upstream source packages (and
> not to the "internal" source packages created by recipes by combining
> multiple upstream sources and patches)
> - map binaries to the source files that have been actually used to
> build them - which usually are a small subset of the whole source
> package
>
> With respect to IP compliance, this would allow to, among other things:
>
> - get the real license text for each binary file, by getting the
> license of the specific source files it has been generated from
> (provided by Fossology, for instance), - and not the main license
> stated in the corresponding recipe (which may be as confusing as
> GPL-2.0-or-later & LGPL-2.1-or-later & BSD-3-Clause & BSD-4-Clause, or
> even worse)

IIUC this is the difference between the "Declared" license and the
"Concluded" license. You can report both, and I think
create-spdx.bbclass can currently do this with its rudimentary source
license scanning. You really do want both and it's a great way to make
sure that the "Declared" license (that is the license in the recipe)
reflects the reality of the source code.

> - automatically check license incompatibilities at the binary file level.
>
> Other possible interesting things could be done also on the security side.
>
> This work intends to add a way to provide additional data that can be
> used by create-spdx, not to replace create-spdx in any way.
>
> The sources with a long README are available at
> https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain/toolchain/tinfoilhat/-/tree/srctracker/srctracker
>
> What do you think of this work? Would it be of interest to integrate
> into YP at some point? Shall we discuss this?

This seems promising as something that could potentially move into
core. I have a few points:
 - The extraction of the sources to a dedicated directory is something
that Richard has been toying around with for quite a while, and I
think it would greatly simplify that part of your process. I would
very much encourage you to look at the work he's done, and work on
that to get it pushed across the finish line as it's a really good
improvement that would benefit not just your source scanning.
 - I would encourage you to not wait to turn this into a bbclass
and/or library functions. You should be able to do this in a new
layer, and that would make it much clearer as to what the path to
being included in OE-core would look like. It also would (IMHO) be
nicer to the users :)

>
> Marta and Alberto

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170660): 
https://lists.openembedded.org/g/openembedded-core/message/170660
Mute This Topic: https://lists.openembedded.org/mt/93678488/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] Adding more information to the SBOM

[Marta Rybczynska]

Dear all,
(cross-posting to oe-core and *-architecture)
In the last months, we have worked in Oniro on using the create-spdx
class for both IP compliance and security.

During this work, Alberto Pianon has found that some information is
missing from the SBOM and it does not contain enough for Software
Composition Analysis. The main missing point is the relation between
the actual upstream sources and the final binaries (create-spdx uses
composite sources).

Alberto has worked on how to obtain the missing data and now has a
POC. This POC provides full source-to-binary tracking of Yocto builds
through a couple of scripts (intended to be transformed into a new
bbclass at a later stage). The goal is to add the missing pieces of
information in order to get a "real" SBOM from Yocto, which should, at
a minimum:

- carefully describe what is found in a final image (i.e. binary files
and their dependencies), since that is what is actually distributed
and goes into the final product;
- describe how such binary files have been generated and where they
come from (i.e. upstream sources, including patches and other stuff
added from meta-layers); provenance is important for a number of
reasons related to IP Compliance and security.

The aim is to become able to:

- map binaries to their corresponding upstream source packages (and
not to the "internal" source packages created by recipes by combining
multiple upstream sources and patches)
- map binaries to the source files that have been actually used to
build them - which usually are a small subset of the whole source
package

With respect to IP compliance, this would allow to, among other things:

- get the real license text for each binary file, by getting the
license of the specific source files it has been generated from
(provided by Fossology, for instance), - and not the main license
stated in the corresponding recipe (which may be as confusing as
GPL-2.0-or-later & LGPL-2.1-or-later & BSD-3-Clause & BSD-4-Clause, or
even worse)
- automatically check license incompatibilities at the binary file level.

Other possible interesting things could be done also on the security side.

This work intends to add a way to provide additional data that can be
used by create-spdx, not to replace create-spdx in any way.

The sources with a long README are available at
https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain/toolchain/tinfoilhat/-/tree/srctracker/srctracker

What do you think of this work? Would it be of interest to integrate
into YP at some point? Shall we discuss this?

Marta and Alberto

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170659): 
https://lists.openembedded.org/g/openembedded-core/message/170659
Mute This Topic: https://lists.openembedded.org/mt/93678488/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [kirkstone][PATCH v2] busybox: add devmem 128-bit support

[Randy MacLeod]

On 2022-09-14 03:20, Yu, Mingli wrote:

From: Mingli Yu

Add devmem 128-bit support [1].


We should merge to master as well but I don't think we need to enable 
devmem, especially in kirkstone.

If a user wants to do that, that's fine.

My notes:

I think that back-porting this commit is low risk since it only adds a 
new, non-default ability in a compatible manner and
more importantly 'devmem' is NOT built by default. Therefore, other 
users are not affected by including this patch.


devmem is a small program that reads and writes from physical memory 
using /dev/mem.


Usage is:

$ busybox devmem --help
...
Usage: devmem ADDRESS [WIDTH [VALUE]]
Read/write from physical address
    ADDRESS    Address to act upon
    WIDTH    Width (8/16/...)
    VALUE    Data to be written

This commit adds 128 bit as a supported WIDTH.

The patch cherry-picks back to 1.35.0 *cleanly* and doesn't require 
pulling in any other commits.
It *does add new functionality* but the patch is written such that if 
_/SIZEOF_INT128/_ isn't define, the code is practically unchanged.


There hasn't been a release of busybox since Dec 2021 so we should 
backport this patch to oe-core/master and oe-core/kirkstone.





[1]https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1

Signed-off-by: Mingli Yu
---
  .../0001-devmem-add-128-bit-width.patch   | 128 ++
  meta/recipes-core/busybox/busybox/defconfig   |   2 +-
  meta/recipes-core/busybox/busybox_1.35.0.bb   |   1 +
  3 files changed, 130 insertions(+), 1 deletion(-)
  create mode 100644 
meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch

diff --git 
a/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch 
b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
new file mode 100644
index 00..985e2bf1d9
--- /dev/null
+++ b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
@@ -0,0 +1,128 @@
+From d432049f288c9acdc4a7caa729c68ceba3c5dca1 Mon Sep 17 00:00:00 2001
+From: Aaro Koskinen
+Date: Thu, 25 Aug 2022 18:47:02 +0300
+Subject: [PATCH] devmem: add 128-bit width
+
+Add 128-bit width if the compiler provides the needed type.
+
+function old new   delta
+devmem_main  405 464 +59
+.rodata   109025  109043 +18
+--
+(add/remove: 0/0 grow/shrink: 2/0 up/down: 77/0)   Total: 77 bytes
+
+Upstream-Status: Backport 
[https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1]
+
+Signed-off-by: Aaro Koskinen
+Signed-off-by: Aaro Koskinen
+Signed-off-by: Denys Vlasenko
+Signed-off-by: Mingli Yu
+---
+ miscutils/devmem.c | 68 ++
+ 1 file changed, 44 insertions(+), 24 deletions(-)
+
+diff --git a/miscutils/devmem.c b/miscutils/devmem.c
+index f9f0276bc..f21621bd6 100644
+--- a/miscutils/devmem.c
 b/miscutils/devmem.c
+@@ -29,7 +29,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+ {
+   void *map_base, *virt_addr;
+   uint64_t read_result;
+-  uint64_t writeval = writeval; /* for compiler */
+   off_t target;
+   unsigned page_size, mapped_size, offset_in_page;
+   int fd;
+@@ -64,9 +63,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+   width = strchrnul(bhwl, (argv[2][0] | 0x20)) - bhwl;
+   width = sizes[width];
+   }
+-  /* VALUE */
+-  if (argv[3])
+-  writeval = bb_strtoull(argv[3], NULL, 0);
+   } else { /* argv[2] == NULL */
+   /* make argv[3] to be a valid thing to fetch */
+   argv--;
+@@ -96,28 +92,46 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+   virt_addr = (char*)map_base + offset_in_page;
+
+   if (!argv[3]) {
+-  switch (width) {
+-  case 8:
+-  read_result = *(volatile uint8_t*)virt_addr;
+-  break;
+-  case 16:
+-  read_result = *(volatile uint16_t*)virt_addr;
+-  break;
+-  case 32:
+-  read_result = *(volatile uint32_t*)virt_addr;
+-  break;
+-  case 64:
+-  read_result = *(volatile uint64_t*)virt_addr;
+-  break;
+-  default:
+-  bb_simple_error_msg_and_die("bad width");
++#ifdef __SIZEOF_INT128__
++  if (width == 128) {
++  unsigned __int128 rd =
++  *(volatile unsigned __int128 *)virt_addr;
++  printf("0x%016llX%016llX\n",
++  (unsigned long long)(uint64_t)(rd >> 64),
++  (unsigned long 

[OE-core] [PATCH] rootfs.py: Drop FAKEROOT support in exec function

[Andrei Gherzan]

From: Andrei Gherzan 

The _exec_shell_cmd function is used by a couple of other functions that
are ultimatelly called from the create_rootfs function. The latter is
used in image bbclass' do_rootfs which is running using the fakeroot
support in bitbake. This makes the fakeroot support in _exec_shell_cmd
redundant and never actually used.

Signed-off-by: Andrei Gherzan 
---
 meta/lib/oe/rootfs.py | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 8587c5db82..890ba5f039 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -173,14 +173,8 @@ class Rootfs(object, metaclass=ABCMeta):
 bb.utils.rename(self.image_rootfs + '-orig', self.image_rootfs)
 
 def _exec_shell_cmd(self, cmd):
-fakerootcmd = self.d.getVar('FAKEROOT')
-if fakerootcmd is not None:
-exec_cmd = [fakerootcmd, cmd]
-else:
-exec_cmd = cmd
-
 try:
-subprocess.check_output(exec_cmd, stderr=subprocess.STDOUT)
+subprocess.check_output(cmd, stderr=subprocess.STDOUT)
 except subprocess.CalledProcessError as e:
 return("Command '%s' returned %d:\n%s" % (e.cmd, e.returncode, 
e.output))
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170657): 
https://lists.openembedded.org/g/openembedded-core/message/170657
Mute This Topic: https://lists.openembedded.org/mt/93676035/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] python3: update 3.10.6 -> 3.11.0rc2

[Alexander Kanavin]

Note, the following items from meta-oe are still not compatible and
fail as of now:

python3-pycocotools
python3-pybluez
python3-gevent
python3-cchardet
python3-grpcio
python3-wxgtk4

Alex

On Wed, 14 Sept 2022 at 14:04, Alexander Kanavin via
lists.openembedded.org 
wrote:
>
> The semaphore fix has landed and is available from 3.11 onwards:
> https://github.com/python/cpython/commit/1ee0f94d16f150356a4b9b0a39d44ba1d2d5b9fc
>
> 3.11 release schedule: https://peps.python.org/pep-0664/
>
> Drop 0001-Mitigate-the-race-condition-in-testSockName.patch
> as it is merged upstream.
>
> Signed-off-by: Alexander Kanavin 
> ---
>  meta/classes-recipe/python3-dir.bbclass   |  2 +-
>  ...ib-termcap-to-linker-flags-to-avoid-.patch |  8 ++--
>  ...-search-system-for-headers-libraries.patch |  6 +--
>  ...-use-prefix-value-from-build-configu.patch | 14 +++---
>  ...e-the-race-condition-in-testSockName.patch | 47 ---
>  ...sts-due-to-load-variability-on-YP-AB.patch | 18 +++
>  ...FLAG_REF-always-for-interned-strings.patch | 33 -
>  ...-detection-of-mips-architecture-for-.patch | 33 ++---
>  ...fig-append-STAGING_LIBDIR-python-sys.patch |  6 +--
>  ...asename-to-replace-CC-for-checking-c.patch | 44 -
>  ...tutils-prefix-is-inside-staging-area.patch | 10 ++--
>  .../python3/avoid_warning_about_tkinter.patch | 21 -
>  .../python/python3/crosspythonpath.patch  | 22 -
>  .../python/python3/makerace.patch | 10 ++--
>  ...python3_3.10.6.bb => python3_3.11.0rc2.bb} | 13 ++---
>  15 files changed, 112 insertions(+), 175 deletions(-)
>  delete mode 100644 
> meta/recipes-devtools/python/python3/0001-Mitigate-the-race-condition-in-testSockName.patch
>  delete mode 100644 
> meta/recipes-devtools/python/python3/0001-Use-FLAG_REF-always-for-interned-strings.patch
>  rename meta/recipes-devtools/python/{python3_3.10.6.bb => 
> python3_3.11.0rc2.bb} (97%)
>
> diff --git a/meta/classes-recipe/python3-dir.bbclass 
> b/meta/classes-recipe/python3-dir.bbclass
> index 912c67253c..d93d337f76 100644
> --- a/meta/classes-recipe/python3-dir.bbclass
> +++ b/meta/classes-recipe/python3-dir.bbclass
> @@ -4,7 +4,7 @@
>  # SPDX-License-Identifier: MIT
>  #
>
> -PYTHON_BASEVERSION = "3.10"
> +PYTHON_BASEVERSION = "3.11"
>  PYTHON_ABI = ""
>  PYTHON_DIR = "python${PYTHON_BASEVERSION}"
>  PYTHON_PN = "python3"
> diff --git 
> a/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
>  
> b/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
> index 62ef6efc28..16a4185704 100644
> --- 
> a/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
> +++ 
> b/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
> @@ -1,4 +1,4 @@
> -From 80f872e4573f542d33f9151453877d566f79 Mon Sep 17 00:00:00 2001
> +From 93ae2ed3fc8be0245e35063c4f63626792f4cd0c Mon Sep 17 00:00:00 2001
>  From: Alexander Kanavin 
>  Date: Fri, 25 Jan 2019 19:04:13 +0100
>  Subject: [PATCH] Do not add /usr/lib/termcap to linker flags to avoid host
> @@ -12,14 +12,14 @@ Signed-off-by: Alexander Kanavin 
>   1 file changed, 1 deletion(-)
>
>  diff --git a/setup.py b/setup.py
> -index 43e807f..11b5cf5 100644
> +index 15d0d45..37ed244 100644
>  --- a/setup.py
>  +++ b/setup.py
> -@@ -1149,7 +1149,6 @@ class PyBuildExt(build_ext):
> +@@ -1109,7 +1109,6 @@ class PyBuildExt(build_ext):
>'termcap'):
>   readline_libs.append('termcap')
>   self.add(Extension('readline', ['readline.c'],
>  -   library_dirs=['/usr/lib/termcap'],
> -extra_link_args=readline_extra_link_args,
>  libraries=readline_libs))
>   else:
> + self.missing.append('readline')
> diff --git 
> a/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
>  
> b/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
> index c790c7b886..f1ffeaebda 100644
> --- 
> a/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
> +++ 
> b/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
> @@ -1,4 +1,4 @@
> -From 7589ab03ad3f7cb4bb092c31273ff22371ac77e4 Mon Sep 17 00:00:00 2001
> +From cc1c59cb7779bab1c20edde9dab9c8b225821095 Mon Sep 17 00:00:00 2001
>  From: Jeremy Puhlman 
>  Date: Wed, 4 Mar 2020 00:06:42 +
>  Subject: [PATCH] Don't search system for headers/libraries
> @@ -11,10 +11,10 @@ Signed-off-by: Jeremy Puhlman 
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
>  diff --git a/setup.py b/setup.py
> -index c3a6b5e..c892537 100644
> +index 7555dcd..b601da2 100644
>  --- a/setup.py
>  +++ b/setup.py
> -@@ -856,8 +856,8 

[OE-core] [RFC PATCH] python3: update 3.10.6 -> 3.11.0rc2

[Alexander Kanavin]

The semaphore fix has landed and is available from 3.11 onwards:
https://github.com/python/cpython/commit/1ee0f94d16f150356a4b9b0a39d44ba1d2d5b9fc

3.11 release schedule: https://peps.python.org/pep-0664/

Drop 0001-Mitigate-the-race-condition-in-testSockName.patch
as it is merged upstream.

Signed-off-by: Alexander Kanavin 
---
 meta/classes-recipe/python3-dir.bbclass   |  2 +-
 ...ib-termcap-to-linker-flags-to-avoid-.patch |  8 ++--
 ...-search-system-for-headers-libraries.patch |  6 +--
 ...-use-prefix-value-from-build-configu.patch | 14 +++---
 ...e-the-race-condition-in-testSockName.patch | 47 ---
 ...sts-due-to-load-variability-on-YP-AB.patch | 18 +++
 ...FLAG_REF-always-for-interned-strings.patch | 33 -
 ...-detection-of-mips-architecture-for-.patch | 33 ++---
 ...fig-append-STAGING_LIBDIR-python-sys.patch |  6 +--
 ...asename-to-replace-CC-for-checking-c.patch | 44 -
 ...tutils-prefix-is-inside-staging-area.patch | 10 ++--
 .../python3/avoid_warning_about_tkinter.patch | 21 -
 .../python/python3/crosspythonpath.patch  | 22 -
 .../python/python3/makerace.patch | 10 ++--
 ...python3_3.10.6.bb => python3_3.11.0rc2.bb} | 13 ++---
 15 files changed, 112 insertions(+), 175 deletions(-)
 delete mode 100644 
meta/recipes-devtools/python/python3/0001-Mitigate-the-race-condition-in-testSockName.patch
 delete mode 100644 
meta/recipes-devtools/python/python3/0001-Use-FLAG_REF-always-for-interned-strings.patch
 rename meta/recipes-devtools/python/{python3_3.10.6.bb => 
python3_3.11.0rc2.bb} (97%)

diff --git a/meta/classes-recipe/python3-dir.bbclass 
b/meta/classes-recipe/python3-dir.bbclass
index 912c67253c..d93d337f76 100644
--- a/meta/classes-recipe/python3-dir.bbclass
+++ b/meta/classes-recipe/python3-dir.bbclass
@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: MIT
 #
 
-PYTHON_BASEVERSION = "3.10"
+PYTHON_BASEVERSION = "3.11"
 PYTHON_ABI = ""
 PYTHON_DIR = "python${PYTHON_BASEVERSION}"
 PYTHON_PN = "python3"
diff --git 
a/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
 
b/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
index 62ef6efc28..16a4185704 100644
--- 
a/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
+++ 
b/meta/recipes-devtools/python/python3/0001-Do-not-add-usr-lib-termcap-to-linker-flags-to-avoid-.patch
@@ -1,4 +1,4 @@
-From 80f872e4573f542d33f9151453877d566f79 Mon Sep 17 00:00:00 2001
+From 93ae2ed3fc8be0245e35063c4f63626792f4cd0c Mon Sep 17 00:00:00 2001
 From: Alexander Kanavin 
 Date: Fri, 25 Jan 2019 19:04:13 +0100
 Subject: [PATCH] Do not add /usr/lib/termcap to linker flags to avoid host
@@ -12,14 +12,14 @@ Signed-off-by: Alexander Kanavin 
  1 file changed, 1 deletion(-)
 
 diff --git a/setup.py b/setup.py
-index 43e807f..11b5cf5 100644
+index 15d0d45..37ed244 100644
 --- a/setup.py
 +++ b/setup.py
-@@ -1149,7 +1149,6 @@ class PyBuildExt(build_ext):
+@@ -1109,7 +1109,6 @@ class PyBuildExt(build_ext):
   'termcap'):
  readline_libs.append('termcap')
  self.add(Extension('readline', ['readline.c'],
 -   library_dirs=['/usr/lib/termcap'],
-extra_link_args=readline_extra_link_args,
 libraries=readline_libs))
  else:
+ self.missing.append('readline')
diff --git 
a/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
 
b/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
index c790c7b886..f1ffeaebda 100644
--- 
a/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
+++ 
b/meta/recipes-devtools/python/python3/0001-Don-t-search-system-for-headers-libraries.patch
@@ -1,4 +1,4 @@
-From 7589ab03ad3f7cb4bb092c31273ff22371ac77e4 Mon Sep 17 00:00:00 2001
+From cc1c59cb7779bab1c20edde9dab9c8b225821095 Mon Sep 17 00:00:00 2001
 From: Jeremy Puhlman 
 Date: Wed, 4 Mar 2020 00:06:42 +
 Subject: [PATCH] Don't search system for headers/libraries
@@ -11,10 +11,10 @@ Signed-off-by: Jeremy Puhlman 
  1 file changed, 2 insertions(+), 2 deletions(-)
 
 diff --git a/setup.py b/setup.py
-index c3a6b5e..c892537 100644
+index 7555dcd..b601da2 100644
 --- a/setup.py
 +++ b/setup.py
-@@ -856,8 +856,8 @@ class PyBuildExt(build_ext):
+@@ -877,8 +877,8 @@ class PyBuildExt(build_ext):
  add_dir_to_list(self.compiler.include_dirs,
  sysconfig.get_config_var("INCLUDEDIR"))
  
diff --git 
a/meta/recipes-devtools/python/python3/0001-Lib-sysconfig.py-use-prefix-value-from-build-configu.patch
 
b/meta/recipes-devtools/python/python3/0001-Lib-sysconfig.py-use-prefix-value-from-build-configu.patch
index 641017edd5..164dc13d20 100644
--- 

[OE-core] [PATCH] icecc.bbclass: fix filename of recipe in comment

[Ulrich Ölmann]

The character '%' is not a valid part of a recipe filename and has been used
here only to suggest a wildcard regarding the recipe's version, see reply [1] in
a companioned thread. Correct that by using the current recipe version instead.

[1] https://lists.yoctoproject.org/g/docs/message/3165

Signed-off-by: Ulrich Ölmann 
---
 meta/classes/icecc.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/icecc.bbclass b/meta/classes/icecc.bbclass
index e478ba488ce0..312e0f17b554 100644
--- a/meta/classes/icecc.bbclass
+++ b/meta/classes/icecc.bbclass
@@ -22,7 +22,7 @@
 # but nothing is sure. ;)
 #
 # If ICECC_ENV_EXEC is set in local.conf, then it should point to the 
icecc-create-env script provided by the user
-# or the default one provided by icecc-create-env_%.bb will be used.
+# or the default one provided by icecc-create-env_0.1.bb will be used.
 # (NOTE that this is a modified version of the needed script and *not the one 
that comes with icecream*).
 #
 # User can specify if specific recipes or recipes inheriting specific classes 
should not use icecc to distribute
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170654): 
https://lists.openembedded.org/g/openembedded-core/message/170654
Mute This Topic: https://lists.openembedded.org/mt/93675187/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] opkg: enable zstd support

[Etienne Cordonnier via lists.openembedded.org]

Also note that zstd's default compression level is 3 per default (from a 1
to 19 scale). I did not test other compression levels.

On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier <
ecordonn...@snapchat.com> wrote:

> I ran a build of core-image-full-cmdline using xz and zstd, using
> pre-populated downloads and sstate-cache directories but with empty tmp
> directory. Here are the numbers:
> zstd:
> bitbake core-image-full-cmdline took 2m52.768s (real), the resulting
> directory tmp/deploy/ipk is 1.6GB big.
> xz:
> bitbake core-image-full-cmdline took 4m14.214s (real), the resulting
> directory tmp/deploy/ipk/ is 996M big
>
> So the build with xz is 47% slower (254/172) than with zstd for this
> "incremental build" use-case.
>
> See the 5 biggest packages, the difference in compression-ratio increases
> with big files:
> build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> total 1.6G
> -rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
> binutils-dbg_2.39-r0_core2-64.ipk
> build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> total 963M
> -rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
> binutils-dbg_2.39-r0_core2-64.ipk
> ...
>
> I think for use-cases where the size of the ipk packages matters, xz is
> better. For use-cases where it does not matter (ipk packages not deployed),
> build-time matters more than compression-ratio and zstd is better.
>
> Regarding the enablement of zstd in opkg per default, I'll send a new
> version of the patch without this line.
> My thinking for enabling zstd per default in opkg was that zstd is already
> enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in
> opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses
> libarchive for doing the actual compression/decompression).
>
> On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart  wrote:
>
>>
>>
>> On 9/13/22 15:20, Alex Feinman wrote:
>> > I do have some numbers. When I was selling this change internally, I
>> > did a comparison on our internal build.
>> >Combined write IPK times (Σ t do_package_write_ipk)
>> > xz 162m 35s
>> > gz 52m 13s
>> > zstd   33m 49s
>> > Compression rate for zstd was closer to xz than to gz but not as good
>> > as xz. For systems that have to cache packages on the device with
>> > limited storage xz might be a better option, but for the bulk of
>> > projects zstd is the best choice
>> > Additionally, zstd offers much faster decompression than xz so the
>> > rootfs build step that includes unpacking all of the ipks, takes 3m
>> > 58s with xz and 2m 44s with zstd.
>> > One other thing of note - if your build includes debug packages, some
>> > may be quite large. E.g. one of our components produces a 2.2 GB debug
>> > package (uncompressed). On large files xz requires a disproportionally
>> > large amount of time resulting in 15 minutes needed to simply write
>> > ipk for the abovementioned packages, whereas zstd took about 45 sec.
>> > For frequent tasks like bitbaking a single package this translates in
>> > a lot of saved time.
>>
>> Those are certainly compelling performance improvements. Assuming that
>> the final data-segment size is within 5%-ish of xz, then I would agree
>> with the rest of the thread that it should probably be the contemporary
>> default.
>>
>> And if we make it the default compressor for OE IPKs, then obviously my
>> criticism in the original PR is satisfied.
>>
>> > Bottom line - I think making xz a default package compressor was not
>> > entirely thought through. gzip or zstd is what the default should be.
>>
>> ZStandard support was only added to opkg last September [1]. Before
>> that, xz was the new hotness that replaced gzip. :)
>>
>> [1]
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294=DwIDaQ=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8=
>>
>>
>> > One final note: I could not find a reasonable explanation for why
>> > opkg-tools require code changes to support a different compressor. BSD
>> > tar and GNU tar both can easily accept compressors that they have no
>> > idea about (via -I option) because all of them provide a unified
>> > command line interface for use in 

[OE-core] Devtool fails if SRCREV is set to autorev #bitbake #poky #devtool #kirkstone

[shibi . cbe]

ello Everyone

We perform yocto build and do devtool modify for one of package it works
if we perform devtool modify for another package without resetting the 
previously modified package we are getting below error in modified package 
during parsing bb step

recipefile:

SRC_URI="git://git.com/pkg/linux;branch=mulberry-5.10;protocol=ssh;name=linux;destsuffix=git

SRCREV = "${AUTOREV}"

PV = "5.10+git${SRCPV}"

Yocto version: Kirkstone
BB_SRCREV_POLICY = "clear"

Error:

ERROR: 
/local/home/test/workspace/poky/../sources/meta-mulberry/recipes-kernel/linux/linux_5.10.bb:
 Error executing a python function in :                                   
                                                                                
| ETA:  --:--:--

The stack trace of python calls that resulted in this exception/failure was:
File: '', lineno: 21, function: 

0017:__anon_56__local_home_test_workspace_poky_meta_classes_kernel_fitimage_bbclass(d)

0018:__anon_795__local_home_test_workspace_poky_meta_classes_kernel_fitimage_bbclass(d)

0019:__anon_7__local_home_test_workspace_poky_meta_classes_kernel_devicetree_bbclass(d)

0020:__anon_717__local_home_test_workspace_poky_meta_classes_kernel_yocto_bbclass(d)

*** 
0021:__anon_145__local_home_test_workspace_poky_meta_classes_externalsrc_bbclass(d)

File: 
'/local/home/shibikri/mulberry_workspace/poky/meta/classes/externalsrc.bbclass',
 lineno: 66, function: 
__anon_145__local_home_test_workspace_poky_meta_classes_externalsrc_bbclass
0062:        else:
0063:            d.setVar('B', '${WORKDIR}/${BPN}-${PV}/')
0064:
0065:        local_srcuri = []
*** 0066:        fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)
0067:        for url in fetch.urls:
0068:            url_data = fetch.ud[url]
0069:            parm = url_data.parm
0070:            if (url_data.type == 'file' or
File: '/local/home/test/workspace/poky/bitbake/lib/bb/fetch2/__init__.py', 
lineno: 1680, function: __init__
1676:
1677:        for url in urls:
1678:            if url not in self.ud:
1679:                try:
*** 1680:                    self.ud[url] = FetchData(url, d, localonly)
1681:                except NonLocalMethod:
1682:                    if localonly:
1683:                        self.ud[url] = None
1684:                        pass
File: '/local/home/test/workspace/poky/bitbake/lib/bb/fetch2/__init__.py', 
lineno: 1317, function: __init__
1313:            logger.warning('Consider updating %s recipe to use "protocol" 
not "proto" in SRC_URI.', d.getVar('PN'))
1314:            self.parm["protocol"] = self.parm.get("proto", None)
1315:
1316:        if hasattr(self.method, "urldata_init"):
*** 1317:            self.method.urldata_init(self, d)
1318:
1319:        if "localpath" in self.parm:
1320:            # if user sets localpath for file, use it instead.\
1321:            self.localpath = self.parm["localpath"]
File: '/local/home/shibikri/test/workspace/poky/bitbake/lib/bb/fetch2/git.py', 
lineno: 249, function: urldata_init
0245:        write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0"
0246:        ud.write_tarballs = write_tarballs != "0" or ud.rebaseable
0247:        ud.write_shallow_tarballs = 
(d.getVar("BB_GENERATE_SHALLOW_TARBALLS") or write_tarballs) != "0"
0248:
*** 0249:        ud.setup_revisions(d)
0250:
0251:        for name in ud.names:
0252:            # Ensure anything that doesn't look like a sha256 
checksum/revision is translated into one
0253:            if not ud.revisions[name] or len(ud.revisions[name]) != 40  or 
(False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
File: '/local/home/test/workspace/poky/bitbake/lib/bb/fetch2/__init__.py', 
lineno: 1347, function: setup_revisions
1343:
1344:    def setup_revisions(self, d):
1345:        self.revisions = {}
1346:        for name in self.names:
*** 1347:            self.revisions[name] = srcrev_internal_helper(self, d, 
name)
1348:
1349:        # add compatibility code for non name specified case
1350:        if len(self.names) == 1:
1351:            self.revision = self.revisions[self.names[0]]
File: '/local/home/test/workspace/poky/bitbake/lib/bb/fetch2/__init__.py', 
lineno: 1212, function: srcrev_internal_helper
1208:
1209:    if srcrev == "INVALID" or not srcrev:
1210:        raise FetchError("Please set a valid SRCREV for url %s (possible 
key names are %s, or use a ;rev=X URL parameter)" % (str(attempts), ud.url), 
ud.url)
1211:    if srcrev == "AUTOINC":
*** 1212:        srcrev = ud.method.latest_revision(ud, d, name)
1213:
1214:    return srcrev
1215:
1216:def get_checksum_file_list(d):
File: '/local/home/test/workspace/poky/bitbake/lib/bb/fetch2/__init__.py', 
lineno: 1624, function: latest_revision
1620:        key = self.generate_revision_key(ud, d, name)
1621:        try:
1622:            return revs[key]
1623:        except KeyError:
*** 1624:            revs[key] = rev = self._latest_revision(ud, d, name)
1625:            return rev
1626:
1627:    def 

[OE-core] [PATCH v2] opkg: add option for zstd support

[Etienne Cordonnier via lists.openembedded.org]

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

Signed-off-by: Alex Feinman 
Signed-off-by: Etienne Cordonnier 
---
 meta/recipes-devtools/opkg/opkg_0.6.0.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb 
b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
index 7b351e8123..4cd589cd29 100644
--- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
+++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
 PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
 PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
 PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
 PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
 EXTRA_OECONF:class-native = 
"--localstatedir=/${@os.path.relpath('${localstatedir}', 
'${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', 
'${STAGING_DIR_NATIVE}')}"
-- 
2.36.1.vfs.0.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170651): 
https://lists.openembedded.org/g/openembedded-core/message/170651
Mute This Topic: https://lists.openembedded.org/mt/93674358/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] opkg: enable zstd support

[Etienne Cordonnier via lists.openembedded.org]

I ran a build of core-image-full-cmdline using xz and zstd, using
pre-populated downloads and sstate-cache directories but with empty tmp
directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting
directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting
directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for this
"incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio increases
with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters, xz is
better. For use-cases where it does not matter (ipk packages not deployed),
build-time matters more than compression-ratio and zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a new
version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already
enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in
opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses
libarchive for doing the actual compression/decompression).

On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart  wrote:

>
>
> On 9/13/22 15:20, Alex Feinman wrote:
> > I do have some numbers. When I was selling this change internally, I
> > did a comparison on our internal build.
> >Combined write IPK times (Σ t do_package_write_ipk)
> > xz 162m 35s
> > gz 52m 13s
> > zstd   33m 49s
> > Compression rate for zstd was closer to xz than to gz but not as good
> > as xz. For systems that have to cache packages on the device with
> > limited storage xz might be a better option, but for the bulk of
> > projects zstd is the best choice
> > Additionally, zstd offers much faster decompression than xz so the
> > rootfs build step that includes unpacking all of the ipks, takes 3m
> > 58s with xz and 2m 44s with zstd.
> > One other thing of note - if your build includes debug packages, some
> > may be quite large. E.g. one of our components produces a 2.2 GB debug
> > package (uncompressed). On large files xz requires a disproportionally
> > large amount of time resulting in 15 minutes needed to simply write
> > ipk for the abovementioned packages, whereas zstd took about 45 sec.
> > For frequent tasks like bitbaking a single package this translates in
> > a lot of saved time.
>
> Those are certainly compelling performance improvements. Assuming that
> the final data-segment size is within 5%-ish of xz, then I would agree
> with the rest of the thread that it should probably be the contemporary
> default.
>
> And if we make it the default compressor for OE IPKs, then obviously my
> criticism in the original PR is satisfied.
>
> > Bottom line - I think making xz a default package compressor was not
> > entirely thought through. gzip or zstd is what the default should be.
>
> ZStandard support was only added to opkg last September [1]. Before
> that, xz was the new hotness that replaced gzip. :)
>
> [1]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294=DwIDaQ=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8=
>
>
> > One final note: I could not find a reasonable explanation for why
> > opkg-tools require code changes to support a different compressor. BSD
> > tar and GNU tar both can easily accept compressors that they have no
> > idea about (via -I option) because all of them provide a unified
> > command line interface for use in pipes. If this were done similar to
> > tar, we could have used any compressor we wanted, including the
> > multithreaded versions (zstdmt)
>
> Well, presumably IPK creation tools can only support the matrix of
> compression algorithms which your opkg binary can decompress. I suppose
> someone could try to implement a plugable compression module system for
> 

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Richard Purdie]

On Wed, 2022-09-14 at 11:09 +0300, Mikko Rapeli wrote:
> Hi,
> 
> Found the root cause. As suggested on #pyco too maybe native openssl
> was mising legacy support.
> It wasn't but loading the on purpose hidden openssl legacy.so was
> failing. It is located in
> recipe-sysroot-native/usr/lib/ossl-modules/legacy.so and only found
> via OPENSSL_MODULES
> variable which wasn't set for python3-native users. These custom
> variables are set in the native openssl
> wrapper script and this also fixes the not found openssl.cnf. Now I
> could send a patch which sets
> the OPENSSL_CONF, OPENSSL_ENGINES and OPENSSL_MODULES paths for python3
> users via python3native.bbclass:
> 
> --- a/meta/classes-recipe/python3native.bbclass
> +++ b/meta/classes-recipe/python3native.bbclass
> @@ -28,3 +28,10 @@ export PYTHONNOUSERSITE = "1"
> 
>  # autoconf macros will use their internal default preference otherwise
>  export PYTHON
> +
> +# find openssl under python, see openssl native wrapper
> +export OPENSSL_CONF="${STAGING_LIBDIR_NATIVE}/ssl-3/openssl.cnf"
> +export SSL_CERT_DIR="${STAGING_LIBDIR_NATIVE}/ssl-3/certs"
> +export SSL_CERT_FILE="${STAGING_LIBDIR_NATIVE}/ssl-3/cert.pem"
> +export OPENSSL_ENGINES="${STAGING_LIBDIR_NATIVE}/engines-3"
> +export OPENSSL_MODULES="${STAGING_LIBDIR_NATIVE}/ossl-modules"
> 
> but that is still a copy of those variables which openssl recipe owns,
> and other users of openssl may
> have similar issues. Is there a way to export these for everyone who
> depends directly or indirectly
> from openssl-native?

Thanks for finding the root cause, this definitely helps a lot.

I'm extremely reluctant to add global exports to the system, they have
nasty effects on sstate checksum files and add overhead I'd prefer not
to have.

I wondered if we could patch openssl to code/find the location of these
files relative to the main library? Presumably that code knows where
the library itself is located so searching a relative path from there
might be something upstream might consider? It is a generic approach
which would work for al the variables too.

It is a patch we'd probably consider carrying if necessary but if there
were upstream buyin, that would obviously be much better.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170649): 
https://lists.openembedded.org/g/openembedded-core/message/170649
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Alexander Kanavin]

On Wed, 14 Sept 2022 at 10:51, Mikko Rapeli  wrote:
> Was there some discussion in the past with openssl upstream about how
> to find config
> files, modules etc, or was there some alignment with e.g. Debian package?

I'm not aware of any such. Please open a ticket, and link to it in any
proposed oe-core patch.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170648): 
https://lists.openembedded.org/g/openembedded-core/message/170648
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2] python3: add PACKAGECONFIG[editline]

[Quentin Schulz]

Hi Etienne,

On 9/13/22 17:25, Etienne Cordonnier via lists.openembedded.org wrote:

editline is a BSD alternative for readline which is GPLv3.

Signed-off-by: Etienne Cordonnier 
---
  meta/recipes-devtools/python/python3_3.10.6.bb | 1 +
  1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/python/python3_3.10.6.bb 
b/meta/recipes-devtools/python/python3_3.10.6.bb
index 1b28728732..1f8b60a7a5 100644
--- a/meta/recipes-devtools/python/python3_3.10.6.bb
+++ b/meta/recipes-devtools/python/python3_3.10.6.bb
@@ -105,6 +105,7 @@ PACKAGECONFIG:class-target ??= "readline gdbm 
${@bb.utils.filter('DISTRO_FEATURE
  PACKAGECONFIG:class-native ??= "readline gdbm"
  PACKAGECONFIG:class-nativesdk ??= "readline gdbm"
  PACKAGECONFIG[readline] = ",,readline"
+PACKAGECONFIG[editline] = "--with-readline=editline,,libedit,,,readline"


I'm wondering if something else isn't missing just below:
https://cgit.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3_3.10.6.bb?h=master#n119

Considering the introducing commit: 
https://cgit.openembedded.org/openembedded-core/commit/?id=79ebbca6943bc66b619671f93ac283ed2cfc8e5c


I'm wondering if there isn't something we can do better here.

I see that the configure.ac script allows to disable readline entirely, 
c.f. https://github.com/python/cpython/blob/v3.10.6/configure.ac#L4945-L5062


--without-readline should work. I hope this can fix the issue the 
aforementioned commit works around by avoiding to always try to build 
the readline module.


Now we'll have another issue is that this without-readline should be 
passed only when both readline and editline PACKAGECONFIG are not used.


I guess we usually do it by having EXTRA_OECONF with the 
--without-readline always set (you'll need --with-readline in the 
readline PACKAGECONFIG though). Then if readline or editline is set, 
--without-readline will be overridden by either --with-readline or 
--with-readline=editline?


Cc'ing Alex as author of the aforementioned commit, hope you don't mind :)

Cheers,
Quentin


  # Use profile guided optimisation by running PyBench inside qemu-user
  PACKAGECONFIG[pgo] = "--enable-optimizations,,qemu-native"
  PACKAGECONFIG[tk] = ",,tk"






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170647): 
https://lists.openembedded.org/g/openembedded-core/message/170647
Mute This Topic: https://lists.openembedded.org/mt/93657764/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Mikko Rapeli]

On Wed, 14 Sept 2022 at 11:45, Alexander Kanavin  wrote:
> But I'm not asking to make wrappers. I'm asking to patch openssl code
> itself to check for an oe-specific environment variable where it's
> making the decision where these things should be looked for. We
> already do this with native python.

Yes, I got this. Was just thinking out loud over email while checking
these details.

Was there some discussion in the past with openssl upstream about how
to find config
files, modules etc, or was there some alignment with e.g. Debian package?

-Mikko

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170646): 
https://lists.openembedded.org/g/openembedded-core/message/170646
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Alexander Kanavin]

But I'm not asking to make wrappers. I'm asking to patch openssl code
itself to check for an oe-specific environment variable where it's
making the decision where these things should be looked for. We
already do this with native python.

Alex

On Wed, 14 Sept 2022 at 10:43, Mikko Rapeli  wrote:
>
> On Wed, 14 Sept 2022 at 11:19, Alexander Kanavin  
> wrote:
> > I can only think of patching openssl to pick up a oe-specific
> > environment variable pointing to staging_libdir_native - making a
> > wrapper for every native binary that sets those variables doesn't seem
> > feasible.
>
> Hmm. I'm a bit worried that things like git, subversion, http clients
> using openssl shared libraries are
> not configured correctly to check certificates.
>
> The default use cases in shared library work, only the legacy module
> is tricky to find. But config
> and certificate store are tricky.
>
> Basically without the cert paths TLS connections don't verify verify:
>
> # OPENSSL_CONF="" OPENSSL_ENGINES="" OPENSSL_MODULES=""
> SSL_CERT_DIR="" SSL_CERT_FILE="" openssl.real s_client -connect
> www.google.com:443 < /dev/null | grep ^Verification
> depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
> verify error:num=20:unable to get local issuer certificate
> verify return:1
> depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
> verify return:1
> depth=0 CN = www.google.com
> verify return:1
> DONE
> Verification error: unable to get local issuer certificate
>
> With the openssl wrapper and if those variables are set correcty for
> the native sysroot paths, then certs get verified:
>
> # openssl s_client -connect www.google.com:443 < /dev/null | grep 
> ^Verification
> depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
> verify return:1
> depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
> verify return:1
> depth=0 CN = www.google.com
> verify return:1
> DONE
> Verification: OK
>
> For example subversion-native doesn't work without the certs and hangs
> on the question:
>
> # OPENSSL_CONF="" OPENSSL_ENGINES="" OPENSSL_MODULES=""
> SSL_CERT_DIR="" SSL_CERT_FILE="" svn co https://www.google.com
> Error validating server certificate for 'https://www.google.com:443':
>  - The certificate is not issued by a trusted authority. Use the
>fingerprint to validate the certificate manually!
> Certificate information:
>  - Hostname: www.google.com
>  - Valid: from Aug 29 08:20:19 2022 GMT until Nov 21 08:20:18 2022 GMT
>  - Issuer: GTS CA 1C3, Google Trust Services LLC, US
>  - Fingerprint: BB:17:2A:6B:F5:43:2C:2B:53:D6:EC:11:21:D3:54:EF:9A:95:19:33
> (R)eject, accept (t)emporarily or accept (p)ermanently?
>
> If the openssl variables are set it works:
>
> # set|grep SSL
> OPENSSL_CONF=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/openssl.cnf
> OPENSSL_ENGINES=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/engines-3
> OPENSSL_MODULES=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ossl-modules
> SSL_CERT_DIR=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/certs
> SSL_CERT_FILE=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/cert.pem
> # svn co https://www.google.com
> svn: E170013: Unable to connect to a repository at URL 
> 'https://www.google.com'
> svn: E175015: The HTTP method 'OPTIONS' is not allowed on '/'
>
> Wrappers here and there doesn't seem like a good idea..
>
> Cheers,
>
> -Mikko

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170645): 
https://lists.openembedded.org/g/openembedded-core/message/170645
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Mikko Rapeli]

On Wed, 14 Sept 2022 at 11:19, Alexander Kanavin  wrote:
> I can only think of patching openssl to pick up a oe-specific
> environment variable pointing to staging_libdir_native - making a
> wrapper for every native binary that sets those variables doesn't seem
> feasible.

Hmm. I'm a bit worried that things like git, subversion, http clients
using openssl shared libraries are
not configured correctly to check certificates.

The default use cases in shared library work, only the legacy module
is tricky to find. But config
and certificate store are tricky.

Basically without the cert paths TLS connections don't verify verify:

# OPENSSL_CONF="" OPENSSL_ENGINES="" OPENSSL_MODULES=""
SSL_CERT_DIR="" SSL_CERT_FILE="" openssl.real s_client -connect
www.google.com:443 < /dev/null | grep ^Verification
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = www.google.com
verify return:1
DONE
Verification error: unable to get local issuer certificate

With the openssl wrapper and if those variables are set correcty for
the native sysroot paths, then certs get verified:

# openssl s_client -connect www.google.com:443 < /dev/null | grep ^Verification
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = www.google.com
verify return:1
DONE
Verification: OK

For example subversion-native doesn't work without the certs and hangs
on the question:

# OPENSSL_CONF="" OPENSSL_ENGINES="" OPENSSL_MODULES=""
SSL_CERT_DIR="" SSL_CERT_FILE="" svn co https://www.google.com
Error validating server certificate for 'https://www.google.com:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: www.google.com
 - Valid: from Aug 29 08:20:19 2022 GMT until Nov 21 08:20:18 2022 GMT
 - Issuer: GTS CA 1C3, Google Trust Services LLC, US
 - Fingerprint: BB:17:2A:6B:F5:43:2C:2B:53:D6:EC:11:21:D3:54:EF:9A:95:19:33
(R)eject, accept (t)emporarily or accept (p)ermanently?

If the openssl variables are set it works:

# set|grep SSL
OPENSSL_CONF=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/openssl.cnf
OPENSSL_ENGINES=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/engines-3
OPENSSL_MODULES=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ossl-modules
SSL_CERT_DIR=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/certs
SSL_CERT_FILE=/home/builder/poky/build_master/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/ssl-3/cert.pem
# svn co https://www.google.com
svn: E170013: Unable to connect to a repository at URL 'https://www.google.com'
svn: E175015: The HTTP method 'OPTIONS' is not allowed on '/'

Wrappers here and there doesn't seem like a good idea..

Cheers,

-Mikko

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170644): 
https://lists.openembedded.org/g/openembedded-core/message/170644
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Alexander Kanavin]

On Wed, 14 Sept 2022 at 10:17, ChenQi  wrote:
> The codes are in: https://github.com/WindRiver-Labs/wrlinux-x
>
> e.g.
>
> ./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)
>
> ./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all
> supported layers, together with their needed tarballs/git repos)
>
> ./wrlinux-x/setup.sh (this pulls down the default list of layers)
>
> ./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list
> of layers + meta-xxx and its dependencies)
>
> And the final list of layers are in
> /config/bblayers.conf.sample, which will be updated if
> needed when re-running the setup.sh script.

May I suggest that the tool skips the bblayers.conf.sample step
altogether, and simply writes the desired layers directly into
bblayers.conf? It can do this with 'bitbake-layers add-layer'. Is
there a need for doing the intermediate step of generating the
template?

Just to be clear where I am coming from: anything in TEMPLATECONF must
not be auto-generated during build setup, and must be under version
control. We are trying to standardize layer setup and configuration
handling, and this would be a necessary step towards that.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170643): 
https://lists.openembedded.org/g/openembedded-core/message/170643
Mute This Topic: https://lists.openembedded.org/mt/93368468/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Alexander Kanavin]

I can only think of patching openssl to pick up a oe-specific
environment variable pointing to staging_libdir_native - making a
wrapper for every native binary that sets those variables doesn't seem
feasible.

Alex

On Wed, 14 Sept 2022 at 10:09, Mikko Rapeli  wrote:
>
> Hi,
>
> Found the root cause. As suggested on #pyco too maybe native openssl
> was mising legacy support.
> It wasn't but loading the on purpose hidden openssl legacy.so was
> failing. It is located in
> recipe-sysroot-native/usr/lib/ossl-modules/legacy.so and only found
> via OPENSSL_MODULES
> variable which wasn't set for python3-native users. These custom
> variables are set in the native openssl
> wrapper script and this also fixes the not found openssl.cnf. Now I
> could send a patch which sets
> the OPENSSL_CONF, OPENSSL_ENGINES and OPENSSL_MODULES paths for python3
> users via python3native.bbclass:
>
> --- a/meta/classes-recipe/python3native.bbclass
> +++ b/meta/classes-recipe/python3native.bbclass
> @@ -28,3 +28,10 @@ export PYTHONNOUSERSITE = "1"
>
>  # autoconf macros will use their internal default preference otherwise
>  export PYTHON
> +
> +# find openssl under python, see openssl native wrapper
> +export OPENSSL_CONF="${STAGING_LIBDIR_NATIVE}/ssl-3/openssl.cnf"
> +export SSL_CERT_DIR="${STAGING_LIBDIR_NATIVE}/ssl-3/certs"
> +export SSL_CERT_FILE="${STAGING_LIBDIR_NATIVE}/ssl-3/cert.pem"
> +export OPENSSL_ENGINES="${STAGING_LIBDIR_NATIVE}/engines-3"
> +export OPENSSL_MODULES="${STAGING_LIBDIR_NATIVE}/ossl-modules"
>
> but that is still a copy of those variables which openssl recipe owns,
> and other users of openssl may
> have similar issues. Is there a way to export these for everyone who
> depends directly or indirectly
> from openssl-native?
>
> Cheers,
>
> -Mikko
>
> On Tue, 13 Sept 2022 at 15:24, Richard Purdie
>  wrote:
> >
> > On Tue, 2022-09-13 at 14:13 +0300, Mikko Rapeli wrote:
> > > On Tue, 13 Sept 2022 at 13:34, Richard Purdie
> > >  wrote:
> > > > Are you using uninative? I'd have expected glibc and pthreads to come
> > > > from there rather than the host.
> > >
> > > Yes, using uninative, not host libc, sorry. Added full list of
> > > openat()'d files to the end of this email, from master branch with
> > > this patch applied.
> > > Delta to without this patch is just a few python modules. I can't see
> > > anything wrong in that list.
> >
> > It looks correct to me too.
> >
> > It is weird it is ubuntu 18.04 as we had a lot of problems with the
> > rust SDK work specifically on that platform. The problem there was rust
> > running things with LD_LIBRARY_PATH set which meant host tools tried to
> > use the SDK sysroot libs which then broke in interesting ways. It was
> > specific to the form of the version mismatch on 18.04.
> >
> > I have no idea if there is a connection, your strace output certainly
> > suggests not.
> > >
> > > > > And then in bitbake -c devshell busybox:
> > > > >
> > > > > # python3 -c  "from OpenSSL import crypto"
> > > > >
> > > > > I guess there is no way to add a test like that for 
> > > > > python3-cryptography-native?
> > > >
> > > > You could probably put that in do_configure to test it?
> > >
> > > Yes, on my layer and recipes I can do this. But I'd rather upstream
> > > the test to python3-cryprography-native or somewhere else but I guess
> > > native recipes don't have selftests or similar.
> >
> > I was wondering about putting something into upstream recipe...
> >
> > We do somehow need to get more information about what is breaking here
> > :/.
> >
> > Cheers,
> >
> > Richard
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170642): 
https://lists.openembedded.org/g/openembedded-core/message/170642
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Chen Qi]

On 9/14/22 16:03, Alexander Kanavin wrote:

On Wed, 14 Sept 2022 at 09:42, ChenQi  wrote:


Yes. The bblayers.conf.sample is generated dynamically according to
project setup.

Can we step back for a moment, I'd like to know more about how you do
this. Is the code that does dynamic generation of the list in bblayers
published somewhere? How a decision to include (or not) a layer is
made?

Alex


The codes are in: https://github.com/WindRiver-Labs/wrlinux-x

e.g.

./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)

./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all 
supported layers, together with their needed tarballs/git repos)


./wrlinux-x/setup.sh (this pulls down the default list of layers)

./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list 
of layers + meta-xxx and its dependencies)


And the final list of layers are in 
/config/bblayers.conf.sample, which will be updated if 
needed when re-running the setup.sh script.


Regards,

Qi



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170641): 
https://lists.openembedded.org/g/openembedded-core/message/170641
Mute This Topic: https://lists.openembedded.org/mt/93368468/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Mikko Rapeli]

Hi,

Found the root cause. As suggested on #pyco too maybe native openssl
was mising legacy support.
It wasn't but loading the on purpose hidden openssl legacy.so was
failing. It is located in
recipe-sysroot-native/usr/lib/ossl-modules/legacy.so and only found
via OPENSSL_MODULES
variable which wasn't set for python3-native users. These custom
variables are set in the native openssl
wrapper script and this also fixes the not found openssl.cnf. Now I
could send a patch which sets
the OPENSSL_CONF, OPENSSL_ENGINES and OPENSSL_MODULES paths for python3
users via python3native.bbclass:

--- a/meta/classes-recipe/python3native.bbclass
+++ b/meta/classes-recipe/python3native.bbclass
@@ -28,3 +28,10 @@ export PYTHONNOUSERSITE = "1"

 # autoconf macros will use their internal default preference otherwise
 export PYTHON
+
+# find openssl under python, see openssl native wrapper
+export OPENSSL_CONF="${STAGING_LIBDIR_NATIVE}/ssl-3/openssl.cnf"
+export SSL_CERT_DIR="${STAGING_LIBDIR_NATIVE}/ssl-3/certs"
+export SSL_CERT_FILE="${STAGING_LIBDIR_NATIVE}/ssl-3/cert.pem"
+export OPENSSL_ENGINES="${STAGING_LIBDIR_NATIVE}/engines-3"
+export OPENSSL_MODULES="${STAGING_LIBDIR_NATIVE}/ossl-modules"

but that is still a copy of those variables which openssl recipe owns,
and other users of openssl may
have similar issues. Is there a way to export these for everyone who
depends directly or indirectly
from openssl-native?

Cheers,

-Mikko

On Tue, 13 Sept 2022 at 15:24, Richard Purdie
 wrote:
>
> On Tue, 2022-09-13 at 14:13 +0300, Mikko Rapeli wrote:
> > On Tue, 13 Sept 2022 at 13:34, Richard Purdie
> >  wrote:
> > > Are you using uninative? I'd have expected glibc and pthreads to come
> > > from there rather than the host.
> >
> > Yes, using uninative, not host libc, sorry. Added full list of
> > openat()'d files to the end of this email, from master branch with
> > this patch applied.
> > Delta to without this patch is just a few python modules. I can't see
> > anything wrong in that list.
>
> It looks correct to me too.
>
> It is weird it is ubuntu 18.04 as we had a lot of problems with the
> rust SDK work specifically on that platform. The problem there was rust
> running things with LD_LIBRARY_PATH set which meant host tools tried to
> use the SDK sysroot libs which then broke in interesting ways. It was
> specific to the form of the version mismatch on 18.04.
>
> I have no idea if there is a connection, your strace output certainly
> suggests not.
> >
> > > > And then in bitbake -c devshell busybox:
> > > >
> > > > # python3 -c  "from OpenSSL import crypto"
> > > >
> > > > I guess there is no way to add a test like that for 
> > > > python3-cryptography-native?
> > >
> > > You could probably put that in do_configure to test it?
> >
> > Yes, on my layer and recipes I can do this. But I'd rather upstream
> > the test to python3-cryprography-native or somewhere else but I guess
> > native recipes don't have selftests or similar.
>
> I was wondering about putting something into upstream recipe...
>
> We do somehow need to get more information about what is breaking here
> :/.
>
> Cheers,
>
> Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170640): 
https://lists.openembedded.org/g/openembedded-core/message/170640
Mute This Topic: https://lists.openembedded.org/mt/93651845/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Alexander Kanavin]

On Wed, 14 Sept 2022 at 09:42, ChenQi  wrote:

> Yes. The bblayers.conf.sample is generated dynamically according to
> project setup.

Can we step back for a moment, I'd like to know more about how you do
this. Is the code that does dynamic generation of the list in bblayers
published somewhere? How a decision to include (or not) a layer is
made?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170639): 
https://lists.openembedded.org/g/openembedded-core/message/170639
Mute This Topic: https://lists.openembedded.org/mt/93368468/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Chen Qi]

On 9/14/22 14:39, Alexander Kanavin wrote:

On Wed, 14 Sept 2022 at 07:49, ChenQi  wrote:

I'm reluctant to agree that this is like machine and distro, because
it's a hard requirement that machine and distro definitions be under
some layer, otherwise how can bitbake get info about it? But
TEMPLATECONF seems to be a different case, it could be everywhere
because it's used by the project setup script.

In our case, the TEMPLATECONF is /config/, and layers are
//. This directory hierarchy has been working for
years until recent changes.

Do you think such directory hierarchy makes sense? How about we deleting
such check if there's no strong technical reason to do so? By 'strong
technical reason', I mean that some codes in oe-core are written based
on this assumption (this is the part I'm sure about).

Again, it's not about only code. It's about humans too: we benefit
from having things where we expect them to be.


$TEMPLATECONF/bblayers.conf.sample has a list of layers. These layers 
may have dependencies on each other or they may not. Which layer do you 
think should this TEMPLATECONF locate?


TEMPLATECONF, by its nature, is a project setup variable. It logically 
does not belong to any layer.


Why would people expect some project level variable to point to some 
directory under a layer?




If your templates are
in meta-layer/conf/templates/ you do not have to document, explain or
support this;


Again, why this meta-layer should have knowledge about the whole 
project? It should be the project that has knowledge about layer, not 
the other way around.




anyone new to the project will simply pick this up from
prior experience or official documentation.


When users see a file in a layer that refers to other layers which this 
layer does not depend on and not been dependent upon, they may ask why.





That said, there is already code that makes this assumption too: both
'bitbake-layers save-template-conf' and upcoming 'oe-setup-build' (the
patch was sent for review here) consider only
meta-layer/conf/templates.


Give it a second thought.


I have to ask the same question to you: is there a strong technical
reason that you cannot move the templates to the new standard
location?


Yes. The bblayers.conf.sample is generated dynamically according to 
project setup.


That said, I could of course create a useless layer that does nothing 
but only holds these sample files to satisfy this sanity check. But I do 
think this sanity check is logically wrong.


Again, project contains layers, project setup could choose to use layers 
the project wants. Forcing a project level variable to point to a 
location under a layer is not reasonable.


Regards,

Qi



Alex




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170638): 
https://lists.openembedded.org/g/openembedded-core/message/170638
Mute This Topic: https://lists.openembedded.org/mt/93368468/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH v2] busybox: add devmem 128-bit support

[Yu, Mingli]

From: Mingli Yu 

Add devmem 128-bit support [1].

[1] 
https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1

Signed-off-by: Mingli Yu 
---
 .../0001-devmem-add-128-bit-width.patch   | 128 ++
 meta/recipes-core/busybox/busybox/defconfig   |   2 +-
 meta/recipes-core/busybox/busybox_1.35.0.bb   |   1 +
 3 files changed, 130 insertions(+), 1 deletion(-)
 create mode 100644 
meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch

diff --git 
a/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch 
b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
new file mode 100644
index 00..985e2bf1d9
--- /dev/null
+++ b/meta/recipes-core/busybox/busybox/0001-devmem-add-128-bit-width.patch
@@ -0,0 +1,128 @@
+From d432049f288c9acdc4a7caa729c68ceba3c5dca1 Mon Sep 17 00:00:00 2001
+From: Aaro Koskinen 
+Date: Thu, 25 Aug 2022 18:47:02 +0300
+Subject: [PATCH] devmem: add 128-bit width
+
+Add 128-bit width if the compiler provides the needed type.
+
+function old new   delta
+devmem_main  405 464 +59
+.rodata   109025  109043 +18
+--
+(add/remove: 0/0 grow/shrink: 2/0 up/down: 77/0)   Total: 77 bytes
+
+Upstream-Status: Backport 
[https://git.busybox.net/busybox/commit/?id=d432049f288c9acdc4a7caa729c68ceba3c5dca1]
+
+Signed-off-by: Aaro Koskinen 
+Signed-off-by: Aaro Koskinen 
+Signed-off-by: Denys Vlasenko 
+Signed-off-by: Mingli Yu 
+---
+ miscutils/devmem.c | 68 ++
+ 1 file changed, 44 insertions(+), 24 deletions(-)
+
+diff --git a/miscutils/devmem.c b/miscutils/devmem.c
+index f9f0276bc..f21621bd6 100644
+--- a/miscutils/devmem.c
 b/miscutils/devmem.c
+@@ -29,7 +29,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+ {
+   void *map_base, *virt_addr;
+   uint64_t read_result;
+-  uint64_t writeval = writeval; /* for compiler */
+   off_t target;
+   unsigned page_size, mapped_size, offset_in_page;
+   int fd;
+@@ -64,9 +63,6 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+   width = strchrnul(bhwl, (argv[2][0] | 0x20)) - bhwl;
+   width = sizes[width];
+   }
+-  /* VALUE */
+-  if (argv[3])
+-  writeval = bb_strtoull(argv[3], NULL, 0);
+   } else { /* argv[2] == NULL */
+   /* make argv[3] to be a valid thing to fetch */
+   argv--;
+@@ -96,28 +92,46 @@ int devmem_main(int argc UNUSED_PARAM, char **argv)
+   virt_addr = (char*)map_base + offset_in_page;
+ 
+   if (!argv[3]) {
+-  switch (width) {
+-  case 8:
+-  read_result = *(volatile uint8_t*)virt_addr;
+-  break;
+-  case 16:
+-  read_result = *(volatile uint16_t*)virt_addr;
+-  break;
+-  case 32:
+-  read_result = *(volatile uint32_t*)virt_addr;
+-  break;
+-  case 64:
+-  read_result = *(volatile uint64_t*)virt_addr;
+-  break;
+-  default:
+-  bb_simple_error_msg_and_die("bad width");
++#ifdef __SIZEOF_INT128__
++  if (width == 128) {
++  unsigned __int128 rd =
++  *(volatile unsigned __int128 *)virt_addr;
++  printf("0x%016llX%016llX\n",
++  (unsigned long long)(uint64_t)(rd >> 64),
++  (unsigned long long)(uint64_t)rd
++  );
++  } else
++#endif
++  {
++  switch (width) {
++  case 8:
++  read_result = *(volatile uint8_t*)virt_addr;
++  break;
++  case 16:
++  read_result = *(volatile uint16_t*)virt_addr;
++  break;
++  case 32:
++  read_result = *(volatile uint32_t*)virt_addr;
++  break;
++  case 64:
++  read_result = *(volatile uint64_t*)virt_addr;
++  break;
++  default:
++  bb_simple_error_msg_and_die("bad width");
++  }
++//printf("Value at address 0x%"OFF_FMT"X (%p): 0x%llX\n",
++//target, virt_addr,
++//(unsigned long long)read_result);
++  /* Zero-padded output shows the width of access just 
done */
++  

Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Alexander Kanavin]

On Wed, 14 Sept 2022 at 07:49, ChenQi  wrote:
> I'm reluctant to agree that this is like machine and distro, because
> it's a hard requirement that machine and distro definitions be under
> some layer, otherwise how can bitbake get info about it? But
> TEMPLATECONF seems to be a different case, it could be everywhere
> because it's used by the project setup script.
>
> In our case, the TEMPLATECONF is /config/, and layers are
> //. This directory hierarchy has been working for
> years until recent changes.
>
> Do you think such directory hierarchy makes sense? How about we deleting
> such check if there's no strong technical reason to do so? By 'strong
> technical reason', I mean that some codes in oe-core are written based
> on this assumption (this is the part I'm sure about).

Again, it's not about only code. It's about humans too: we benefit
from having things where we expect them to be. If your templates are
in meta-layer/conf/templates/ you do not have to document, explain or
support this; anyone new to the project will simply pick this up from
prior experience or official documentation.

That said, there is already code that makes this assumption too: both
'bitbake-layers save-template-conf' and upcoming 'oe-setup-build' (the
patch was sent for review here) consider only
meta-layer/conf/templates.

I have to ask the same question to you: is there a strong technical
reason that you cannot move the templates to the new standard
location?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170636): 
https://lists.openembedded.org/g/openembedded-core/message/170636
Mute This Topic: https://lists.openembedded.org/mt/93368468/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH][dunfell 1/2] gst-plugins-good: fix several CVE

[Lee Chee Yang]

From: Chee Yang Lee 

backport fix for:
CVE-2022-1920
CVE-2022-1921
CVE-2022-1922
CVE-2022-1923
CVE-2022-1924
CVE-2022-1925
CVE-2022-2122

also set ignore at gstreamer1.0_1.16.3.bb

Signed-off-by: Chee Yang Lee 
---
 .../CVE-2022-1920.patch   |  59 +
 .../CVE-2022-1921.patch   |  69 ++
 .../CVE-2022-1922-1923-1924-1925.patch| 214 ++
 .../CVE-2022-2122.patch   |  60 +
 .../gstreamer1.0-plugins-good_1.16.3.bb   |   4 +
 .../gstreamer/gstreamer1.0_1.16.3.bb  |   7 +
 6 files changed, 413 insertions(+)
 create mode 100644 
meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1920.patch
 create mode 100644 
meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1921.patch
 create mode 100644 
meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1922-1923-1924-1925.patch
 create mode 100644 
meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-2122.patch

diff --git 
a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1920.patch
 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1920.patch
new file mode 100644
index 00..ee33c5564d
--- /dev/null
+++ 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1920.patch
@@ -0,0 +1,59 @@
+From cf887f1b8e228bff6e19829e6d03995d70ad739d Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Sebastian=20Dr=C3=B6ge?= 
+Date: Wed, 18 May 2022 10:23:15 +0300
+Subject: [PATCH] matroskademux: Avoid integer-overflow resulting in heap
+ corruption in WavPack header handling code
+
+blocksize + WAVPACK4_HEADER_SIZE might overflow gsize, which then
+results in allocating a very small buffer. Into that buffer blocksize
+data is memcpy'd later which then causes out of bound writes and can
+potentially lead to anything from crashes to remote code execution.
+
+Thanks to Adam Doupe for analyzing and reporting the issue.
+
+CVE: CVE-2022-1920
+
+https://gstreamer.freedesktop.org/security/sa-2022-0004.html
+
+Fixes https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1226
+
+Part-of: 

+
+https://gitlab.freedesktop.org/gstreamer/gstreamer/-/commit/0df0dd7fe388174e4835eda4526b47f470a56370
+Upstream-Status: Backport
+Signed-off-by: Chee Yang Lee 
+---
+ .../gst/matroska/matroska-demux.c | 10 +-
+ 1 file changed, 9 insertions(+), 1 deletion(-)
+
+diff --git a/gst/matroska/matroska-demux.c b/gst/matroska/matroska-demux.c
+index 64cc6be60be..01d754c3eb9 100644
+--- a/gst/matroska/matroska-demux.c
 b/gst/matroska/matroska-demux.c
+@@ -3933,7 +3933,8 @@ gst_matroska_demux_add_wvpk_header (GstElement * element,
+   } else {
+ guint8 *outdata = NULL;
+ gsize buf_size, size;
+-guint32 block_samples, flags, crc, blocksize;
++guint32 block_samples, flags, crc;
++gsize blocksize;
+ GstAdapter *adapter;
+ 
+ adapter = gst_adapter_new ();
+@@ -3974,6 +3975,13 @@ gst_matroska_demux_add_wvpk_header (GstElement * 
element,
+ return GST_FLOW_ERROR;
+   }
+ 
++  if (blocksize > G_MAXSIZE - WAVPACK4_HEADER_SIZE) {
++GST_ERROR_OBJECT (element, "Too big wavpack buffer");
++gst_buffer_unmap (*buf, );
++g_object_unref (adapter);
++return GST_FLOW_ERROR;
++  }
++
+   g_assert (newbuf == NULL);
+ 
+   newbuf =
+-- 
+GitLab
+
diff --git 
a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1921.patch
 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1921.patch
new file mode 100644
index 00..99dbb2b1b0
--- /dev/null
+++ 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good/CVE-2022-1921.patch
@@ -0,0 +1,69 @@
+From f503caad676971933dc0b52c4b313e5ef0d6dbb0 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Sebastian=20Dr=C3=B6ge?= 
+Date: Wed, 18 May 2022 12:00:48 +0300
+Subject: [PATCH] avidemux: Fix integer overflow resulting in heap corruption
+ in DIB buffer inversion code
+
+Check that width*bpp/8 doesn't overflow a guint and also that
+height*stride fits into the provided buffer without overflowing.
+
+Thanks to Adam Doupe for analyzing and reporting the issue.
+
+CVE: CVE-2022-1921
+
+See https://gstreamer.freedesktop.org/security/sa-2022-0001.html
+
+Fixes https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1224
+
+Part-of: 

+
+https://gitlab.freedesktop.org/gstreamer/gstreamer/-/commit/f503caad676971933dc0b52c4b313e5ef0d6dbb0
+Upstream-Status: Backport
+Signed-off-by: Chee Yang Lee 
+---
+ .../gst/avi/gstavidemux.c  | 17 ++---
+ 1 file changed, 14 insertions(+), 3 deletions(-)
+
+diff --git a/gst/avi/gstavidemux.c b/gst/avi/gstavidemux.c
+index eafe865494c..0d18a6495c7 100644
+--- a/gst/avi/gstavidemux.c
 b/gst/avi/gstavidemux.c
+@@ -4973,8 +4973,8 @@ 

[OE-core] [PATCH][dunfell 2/2] qemu: fix and ignore several CVEs

[Lee Chee Yang]

From: Chee Yang Lee 

backport fixes:
CVE-2020-13754, backport patches as debian security tracker notes
  https://security-tracker.debian.org/tracker/CVE-2020-13754

CVE-2021-3713
CVE-2021-3748
CVE-2021-3930
CVE-2021-4206
CVE-2021-4207
CVE-2022-0216, does not include qtest in patches, the qtest code were not 
available in v4.2.

Ignore:
CVE-2020-27661, issue introduced in v5.1.0-rc0
https://security-tracker.debian.org/tracker/CVE-2020-27661

Signed-off-by: Chee Yang Lee 
---
 meta/recipes-devtools/qemu/qemu.inc   |  14 ++
 .../qemu/qemu/CVE-2020-13754-1.patch  |  91 +
 .../qemu/qemu/CVE-2020-13754-2.patch  |  69 ++
 .../qemu/qemu/CVE-2020-13754-3.patch  |  65 +
 .../qemu/qemu/CVE-2020-13754-4.patch  |  39 ++
 .../qemu/qemu/CVE-2021-3713.patch |  67 ++
 .../qemu/qemu/CVE-2021-3748.patch | 124 ++
 .../qemu/qemu/CVE-2021-3930.patch |  53 
 .../qemu/qemu/CVE-2021-4206.patch |  89 +
 .../qemu/qemu/CVE-2021-4207.patch |  43 ++
 .../qemu/qemu/CVE-2022-0216-1.patch   |  42 ++
 .../qemu/qemu/CVE-2022-0216-2.patch   |  52 
 12 files changed, 748 insertions(+)
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2020-13754-1.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2020-13754-2.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2020-13754-3.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2020-13754-4.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2021-3713.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2021-3748.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2021-3930.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2021-4206.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2021-4207.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2022-0216-1.patch
 create mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2022-0216-2.patch

diff --git a/meta/recipes-devtools/qemu/qemu.inc 
b/meta/recipes-devtools/qemu/qemu.inc
index a773068499..c1db723e90 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -100,6 +100,17 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
file://CVE-2020-13791.patch \
file://CVE-2022-35414.patch \
file://CVE-2020-27821.patch \
+   file://CVE-2020-13754-1.patch \
+   file://CVE-2020-13754-2.patch \
+   file://CVE-2020-13754-3.patch \
+   file://CVE-2020-13754-4.patch \
+   file://CVE-2021-3713.patch \
+   file://CVE-2021-3748.patch \
+   file://CVE-2021-3930.patch \
+   file://CVE-2021-4206.patch \
+   file://CVE-2021-4207.patch \
+   file://CVE-2022-0216-1.patch \
+   file://CVE-2022-0216-2.patch \
"
 UPSTREAM_CHECK_REGEX = "qemu-(?P\d+(\.\d+)+)\.tar"
 
@@ -117,6 +128,9 @@ CVE_CHECK_WHITELIST += "CVE-2007-0998"
 # https://bugzilla.redhat.com/show_bug.cgi?id=1609015#c11
 CVE_CHECK_WHITELIST += "CVE-2018-18438"
 
+# the issue introduced in v5.1.0-rc0
+CVE_CHECK_WHITELIST += "CVE-2020-27661"
+
 COMPATIBLE_HOST_mipsarchn32 = "null"
 COMPATIBLE_HOST_mipsarchn64 = "null"
 
diff --git a/meta/recipes-devtools/qemu/qemu/CVE-2020-13754-1.patch 
b/meta/recipes-devtools/qemu/qemu/CVE-2020-13754-1.patch
new file mode 100644
index 00..fdfff9d81d
--- /dev/null
+++ b/meta/recipes-devtools/qemu/qemu/CVE-2020-13754-1.patch
@@ -0,0 +1,91 @@
+From 5d971f9e672507210e77d020d89e0e89165c8fc9 Mon Sep 17 00:00:00 2001
+From: "Michael S. Tsirkin" 
+Date: Wed, 10 Jun 2020 09:47:49 -0400
+Subject: [PATCH] memory: Revert "memory: accept mismatching sizes in
+ memory_region_access_valid"
+
+Memory API documentation documents valid .min_access_size and .max_access_size
+fields and explains that any access outside these boundaries is blocked.
+
+This is what devices seem to assume.
+
+However this is not what the implementation does: it simply
+ignores the boundaries unless there's an "accepts" callback.
+
+Naturally, this breaks a bunch of devices.
+
+Revert to the documented behaviour.
+
+Devices that want to allow any access can just drop the valid field,
+or add the impl field to have accesses converted to appropriate
+length.
+
+Cc: qemu-sta...@nongnu.org
+Reviewed-by: Richard Henderson 
+Fixes: CVE-2020-13754
+Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1842363
+Fixes: a014ed07bd5a ("memory: accept mismatching sizes in 
memory_region_access_valid")
+Signed-off-by: Michael S. Tsirkin 
+Message-Id: <20200610134731.1514409-1-...@redhat.com>
+Signed-off-by: Paolo Bonzini 
+
+https://git.qemu.org/?p=qemu.git;a=patch;h=5d971f9e672507210e77d020d89e0e89165c8fc9
+CVE: CVE-2020-13754
+Upstream-Status: Backport
+Signed-off-by: Chee Yang Lee 
+---
+ memory.c | 29 +
+ 1 file