Re: [Openembedded-architecture] Cleanup of WORKDIR by separating do_unpack

[Richard Purdie]

On Mon, 2024-04-29 at 09:27 +0200, Esben Haabendal wrote:
> "Richard Purdie"  writes:
> 
> > ...
> > 
> > b) Change recipes that use S = "${WORKDIR}" to
> > 
> > S = "${WORKDIR}/sources"
> > UNPACKDIR = "${S}"
> > 
> > ...
> > 
> > This in theory then sets the stage for us to change UNPACKDIR to be
> > something like ${WORKDIR}/unpack-sources and we could put links in
> > for
> > "${BPN}-${PV}" and 'git' into WORKDIR. We could then always clean
> > UNPACKDIR.
> 
> I might just a bit thickheaded, but exactly how would the end result
> then look like?
> 
> Will there be both a ${WORKDIR}/sources and ${WORKDIR}/unpack-sources
> ?
> 
> And if so, what would be the relation between these?

There would just be a ${WORKDIR}/sources since UNPACKDIR is being
overridden.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2001): 
https://lists.openembedded.org/g/openembedded-architecture/message/2001
Mute This Topic: https://lists.openembedded.org/mt/105739556/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] CVE program statement on the use of the new data format

[Richard Purdie]

On Mon, 2024-04-29 at 15:29 +0200, Marta Rybczynska wrote:
> https://medium.com/@cve_program/new-cve-record-format-enables-additional-data-fields-at-time-of-disclosure-82eef1d4035e
> 
> Quote:
> 
> The CVE Board is proud to announce that the CVE Program has evolved
> its record format to enhance automation capabilities and data
> enrichment. This format, utilized by CVE Services, facilitates the
> reservation of CVE IDs and the inclusion of data elements like CVSS,
> CWE, CPE, and other data into the CVE Record at the time of issuing a
> security advisory. This means the authoritative source (within their
> CNA scope) of vulnerability information — those closest to the
> products themselves — can accurately report enriched data to CVE
> directly and contribute more substantially to the vulnerability
> management process.

Thanks Marta.

This is definitely a step in the right direction. Do we know if they
plan to make it easy for CNAs to add such information later to existing
CVEs?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2000): 
https://lists.openembedded.org/g/openembedded-architecture/message/2000
Mute This Topic: https://lists.openembedded.org/mt/105800359/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Cleanup of WORKDIR by separating do_unpack

[Richard Purdie]

One interesting side effect I've noticed with testing the a-c patches
was a failure in meta-virtualization.

The issue is a bbappend to sysvinit which does:

"""
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"

SRC_URI += "file://getty-wrapper"

do_install:append() {
install -d ${D}${base_sbindir}
install -m 0755 ${WORKDIR}/getty-wrapper ${D}${base_sbindir}/getty-wrapper
}
FILES:${PN} += "${base_sbindir}/getty-wrapper"
"""

which breaks after I change S to be != WORKDIR.

The correct fix here is to use UNPACKDIR instead of WORKDIR so it is an
easy fix, just annoying. It does suggest we may need to start with the
patches making that substitution before we can start on the S = WORKDIR
removal.

I'm trying to decide how upset people are going to get if I start
making these changes as I realise that whilst core is effectively done
with scarthgap, other layers will still be working on that.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1997): 
https://lists.openembedded.org/g/openembedded-architecture/message/1997
Mute This Topic: https://lists.openembedded.org/mt/105739556/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Cleanup of WORKDIR by separating do_unpack

[Richard Purdie]

On Sun, 2024-04-28 at 11:10 +0100, Paul Barker wrote:
> On 25/04/2024 22:38, Richard Purdie via lists.openembedded.org wrote:
> > As people probably realise, WORKDIR is a bit of a mess at the moment.
> > do_unpack writes things directly into there and we don't keep track of
> > what it writes there very easily. I know there are source tracer
> > solutions but the whole situation is messy.
> > 
> > I'd like to try and improve things somehow but it isn't anywhere near
> > as easy as people think. The key hidden issues are that:
> > 
> > * release tarballs create the subdir under WORKDIR (BPN-PV)
> > * the git fetcher fetches into a "git" subdir by default
> > * plain file:// referenced files end up in WORKDIR directly.
> > 
> > This means:
> > 
> > * git recipes set S = "${WORKDIR}/git"
> > * tarball release recipes don't usually set S as the default works
> > * do_compile/do_install often reference file from SRC_URI in WORKDIR
> > 
> > Why is any of this a problem? If you have a recipe with SRC_URI =
> > "file://myfile1 file://myfile2", run a build, then remove myfile2 from
> > SRC_URI and rebuild, myfile2 will still be in WORKDIR. There are
> > situations that breaks things.
> > 
> > Put another way, the fetcher can't undo the result of do_unpack, there
> > is no directory it can clean unless S != WORKDIR.
> > 
> > It also results in lots of files in WORKDIR which can be very confusing
> > to people.
> > 
> > I think it is time we tried to start and fix this. It is however going
> > to be a little painful.
> > 
> > My initial proposed steps are:
> > 
> > a) Add UNPACKDIR as a variable, defaulting to WORKDIR and switch
> > do_unpack to unpack there instead of WORKDIR directly.
> > 
> > b) Change recipes that use S = "${WORKDIR}" to
> > 
> > S = "${WORKDIR}/sources"
> > UNPACKDIR = "${S}"
> >  
> > instead. Their do_compile/do_install tasks would need to change
> > ${WORKDIR} -> ${S} if they don't have that already.
> 
> To avoid having to make another change later, we should instead set
> 
> UNPACKDIR = "${WORKDIR}/unpack-sources"
> S = "${UNPACKDIR}"
> 
> Then when the default UNPACKDIR is changed to the above, we can simply
> drop the UNPACKDIR assignment in these recipes.

I did wonder about that. In the end I decided it was probably clearer
leaving things long term as:

"""
S = "${WORKDIR}/sources"
UNPACKDIR = "${S}"
"""

instead of:

"""
S = "${UNPACKDIR}"
"""

since otherwise you get recipes building in UNPACKDIR which I didn't
really want to encourage. This way, if recipes are building under
UNPACKDIR, it at least gets renamed from the default so we have some
kind of marker for that.

To be clear, I worry less about the fact the build happens there but
more how the code in the tasks is written. We want to encourage the
correct usage of UNPACKDIR vs S.

> > c) Make S = "${WORKDIR}" a fatal error
> > 
> > d) Audit do_compile/do_install usage of ${WORKDIR} and change to
> > ${UNPACKDIR}.
> > 
> > e) Add sanity checks to make ${WORKDIR} references in compile/install
> > fatal
> > 
> > f) make do_unpack clean ${UNPACKDIR} before it reruns if it isn't equal
> > to WORKDIR
> > 
> > g) We could consider passing config into the fetcher to change the
> > "git" default to match BPN-PV to simplify things.
> > 
> > 
> > This in theory then sets the stage for us to change UNPACKDIR to be
> > something like ${WORKDIR}/unpack-sources and we could put links in for
> > "${BPN}-${PV}" and 'git' into WORKDIR. We could then always clean
> > UNPACKDIR.
> 
> We could instead change the default for S to "${UNPACKDIR}/${BP}", then
> we shouldn't need links in ${WORKDIR}.

This is going to encourage people to "build" inside UNPACKDIR though
and I'm not convinced that is something we should do. I'd prefer to
keep the directory with a clear purpose.

> At this point, any recipe manually setting UNPACKDIR = "${WORKDIR}"
> should trigger an error.
> 
> > 
> > Every time I think about all this, my head hurts, people complain about
> > having breaking changes and I end up trying to ignore it a bit more. I
> > don't think it is possible to have a completely scripted transition
> > where the end result is actually nice looking, to get better looking
> > layout and recipes, we're going to have to inject a variable like
> > UNPACKDIR.
> > 
>

[Openembedded-architecture] Cleanup of WORKDIR by separating do_unpack

[Richard Purdie]

As people probably realise, WORKDIR is a bit of a mess at the moment.
do_unpack writes things directly into there and we don't keep track of
what it writes there very easily. I know there are source tracer
solutions but the whole situation is messy.

I'd like to try and improve things somehow but it isn't anywhere near
as easy as people think. The key hidden issues are that:

* release tarballs create the subdir under WORKDIR (BPN-PV)
* the git fetcher fetches into a "git" subdir by default
* plain file:// referenced files end up in WORKDIR directly.

This means:

* git recipes set S = "${WORKDIR}/git"
* tarball release recipes don't usually set S as the default works
* do_compile/do_install often reference file from SRC_URI in WORKDIR

Why is any of this a problem? If you have a recipe with SRC_URI =
"file://myfile1 file://myfile2", run a build, then remove myfile2 from
SRC_URI and rebuild, myfile2 will still be in WORKDIR. There are
situations that breaks things.

Put another way, the fetcher can't undo the result of do_unpack, there
is no directory it can clean unless S != WORKDIR.

It also results in lots of files in WORKDIR which can be very confusing
to people.

I think it is time we tried to start and fix this. It is however going
to be a little painful.

My initial proposed steps are:

a) Add UNPACKDIR as a variable, defaulting to WORKDIR and switch
do_unpack to unpack there instead of WORKDIR directly.

b) Change recipes that use S = "${WORKDIR}" to

S = "${WORKDIR}/sources"
UNPACKDIR = "${S}"
 
instead. Their do_compile/do_install tasks would need to change
${WORKDIR} -> ${S} if they don't have that already.

c) Make S = "${WORKDIR}" a fatal error

d) Audit do_compile/do_install usage of ${WORKDIR} and change to
${UNPACKDIR}.

e) Add sanity checks to make ${WORKDIR} references in compile/install
fatal

f) make do_unpack clean ${UNPACKDIR} before it reruns if it isn't equal
to WORKDIR

g) We could consider passing config into the fetcher to change the
"git" default to match BPN-PV to simplify things.


This in theory then sets the stage for us to change UNPACKDIR to be
something like ${WORKDIR}/unpack-sources and we could put links in for
"${BPN}-${PV}" and 'git' into WORKDIR. We could then always clean
UNPACKDIR.

Every time I think about all this, my head hurts, people complain about
having breaking changes and I end up trying to ignore it a bit more. I
don't think it is possible to have a completely scripted transition
where the end result is actually nice looking, to get better looking
layout and recipes, we're going to have to inject a variable like
UNPACKDIR.

Thoughts? Any better ideas?

I do have some patches doing a-c for OE-Core.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1992): 
https://lists.openembedded.org/g/openembedded-architecture/message/1992
Mute This Topic: https://lists.openembedded.org/mt/105739556/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Open letter to the CVE Project/CNAs

[Richard Purdie]

The recent NVD changes have impacted a number of projects and there are
quite a few uneasy developers out there, not just in the Yocto Project
community but much more widely. There could be some simple changes or
process improvements that the CVE Project could make which would
massively help things.

The key thing for many of us is to have accurate product/vendor
identification and useful version constraints. A secondary issue is an
easy process to allow updates/addition of that information.

The intent would be to have accurate CVE data at source, freeing NVD
andothers to validate/improve/vet that information.

As such I'm inviting our community to sign this open letter, the intent
of which is show the demand for these improvements:

https://github.com/yoctoproject/cve-cna-open-letter

I've chosen to use github pull requests for this for ease.

Please feel free to share this widely as I believe the ideas here would
benefit many projects as we all face a similar challenge.

What we don't want to see is "pay for access" data, or a fragmented
data ecosystem which is becoming a real risk.

The letter deliberately doesn't dive into implementation details as it
would complicate a simple message, those details are a solvable problem
if the desire is there. The version 5 schema already hints at some of
this path, this proposal would just take it a step further again by
allowing buy in from the CNAs themselves.

If there is a good process and tooling to allow updates of CVE entries
with information, even if it is optional, it will get adopted if it
works well and is helpful.

Cheers,

Richard
(on behalf of the Yocto Project/OpenEmbedded TSCs)





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1990): 
https://lists.openembedded.org/g/openembedded-architecture/message/1990
Mute This Topic: https://lists.openembedded.org/mt/105676860/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] New mailing list for layer patches

[Richard Purdie]

On Tue, 2024-04-02 at 22:33 +, Peter Kjellerstedt wrote:
> 
> To whoever is responsible for updating 
> https://www.yoctoproject.org/community/mailing-lists/: the links for
> the two new lists both refer to yocto-announce rather than yocto-
> status and yocto-patches.

That should be fixed now, thanks.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1986): 
https://lists.openembedded.org/g/openembedded-architecture/message/1986
Mute This Topic: https://lists.openembedded.org/mt/105197686/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Feature freeze status

[Richard Purdie]

On Fri, 2024-02-23 at 11:02 +0100, Alexander Kanavin wrote:
> As someone completely out of this, I want to ask: should genericarm64
> definition go to a mixin layer until it meets the criteria? Could it
> even be backported to LTS proper when it does? It's a machine
> definition, and so won't affect users not using the machine.
> 
> I also object to the notion that things 'must' happen now, with
> overtime and stress even, because someone doesn't use inter-LTS
> releases or feature mixins, and they made plans to use this LTS with
> the assumption it will have those things. One should always have a
> plan B, that corrects for features that had missed the deadline.

There are two sides to this. My time and the LTS are both supported by
the project members. Those members have a clear desire to have this in
the LTS and it ultimately would be good for the project overall for a
number of reasons.

Unfortunately it hasn't worked according to the timelines and I'm
unhappy about that but we are in the position we're in. That support
from the members doesn't write a blank cheque however it does need to
be factored in alongside the community side to things.

The TSC made a very clear policy about backports and letting in some
things but not others would create a very difficult position as who
gets to decide which things should and should not be added
retrospectively. The TSC went into the LTS policies very much aware of
how problematic that would be and therefore set out to take a strong
line on it.

For any given deadline and featureset, there are basically three
options when running into problems. Reduce the features, slip the
deadline or skip it entirely. At the moment I'm seeing discussion which
suggest that if we do give a little more time, we should be able to
resolve this. As such I probably need to give that a chance to happen.

I plan to use that time to try and resolve a chunk of other problems
which we need to fix for the release one way or another regardless. We
can then hopefully make some time "back" in M4.

I have been asked about my intent of my original email. I think I
wanted to make clear to everyone some of the dilemmas faced when trying
to make decisions like this. I appreciate not all the factors are
easily visible to everyone and thought it may be helpful/useful to give
a bit more insight.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1979): 
https://lists.openembedded.org/g/openembedded-architecture/message/1979
Mute This Topic: https://lists.openembedded.org/mt/104506925/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Feature freeze status

[Richard Purdie]

I'm now in a pretty horrible position.

We've now passed feature freeze. This freeze is pretty critical since
it concerns the LTS so if things miss this, they're out for the next
two years. The deadline shouldn't be a surprise to anyone.

I'd "committed" to genericarm64 being a feature for this release. This
is wanted by a number of Yocto Project member companies. This
commitment was made months ago and has been the subject of
presentations too.

It relates to BSP support and there is a list of criteria for
"official" platforms. That criteria is clear and has been available for
a long time, particularly to the project members and in the context of
discussion of genericarm64.

Recently the proposed format of genericarm64 changed radically and
competitively by surprise.

Various maintainers/developers:

* aren't interested in re-discussing things
* may be on vacation, I really don't know
* feel undervalued
* don't want to engage in any discussion at all
* feel their experience is being ignored
* are really upset at the current situation

I'm not naming names, the people above could easily be me for some of
them (except for vacation, that clearly can't be me). I can see various
sides of this and do understand people's positions.

I have tried to talk to maintainers, tried to move the discussion
forward and I've also tried to talk to some of the companies. The
number of people on sabbatical, extended vacation and so on makes this
really challenging.

I'm being asked "when are you building the milestone?". I have no
control over the people, the proposal or anything really, so I have no
idea. I can say this really isn't fair on me.

Technically, what I should do is say it didn't make the deadline and
move on, it isn't included. Unfortunately, that could result in a
number of companies becoming very upset with me and potentially puts my
income at risk.

I have been willing to try and be flexible, however the fact that
nobody is talking to me, or trying to solve this is probably the thing
which really annoys and frustrates me. I do a lot to make things come
together despite some huge challenges and yet here, we're just left
with this impasse with nobody willing to try and resolve it. I do not
see a path out right now.

Perhaps there are discussions happening and I just don't know about
them, if that is the case you may want to give me a hint.

Failing that and if there is no movement on this, I will have no other
option than to rule this out the release and LTS.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1977): 
https://lists.openembedded.org/g/openembedded-architecture/message/1977
Mute This Topic: https://lists.openembedded.org/mt/104506925/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [RFC PATCH] Add genericarm64 MACHINE using upstream defconfig

[Richard Purdie]

On Wed, 2024-02-21 at 10:57 +, Ross Burton wrote:
> From: Ross Burton 
> 
> This is a new 64-bit "generic" Arm machine, that expects the hardware to
> be SystemReady IR compatible. This is slightly forward-leaning as there's
> not a _lot_ of SystemReady hardware in the wild, but most modern boards
> are and the number will only grow.  Also, this is the only way to have a
> 'generic' machine as without standardised bootloaders and firmware it
> would be impossible.
> 
> The base machine configuration isn't that exciting: it's a fully featured
> machine that supports most things, booting via UEFI and an initramfs.
> 
> However, the kernel is more interesting.  This RFC uses the upstream defconfig
> because unlike some other platforms, the arm64 defconfig is actively
> maintained with the goal of being a 'boots on most hardware' configuration.
> My argument is: why would we duplicate that effort?

Can you point at the policy/process which decides how a config option
makes it in there?

> The "linux-yocto way" is configuration fragments and after a week of
> hair-pulling I do actually have fragments that boot on a BeaglePlay, but
> to say this was a tiresome and frustrating exercise would be understating it.
> 
> So, a request for comments: is it acceptable to use the upstream defconfig in
> a reference BSP?  Personally I'm torn: the Yocto way is fragments not 
> monolithic
> configs, but repeating the effort to fragmentise the configuration and then
> also have it sufficiently modular that it can be used in pieces - instead of
> just being a large file split up into smaller files - is a lot of effort for
> what might end up being minimal gain.  My fear is we end up with a fragmented
> configuration that can't be easily modified without breaking some platforms,
> and badly copies what the defconfig already does.

Let me play devils advocate.

I do understand it is a pain, equally, once you do have it working, it
is something you rarely have to touch again for a given board.

In the context of linux-yocto, I suspect we'd end up with a few board
specific lists of config options (fragments) that board *really* needs.
genericarm64 would end up as the sum of all those configs, plus any
other bits which you really want in a generic machine, presumably the
genericx86 machines have an idea of that.

The advantage here is that you give the users three options:

a) the kitchen sink upstream "systemready" defconfig
b) the genericarm64 machine defconfig which works most places and 
   is 'guaranteed' on our target test platforms
c) a machine targeted defconfig with only what a board needs

Developers/users end up in different places at different points in
there lifecycles. 

For in depth development, you might like to be able to cut the kernel
down so you'd not rebuilding tons of stuff you don't need/use/care
about each time, nor transferring it to a device during updates.

For a specific product release, you again might want to trim it down to
what it needs.

If you're shipping demo software, a full systemready image would be
much more appropriate.

I think it comes down to whether the fragments are usable and testable.
We have a list of targets we want this new machine to run on so lets
start with those, define genericarm64 as that set of fragments combined
plus the generic pieces linux-yocto adds, then go from there. If you
add a new machine to the test matrix, we add a new fragment. If someone
wants to add new config, they need to show a machine using it.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1967): 
https://lists.openembedded.org/g/openembedded-architecture/message/1967
Mute This Topic: https://lists.openembedded.org/mt/104485828/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] proposal: config fragments

[Richard Purdie]

On Wed, 2024-02-21 at 11:06 +0100, Alexander Kanavin wrote:
> the next missing piece in the official setup and configuration
> management is configuration fragments, so I'd like to describe how I
> see them:
> 
> 1. They are regular .inc files, residing in
> meta-somelayer/conf/fragments/category/. No special tooling is
> needed:
> you can 'cat fragment.inc > local.conf', or 'echo "require
> conf/fragments/category/fragment.inc" > local.conf'
> 
> 2. There would be however a tool that allows standardized operations:
> add/remove/list/search. I'd suggest 'bitbake-config' or similar,
> which
> would basically reuse bitbake-layers infrastructure as much as
> possible. Bitbake-layers itself is already over-stuffed with various
> commands, and is more about layer management.

I do have some concerns about "bitbake-config" since it sounds
potentially a lot more general purpose and we need to be careful about
that. Naming is important and hard and I don't think the name should
block this but I do want to be clear up front that this is a tricky
area.

> 3. 'add' would simply add 'require fragment.inc' to local.conf, if it
> isn't already there. 'remove' would undo that, if it is there.
> 
> 4. Each fragment must contain a description at the start, and the
> tooling would refuse to work with fragments without descriptions.
> Similar to machine configurations in oe-core.

I'm concerned about the machine configuration descriptions as they
invent a new syntax which the rest of the system can't read. With
layer.conf we deliberately tried not to add magic comment syntax.

Ideally here we'd wither use normal variables, or we write some kind of
dedicated markup which bitbake itself can understand.

A key element of this will be knowing how to identify and specify a
specific fragment. This implies they should have "names" but we also
probably need to make it easy to know where a fragment comes from even
out of context of the layer. I don't have a specific proposal, just
mentioning thoughts at the moment.

> 5. The eventual goal is to reduce a typical local.conf to
> distro+machine+fragments. Any other settings in it should be seen as
> 'local hacks' that never leave a developer's machine.
> 
> 6. The testing ground for all of this would be transferring the
> fragments from
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json
> into oe-core/poky. So that when someone gets feedback that their
> change failed on multilib, they don't have to go through complicated
> investigation of how to reproduce that. There are also fragments all
> over selftest that could be consolidated this way.
> 
> 7. Layers are welcome and encouraged to publish their own fragments
> inside the layer. I love copy-paste-tweak programming without having
> to fully understand what is being tweaked, and I find it superior to
> 'reading documentation', especially when such documentation spends
> pages after pages describing design principles, syntax intricacies or
> rarely used command line options, and doesn't get to something I can
> copy paste until the end (and in worst cases doesn't offer that at
> all). Such fragments don't have to be directly usable; they can serve
> as a set of starting points.
> 
> Anything else? Anything controversial above?

I'd like to understand more about the syntax used in the files and the
identification/naming of fragments.

In the back of my mind is the idea that in some ways we need a table
operation. layer.conf files, machine conf files, distro configs and so
on are effectively rows in a "table" of data and this feels similar.
I've wondered if there is a better way to handle such things.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1965): 
https://lists.openembedded.org/g/openembedded-architecture/message/1965
Mute This Topic: https://lists.openembedded.org/mt/104485443/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Kernel versions in our next LTS release

[Richard Purdie]

The project is aware that the kernel versions that ship in our next LTS
in April will likely not have support for the full lifetime of our LTS
(four years). We've been asked a few questions about this so we wanted
to write down what our plan is.

Our intent is to support the versions we ship with for as long as is
practical. If they become end of life around the mid point of the LTS
as expected, our plan is to backport a current LTS kernel at that time
into the release and make it the default, leaving the older recipes
there for anyone who still needs to use them. It would be based on the
kernel in one of our future releases, as appropriate.

The linux-libc-headers version would remain at the released version.
This is used to control the features in the toolchain/userspace and
there is no need to upgrade this to work with a newer kernel version.
Upgrading it would potentially change the toolchain and userspace
functionality and this is not desirable in the LTS. To be clear, this
does mean bleeding edge features in the newer kernel may not be
available but that is desirable in the context of the purpose of the
LTS.

The only reason the headers version would change would be to address a
security issue in the toolchain or kernel/userspace interface.

Since we don't know what the upstream kernel will do, or when exactly,
it is hard to make a definitive plan beyond this.

We appreciate this isn't our usual policy of "no upgrades", however if
everyone knows this is the plan from the start, we believe that is
different and everyone can plan accordingly, hence wanting to make this
clear now.

Since the project is planning to execute the full QA process against
this new version, it is putting a plan in place to switch from the
start and it plans not to test against any other version beyond that
point, it makes most sense to directly integrate the changes rather
than the mix in layer approach which is there for unplanned changes.

To be clear, we may also update this plan if/as things develop in the
other dependent ecosystems such as kernel as we can't know what will
happen in these on the timescales involved.

Cheers,

Richard
(on behalf of the Yocto Project TSC, LTS and kernel maintainers)


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1911): 
https://lists.openembedded.org/g/openembedded-architecture/message/1911
Mute This Topic: https://lists.openembedded.org/mt/103681177/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Proposal for adding a "inherit_deferred" directive to bitbake

[Richard Purdie]

On Mon, 2024-01-08 at 17:08 +, chris.lapla...@agilent.com wrote:
> > > Having thought a bit more about this, I've wondered if
> > > "inherit_defer"
> > > is slightly better phrasing than "inherit_deferred".
> > > 
> > 
> > FWIW. I like the new syntax, as the semantics of variable use in
> > the inherits was
> > not obvious to us plebs (me) in the past (since not a lot of deeper
> > reflection
> > usually goes on when creating a recipe).
> > 
> > This makes it clear that something different is going on, even if
> > exactly the
> > point to which things are deferred is not front of mind for most.
> > It gives
> > someone a thread to pull on to learn more.
> > 
> > I can definitely leverage it in meta-virt.
> > 
> > As for the tense of the syntax, I can see an argument for either. I
> > lean towards
> > "inherit_defer" as it is shorter and the defer matches the tense of
> > inherit.
> > ("inherited_deferred" anyone ? :)
> 
> Not to bikeshed, but what about defer_inherit?
> 
> Makes it easier when quickly scanning a recipe to pick out the
> 'inherit' vs 'defer_inherit' since you don't need to read more than
> the first character.

I did wonder about that but I think I'd prefer it to be mentally
associated with inherit more closely...

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1910): 
https://lists.openembedded.org/g/openembedded-architecture/message/1910
Mute This Topic: https://lists.openembedded.org/mt/103493127/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Proposal for adding a "inherit_deferred" directive to bitbake

[Richard Purdie]

On Tue, 2024-01-02 at 23:47 +, Richard Purdie via
lists.openembedded.org wrote:
> It isn't every day I propose new syntax and the bar for doing so is
> high but I think we have a good case for making his change.
> 
> I've put together a proof of concept patch here:
> 
> https://lists.openembedded.org/g/bitbake-devel/message/15738
> 
> The idea is to add syntax for:
> 
> inherit_deferred ${SOME_VARIABLE}
> 
> which would handle the inherit at the end of parsing rather than
> immediately.
> 
> The reason is that currently people don't realise that in order for
> bitbake to parse that, it has to immediately expand the variable at
> that point in parsing. This means:
> 
> SOME_VARIABLE = "a"
> inherit ${SOME_VARIABLE}
> SOME_VARIABLE = "b"
> 
> will inherit class a, whilst with inherit_deferred, it would inherit
> class b.
> 
> SOME_VARIABLE = "a"
> inherit_deferred ${SOME_VARIABLE}
> SOME_VARIABLE += "b"

Having thought a bit more about this, I've wondered if "inherit_defer"
is slightly better phrasing than "inherit_deferred".

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1907): 
https://lists.openembedded.org/g/openembedded-architecture/message/1907
Mute This Topic: https://lists.openembedded.org/mt/103493127/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Proposal for adding a "inherit_deferred" directive to bitbake

[Richard Purdie]

It isn't every day I propose new syntax and the bar for doing so is
high but I think we have a good case for making his change.

I've put together a proof of concept patch here:

https://lists.openembedded.org/g/bitbake-devel/message/15738

The idea is to add syntax for:

inherit_deferred ${SOME_VARIABLE}

which would handle the inherit at the end of parsing rather than
immediately.

The reason is that currently people don't realise that in order for
bitbake to parse that, it has to immediately expand the variable at
that point in parsing. This means:

SOME_VARIABLE = "a"
inherit ${SOME_VARIABLE}
SOME_VARIABLE = "b"

will inherit class a, whilst with inherit_deferred, it would inherit
class b.

SOME_VARIABLE = "a"
inherit_deferred ${SOME_VARIABLE}
SOME_VARIABLE += "b"

would inherit a and b.

When the inherit is conditional on a variable like PACKAGECONFIG, this
starts to make a lot more sense to users as you can change
PACKAGECONFIG in a bbappend and the behaviour the user expects will
occur.

It isn't as simple to implement as you'd first think as we require
native/nativesdk classes to be last and a deferred class can add
further deferred inherits so we have to be very careful about ordering.

I did run some tests against OE-Core and whilst I ran into some
horrible variable ordering issues (TARGET_HOST_TASK for example), I
have patches in master-next (commit messages and cleanup still pending)
which do allow all our current tests to pass.

If the warning in the patch is enabled, it will show warnings in meta-
oe, meta-agl and meta-mingw but nothing which can't be fixed relatively
easily. Given that I think I'd be in favour of a fairly fast removal of
support for non-deferred conditional inherits, probably landing quickly
so we can include in the next release.

Just to be clear, I'm in favour of this new syntax because:

a) it is pretty simple
b) it should be clear to users what it is doing
c) it solves a common real world usability issue people run into
d) we can clearly detect and error for the corner case we want 
   to deprecate/remove
e) we can give the user a specific error message about how to fix it

I have wondered about a INHERIT_DEFERRED variable instead. There are
pros and cons to it since it would allow easier editing of the classes
(e.g. using append, += and remove) but it would also potentially add
confusion since INHERIT only works in configuration files and is
actually a deferred inherit for them. It also still leaves an "arms
race" since you couldn't undo a remove. Using the syntax proposed above
doesn't preclude allowing editing of the list via a variable at a later
date if we did want to go that route. It has the benefit of matching
the current inherit people expect in recipes/classes.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1906): 
https://lists.openembedded.org/g/openembedded-architecture/message/1906
Mute This Topic: https://lists.openembedded.org/mt/103493127/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Weak PACKAGECONFIG removal

[Richard Purdie]

On Tue, 2024-01-02 at 14:28 +0100, Adrian Freihofer wrote:
> On Wed, 2023-12-27 at 07:03 +, Peter Kjellerstedt via
> lists.openembedded.org wrote:
> > > -Original Message-
> > > From:
> > > openembedded-architecture@lists.openembedded.org  > > architect...@lists.openembedded.org> On Behalf Of Joshua Watt
> > > Sent: den 22 december 2023 15:25
> > > To: Adrian Freihofer 
> > > Cc: openembedded-architecture@lists.openembedded.org
> > > Subject: Re: [Openembedded-architecture] Weak PACKAGECONFIG removal
> > > 
> > > On Fri, Dec 22, 2023 at 3:46 AM Adrian Freihofer
> > >  wrote:
> > > > 
> > > > On Fri, 2023-12-22 at 10:25 +, Ross Burton via
> > > > lists.openembedded.org wrote:
> > > > > On 22 Dec 2023, at 04:59, Chen Qi via lists.openembedded.org
> > > > >  wrote:
> > > > > > > One of the major things that comes up frequently is that
> > > > > > > the
> > > > > > > PACKAGECONFIG for a recipe needs to be modified to remove
> > > > > > > some
> > > > > > > incompatible flag from the default options. This particular
> > > > > > > option is
> > > > > > > not very simple though, because most recipes set
> > > > > > > PACKAGECONFIG
> > > > > > > using
> > > > > > > the "deferred weak" assignment, as in `PACKAGECONFIG ??=
> > > > > > > "foo
> > > > > > > bar"`.
> > > > > > > Because of this, any operation that attempts to modify this
> > > > > > > variable
> > > > > > > will overwrite the default; commonly one might like to do
> > > > > > > something
> > > > > > > like `PACKAGECONFIG:remove = "foo"`, but since this happens
> > > > > > > before
> > > > > > > weak assignment, the result is an empty string instead of
> > > > > > > just
> > > > > > > the
> > > > > > > value "bar".
> > > > > > 
> > > > > > Checking the master branch, and I get a different result.
> > > > > 
> > > > > Indeed. With this in local.conf, or split across a
> > > > > recipe/bbappend:
> > > > > 
> > > > > FOOBAR ??= "foo bar"
> > > > > FOOBAR:remove = “foo"
> > > > > 
> > > > > $ bitbake-getvar  FOOBAR
> > > > > #
> > > > > # $FOOBAR [2 operations]
> > > > > #   set /yocto/ross/build/conf/local.conf:495
> > > > > # [_defaultval] "foo bar"
> > > > > #   :remove /yocto/ross/build/conf/local.conf:496
> > > > > # "foo"
> > > > > # pre-expansion value:
> > > > > #   "foo bar"
> > > > > FOOBAR=" bar”
> > > > > 
> > > > > $ bitbake-getvar  FOOBAR -r psplash
> > > > > #
> > > > > # $FOOBAR [2 operations]
> > > > > #   set /home/ross/Yocto/poky/meta/recipes-
> > > > > core/psplash/psplash_git.bb:133
> > > > > # [_defaultval] "foo bar"
> > > > > #   :remove /home/ross/Yocto/poky/meta-poky/recipes-
> > > > > core/psplash/psplash_git.bbappend:3
> > > > > # "foo"
> > > > > # pre-expansion value:
> > > > > #   "foo bar"
> > > > > FOOBAR=" bar”
> > > > > 
> > > > > Josh: can you provide a minimal reproducer, or did this quietly
> > > > > change behaviour at some point?
> > > > 
> > > > When I read this discussion a few days ago, I also wondered. For
> > > > example, we have systemd on kirkstone which does:
> > > > 
> > > > PACKAGECONFIG ??= " \
> > > >     ${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi...
> > > >     ...
> > > > "
> > > > 
> > > > and in the distro there is for example:
> > > > 
> > > > PACKAGECONFIG:append:pn-systemd = " cgroupv2"
> > > > PACKAGECONFIG:remove:pn-systemd = "sysvinit"
> > > > 
> > > > The append and the remove operators do not reset the
> > > > PACKAGECONFIG to
> > > > "".
> > > > 
> > > > I see that e.g. += can be misleading. But append and remove seam
> > > > to
> > > > work as expected (by me) on kirkstone.
> > > 
> > > Yes,
> > > 
> > > You and Ross are correct. For some reason, I was mentally hung up
> > > on
> > > PACKAGECONFIG:pn-systemd:remove being the correct way to do this,
> > > which is very obviously wrong after you think about it.
> > > 
> > > AFAICT, there is no reason to make any changes, and I sincerely
> > > apologize for wasting everyone's time and sanity on this e-mail
> > > chain.
> > 
> > Actually, I do not think it was a waste of time, because even if
> > :append 
> > and :remove do work with ??=, you are forced to use :append if you
> > want to 
> > add to PACKAGECONFIG in a bbappend. You cannot use +=. Changing the
> > recipes 
> > to use = instead of ??= would allow this, without changing anything
> > else 
> > (except for the case that Ross noted, which was wrong to begin with).
> > 
> 
> Could some additional operators such as ?+=, ??+=, ?=+, ??=+, -=, ?-=,
> ??-=, =-, ?=-, ??=- make this behavior more obvious?

The challenge is how do these interact with the other operators and
existing conventions. Can a user reasonable predict what they might do
just from seeing them?

I've wondered about some other kinds of operator but as others have
said, people barely understand how the existing ones interact with each
other so adding more of them to the mix may make the problem worse, not
better :/.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all 

[Openembedded-architecture] Removal/disable of testimage dumper code in LTS branches?

[Richard Purdie]

I wanted to share/summarise my findings on the recent qemurunner
changes as the patches themselves are quite dry and the shear horror of
what is happening is at risk of getting lost.

Whilst I'm fully aware of our stable branch policies, there is probably
a case for making a more invasive change than usual in this area due
top the scope of the issues.

The target dumper runs (or ran for master) the following commands:

top -bn1
ps
free
df
export DEFAULT_GATEWAY=$(ip route | awk '/default/ { print $3}')
ping -c3 $DEFAULT_GATEWAY
dmesg
netstat -an
ip address
# Next command will dump logs from /var/log/
find /var/log/ -type f 2>/dev/null -exec echo "" \; -exec 
echo {} \; -exec echo "" \; -exec cat {} \; -exec echo "" \;

and it was running them whenever the ssh command failure or there was a
"No route to host" message in a command output.

Some of these are a lot of data (entire dmesg). Some were a little
broken and cat'd a binary file to the serial such as /var/log/wtmp.

This meant large files were being sent at 115200bps over the serial
line which is plain slow just to start with. The qemurunner code was
looking for "\r\n" sequences to find the end of command output and with
the binary data, was getting horribly confused, potentially leaving
large chunks of data sitting in buffers unread.

If data sits unread in those buffers, it can lock up qemu and/or it can
cause rcu stalls in guest kernel in the serial TX paths inside qemu.

On my local test run, I was seeing that whole set of commands run
around 5 times on a single testimage run. At best this is slowing
things down a lot and I suspect there could be many more invocations
for some other workloads/images.

There was also the possibility that if qemu had output on stdout or
stderr, things could lockup too.

This is compounded by the qmp "dumper" commands that were also being
run multiple times. If you had large amounts of memory for qemu, it
meant large dumps of data (GBs) from qemu's qmp interface, also often
repeatedly. These IO storms wouldn't do workloads on the autobuilder
any good at all.

Given the intermittent issues we're struggling with and the need to
have tests complete in a sensible amount of time, I feel that there is
a case for cleaning up these issues even on the stable release
branches. Emptying the list of commands is probably the least invasive
change, removing things as we did on master is also straight forward
and I doubt anyone is using these that much.

The other half of the issue, the qemurunner serial handling is quite
invasive and I'm less sure what we want to do there as the changes are
potentially quite disruptive.

Regardless, I wanted to document things so people understand how
horrible the potential issues here are.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1888): 
https://lists.openembedded.org/g/openembedded-architecture/message/1888
Mute This Topic: https://lists.openembedded.org/mt/103301450/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Weak PACKAGECONFIG removal

[Richard Purdie]

On Tue, 2023-12-19 at 13:19 -0700, Joshua Watt wrote:
> This message is to start a discussion about removal of undesired
> recipe PACKAGECONFIG options.
> 
> Background:
> Since the dawn of using OpenEmbedded as part of my day job, we've used
> meta-gplv2 to avoid GPLv3 code in our products. However, I recently
> decided that it was past time to excise this particular wart from our
> code base, so I've been looking at how to do this. Ideally, I would
> like to make it some sort of shared .inc file that can be included and
> we can all collaborate on to make it easier for this fairly common use
> case.
> 
> 
> One of the major things that comes up frequently is that the
> PACKAGECONFIG for a recipe needs to be modified to remove some
> incompatible flag from the default options. This particular option is
> not very simple though, because most recipes set PACKAGECONFIG using
> the "deferred weak" assignment, as in `PACKAGECONFIG ??= "foo bar"`.
> Because of this, any operation that attempts to modify this variable
> will overwrite the default; commonly one might like to do something
> like `PACKAGECONFIG:remove = "foo"`, but since this happens before
> weak assignment, the result is an empty string instead of just the
> value "bar".
> 
> I've come up with a number of solutions for this, but I'm unsure which
> ones make sense to explore as long term options for the project:
> 
> 1) Do nothing and respecify the complete set of PACKAGECONFIG options
> when overriding. This basically means instead of PACKAGECONFIG:remove
> = "foo", you would explicitly set to the remaining items like
> PACKAGECONFIG = "bar". If no other solution is appealing, this is
> probably fine, and what I'll do. The main reason I don't like this is
> because package configs are frequently added to recipes and it's
> annoying to keep them in sync (mostly, because I'm lazy :) ).
> 
> 2) Add a PACKAGECONFIG_REMOVE variable that is evaluated by the
> package config code and causes it to ignore any matching items. This
> one is pretty easy in the packageconfig code, and I think users would
> find it pretty easy to understand. The downside is that it's very
> specific to packageconfig; there are other variables that probably
> would want similar treatment, but it may not make as much intuitive
> sense to do for them also (e.g. IMAGE_FEATURES, DISTRO_FEATURES).
> Also, much like :remove, you wouldn't want this used in actual
> recipes; it should only be set by end users.
> 
> 3) Change the order of weak assignment vs. remove. This one is a
> little trickier but the idea is that if ??= and :remove don't make
> sense in their current order, flip around the order of evaluation so
> that it does make sense (e.g. apply :remove after ??=). I'm not sure
> how easy this would be and if it could be done all the time, only when
> a variable was a "list", or some other criteria
> 
> 4) Stop using ??= for default PACKAGECONFIG. I'm not sure the history
> as to why it's ??=, but switching it to ?= (or maybe even =) would
> solve the problem because then :append, :remove, +=, et. al. would
> work as expected. The only case I can think of where this wouldn't
> work is if a bbappend was doing a ?= to set a completely new set of
> default PACKAGECONFIG flags; I'm not sure why that would be necessary
> though. Are there any other cases I've not thought of?
> 
> 5) Add "filters" to variables. The basic idea behind this one is that
> variables would get a new flag called "filter" where pipelines of
> limited expressions could be constructed. These filter expressions
> would be applied after the final value of the variable (and thus,
> after weak assignment). For this specific case, this might look
> something like: `PACKAGECONFIG[filter] = "remove(v, 'foo')"`. This is,
> admittedly, a _very_ general solution for this particular problem.
> However, this concept has been discussed in other contexts such as
> multilib and BBCLASSEXTEND, so if it is something that would help
> there, this problem could leverage that solution also. If you want a
> peek at what this might look like, there is a jpew/bb-filters branch
> in poky-contrib
> 
> 
> If you have any thoughts or other possible solutions, please let me know.

I've some other 'random' thoughts.

I've wondered about making the default value accessible in some way
from the metadata and syntax. The most obvious is via a flag:

PACKAGECONFIG[defaultval] = "somedefault"

The downside is that flags are quite poor syntax wise compared to full
variables, you can't change them with overrides for example so:

PACKAGECONFIG:class-native[defaultval] = "somenativedefault"

doesn't work.

When you think about the bitbake internal ramifications of making flags
support overrides, it makes me want to run away screaming so that isn't
an option.

That then brings me to:

PACKAGECONFIG:defaultval = "somedefault"

i.e. some kind of "special" override. How special it might need to me
I've not really thought through.

Re: [Openembedded-architecture] Weak PACKAGECONFIG removal

[Richard Purdie]

On Wed, 2023-12-20 at 22:07 +0100, Alexander Kanavin wrote:
> FWIW, I think that the right way out is to convert all recipes to use
> = for default PACKAGECONFIG sets., and drop the confusing ?=/??=
> stuff. What we should offer is a sane default, and an easy, obvious
> way to add and subtract from it. Is there a real need to completely
> reset the overall value?
> 
> If you disagree, please speak up now.

This is definitely not a time/place for "if nobody speaks we can do
it".

I know I for one am burnt out and struggling and I don't have the time
to give this the thought it needs. Many people are not going to be
paying attention due to the holiday season.

I appreciate Joshua has run into this, like several of us have before,
been horrified and is trying to find a way to improve things. I've
given this a lot of thought in the past and posted various things about
ideas. I've never pushed forward with them as they either weren't right
or didn't have the level of community buy in needed.

I've very reluctant to either make hard/fast "rules" about variable X
needs to use operator Y, or make wholesale changes from one format to
another without careful thought and a plan.

I've been traditionally very reluctant to try and move forward with
changing this without a way to improve things with the other problem
variables like IMAGE_INSTALL and DISTRO_FEATURES too.

It is really tempting to say "?=" is better or "=" is better and for
some scenarios they are, some they aren't. The reality is that what we
have today is what everyone's distros/configs work with and if we want
to change from what people have working, we need a good plan.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1886): 
https://lists.openembedded.org/g/openembedded-architecture/message/1886
Mute This Topic: https://lists.openembedded.org/mt/103269425/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Removing moving "early boot" binaries to /bin

[Richard Purdie]

On Thu, 2023-12-07 at 07:59 -0700, Joshua Watt wrote:
> On Thu, Dec 7, 2023 at 5:36 AM Ross Burton 
> wrote:
> > 
> > For many years we’ve been carrying patches to recipes to move some
> > libraries or binaries into /bin and /lib, on the argument that
> > people might have /usr on another mount.
> > 
> > I don’t believe this has been the case for a long time.
> 
> AFAICT, the usrmerge distro feature (effectively) forces this
> already,
> so I don't really see a problem with making it the standard.
> Presumably, that will make it so that usrmerge is always enabled and
> the actual distro feature will be removed or a noop?

No, this is something different. Ross is referring to the patches which
allow you to mount /usr late in the boot. We've slowly been dropping
these over time and I very much doubt it works now.

The usrmerge thing is a bit different.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1879): 
https://lists.openembedded.org/g/openembedded-architecture/message/1879
Mute This Topic: https://lists.openembedded.org/mt/103033053/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Why are sstate task interdependencies a bad idea?

[Richard Purdie]

I keep seeing people argue that adding dependencies between setscene
tasks "is fine" and "just works" and isn't a problem.

It is probably apparent that I get really nervous about doing this and
yet I've not been able to articulate clearly why this is such a really
bad idea. Having dug into it again I thought I'd try and
explain/document it better.

The issue is the way setscene tasks are designed to work, basically
they work on priority/dependency inversion.

bitbake starts at the end results, for example the image target. To
build an image, you need packages and tools so it would work out you
want the do_package_write_ipk's in the dependency chain and any -native
do_populate_sysroot for the tools.

bitbake knows that even though do_package_write_ipk depends on
do_package and other do_populate_sysroot native tools, those aren't
needed so it stops recursing those chains. It knows that any *native*
do_populate_sysroot tasks that are directly depended upon by the image
process are needed. If a native do_populate_sysroot is pulled in, those
can link against things for shared libs so it follows those chains.

That is bad enough to work out but it is more complicated. If for
example, bash:do_package_write_ipk isn't available, it would then have
to check for bash:do_package is available as it can use that to build
the ipk step. That may mean adding the sstate for opkg-utils-native 
do_populate_sysroot and since that is a native, any of its
dependencies.

All these dependencies have to be processed "live" too, i.e. we don't
know up front if any given setscene is going to work.

The problem comes when you start requiring reverse dependencies, i.e.
that pseudo-native is needed to unpack do_package sstate objects, or
shadow-native is needed for do_package. The key problem is that you
want pseudo-native to be available *first*. Normally bitbake works
through the dependency chains for setscene backwards, i.e. it would
handle the do_package, then think about pseudo-native as it might not
actually be required as a "normal" dependency.

The code in bitbake which considers setscene interdependencies is here:

https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/runqueue.py#n2901

There is a insignificant looking note:

"""
  # Note that anything explicitly depended upon will have its reverse
dependencies removed to avoid circular dependencies
"""

followed by an innocent looking
"""
# Have to zero this to avoid circular dependencies
sq_revdeps_squash[deptid] = set()
"""

This hack has pretty profound implications. It is basically what allows
the pseudo-native dependency needed by target tasks and the base-
passwd/shadow-native useradd dependencies to work at all but it does it
by hacking the dependency chain.


Pidge has been looking into some of the useradd bugs and suggested we
should use DEPENDS in the useradd code. The trouble is that DEPENDS are
so extensive, it basically smashes and inverts large chunks of the
dependency tree to the point it totally destroys the effectiveness of
sstate. For the image I mention above, the setscene dependencies could
end up lost and it would end up installing not just the
do_package_write_ipk tasks but do_package and target
do_populate_sysroot which aren't needed.

I did try my own experiments with just having creategroup2 using a
group from creategroup1 have a simple setscene dependency but the
smashing of the dependency chain that results basically doesn't fix the
issue and creates new ones. If you try that you see
creategroup2:do_package_setscene fail since
creategroup1:do_populate_sysroot_setscene didn't run, then
creategroup2:do_prepare_recipe_sysroot fails since it also can't find
creategroup1:do_populate_sysroot.

The bitbake code is a mess in this area, we retrofitted running of
"real" and "setscene" tasks in parallel and dynamic "rehashing" of the
tasks from has equivalence. Having such a mix of normal dependencies,
setscene dependencies and then inverting bits of them makes for some
really nasty corner cases. The code probably could take a better
approach to some of it given new capabilities it now has that the
original design could never have considered. Exactly what/where those
are isn't clear though and would need a great deal of thought before we
tried to change it.

My aim in writing this down is mainly to explain to people why setscene
task dependencies are a really bad idea given the current code. Yes, we
probably should improve it but who/when is unclear.

Meanwhile we really do need to find a way to fix useradd.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1873): 
https://lists.openembedded.org/g/openembedded-architecture/message/1873
Mute This Topic: https://lists.openembedded.org/mt/102848654/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]

Re: [Openembedded-architecture] [oe] [RFC] Support for alternative init systems

[Richard Purdie]

On Tue, 2023-11-07 at 06:30 -0800, Mathieu Anquetin wrote:
> 
> > 
> > Also, the packages wouldn't be that useful since if you change 
> > VIRTUAL-RUNTIME_init-manager, all the packages are going to end up
> > being rebuilt anyway.
> > 
> > As such, I think the proposed solution would need a lot more careful
> > thought and detail before we'd be able to say whether it would make
> > sense.
> 
> I am trying to port my patches on master, so we can discus on something
> more concrete.

The key thing to understand is that VIRTUAL-RUNTIME_init-manager is
fixed at build time. We do not support runtime resolution of these
values, they're fixed at package creation time. To resolve them at
runtime (image creation time) you'd have to have package manager
specific support and we don't have that.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1838): 
https://lists.openembedded.org/g/openembedded-architecture/message/1838
Mute This Topic: https://lists.openembedded.org/mt/102442132/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [oe] [RFC] Support for alternative init systems

[Richard Purdie]

On Tue, 2023-11-07 at 09:30 +, ANQUETIN Mathieu wrote:
> Hello all,
> 
> I would like to discuss an architecture solution to ease support for
> alternative init systems.
> 
> As of now, OpenEmbedded supports sysvinit and systemd as first-class
> citizens but does so by including required files in the main package
> based on the value of the VIRTUAL-RUNTIME_init-manager variable. This
> forces layers, like meta-openrc for example, to remove files
> generated by other layers before providing their own. This increases
> the maintenance burden for layer maintainers of these alternative
> init systems while making them always feel like second-class
> citizens.
> 
> My solution would be to generate specific packages for each init
> system (${PN}-sysvinit, ${PN}-systemd, ...) and RDEPENDS on them
> given the value of the VIRTUAL-RUNTIME_init-manager variable. This
> would simplify recipes because filtering files would no longer be
> required since all packages would be generated and other layers would
> simply provide their ${PN}-altinit package through bbappends.
> I have made a PoC on the 'kirkstone' branch but this kind of
> modification requires all recipes to adapt to the new architecture
> and therefore I would like to know if you would accept such
> modifications.

I'd note that both systemd and sysvinit end up with code which does
manipulate the generated packages depending on which is enabled.
Packages can be generated for systemd only which would result in the
packages not containing sysvinit scripts.

As such, ${PN}-sysvinit and ${PN}-systemd would never exist together
since you have to choose an init system and the ability to swap between
them is limited.

I'd note that you can have init scripts for multiple packages within a
given recipe. I'd also note that packages sometimes aren't actually
useful without their init script.

Also, the packages wouldn't be that useful since if you change 
VIRTUAL-RUNTIME_init-manager, all the packages are going to end up
being rebuilt anyway.

As such, I think the proposed solution would need a lot more careful
thought and detail before we'd be able to say whether it would make
sense.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1835): 
https://lists.openembedded.org/g/openembedded-architecture/message/1835
Mute This Topic: https://lists.openembedded.org/mt/102442132/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Sat, 2023-11-04 at 11:29 +0100, adrian.freiho...@gmail.com wrote:
> Hi Alex, hi Richard
> 
> After some internal discussions, I would like to clarify my previous
> answers on this topic.
> 
>  * Usually there are two different workflows
> - application developers: could use an SDK with a locked sstate-cache.
> - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
>  * A locked SDK
> - can work with setscene from SSTATE_MIRRORS
> - setscene does caching in the SSTATE_DIR (no issue about that)
> - But network problems can occur during the initial build because
>   bitbake executes many independent setscene tasks. Opening so many
>   independent connections slows down the build, especially if the
>   server treats them as a denial of service attack.
> - The denial of service problem is difficult to solve because each
>   setscene task runs in its own bibtake task. Reusing a connection to
>   download multiple sstate artifacts seems almost impossible.
>   This is much easier to solve with separate sstate download script.

FWIW, we did have a similar issue with do_fetch overloading
servers/proxies/ISPs and added:

do_fetch[number_threads] = "4"

Finding the right place to put a thread limit on overall setscene tasks
is harder but in theory possible. Or perhaps a "network capable tasks"
thread limit?

Is the overload caused by the initial query of sstate presence, or,
does it happen when the setscene tasks themselves run?


>  * An unlocked SDK
> - Tries to download the sstate cache for changed recipes and their
>   dependencies, which obviously can't work.
> - The useless download requests slow down the build considerably and
>   cause a high load on the servers without any benefit.

Is this sstate over http(s) or something else? I seem to remember you
mentioning sftp. If this were using sftp, it would be horribly slow as
it was designed for a light overhead "does this exist?" check which
http(s) can manage well.

Recently we've been wondering about teaching the hashequiv server about
"presence", which would then mean the build would only query things
that stood a good chance of existing.

> - A script which gets a list of sstate artifacts from bitbake and then
>   does a upfront download works much better
>+ The script runs only when the user calls it or the SDK gets boot-
>  strapped
>+ The script uses a reasonable amount of parallel connections which
>  are re-used for more then one artifact download

Explaining to users they need to do X before Y quickly gets tiring,
both for people explaining it and the people doing it trying to
remember. I'd really like to get to a point where the system "does the
right thing" if we can.

I don't believe the problems you describe are insurmountable. If you
are using sftp, that is going to be a big chunk of the problem as the
system assumes something faster is available. Yes, I've taken patches
to make sftp work but it isn't recommended at all. I appreciate there
would be reasons why you use sftp but if it is possible to get a list
of "available sstate" via other means, it would improve things.

>  * Idea for a smart lock/unlock implementation
> - Form a user's perspective a locked vs. an unlocked SDK does not make
>   much sense. It makes more sense if the SDK would automatically
>   download the sstate-cache if it is expected to be available.
>   Lets think about an implementation (which allows to override the
>   logic) to switch from automatic to manual mode:
>   
>   SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"

What determines this availability? I worry that is something very
fragile and specific to your use case. It is also not an all or nothing
binary thing.

>   In our case the sstate mirror is expected to provide all artifacts
>   for tagged commits and for some git branches of the layer
>   repositories.
>   The sstate is obviousely not usable for a "dirty" git layer
>   repository.

That isn't correct and isn't going to work. If I make a single change
locally, there is a good chance that 99.9% of the sstate could still be
valid in some cases. Forcing the user through 10 hours of rebuild when
potentially that much was available is a really really bad user
experience.

>  That's what the is_sstate_mirror_available function
>   could check to automatically enable and disable lazy downloads.
>   
> - If is_sstate_mirror_available() returns false, it should still be
>   possible to initiate a sstate-cache download manually.
>   
>  * Terminology
> - Older Yocto Releases:
>+ eSDK means an installer which provides a different environment with
>  different tools
>+ The eSDK was static, with a locked sstate cache
>+ Was for one MACHINE, for one image...
> - Newer Yocto Releases:
>+ The bitbake environment offers all 

Re: [Openembedded-architecture] [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Tue, 2023-10-31 at 13:08 +0100, Alexander Kanavin wrote:
> On Mon, 30 Oct 2023 at 16:02, Alexander Kanavin via
> lists.openembedded.org 
> wrote:
> > So here's what could be done:
> > 
> > - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
> > environment script puts that in PATH, rather than some custom
> > esdk-specific location (the code to generate that can then be
> > dropped).
> 
> This is now implemented (needs to be tested on AB).
> 
> > - esdk tweaks to local.conf move into a dedicated include file, which
> > can be static and under version control, except for perhaps
> > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
> > I need to check why is it there and how that is used.
> 
> It's actually more complicated. The code to generate esdk-specific
> local.conf with all the tweaks has too much dynamic stuff in it which
> is subject to what various variables are set to. So I'm thinking of
> extracting that to a dedicated function, then attaching a bitbake task
> to that function.
> 
> Then we can pull all of it together into 'devtool esdk '
> command (or similar), which would enter the esdk environment directly
> via:
> - running 'bitbake  meta-ide-support'
> - running the above mentioned bitbake local.conf task to generate the
> esdk-specific local.conf
> - sourcing the environment script produced by meta-ide-support
> - rewriting PATH to provide only the curated esdk tools and not
> everything plus bitbake.
> - writing a custom devtool.conf similar to that of standalone esdk so
> that devtool can find bitbake and bitbake can use the esdk-specific
> local.conf
> 
> And it would be tested in the same way standalone esdks are.
> 
> Thoughts? Anything missing from the above list?

That sounds like a good way to handle this to me!

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1818): 
https://lists.openembedded.org/g/openembedded-architecture/message/1818
Mute This Topic: https://lists.openembedded.org/mt/102295495/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > There are design elements to this work. We need to work out how we can
> > make eSDK and "normal" builds more similar and less of an overhead to
> > switch between one and the other. A "bblock all" command does partly
> > get you to an eSDK effectively so part of this may be switching eSDK to
> > use the new lock command. What other differences are there? What other
> > differences are necessary or make sense for the use cases eSDK was
> > designed for? How would you turn an existing build into an eSDK like
> > one? Could you provide a copy of a local build to someone else easily
> > using something like eSDK's tooling? What does the eSDK look like at
> > the end of this. One section we don't have good answers to yet is setup
> > and configuration although I know you've started on some of that.
> 
> So I see the following differences between esdk and normal modes:
> 
> 1. Environment and tooling availability.
> 
> a) esdk sets a number of variables from its initialization script that
> aid with cross-compiling components directly (e.g. the core use case
> of SDKs). Normal mode doesn't do that, but recently added
> meta-ide-support will generate a similar initialization script that
> will set up the same environment from the normal mode. There are tests
> and documentation for it.

In that case, this one is something we can document as how to make the
functionality available in the normal build.

> b) PATH. eSDK has a number of items in PATH that point to various
> locations inside tmp/sysroots/, collectively they provide the
> cross-toolchain.
> 
> eSDK also puts a selection of yocto tools into path - wic, devtool but
> not bitbake:
> 
> 
> alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
> total 48
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
> ../../../../layers/poky/scripts/devtool
> lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
> ../../../../layers/poky/scripts/oe-find-native-sysroot
> lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
> ../../../../layers/poky/scripts/recipetool
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
> ../../../../layers/poky/scripts/runqemu
> lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
> ../../../../layers/poky/scripts/runqemu-addptable2image
> lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
> ../../../../layers/poky/scripts/runqemu-export-rootfs
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
> ../../../../layers/poky/scripts/runqemu-extract-sdk
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
> ../../../../layers/poky/scripts/runqemu-gen-tapdevs
> lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
> ../../../../layers/poky/scripts/runqemu-ifdown
> lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
> ../../../../layers/poky/scripts/runqemu-ifup
> lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
> lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
> ../../../../layers/poky/scripts/wic
> ==
> 
> 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
> Cross-toolchain can be added by the same environment script made by
> meta-ide-support as mentioned in 1a.

Right, so in theory we can change PATH and change this which can also
easily be documented.

> 2. Configuration (e.g. local.conf).
> 
> eSDK local.conf is local.conf from the normal mode that was used to
> produce eSDK, stripped of all comments, and with a bunch of extra
> settings:
> 
> 
> INHERIT:remove = "buildhistory icecc"
> CONNECTIVITY_CHECK_URIS = ""
> 
> SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
> 
> SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
> 
> BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
> 
> BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
> *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
> 
> BUILDCFG_HEADER = ""
> 
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
> 
> # Provide a flag to indicate we are in the EXT_SDK Context
> WITHIN_EXT_SDK = "1"
> 
> SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
> file://universal-4.9/(.*) file://universal-4.8/\1"
> 

Perhaps some of this should become a generic include file which is
included w

Re: [Openembedded-architecture] Question: supported layers for security processes

[Richard Purdie]

> 
> On 10/20/23 05:49, Richard Purdie via lists.openembedded.org wrote:
> > On Fri, 2023-10-20 at 10:56 +0200, Marta Rybczynska wrote:
> > > While working on multiple aspects of security processes, one question
> > > comes back frequently: which are the layers we support with those
> > > processes? This has impact on the number of SECURITY.md I will be
> > > submitting, of configuring any CVE synchronization work etc.
> > > 
> > > The YP download page offers a download of poky. The release
> > > documentation 
> > > https://docs.yoctoproject.org/migration-guides/index.html#release-information
> > > nor the Release page (https://wiki.yoctoproject.org/wiki/Releases)
> > > does not exactly list layers covered.
> > > 
> > > Is it poky only? Poky + meta-openemedded? With some BSP layers?
> > > 
> > > This has a general impact, because I assume that layers maintained
> > > "externally" might have different security contacts, for example.
> > > 
> > > Do we have that documented somewhere so that we can reference in the
> > > security documentation?
> > It will be for the layer maintainers to decide what to do about this
> > file. From the Yocto Project perspective, we should cover bitbake,
> > meta-yocto, openembedded-core (done) and yocto-docs.
> > 
> > Looking over https://git.yoctoproject.org/ we should add one to meta-
> > mingw as a tested layer. I've asked meta-gplv2 move to other layers.
> > 
> > We should probably mention this issue to the other layer maintainers,
> > maybe on the architecture list and perhaps also open a bug to make
> > SECURITY.md a requirement for Yocto Project Compatible status?
> > 
> > We should also add it to some of the code/tools repositories, in
> > particular:
> > 
> > auto-upgrade-helper,  buildhistory-web, error-report-web, git-refinery-
> > web, layerindex-web, pseudo, psplash, ptest-runner2, update-rc.d,
> > swatbot, yocto-autobuilder-helper, yocto-autobuilder2.
> > 
> > If we're happy with the test in OE-Core, I can update several of these
> > to make the work a little easier?
> > 
> > We should email the maintainers for opkg/opkg-utils as well (opkg
> > mailing list).
> 
> That's me. :)
> 
> The request here is that I add a SECURITY.md with instructions for how 
> to file security issues against opkg, a la the same document that is 
> already in OE-core; right?
> 
> Would y'all prefer if private security emails for opkg went to 
> `secur...@yoctoproject.org`? Otherwise, I'll default to my email directly.

I've just been trying to work out what we're doing with other repos
before replying.

For "tools", I've gone with simply:

"""
How to Report a Potential Vulnerability?


If you would like to report a public issue (for example, one with a released
CVE number), please report it using the
[https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Security Security 
Bugzilla].
If you have a patch ready, submit it following the same procedure as any other
patch as described in README.md.

If you are dealing with a not-yet released or urgent issue, please send a
message to security AT yoctoproject DOT org, including as many details as
possible: the layer or software module affected, the recipe and its version,
and any example code, if available.

"""
e.g.

https://git.yoctoproject.org/swatbot/commit/?id=961b8c10da89f011e79834c160196057a4233170

There is a second paragraph about release but it only makes sense in
metadata repositories (e.g. meta-yocto or meta-mingw).

You would be more than welcome to put your name as the maintainer
there. We've gone with the security list/bugzilla as the project
defaults but the maintainer makes sense when they're willing/able as
they're better placed to handle this.

The key thing is to get a SECURITY file in place.

If you could take care of opkg/opkg-utils that would be great!

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1805): 
https://lists.openembedded.org/g/openembedded-architecture/message/1805
Mute This Topic: https://lists.openembedded.org/mt/102077441/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [yocto-security] Question: supported layers for security processes

[Richard Purdie]

On Fri, 2023-10-20 at 11:25 -0400, akuster808 wrote:
> 
> On 10/20/23 5:49 AM, Richard Purdie wrote:
> > On Fri, 2023-10-20 at 10:56 +0200, Marta Rybczynska wrote:
> > > While working on multiple aspects of security processes, one question
> > > comes back frequently: which are the layers we support with those
> > > processes? This has impact on the number of SECURITY.md I will be
> > > submitting, of configuring any CVE synchronization work etc.
> > > 
> > > The YP download page offers a download of poky. The release
> > > documentation 
> > > https://docs.yoctoproject.org/migration-guides/index.html#release-information
> > > nor the Release page (https://wiki.yoctoproject.org/wiki/Releases)
> > > does not exactly list layers covered.
> > > 
> > > Is it poky only? Poky + meta-openemedded? With some BSP layers?
> > > 
> > > This has a general impact, because I assume that layers maintained
> > > "externally" might have different security contacts, for example.
> > > 
> > > Do we have that documented somewhere so that we can reference in the
> > > security documentation?
> > It will be for the layer maintainers to decide what to do about this
> > file. From the Yocto Project perspective, we should cover bitbake,
> > meta-yocto, openembedded-core (done) and yocto-docs.
> > 
> > Looking over https://git.yoctoproject.org/ we should add one to meta-
> > mingw as a tested layer. I've asked meta-gplv2 move to other layers.
> > 
> > We should probably mention this issue to the other layer maintainers,
> > maybe on the architecture list and perhaps also open a bug to make
> > SECURITY.md a requirement for Yocto Project Compatible status?
>
> Why? My layers only have upstream components. I would just tell an 
> individual to go to the upstream source and deal with those maintainers. 
> I bring no value being in the middle.

Your layers always have every component up to date with latest upstream
releases?

I can imagine reasons why someone may need to report a security issue
to the maintainer...

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1802): 
https://lists.openembedded.org/g/openembedded-architecture/message/1802
Mute This Topic: https://lists.openembedded.org/mt/102083332/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Question: supported layers for security processes

[Richard Purdie]

On Fri, 2023-10-20 at 10:56 +0200, Marta Rybczynska wrote:
> While working on multiple aspects of security processes, one question
> comes back frequently: which are the layers we support with those
> processes? This has impact on the number of SECURITY.md I will be
> submitting, of configuring any CVE synchronization work etc.
> 
> The YP download page offers a download of poky. The release
> documentation 
> https://docs.yoctoproject.org/migration-guides/index.html#release-information
> nor the Release page (https://wiki.yoctoproject.org/wiki/Releases)
> does not exactly list layers covered.
> 
> Is it poky only? Poky + meta-openemedded? With some BSP layers?
> 
> This has a general impact, because I assume that layers maintained
> "externally" might have different security contacts, for example.
> 
> Do we have that documented somewhere so that we can reference in the
> security documentation?

It will be for the layer maintainers to decide what to do about this
file. From the Yocto Project perspective, we should cover bitbake,
meta-yocto, openembedded-core (done) and yocto-docs.

Looking over https://git.yoctoproject.org/ we should add one to meta-
mingw as a tested layer. I've asked meta-gplv2 move to other layers.

We should probably mention this issue to the other layer maintainers,
maybe on the architecture list and perhaps also open a bug to make
SECURITY.md a requirement for Yocto Project Compatible status?

We should also add it to some of the code/tools repositories, in
particular:

auto-upgrade-helper,  buildhistory-web, error-report-web, git-refinery-
web, layerindex-web, pseudo, psplash, ptest-runner2, update-rc.d,
swatbot, yocto-autobuilder-helper, yocto-autobuilder2.

If we're happy with the test in OE-Core, I can update several of these
to make the work a little easier?

We should email the maintainers for opkg/opkg-utils as well (opkg
mailing list).

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1798): 
https://lists.openembedded.org/g/openembedded-architecture/message/1798
Mute This Topic: https://lists.openembedded.org/mt/102077441/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] YP Security team document

[Richard Purdie]

On Tue, 2023-10-17 at 12:09 +0200, Marta Rybczynska wrote:
> Hello all,
> This has been a moment working on the YP security team and I think
> that the current version of the document contains everything that it
> should have to get it a go. Please have a look and let me know if you
> have any last comments:
> https://wiki.yoctoproject.org/wiki/Security_private_reporting
> 
> I will then start transcription to the docs.

I made a few small wording tweaks and added a guide of what we'd like
to cover with security team membership. Hope that is all ok!

Otherwise looks good to me.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1791): 
https://lists.openembedded.org/g/openembedded-architecture/message/1791
Mute This Topic: https://lists.openembedded.org/mt/102014565/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-29 at 14:06 +0200, Alexander Kanavin wrote:
> On Thu, 28 Sept 2023 at 18:49, Richard Purdie
>  wrote:
> 
> > I'm curious to see what you find with analysis of bitbake-whatchanged.
> 
> I've taken a look a the script. It obtains the current location of
> STAMPS_DIR, then runs this:
> 
> # Generate the new stamps dir
> print("Generating the new stamps ... (need several minutes)")
> cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
> args.recipe)
> 
> Then it walks both trees, matching up file names with a regex:
> 
> # Match the stamp's filename
> # group(1): PE_PV (may no PE)
> # group(2): PR
> # group(3): TASK
> # group(4): HASH
> stamp_re = 
> re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)")
> 
> Then there's some code that finds out what changed in the above
> between the two sets.
> 
> I don't see a way to make it work: messing about with STAMPS_DIR like
> that isn't supported, and will either do nothing, or remove the
> original stamps. Also stamp filenames aren't really a 'public API',
> are they?
> 
> Should the script simply be removed, or is there some better way to
> re-implement answering the 'what has changed' question in a way that
> doesn't flood the console with task hashes? I'd be glad to get
> suggestions for this.

I'd prefer to see some dedicated bitbake API used even if we need to
create/add it. tinfoil and some of the bblock/unlock work shows we can
get stamp data, the question would be how to get it without
"disturbing" the existing build.

By using dedicated API, we'd be able to control the console output.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1784): 
https://lists.openembedded.org/g/openembedded-architecture/message/1784
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-28 at 18:43 +0200, Alexander Kanavin wrote:
> On Fri, 22 Sept 2023 at 12:42, Richard Purdie
>  wrote:
> 
> > Things which used to be problematic:
> > 
> > a) changes involving changes to gcc-source since it uses a shared
> > sources stamps which confused the tools (at least used to). That may
> > have been before gcc-source became a recipe?
> > b) changes to a very common component (e.g. autoconf-native's
> > do_configure) which make it hard to understand where the root cause of
> > the changes came from
> > c) changes which affect many recipes at once, e.g. the do_configure
> > function in base.bbclass
> > 
> > It might be helpful to write test cases for the scenario you showed as
> > working above and some of the ones I mention above, then we can
> > document they work and have an easier way to add tests for issues
> > if/as/when we identify the problematic scenarios in future.
> 
> I've now written down the tests for these three scenarios and got them
> to pass (in oe-selftest too \0/):
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
> (check the commit message too)
> 
> I am going to look closer at bitbake-whatchanged, what it aims to do
> and why it doesn't work. I have a hunch it can produce useful high
> level reports, and so shouldn't be simply thrown away. 'bitbake -S
> printdiff' is too techy and verbose for some use cases. Maybe we can
> fold that functionality into 'bitbake -S whatchanged'.

I've wondered if we should split bitbake -S printdiff into a separate
utility? It exists from a time before we had bitbake command APIs.

I'm curious to see what you find with analysis of bitbake-whatchanged.
I'm also somewhat surprised the scenarios you're testing all work!

I'm guess one of the commits I pointed to must have fixed them (the
removal of paths from the sig files)?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1781): 
https://lists.openembedded.org/g/openembedded-architecture/message/1781
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-22 at 11:17 +0200, Alexander Kanavin wrote:
> On Thu, 21 Sept 2023 at 16:39, chris.lapla...@agilent.com
>  wrote:
> 
> > That is very impressive and I'd also love to hear about what heuristics it 
> > uses.
> 
> It's actually rather simple. It uses glob.glob on stamps in tmp/, then
> on local sstate to find possible matches, then sorts them by mtime and
> takes the most recent. It's what would work most of the time, but we
> could add printdiff-all (print difference with all sstate matches) or
> printdiff-N (N most recent). It also could abstain from dumping
> locked-sigs.inc into cwd with both -S none and -S printdiff, unless
> explicitly asked
> 
> I just discovered there's also scripts/bitbake-whatchanged (that
> hasn't seen activity in years and is neither documented nor tested).
> Unsurprisingly then, it doesn't work in the same scenario:
> 
> 
> alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
> libsolv-native
> Figuring out the STAMPS_DIR ...
> Generating the new stamps ... (need several minutes)
> 
> === Summary: (0 changed, 0 unchanged)
> Newly added: 0
> PV changed: 0
> PR changed: 0
> Dependencies changed: 0
> 
> Removing the newly generated stamps dir ...
> 
> 
> Maybe this is what RP was referring to when he said the tools don't
> work properly?

No, I've believed that should probably be removed. I think there was a
recent change to it.

I think we had a major step change in this functionality working when
this was fixed:

https://git.yoctoproject.org/poky/commit/?id=84a7485025dd4473403b8da36a0c979a3afd5e93

and this test case was added:

https://git.yoctoproject.org/poky/commit/?id=1bdcd76d2968c3cc6ec2815afceba1cf98efd6d5

Things which used to be problematic:

a) changes involving changes to gcc-source since it uses a shared
sources stamps which confused the tools (at least used to). That may
have been before gcc-source became a recipe?
b) changes to a very common component (e.g. autoconf-native's
do_configure) which make it hard to understand where the root cause of
the changes came from
c) changes which affect many recipes at once, e.g. the do_configure
function in base.bbclass

It might be helpful to write test cases for the scenario you showed as
working above and some of the ones I mention above, then we can
document they work and have an easier way to add tests for issues
if/as/when we identify the problematic scenarios in future.

As you mention, it also uses mtime so perhaps issues happen if you run
a different build, then try and go back to the other config? I suspect
once you understand the algorithm the code uses, you can pick holes in
it.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1773): 
https://lists.openembedded.org/g/openembedded-architecture/message/1773
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Sun, 2023-09-17 at 10:54 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Fri, 2023-09-15 at 22:08 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > Just to follow up with where I got to with this.
> > 
> > The problem is that inotify isn't delivering change notifications in a
> > timely fashion.
> > 
> > With the patch applied and BB_SERVER_TIMEOUT = "60", you can show an
> > issue with something like:
> > 
> > rm bitbake-cookerdaemon.log
> > bitbake-layers add-layer ../meta-virtualization/
> > bitbake-layers add-layer ../meta-openembedded/meta-oe/
> > bitbake -m
> > 
> > where the second add-layer fails with the same error as the first with
> > a message missing meta-virt dependencies. It doesn't notice
> > bblayers.conf has changed and hence doesn't re-parse the base
> > configuration. I've locally proven this is due to the inotify events
> > not arriving until after the parsing commands.
> > 
> > os.sync() isn't enough to flush the inotify backlog. os.system("sync")
> > does but that may be the fork overhead. sys.stdout.flush() does too,
> > probably as side effects in the kernel IO workings.
> > 
> > This basically brings me to the conclusion that it is hard to know when
> > the inotify events are in sync with the incoming command stream and the
> > current approach is flawed.
> > 
> > I'm starting to think that we should drop inotify and go back to stat
> > calls and maybe require that UIs send "re-scan cache" events so we can
> > avoid the scanning when we don't need to. The client code does have a
> > pretty good idea of when it changes metadata, it just doesn't share
> > that at the moment. We can rescan at reconnection so it would just
> > affect tinfoil usecases.
> 
> There are patches in master-next which replace inotify usage with
> stat() calls and mtime comparisons.
> 
> For "bitbake" commands, I've made it revalidate the cache upon new
> commands so that part is relatively easy/safe.
> 
> The tricky part is that we'd now require tinfoil usage to explicitly
> say when the client has modified metadata (or run all the stat calls
> between every tinfoil command which would have crazy overhead).
> 
> This means we need to audit the tinfoil command usage to markup the
> places where it changes files and expects bitbake to notice.
> 
> In master-next, I did audit bitbake-layers and recipetool but not
> devtool. I've run an autobuilder build just to get an idea of how close
> this gets us to a solution. Unsurprisingly devtool is exploding there
> but other areas look more promising.
> 
> Ultimately we need to make a few more invasive changes to the way we
> control the server from the UI and be a bit more explicit about the
> command flow but that should wait until after release.

Final follow up to this. After some testing and sanity checking
my change with others I've merged a patch removing inotify usage in
bitbake. This then allowed the flush removal to merge too.

There are further things we should really fix but those are less urgent
and this removes the problems identified. It should hopefully help some
of the AB-INT bugs.

Anyone with custom tinfoil usage where they modify files will need to
add a function call but I doubt there are too many people doing that
(bitbake-layers, devtool and recipetool did).

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1766): 
https://lists.openembedded.org/g/openembedded-architecture/message/1766
Mute This Topic: https://lists.openembedded.org/mt/101375777/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Fri, 2023-09-15 at 22:08 +0100, Richard Purdie via
lists.openembedded.org wrote:
> Just to follow up with where I got to with this.
> 
> The problem is that inotify isn't delivering change notifications in a
> timely fashion.
> 
> With the patch applied and BB_SERVER_TIMEOUT = "60", you can show an
> issue with something like:
> 
> rm bitbake-cookerdaemon.log
> bitbake-layers add-layer ../meta-virtualization/
> bitbake-layers add-layer ../meta-openembedded/meta-oe/
> bitbake -m
> 
> where the second add-layer fails with the same error as the first with
> a message missing meta-virt dependencies. It doesn't notice
> bblayers.conf has changed and hence doesn't re-parse the base
> configuration. I've locally proven this is due to the inotify events
> not arriving until after the parsing commands.
> 
> os.sync() isn't enough to flush the inotify backlog. os.system("sync")
> does but that may be the fork overhead. sys.stdout.flush() does too,
> probably as side effects in the kernel IO workings.
> 
> This basically brings me to the conclusion that it is hard to know when
> the inotify events are in sync with the incoming command stream and the
> current approach is flawed.
> 
> I'm starting to think that we should drop inotify and go back to stat
> calls and maybe require that UIs send "re-scan cache" events so we can
> avoid the scanning when we don't need to. The client code does have a
> pretty good idea of when it changes metadata, it just doesn't share
> that at the moment. We can rescan at reconnection so it would just
> affect tinfoil usecases.

There are patches in master-next which replace inotify usage with
stat() calls and mtime comparisons.

For "bitbake" commands, I've made it revalidate the cache upon new
commands so that part is relatively easy/safe.

The tricky part is that we'd now require tinfoil usage to explicitly
say when the client has modified metadata (or run all the stat calls
between every tinfoil command which would have crazy overhead).

This means we need to audit the tinfoil command usage to markup the
places where it changes files and expects bitbake to notice.

In master-next, I did audit bitbake-layers and recipetool but not
devtool. I've run an autobuilder build just to get an idea of how close
this gets us to a solution. Unsurprisingly devtool is exploding there
but other areas look more promising.

Ultimately we need to make a few more invasive changes to the way we
control the server from the UI and be a bit more explicit about the
command flow but that should wait until after release.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1765): 
https://lists.openembedded.org/g/openembedded-architecture/message/1765
Mute This Topic: https://lists.openembedded.org/mt/101375777/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Fri, 2023-09-15 at 15:29 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Fri, 2023-09-15 at 13:59 +, Peter Kjellerstedt wrote:
> > 
> > While the problem the patch tried to fix is real, it seems a lot less
> > problematic than the result of applying the patch. So from a release 
> > perspective, I would keep the current code and possibly add it as a 
> > known bug to the release notes. Then, after the release is out, start
> > the work of solving the fallout from applying the patch, with the 
> > intention to have it solved for the first point release of Nanbield.
> > 
> > While not ideal, it sounds a lot less stressful than applying the 
> > patch at this stage.
> 
> My concern is the cache coherency issues this exposes.
> 
> The 'easy' fix for release is explicit os.sync() calls in most command
> paths (everything except ping).
> 
> I'm not sure I should really allow such a patch through review though
> as it would raise a lot of serious questions and the underlying issues
> need to be addressed.

Just to follow up with where I got to with this.

The problem is that inotify isn't delivering change notifications in a
timely fashion.

With the patch applied and BB_SERVER_TIMEOUT = "60", you can show an
issue with something like:

rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m

where the second add-layer fails with the same error as the first with
a message missing meta-virt dependencies. It doesn't notice
bblayers.conf has changed and hence doesn't re-parse the base
configuration. I've locally proven this is due to the inotify events
not arriving until after the parsing commands.

os.sync() isn't enough to flush the inotify backlog. os.system("sync")
does but that may be the fork overhead. sys.stdout.flush() does too,
probably as side effects in the kernel IO workings.

This basically brings me to the conclusion that it is hard to know when
the inotify events are in sync with the incoming command stream and the
current approach is flawed.

I'm starting to think that we should drop inotify and go back to stat
calls and maybe require that UIs send "re-scan cache" events so we can
avoid the scanning when we don't need to. The client code does have a
pretty good idea of when it changes metadata, it just doesn't share
that at the moment. We can rescan at reconnection so it would just
affect tinfoil usecases.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1764): 
https://lists.openembedded.org/g/openembedded-architecture/message/1764
Mute This Topic: https://lists.openembedded.org/mt/101375777/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Fri, 2023-09-15 at 10:58 -0400, Philip Balister wrote:
> On 9/15/23 09:59, Peter Kjellerstedt wrote:
> > > -Original Message-
> > > From: bitbake-de...@lists.openembedded.org 
> > >  On Behalf Of Richard Purdie
> > > Sent: den 15 september 2023 10:41
> > > To: bitbake-de...@lists.openembedded.org; openembedded-architecture 
> > > 
> > > Cc: Philip Balister 
> > > Subject: Re: [bitbake-devel] [PATCH] server/process: Disable the flush() 
> > > call in server logging
> > > 
> > > On Thu, 2023-09-14 at 17:28 +0100, Richard Purdie via 
> > > lists.openembedded.org wrote:
> > > > We've been chasing bitbake timeouts for a while and it was unclear 
> > > > where things
> > > > were blocking on IO. It appears the flush() call in server logging can 
> > > > cause
> > > > pauses up to minutes long on systems with slow (spinning) disks that 
> > > > are heavily
> > > > loaded with IO.
> > > > 
> > > > Since the flush() was added to aid debugging of other timing issues, we 
> > > > shouldn't
> > > > need it now and it can be disabled. Leave a comment as a reminder of 
> > > > the pain this
> > > > can cause.
> > > > 
> > > > Signed-off-by: Richard Purdie 
> > > > ---
> > > >   lib/bb/server/process.py | 3 ++-
> > > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
> > > > index 40cb99bc97..94e93bc3a6 100644
> > > > --- a/lib/bb/server/process.py
> > > > +++ b/lib/bb/server/process.py
> > > > @@ -43,7 +43,8 @@ def currenttime():
> > > > 
> > > >   def serverlog(msg):
> > > >   print(str(os.getpid()) + " " +  currenttime() + " " + msg)
> > > > -sys.stdout.flush()
> > > > +#Seems a flush here triggers filesytem sync like behaviour and 
> > > > long hangs in the server
> > > > +#sys.stdout.flush()
> > > > 
> > > >   #
> > > >   # When we have lockfile issues, try and find infomation about which 
> > > > process is
> > > 
> > > Whilst this patch looks simple, it has all the signs of turning into
> > > weeks of pain after looking at the results from testing. It looks like
> > > we have a race in the system which this flush is masking.
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio
> > > 
> > > Looking at just one of the failures there (of the 6), we see this test
> > > pass:
> > > 
> > > 1: 20/57 355/537 (0.12s) (0 failed)
> > > (recipetool.RecipetoolTests.test_recipetool_split_pkg_licenses)
> > > 
> > > then the next test to run on that build directory is:
> > > 
> > > 1: 21/57 359/537 (137.64s) (0 failed)
> > > (signing.Signing.test_signing_packages)
> > > 
> > > with an error which boils down to:
> > > 
> > > FileNotFoundError: [Errno 2] No such file or directory:
> > > '/tmp/recipetoolqaizvjzf8h'
> > > 
> > > from
> > > 
> > > run(cmd='git rev-parse --abbrev-ref HEAD', input=None, log=None,
> > > extrafiles=[], **options={'cwd': '/tmp/recipetoolqaizvjzf8h', 'shell':
> > > True})
> > > 
> > > i.e. it is trying to run a command in a directory which doesn't exist.
> > > 
> > > Without looking, I know "recipetoolqa" tmpfiles will come from
> > > RecipetoolTests so the previous build directory is being removed and
> > > yet still is cwd for the server.
> > > 
> > > I've not looked much into any of the other failures yet.
> > > 
> > > Looking at the 5 different selftest failures:
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio
> > > 
> > > 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> > > archiver.Archiver.test_archiver_srpm_mode: FAILED (237.00s)
> > > 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> > > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: FAILED
> > > (13.25s)
> > > 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> > > prservice.BitbakePrTests.test_pr_service_deb_arch_dep: FAILED (126.30s)
> > > 202

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Fri, 2023-09-15 at 13:59 +, Peter Kjellerstedt wrote:
> 
> While the problem the patch tried to fix is real, it seems a lot less
> problematic than the result of applying the patch. So from a release 
> perspective, I would keep the current code and possibly add it as a 
> known bug to the release notes. Then, after the release is out, start
> the work of solving the fallout from applying the patch, with the 
> intention to have it solved for the first point release of Nanbield.
> 
> While not ideal, it sounds a lot less stressful than applying the 
> patch at this stage.

My concern is the cache coherency issues this exposes.

The 'easy' fix for release is explicit os.sync() calls in most command
paths (everything except ping).

I'm not sure I should really allow such a patch through review though
as it would raise a lot of serious questions and the underlying issues
need to be addressed.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1762): 
https://lists.openembedded.org/g/openembedded-architecture/message/1762
Mute This Topic: https://lists.openembedded.org/mt/101375777/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Bitbake server file coherency problems

[Richard Purdie]

On Fri, 2023-09-15 at 09:40 +0100, Richard Purdie via lists.openembedded.org 
wrote:
> The question at this point is what do people want me to do. We clearly
> have a really nasty bug in here. The patch is "right" and we do need to
> fix this. If I merge it, I suspect I'm going to end up having to chase
> this down before we can release and I am going to struggle to find the
> time to do it and I suspect my sanity will suffer. This does look to be
> a significant issue though.

I've spent some time digging into what is going on and I really don't
like what I'm finding.

The existing .flush() call in the server logging path is basically
injecting a "sync" equivalent in the main command loop within bitbake.

That "sync" is effectively by accident maintaining our cache coherence
and it is also significantly damaging certain kinds of build
performance.

If we remove it, bitbake fails to see file changes. In theory nothing
should be changing files when builds are running however tinfoil users
have already assumed bitbake manages it's cache correctly and with
memory resident bitbake, there are signs of cache invalidation misses.
Exactly what/where/why I still haven't quite worked out. Since we run
oe-selftest with memory resident bitbake, it is particularly badly
effected but back to back normal builds are also failing.

The challenge is that the sync flushes the inotify watches and without
that, writes to files may have not triggered the inotify report which
we rely on to invalidate the caches. Even adding os.sync() calls into
bitbake isn't solving the problem as we only want to do this when
needed for performance and it isn't clear when it is actually needed.

What is really needed is a step back and a re-design of the cache
management within cooker, there is too many weird code paths that
aren't actually that useful any more and we've tried to retrofit cache
handling to something which never had it originally which isn't working
out so well. Given the release I'm worried about undertaking something
like this at such a time, equally, now I've seen the sheer amount of
problems and the fact it is just luck things happen to work, I'm very
very worried.

I've been asked who knows this area of code with a view to working out
who we could lean on to help fix it. Sadly, I think I'm probably the
only one who does.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1761): 
https://lists.openembedded.org/g/openembedded-architecture/message/1761
Mute This Topic: https://lists.openembedded.org/mt/101380274/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Richard Purdie]

On Thu, 2023-09-14 at 17:28 +0100, Richard Purdie via
lists.openembedded.org wrote:
> We've been chasing bitbake timeouts for a while and it was unclear where 
> things
> were blocking on IO. It appears the flush() call in server logging can cause
> pauses up to minutes long on systems with slow (spinning) disks that are 
> heavily
> loaded with IO.
> 
> Since the flush() was added to aid debugging of other timing issues, we 
> shouldn't
> need it now and it can be disabled. Leave a comment as a reminder of the pain 
> this
> can cause.
> 
> Signed-off-by: Richard Purdie 
> ---
>  lib/bb/server/process.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
> index 40cb99bc97..94e93bc3a6 100644
> --- a/lib/bb/server/process.py
> +++ b/lib/bb/server/process.py
> @@ -43,7 +43,8 @@ def currenttime():
>  
>  def serverlog(msg):
>  print(str(os.getpid()) + " " +  currenttime() + " " + msg)
> -sys.stdout.flush()
> +#Seems a flush here triggers filesytem sync like behaviour and long 
> hangs in the server
> +#sys.stdout.flush()
>  
>  #
>  # When we have lockfile issues, try and find infomation about which process 
> is

Whilst this patch looks simple, it has all the signs of turning into
weeks of pain after looking at the results from testing. It looks like
we have a race in the system which this flush is masking.

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio

Looking at just one of the failures there (of the 6), we see this test
pass:

1: 20/57 355/537 (0.12s) (0 failed) 
(recipetool.RecipetoolTests.test_recipetool_split_pkg_licenses)

then the next test to run on that build directory is:

1: 21/57 359/537 (137.64s) (0 failed) (signing.Signing.test_signing_packages)

with an error which boils down to:

FileNotFoundError: [Errno 2] No such file or directory: 
'/tmp/recipetoolqaizvjzf8h'

from 

run(cmd='git rev-parse --abbrev-ref HEAD', input=None, log=None, extrafiles=[], 
**options={'cwd': '/tmp/recipetoolqaizvjzf8h', 'shell': True})

i.e. it is trying to run a command in a directory which doesn't exist.

Without looking, I know "recipetoolqa" tmpfiles will come from
RecipetoolTests so the previous build directory is being removed and
yet still is cwd for the server.

I've not looked much into any of the other failures yet.

Looking at the 5 different selftest failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio

2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
archiver.Archiver.test_archiver_srpm_mode: FAILED (237.00s)
2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: FAILED 
(13.25s)
2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
prservice.BitbakePrTests.test_pr_service_deb_arch_dep: FAILED (126.30s)
2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
signing.Signing.test_signing_packages: FAILED (137.64s)
2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
sstatetests.SStateHashSameSigs.test_sstate_32_64_same_hash: FAILED (541.51s)
2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS - 
sstatetests.SStateHashSameSigs3.test_sstate_multilib_or_not_native_samesigs: 
FAILED (224.05s)

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5710/steps/14/logs/stdio

2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS - 
archiver.Archiver.test_archiver_srpm_mode: FAILED (59.78s)
2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS - 
bblogging.BitBakeLogging.test_shell_loggingA: FAILED (15.71s)
2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS - 
multiconfig.MultiConfig.test_multiconfig_reparse: FAILED (6.39s)
2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS - 
prservice.BitbakePrTests.test_pr_service_rpm_arch_indep: FAILED (0.84s)

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5746/steps/14/logs/stdio

2023-09-14 23:33:02,720 - oe-selftest - INFO - RESULTS - 
package.VersionOrdering.test_dpkg: ERROR
2023-09-14 23:33:02,720 - oe-selftest - INFO - RESULTS - 
package.VersionOrdering.test_opkg: ERROR
2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS - 
package.VersionOrdering.test_rpm: ERROR
2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS - 
archiver.Archiver.test_archiver_srpm_mode: FAILED (129.04s)
2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS - 
bbtests.BitbakeTests.test_postfile: FAILED (2.06s)
2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS - 
bbtests.BitbakeTests.test_prefile: FAILED (1.73s)
2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS - 
devtool.DevtoolModifyTests.test_devtool_buildclean: FAILED (265.26s)
2023-09-14 23:33:02,721 - oe-selftest - INFO 

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > For the task signatures, we need to think about some questions. If I
> > make a change locally, can I query how much will rebuild and how much
> > will be reused? There is bitbake --dry-run but perhaps it is time for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps should
> > define some existing scenarios where it is currently very difficult to
> > work this out and then work out how we can report that information to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time.

Right, what we lack is a way for the user to ask this and see the
result easily. As you say, bitbake can do this already.

>  Is there something else that's easily done and useful to print?

I think there is also the scenario of:
"I've run a build and have an existing TMPDIR and stamp info. I've now
pulled in a change. How much is going to rebuild and more importantly
*why*?"

This is different to a remote sstate situation as you have the stamp
info of the previous build already there to compare against.

> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 
> Existing cache objects are not suitable because:
>  was built on  and has a mismatching SRCREV
>  was built on  and has a different do_compile()

In theory you can do an:

ls 
sstate/*/*/sstate:curl:corei7-64-poky-linux:::corei7-64:10:*_package.tar.zst

and get a list of possible objects. Some key information was put into
the cache file names for this reason. Obviously this becomes much
trickier when the sstate is remote over http though.

"newest" is problematic in this context as you really want the closest
match. There is no concept of build date in the cache as it often isn't
relevant (building an old release for example). The only point a
date/time is used is for cleaning out the cache for files which haven't
been accessed in a long time.

The tools are already supposed to support doing this with local file
sstate sources, they just do a bad job at getting the diffs right. One
intent of this work item was to try and understand why they don't work
and address that so at least for filesystem sstate mirrors, you can get
better results. I don't know how we solve the remote http issue as yet.

> > One of the big problems in the past was that we lost much of the hash
> > information after parsing completed. This meant that if the hashes then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain more of the information
> > so that we didn't have to recompute it every time to be able to do
> > processing with it. I have to admit I've totally lost track of where I
> > got to with that.
> 
> Here's an idea I can't get out of my head. Right now, the cache is
> simply an amorphous mass of objects, with no information regarding how
> they were created. How about storing complete build confgurations as
> well into the same directory? There would be a dedicated, separate
> area for each configuration that placed objects into the cache,
> containing:
> - list of layers and revisions
> - config template used
> - complete content of build/conf
> - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
> - complete list of sstate objects that were produced as a result, so
> they can be checked for existence
> 
> This would be written into the cache dir at the very end of the build
> when everything else is already there.

I'm not sure this helps as much as you'd like. For example I build
core-image-sato-sdk on the autobuilder and populate this but you want
to build core-image-sato locally. There would be no info here that
would help with informa

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote:
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> > Alexander Kanavin will be working on the core workflow topic
> 
> I am now ready to start doing this, but before I do, I'd like to
> decompose the subject into manageable tasks with a bit of help from RP
> and the community:
> 
> https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/
> 
> 
> Core Workflow – Process Improvements
> 
> Background
> 
> The project builds everything from source by default. This means it
> has a reputation for being slow and heavy. There are ways the project
> can accelerate this which means faster workflows and improved
> developer experience but these are currently cumbersome to use.
> 
> Rationale
> 
> The project aims to defragment customised embedded Linux. This is
> important as if we succeed at this, it gives benefits to the wider
> ecosystem through making it easier to inject security fixes and an
> ability to share and collaborate without re-inventing the wheel.
> 
> To do this, we need to provide best-in-class support and compete
> against binary distributions for usability and speed. One way we can
> do this is provide better support for binary artifacts via our sstate
> mechanism. We do already have some of this functionality in our
> “extensible SDK” or “eSDK”.
> 
> Deliverables
> 
> Enable a public sstate mirror via a content delivery network (CDN) and
> populate using the autobuilder
> Ensure CDN sstate is reused under normal use case scenarios,
> particularly for slow components like rust-native. Identify any common
> sstate mismatch causes. Ensure test cases are added to cover the use
> cases and prevent regressions.
> Add lock and unlock commands to allow specific components to be locked
> down to specific sstate checksums or allow them to vary
> Allow switching between eSDK and non-eSDK modes
> Add tooling so we can understand why something is rebuilding when it
> isn’t expected to.
> ===
> 
> So: where to start? Do we need to 'design' something, or maybe time
> should go directly into addressing specific sore points? All feedback
> welcome.
> 
> There's also an unfinished patchset for adding bblock/bbunlock, which
> I would prefer to *not* 'take over and finish' but rather see the
> author get it merged:
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

To start with I'll try and write down and give a random walk through my
thoughts.

Certainly we need to get something like that patchset over the line. I
think it was blocked on a reply to this email:

https://lists.openembedded.org/g/openembedded-core/message/186497

which has suffered whilst I was pulled into the qemuppc mess. I feel
bad for not getting to a reply to that.

There are design elements to this work. We need to work out how we can
make eSDK and "normal" builds more similar and less of an overhead to
switch between one and the other. A "bblock all" command does partly
get you to an eSDK effectively so part of this may be switching eSDK to
use the new lock command. What other differences are there? What other
differences are necessary or make sense for the use cases eSDK was
designed for? How would you turn an existing build into an eSDK like
one? Could you provide a copy of a local build to someone else easily
using something like eSDK's tooling? What does the eSDK look like at
the end of this. One section we don't have good answers to yet is setup
and configuration although I know you've started on some of that.

For the task signatures, we need to think about some questions. If I
make a change locally, can I query how much will rebuild and how much
will be reused? There is bitbake --dry-run but perhaps it is time for a
an option (or dedicated separate command?) to give some statistics
about what bitbake would do? How much sstate would be reused?

That then logically leads into the questions, can we tell what has
changed? Why isn't my sstate being reused? For that we perhaps should
define some existing scenarios where it is currently very difficult to
work this out and then work out how we can report that information to
the user. These could become test cases?

One of the big problems in the past was that we lost much of the hash
information after parsing completed. This meant that if the hashes then
didn't match, we couldn't tell why as the original computation was
lost. I did some work on allowing us to retain more of the information
so that we didn't have to recompute it every time to be able to do
processing with it. I have to admit I've totally lost track of where I
got to with that.

Michael Halstead will be working on setting the CDN up so I'll let him
comment on when we'll have things ready for testing with that. We do
already have sstate shared from the autobuilder so some basic tests to
make sure our "base" shared files do work as expected is something
which can happen there already.

Another interesting 

[Openembedded-architecture] Reducing mips/powerpc testing

[Richard Purdie]

It is clear that mips and powerpc usage is in decline. I'm reluctant to
drop support completely as I think it is a valuable asset for OE but I
think the time has come to cut back on the testing we're doing. A lot
of the burden for keeping these alive is falling to a small group of
people who aren't in a position to keep doing it.

As such I'm planning to cut back on the "on target" heavier testing
we're doing, basically the long running pieces and focus on what we can
do minimally.

As part of this I'd probably put qemuppc64 pieces into the mix as that
is much more maintained than the 32 bit side of things and we do get
reports when host support for qemuppc64 breaks.

I need to run this past various groups including the TSC but plan to
move quickly as I'm intending this to go into the current release. The
recent experiences with apathy over qemuppc issues is what is pushing
me to decide to do this and if we do it, it is best done now rather
than closer to the LTS.

If anyone strongly objects to this let me know and we can discuss how
you can help maintain it!

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1746): 
https://lists.openembedded.org/g/openembedded-architecture/message/1746
Mute This Topic: https://lists.openembedded.org/mt/101146524/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Changes needing documentation - a new procedure

[Richard Purdie]

I've been concerned for a while about the process of adding
documentation influencing changes. Our docs are in a separate repo and
this does have advantages. They do span multiple areas of the project
and in general this does work well for us. It does mean the docs are
consistent and have a common style/voice.

Equally, it is hard to ask users to add documentation changes in their
commits and this leads to the docs having to play catchup and pushes a
lot of work to the docs maintainers.

We're therefore going to try something new where I'll ask people to add
notes to a file in OE-Core, DOCS-WIP when they make a change which
needs docs updates. This will include changes which need a release
notes or migration guide entry. The docs maintainers will then know
what needs to be done and hopefully have at least the basic information
they need to make the docs changes needed.

This is an experiment but hopefully one that will be successful and let
our docs continue to grow.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1743): 
https://lists.openembedded.org/g/openembedded-architecture/message/1743
Mute This Topic: https://lists.openembedded.org/mt/101109973/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] New Contributors Guide

[Richard Purdie]

Hi All,

The OE TSC has been aware of the feedback around our contributions
process and the state of our docs on the subject, with many different
wiki pages, some of which disagree with each other.

It isn't something a newcomer to the project can untangle, it needed
people with authority and in some ways "bravery" to do something
radical to improve the situation.

After discussion, the OE TSC has decided to do just that. Along with
the YP TSC, we agreed to take the various "contributions" wiki pages
and docs and pull them together into a new "Contributors Guide" manual
as part of the rest of "our" docs bookshelf.

Rather than discuss this, we've done it, it is here:

https://docs.yoctoproject.org/dev/contributor-guide/index.html

This is the work of a number of us and we did pull in help from Michael
Opdenacker to help us make this happen, thanks to all involved!

We've gone and made a number of wiki pages point at this where we
believe the content was migrated, duplicated or obsolete. I've also
just sent out patches to some of the core repository README files to
update those to match.

We've tried to add sections to the docs to cover things which we could
assume 10 years ago but which are perhaps not common knowledge to a
newcomer now.

We're aware this won't be perfect and there may be areas that need
improvement or maybe where people disagree. As ever, the docs are a
living entity and evolve over time so patches welcome, as ever. This
effort by the TSCs was mainly to get us over the hump of starting
something, having a fairly ruthless cleanup of the wiki and giving us a
basic canvas to work on together.

If anyone has new people working on the project, please test them out
on the new guide, we'd love to know what is clear, what isn't and if
there is anything we should improve.

Also, if anyone spots wiki docs or other information that should be in
the manual, please let us know too.

Cheers,

Richard
(channelling both TSCs)


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1742): 
https://lists.openembedded.org/g/openembedded-architecture/message/1742
Mute This Topic: https://lists.openembedded.org/mt/101061316/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Fri, 2023-08-18 at 16:01 +, Peter Kjellerstedt wrote:
> 
> I think we can live without it. If I am not mistaken we can always 
> inject our extra information by doing something like 
> PKGV:append = "${MAINTPV}" in recipes where we need it and then rely on 
> your code to add the SHA-1 if the output from ${MAINTPV} includes a "+".
> 
> > I'm deliberately trying to avoid putting
> > the function call into a variable as it then becomes part of the task
> > hash computations. This means there are calls into it during parsing
> > rather than at runtime. That was part of the pain in the previous
> > approach and something I'm keen to avoid this time around to simplify
> > things if we can. It does get included in the taskhashes after the
> > changes but at points we control (vardepvalue) rather than 'random'
> > expansion points of a common variable like PV during parsing.
> 
> I am a bit unsure about the trying to avoid calls to the get_pkgv_string() 
> function during parsing, because it seems to happen with the current 
> code. When I first added your changes, I got errors for a number of 
> recipes in meta-oe that failed to set SRCREV_FORMAT (Khem has since 
> corrected those recipes). E.g., if you add meta-oe to the build and 
> revert commit 4228656bae, then you will get the following error if 
> you run `bitbake -p`:
> 
> WARNING: .../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb: Exception 
> during build_dependencies for fetcher_hashes_dummyfunc
> WARNING: .../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb: Error during 
> finalise of 
> /home/pkj/dists/poky-master/meta-oe/recipes-extended/sysdig/sysig_0.28.0.bb
> ERROR: ExpansionError during parsing 
> .../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb

It will happen, we're just making it happen at a point of our choosing
rather than the current situation where it happens when PV happens to
be expanded, which could be anywhere during parsing.

> This is causing problems for me in our code base because we have a 
> number of recipes that fetch from Git repositories with restricted 
> access that I cannot fetch (we have alternative recipes that install 
> prebuilt binaries for those of us without access to the source code).
> Now all those recipes fail during parsing because bitbake cannot run 
> `git ls-remote`, even if they will never be built in my builds.

I suspect that isn't from get_pkgv_string but from get_hashvalue?

If so, I'd suggest your "internal only" recipes need to raise
SkipRecipe() if they can't function and that should avoid the issue.


Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1739): 
https://lists.openembedded.org/g/openembedded-architecture/message/1739
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Tue, 2023-08-15 at 15:11 +, chris.lapla...@agilent.com wrote:
> Hi Richard,
> 
> > I tried some experiments and this doesn't actually turn out to be so 
> > difficult
> > to do. I've included a patch below which basically drops the SRCPV pieces
> > into PKGV, the conditional can probably be simplified further.
> > 
> > This patch isn't quite right in that we need to split get_srcrev into two
> > versions, one which returns the SRCREV_FORMAT string and one which
> > returns the full hashes for taskhash purposes.
> > 
> > The nice thing about this approach is that SRCPV can simply be removed over
> > time, it no longer does anything. That means compatibility is retained with
> > existing recipes.
> > 
> > I've not tested this extensively yet with AUTOREV but normal builds appear
> > to work ok with it and the WORKDIR directory naming simplification is
> > probably a win.
> 
> This sounds interesting; thank you for your tiresome work! I will try
> to find some time to play with the patch.
> 
> Just to add another dimension to the discussion, I wonder how (if at
> all) this could affect how cve-check.bbclass works? One thing I don't
> like about that class is that the CVE checking happens after
> do_fetch, when really all do_cve_check cares about is the resolved
> revision. Is there an opportunity here to optimize do_cve_check?
> 
> Over the past few months (maybe even longer), I've had an idea in my
> head for a task that happens before do_fetch that actually performs
> revision resolution. Maybe do_resolve_autorev or something. But I
> think your recent work obsoletes that idea. 

FWIW I suspect there are ways to make this work without a task at all,
yes. Definitely worth exploring. 

I can try and provide help if needed if someone wants to experiment
with it but I'd note the PKGV idea still isn't quite right or merged
yet!

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1737): 
https://lists.openembedded.org/g/openembedded-architecture/message/1737
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Tue, 2023-08-15 at 21:27 +, Peter Kjellerstedt wrote:
> 
> I decided to give the related changes you have in master-next a try. I 
> cherry-picked the following three commits:
> 
>  bitbake: fetch2: Add new srcrev fetcher API
>  recipes/classes/scripts: Drop SRCPV usage in OE-Core
>  base/package: Move source revision information from PV to PKGV
> 
> I am of course aware that the above changes are not final, but I have
> some concerns.
> 
> I tested building "iso-codes" and "xcursor-transparent-theme". They are 
> both allarch recipes with few dependencies. The former has a SRCREV that 
> matches PV, and the latter does not (and thus uses PV = "0.1.1+git${SRCPV}").
> 
> Here are the names of the resulting RPM packages before the changes are 
> applied:
> 
>   iso-codes-4.15.0-r0.4.noarch.rpm
>   xcursor-transparent-theme-0.1.1+git0+23c8af5ba4-r0.0.noarch.rpm
> 
> And here they are afterwards:
> 
>   iso-codes-4.15.0AUTOINC+69ba16daef-r0.0.noarch.rpm
>   xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4-r0.0.noarch.rpm
> 
> The first problem is of course that AUTOINC is not replaced as intended 
> when a PR server is active. This is due to the code in package_get_auto_pr() 
> that should update PRSERV_PV_AUTOINC not being triggered and thus it 
> retains its default value "AUTOINC". The code isn't triggered because it 
> looks for "AUTOINC" in PKGV, but it isn't there (yet) since ${SRCPV} is 
> no longer used. The following patch solves it:
> 
> diff --git a/meta/classes-global/package.bbclass 
> b/meta/classes-global/package.bbclass
> index 7f55b123c4..bf824b0b83 100644
> --- a/meta/classes-global/package.bbclass
> +++ b/meta/classes-global/package.bbclass
> @@ -266,6 +266,9 @@ python package_get_auto_pr() {
>  d.setVar("PRSERV_HOST", host)
> 
>  pkgv = d.getVar("PKGV")
> +srcpv = bb.fetch.get_pkgv_string(d)
> +if srcpv:
> +pkgv += srcpv
> 
>  # PR Server not active, handle AUTOINC
>  if not d.getVar('PRSERV_HOST'):
> 
> The second problem is that the value from bb.fetch.get_pkgv_string() is 
> concatenated to the version without any separator. So if the AUTOINC 
> part had been replaced as expected, the RPM name for the iso-codes 
> package would have become iso-codes-4.15.00+69ba16daef-r0.0.noarch.rpm, 
> which I doubt was intended... The easiest solution should be to make 
> get_pkgv_string() return "+AUTOINC+" instead of the current 
> "AUTOINC+".
> 
> Third, I am not sure if it is a good idea to always include the SHA-1
> in the package name. Before it was typically only done when the SRCREV 
> did not match PV. For most packages it is redundant. It also means that 
> the PR server needs to keep track of the version for all Git recipes,
> not just the ones where ${SRCPV} is used today. However, I understand
> that it is probably not easily solvable to automatically detect when 
> it is needed and when it is not. So it is either do it manually as 
> today and risk missing to do it when it is needed, or always do it and 
> have it be redundant in most cases.
> 
> One thing that I would prefer is if there is a variable, similar to 
> the old SRCPV, that contains "${@bb.fetch.get_pkgv_string(d)}" by 
> default and is used from package_convert_pr_autoinc() instead of 
> calling the method directly. The reason for that is to make it 
> possible to use a different method to determine the value. E.g., in 
> our case we have a variable called MAINTPV, which is similar to 
> SRCPV. The reason for it is that it handles Gerrit references and 
> expands them in a more suitable way than just a SHA-1. Thus I would 
> like to be able to use its underlying function instead of 
> bb.fetch.get_pkgv_string(d) to generate the string for PKGV.

I think this patch might address some of the issues:

diff --git a/meta/classes-global/package.bbclass 
b/meta/classes-global/package.bbclass
index 7f55b123c4..0338a5c690 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -315,17 +315,21 @@ python package_get_auto_pr() {
 # Package functions suitable for inclusion in PACKAGEFUNCS
 #
 
-python package_convert_pr_autoinc() {
+python package_setup_pkgv() {
+pkgv = d.getVar("PKGV")
 # Expand SRCPV into PKGV if not present
 srcpv = bb.fetch.get_pkgv_string(d)
-if srcpv:
+if srcpv and "+" in pkgv:
 d.appendVar("PKGV", srcpv)
+pkgv = d.getVar("PKGV")
 
-pkgv = d.getVar("PKGV")
 # Adjust pkgv as necessary...
 if 'AUTOINC' in pkgv:
 d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+}
+
 
+python package_convert_pr_autoinc() {
 # Change PRSERV_PV_AUTOINC and EXTENDPRAUTO usage to special values
 d.setVar('PRSERV_PV_AUTOINC', '@PRSERV_PV_AUTOINC@')
 d.setVar('EXTENDPRAUTO', '@EXTENDPRAUTO@')
@@ -498,6 +502,7 @@ python do_package () {
 oe.qa.handle_error("var-undefined", msg, d)
 return
 
+bb.build.exec_func("package_setup_pkgv", d)
 

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Tue, 2023-08-15 at 21:27 +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From: openembedded-architecture@lists.openembedded.org
> >  On Behalf Of
> > Richard Purdie
> > Sent: den 10 augusti 2023 18:53
> > To: openembedded-architecture
> > 
> > Subject: Re: [Openembedded-architecture] Workflow improvements -
> > incremental source updates?
> > 
> > On Tue, 2023-08-08 at 10:10 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > 
> 
> I decided to give the related changes you have in master-next a try. I 
> cherry-picked the following three commits:
> 
>  bitbake: fetch2: Add new srcrev fetcher API
>  recipes/classes/scripts: Drop SRCPV usage in OE-Core
>  base/package: Move source revision information from PV to PKGV
> 
> I am of course aware that the above changes are not final, but I have
> some concerns.

Thanks, I'm happy someone else is looking at this and you raise some
issues I hadn't thought about as I've been too busy trying to even get
the test suites and devtool to work correctly with the changes.

> 
> I tested building "iso-codes" and "xcursor-transparent-theme". They are 
> both allarch recipes with few dependencies. The former has a SRCREV that 
> matches PV, and the latter does not (and thus uses PV = "0.1.1+git${SRCPV}").
> 
> Here are the names of the resulting RPM packages before the changes are 
> applied:
> 
>   iso-codes-4.15.0-r0.4.noarch.rpm
>   xcursor-transparent-theme-0.1.1+git0+23c8af5ba4-r0.0.noarch.rpm
> 
> And here they are afterwards:
> 
>   iso-codes-4.15.0AUTOINC+69ba16daef-r0.0.noarch.rpm
>   xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4-r0.0.noarch.rpm


I've definitely been focusing on the latter, to the expense of the
former.

> The first problem is of course that AUTOINC is not replaced as intended 
> when a PR server is active. This is due to the code in package_get_auto_pr() 
> that should update PRSERV_PV_AUTOINC not being triggered and thus it 
> retains its default value "AUTOINC". The code isn't triggered because it 
> looks for "AUTOINC" in PKGV, but it isn't there (yet) since ${SRCPV} is 
> no longer used. The following patch solves it:
> 
> diff --git a/meta/classes-global/package.bbclass 
> b/meta/classes-global/package.bbclass
> index 7f55b123c4..bf824b0b83 100644
> --- a/meta/classes-global/package.bbclass
> +++ b/meta/classes-global/package.bbclass
> @@ -266,6 +266,9 @@ python package_get_auto_pr() {
>  d.setVar("PRSERV_HOST", host)
> 
>  pkgv = d.getVar("PKGV")
> +srcpv = bb.fetch.get_pkgv_string(d)
> +if srcpv:
> +pkgv += srcpv
> 
>  # PR Server not active, handle AUTOINC
>  if not d.getVar('PRSERV_HOST'):

Thanks, I'll have to look into the ordering of this. The original code
changes were in this function but that didn't work for other reasons.

> The second problem is that the value from bb.fetch.get_pkgv_string() is 
> concatenated to the version without any separator. So if the AUTOINC 
> part had been replaced as expected, the RPM name for the iso-codes 
> package would have become iso-codes-4.15.00+69ba16daef-r0.0.noarch.rpm, 
> which I doubt was intended... The easiest solution should be to make 
> get_pkgv_string() return "+AUTOINC+" instead of the current 
> "AUTOINC+".

That would then add a separator to strings where there currently isn't
one. The intent wasn't to add to fixed versions but I've over
simplified one of the patches.

> Third, I am not sure if it is a good idea to always include the SHA-1
> in the package name. Before it was typically only done when the SRCREV 
> did not match PV. For most packages it is redundant. It also means that 
> the PR server needs to keep track of the version for all Git recipes,
> not just the ones where ${SRCPV} is used today. However, I understand
> that it is probably not easily solvable to automatically detect when 
> it is needed and when it is not. So it is either do it manually as 
> today and risk missing to do it when it is needed, or always do it and 
> have it be redundant in most cases.

I think we can probably come up with a heuristic like a "+" in PV
triggers the addition which should work in most cases.

> One thing that I would prefer is if there is a variable, similar to 
> the old SRCPV, that contains "${@bb.fetch.get_pkgv_string(d)}" by 
> default and is used from package_convert_pr_autoinc() instead of 
> calling the method directly. The reason for that is to make it 
> possible to use a different method to determine the value. E.g., in 
> our case we have a variable called MAINTPV, which is similar to 
> SRC

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Tue, 2023-08-08 at 10:10 +0100, Richard Purdie via lists.openembedded.org 
wrote:
> There is a recent discussion about AUTOREV which has got me thinking a
> bit about the big picture and incremental development.
> 
> Currently AUTOREV injects into PV. The reasons for that are historical,
> when OE was not taskhash based we had to do that to trigger updates.
> There are now different ways it could be done with the revision just
> needing injection into PKGPV for packaging purposes.
> 
> I've also been looking at externalsrc bugs.
> 
> This got me thinking, if we did make AUTOREV work without messing with
> PV, we could probably go a step further and allow the fetcher to
> incrementally update an existing git checkout rather than remove and
> start again from scratch. Patches may or may not be an issue, we could
> mandate a clean single git:// srcuri to start with although the patch
> code can remove the patch stack already. This would need to hook in to
> disable some of the cleanup/removal code.
> 
> The usecase would be to allow the developer to pull updates from a
> remote repo, or even commit changes locally and then have the system
> "flow" the changes through from there. This would remove the annoying
> clean and re-checkout process which drives some developers crazy. It
> might also allow externalsrc to change to a new model with less
> deviation from the normal workflow.

I tried some experiments and this doesn't actually turn out to be so
difficult to do. I've included a patch below which basically drops the
SRCPV pieces into PKGV, the conditional can probably be simplified
further.

This patch isn't quite right in that we need to split get_srcrev into
two versions, one which returns the SRCREV_FORMAT string and one which
returns the full hashes for taskhash purposes.

The nice thing about this approach is that SRCPV can simply be removed
over time, it no longer does anything. That means compatibility is
retained with existing recipes.

I've not tested this extensively yet with AUTOREV but normal builds
appear to work ok with it and the WORKDIR directory naming
simplification is probably a win.

Cheers,

Richard

From: Richard Purdie 
Subject: fetch: Move SRCPV from PV to PKGV

Signed-off-by: Richard Purdie 
---
 bitbake/lib/bb/fetch2/__init__.py   |  5 -
 meta/classes-global/base.bbclass| 26 +-
 meta/classes-global/package.bbclass |  6 +-
 meta/conf/bitbake.conf  |  5 +
 4 files changed, 11 insertions(+), 31 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py 
b/bitbake/lib/bb/fetch2/__init__.py
index ad9d8a31a7e..ea51a009e1d 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -753,7 +753,7 @@ def get_autorev(d):
 d.setVar("__BBAUTOREV_SEEN", True)
 return "AUTOINC"
 
-def get_srcrev(d, method_name='sortable_revision'):
+def get_srcrev(d, method_name='sortable_revision', allow_none=False):
 """
 Return the revision string, usually for use in the version string (PV) of 
the current package
 Most packages usually only have one SCM so we just pass on the call.
@@ -781,6 +781,9 @@ def get_srcrev(d, method_name='sortable_revision'):
 scms.append(u)
 
 if not scms:
+if allow_none:
+d.delVar("__BBINSRCREV")
+return ""
 raise FetchError("SRCREV was used yet no valid SCM was found in 
SRC_URI")
 
 if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index cbda8d12f09..2567b7d24be 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -130,7 +130,7 @@ addtask fetch
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
-do_fetch[vardeps] += "SRCREV"
+do_fetch[vardepvalue] += "${@bb.fetch.get_srcrev(d, allow_none=True)}"
 do_fetch[network] = "1"
 python base_do_fetch() {
 
@@ -606,7 +606,6 @@ python () {
 bb.debug(1, "Skipping recipe %s because of incompatible 
license(s): %s" % (pn, ' '.join(incompatible_lic)))
 raise bb.parse.SkipRecipe("it has incompatible license(s): 
%s" % ' '.join(incompatible_lic))
 
-needsrcrev = False
 srcuri = d.getVar('SRC_URI')
 for uri_string in srcuri.split():
 uri = bb.fetch.URI(uri_string)
@@ -619,24 +618,17 @@ python () {
 
 # Svn packages should DEPEND on subversion-native
 if uri.scheme == "svn":
-needsrcrev = True
 d.appendVarFlag('do_fetch', 'depends', ' 
subversion-native:do_populate_sysroot')
 
 # Git packages should DEPEND 

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Wed, 2023-08-09 at 22:49 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Wed, 2023-08-09 at 14:03 +, Peter Kjellerstedt wrote:
> > 
> > One question though: would any of this make it possible to use 
> > ${AUTOREV} together with BB_SRCREV_POLICY = "cache"?
> 
> No, since those two things are mutually exclusive.

Let me put this another way. I don't think those two options can ever
work together. 

It may be possible to add a different policy to do what you're thinking
of. The main issue with that is documentation, automated tests and
making it easy for users to know which they want. That is probably
possible but not easy.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1730): 
https://lists.openembedded.org/g/openembedded-architecture/message/1730
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

On Wed, 2023-08-09 at 14:03 +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From: openembedded-architecture@lists.openembedded.org
> >  > architect...@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 8 augusti 2023 11:11
> > To: openembedded-architecture  > architect...@lists.openembedded.org>
> > Subject: [Openembedded-architecture] Workflow improvements -
> > incremental
> > source updates?
> > 
> > There is a recent discussion about AUTOREV which has got me
> > thinking a
> > bit about the big picture and incremental development.
> > 
> > Currently AUTOREV injects into PV. The reasons for that are
> > historical,
> > when OE was not taskhash based we had to do that to trigger
> > updates.
> > There are now different ways it could be done with the revision
> > just
> > needing injection into PKGPV for packaging purposes.
> > 
> > I've also been looking at externalsrc bugs.
> > 
> > This got me thinking, if we did make AUTOREV work without messing
> > with
> > PV, we could probably go a step further and allow the fetcher to
> > incrementally update an existing git checkout rather than remove
> > and
> > start again from scratch. Patches may or may not be an issue, we
> > could
> > mandate a clean single git:// srcuri to start with although the
> > patch
> > code can remove the patch stack already. This would need to hook in
> > to
> > disable some of the cleanup/removal code.
> > 
> > The usecase would be to allow the developer to pull updates from a
> > remote repo, or even commit changes locally and then have the
> > system
> > "flow" the changes through from there. This would remove the
> > annoying
> > clean and re-checkout process which drives some developers crazy.
> > It
> > might also allow externalsrc to change to a new model with less
> > deviation from the normal workflow.
> > 
> > Whilst on the subject, I have also been wondering about a new image
> > target, effectively one that would build a new image but then allow
> > an
> > rsync over to a running target. Obviously this may or may not be
> > advisable depending on what was actually running but for some
> > development, it could be useful.
> > 
> > Anyway, just thinking out loud but wanted to document the idea. I
> > might
> > look into at least improving the way AUTOREV works...
> > 
> > Cheers,
> > 
> > Richard
> 
> It sounds like there are a lot of good possibilities here.
> 
> One question though: would any of this make it possible to use 
> ${AUTOREV} together with BB_SRCREV_POLICY = "cache"?

No, since those two things are mutually exclusive.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1729): 
https://lists.openembedded.org/g/openembedded-architecture/message/1729
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] git.openembedded.org

[Richard Purdie]

On Wed, 2023-08-09 at 08:21 -0500, Ryan Eatmon via
lists.openembedded.org wrote:
> We are seeing a lot failures with the git.openembedded.org server while 
> running our regular CICD builds.  Does the server need to be kicked or 
> something?  Also, is the right list to report/discuss issues like this?

yocto-infrastruct...@lists.yoctoproject.org is perhaps the right place,
and/or helpd...@yoctoproject.org.

I've copied them on my reply.

I know we have seen a couple of glitches on the autobuilder as well
related to this and there is a server transition plan in progress for
this service.

Cheers,

Richard

> 08:11:29 Cloning into 
> '/scratch/jenkins_builds/arago-master-next/build/sources/oe-core'...
> 08:11:48 error: Unable to get pack index 
> https://git.openembedded.org/openembedded-core/objects/pack/pack-c60b12d26c1fad6b8805af480670be31735e7f8f.idx
> 08:11:48 error: Unable to get pack index 
> https://git.openembedded.org/openembedded-core/objects/pack/pack-fd33b8f607375a7c61da83b0cf00318d21726dae.idx
> 08:11:49 error: Unable to get pack index 
> https://git.openembedded.org/openembedded-core/objects/pack/pack-95788bc2bc228ac1b0d1d505522830a98a3bb8e2.idx
> 08:12:02 error: Unable to get pack index 
> https://git.openembedded.org/openembedded-core/objects/pack/pack-00d508e7dcfde948435d430584e129a938f70677.idx
> 08:14:30 error: Unable to find 3e64884e615e76e97429b0aa38637254dd5a99d5 
> under https://git.openembedded.org/openembedded-core
> 08:14:30 Cannot obtain needed tree 3e64884e615e76e97429b0aa38637254dd5a99d5
> 08:14:30 while processing commit bff17171841fd15fb6084262e397c736005f7e3b.
> 08:14:30 error: fetch failed.
> 08:14:31 git failed... remaining attempts: 4sleeping 15 seconds
> 08:14:46 Cloning into 
> '/scratch/jenkins_builds/arago-master-next/build/sources/oe-core'...
> 08:15:01 error: Unable to get pack index 
> https://git.openembedded.org/openembedded-core/objects/pack/pack-4e27dc9c1f2589302f976de7fd3550aef1f9f4d5.idx
> 08:18:40 error: Unable to find 7d148142fc7828abbc8c360e6890d841fd5b4b68 
> under https://git.openembedded.org/openembedded-core
> 08:18:40 Cannot obtain needed blob 7d148142fc7828abbc8c360e6890d841fd5b4b68
> 08:18:40 while processing commit f13fe33403f077802640e70a84596546d555c3b9.
> 08:18:40 error: fetch failed.
> 08:18:40 git failed... remaining attempts: 3sleeping 30 seconds
> 
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1727): 
https://lists.openembedded.org/g/openembedded-architecture/message/1727
Mute This Topic: https://lists.openembedded.org/mt/100642555/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Workflow improvements - incremental source updates?

[Richard Purdie]

There is a recent discussion about AUTOREV which has got me thinking a
bit about the big picture and incremental development.

Currently AUTOREV injects into PV. The reasons for that are historical,
when OE was not taskhash based we had to do that to trigger updates.
There are now different ways it could be done with the revision just
needing injection into PKGPV for packaging purposes.

I've also been looking at externalsrc bugs.

This got me thinking, if we did make AUTOREV work without messing with
PV, we could probably go a step further and allow the fetcher to
incrementally update an existing git checkout rather than remove and
start again from scratch. Patches may or may not be an issue, we could
mandate a clean single git:// srcuri to start with although the patch
code can remove the patch stack already. This would need to hook in to
disable some of the cleanup/removal code.

The usecase would be to allow the developer to pull updates from a
remote repo, or even commit changes locally and then have the system
"flow" the changes through from there. This would remove the annoying
clean and re-checkout process which drives some developers crazy. It
might also allow externalsrc to change to a new model with less
deviation from the normal workflow.

Whilst on the subject, I have also been wondering about a new image
target, effectively one that would build a new image but then allow an
rsync over to a running target. Obviously this may or may not be
advisable depending on what was actually running but for some
development, it could be useful.

Anyway, just thinking out loud but wanted to document the idea. I might
look into at least improving the way AUTOREV works...

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1725): 
https://lists.openembedded.org/g/openembedded-architecture/message/1725
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OpenEmbedded Architecture] Moving the Opkg mailing list

[Richard Purdie]

On Mon, 2023-07-24 at 10:21 -0400, Alex Stewart wrote:
> 
> On 7/24/23 09:37, Richard Purdie wrote:
> > On Mon, 2023-07-24 at 09:23 -0400, Alex Stewart wrote:
> > > Thanks for the feedback. Does anyone on the list know who the authority
> > > would be to approve (or deny) this move would be? I can make a
> > > recommendation as the repo maintainer, but I'm not sure I have the
> > > *ownership* to change the canon. Do I need to take this to the Yocto
> > > steering committee?
> > I suspect the Yocto Project TSC is the decision making "authority" for
> > this although it obviously needs to be done in conjunction with the
> > maintainer.
> 
> Sounds like I should plan to stop by the next TSC meeting then.
> 
> I don't see any details about when/where the next meeting is on either 
> the latest project status email, or the wiki (unless I'm just blind). Is 
> there an invite for that meeting series that I can subscribe to?

The Yocto Project TSC works differently to the OE TSC and doesn't meet
publicly. The TSC members are public, easily contacted and the TSC
itself can be contacted through t...@lists.yoctoproject.org.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1722): 
https://lists.openembedded.org/g/openembedded-architecture/message/1722
Mute This Topic: https://lists.openembedded.org/mt/100278606/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OpenEmbedded Architecture] Moving the Opkg mailing list

[Richard Purdie]

On Mon, 2023-07-24 at 09:23 -0400, Alex Stewart wrote:
> Thanks for the feedback. Does anyone on the list know who the authority 
> would be to approve (or deny) this move would be? I can make a 
> recommendation as the repo maintainer, but I'm not sure I have the 
> *ownership* to change the canon. Do I need to take this to the Yocto 
> steering committee?

I suspect the Yocto Project TSC is the decision making "authority" for
this although it obviously needs to be done in conjunction with the
maintainer.

> On 7/23/23 07:55, Alexander Kanavin wrote:
> > May I suggest trying out sourcehut as the hosting provider for opkg?
> > 
> > https://sourcehut.org
> > 
> > It has all the web niceties, but it's also been built from the ground
> > up to integrate with the email workflow, unlike any of the 'big' SCM
> > providers, so if that works out well for opkg, we could consider using
> > it elsewhere (yes, the core yocto repos :)
> 
> That's an idea. I don't think I have enough knowledge about how the 
> existing mailing-list based workflows are designed to do an immediate 
> eval of sourcehut. But I can test it out as an option, if I can get some 
> traction on the migration.

Moving the list to groups.io where the rest of the Yocto Project/OE
lists are is straight forward and easy to do, if you request that we
can just make that happen.

As has been mentioned, "we" have mixed feelings on github/gitlab. There
are some layers that use those workflows and our policy is that it
needs to be documented in the project's README, whatever the submission
process is. That decision really comes down to the layer maintainer.

sourcehut does sound interesting. Looks like you can run your own
instance or their servers, the latter eventually encouraging you to
support them (which is fair enough). Yocto Project probably can't use
their servers without some kind of support in place. I really don't
know enough about it to know how much it makes sense. It is described
as "alpha" which probably isn't ready for us to depend on too. I am
curious though...

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1720): 
https://lists.openembedded.org/g/openembedded-architecture/message/1720
Mute This Topic: https://lists.openembedded.org/mt/100278606/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] create-spdx support in dunfell

[Richard Purdie]

Hi All,

I'm sending this to oe-arch as it seems the best place to
mention/discuss it.

After reading an email about encouraging best practises when supporting
older software releases, it got me thinking about our LTS and spdx/sbom
support. We don't have create-spdx there and our official policy says
no feature backports or breaking changes.

If you understand LTS as "support" instead of "stable", there is a
compelling argument that we should be supporting people of older
releases to meet things like the new legislation around SBoMs. I think
the world is learning that difference and being more open to it too.

I discussed this with the YP TSC and we agreed we probably should
evaluate the options for adding create-spdx to dunfell. There are a lot
of questions about it and it is something a lot of users would like. I
think it would better reflect on the project to support it.

There is some risk to this. We need the bb.compress zstd code and some
changes to package.bbclass, specially:

https://git.yoctoproject.org/poky/commit/?id=7ec54b174304e940ec66f21ac512f7b50fa637b3

The class itself isn't problematic as it is standalone.

This doesn't mean any policy will change, it would be a one off
exception granted by the TSC for an initiative to support older
releases in a changing world.

I think Joshua said he'd be willing to have a look at what was needed,
I wanted to put the idea out there and say the TSC is open minded to
the idea assuming testing works out ok and so on. We need to look
carefully at the zstd requirement in particular. I believe our LTS
maintainer is open to the idea too if there is support from the
community in working out the details.

I guess I'm asking if there is that support? :)

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1706): 
https://lists.openembedded.org/g/openembedded-architecture/message/1706
Mute This Topic: https://lists.openembedded.org/mt/97311397/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Empty packages and bogus dependencies - what to do?

[Richard Purdie]

On Tue, 2023-02-28 at 22:28 -0500, Denys Dmytriyenko wrote:
> On Sat, Feb 25, 2023 at 10:24:17AM +0000, Richard Purdie wrote:
> > Hi All,
> 
> 
> 
> > Reasons vary but the easy to understand case is where the code is just
> > development headers and all ends up in ${PN}-dev.
> > 
> > We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839
> > 
> > We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
> > turned out to be a bad idea and we relaxed it to a RRECOMENDS:
> > 
> > https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b
> > 
> > I did at least mark up the recipes with this issue:
> > 
> > https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81
> 
> Independent of this discussion - does it make sense to backport the above 2 
> changes to Kirkstone?

I suspect not since it is a significant change in behaviour which we're
not seeing a pressing need for from kirkstone users?

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1705): 
https://lists.openembedded.org/g/openembedded-architecture/message/1705
Mute This Topic: https://lists.openembedded.org/mt/97224039/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Empty packages and bogus dependencies - what to do?

[Richard Purdie]

Hi All,

There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.

I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.

The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:

bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.

As I see it there are a few options:

a) Do nothing and continue with the above   
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else

I think I'm in favour of c), removing PN from PACKAGES and not even
pretending it exists. We could add an RPROVIDES:${PN}-dev += "${PN} so
that assumptions about naming work, not sure about that.


That handles one specific corner case but there are others as bug #8222
talks about for example
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8222

Here, the proposal is to change the globally inserted RDEPENDS (now a
RRECOMMEND) from PN-dev to PN to become conditional upon (roughly hand
waving), there being libraries in other packages which PN-dev therefore
needs to depend upon. I think that idea is worthy of exploration. Are
there any cases where the -dev package depends on ${PN} for things
other than shared libraries?

Thinking out loud, our symlink dependency handling code in do_package
might even be enough to detect and handle injecting that?


So, turning this into a set of actions, my personal view is that we
should probably:

a) Add warnings for dependencies (RDEPENDS or RRECOMMENDS) on packages
which don't get created for some reason (either not in PACKAGES or not
ALLOW_EMPTY). We should be able to properly check this using
packagedata which never used to exist as it does now when these bugs
were first created! We can leave the warnings disabled at first until
we clean things up for core and our key layers

b) Work through those warnings to work out if we can silence them at
source. I think the dependency creation code for -dbg and -dev packages
will need to be taught to check if a target dependency exists to fix
the bulk of them.

c) Explore whether we should drop PN from PACKAGES in some of the above
cases.

d) Explore whether we can replace the current generic RRECOMMENDS from
PN-dev -> PN with something autogenerated based on the presence of
shared libs

e) For a long time I've wondered if we should split the -dbg packages
from one per recipe to one per package with binaries in it. This would
solve some of the problems that we'll likely encounter in b) but might 













new ones.

I've given this some thought but not thought through the ideas in full.

I appreciate some of this was set as a newcommer bug in bugzilla and in
hindsight this perhaps shouldn't have been the case. I suspect taking
d) in isolation was the idea and that should still be possible.

I'd note that we're at feature freeze so more invasive changes like
this would likely need to be post release unless any become bug fixes.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1701): 
https://lists.openembedded.org/g/openembedded-architecture/message/1701
Mute This Topic: https://lists.openembedded.org/mt/97224039/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] WORKDIR fetcher interaction issue

[Richard Purdie]

On Sat, 2022-12-31 at 10:47 -0600, Mark Hatle wrote:
> 
> On 12/29/22 9:06 AM, Richard Purdie wrote:
> > On Thu, 2022-12-29 at 08:50 -0600, Joshua Watt wrote:
> > > On Thu, Dec 29, 2022 at 7:56 AM Richard Purdie
> > >  wrote:
> > > > 
> > > > I was asked about a WORKDIR/fetcher interaction problem and the bugs it
> > > > results in. I've tried to write down my thoughts.
> > > > 
> > > > The unpack task writes it's output to WORKDIR as base.bbclass says:
> > > > 
> > > >  fetcher = bb.fetch2.Fetch(src_uri, d)
> > > >  fetcher.unpack(d.getVar('WORKDIR')
> > > > 
> > > > We historically dealt with tarballs which usually have a NAME-VERSION
> > > > directory within them, so when you extract them, they go into a sub
> > > > directory which tar creates. We usually call that subdirectory "S".
> > > > 
> > > > When we wrote the git fetcher, we emulated this by using a "git"
> > > > directory to extract into rather than WORKDIR.
> > > > 
> > > > For local files, there is no sub directory so they go into WORKDIR.
> > > > This includes patches, which do_patch looks for in WORKDIR and applies
> > > > them from there.
> > > > 
> > > > What issues does this cause? If you have an existing WORKDIR and run a
> > > > build with:
> > > > 
> > > > SRC_URI = "file://a file://b"
> > > > 
> > > > then change it to:
> > > > 
> > > > SRC_URI = "file://a"
> > > > 
> > > > and rebuild the recipe, the fetch and unpack tasks will rerun and their
> > > > hashes will change but the file "b" is still in WORKDIR. Nothing in the
> > > > codebase knows that it should delete "b" from there. If you have code
> > > > which does "if exists(b)", which is common, it will break.
> > > > 
> > > > There are variations on this, such as a conditional append on some
> > > > override to SRC_URI but the fundamental problem is one of cleanup when
> > > > unpack is to rerun.
> > > > 
> > > > The naive approach is then to think "lets just delete WORKDIR" when
> > > > running do_unpack. There is the small problem of WORKDIR/temp with logs
> > > > in. There is also the pseudo database and other things tasks could have
> > > > done. Basically, whilst tempting, it doesn't work out well in practise
> > > > particularly as that whilst unpack might rerun, not all other tasks
> > > > might.
> > > > 
> > > > I did also try a couple of other ideas. We could fetch into a
> > > > subdirectory, then either copy or symlink into place depending on which
> > > > set of performance/usabiity challenges you want to deal with. You could
> > > > involve a manifest of the files and then move into position so later
> > > > you'd know which ones to delete.
> > > > 
> > > > Part of the problem is that in some cases recipes do:
> > > > 
> > > > S = "${WORKDIR}"
> > > > 
> > > > for simplicity. This means that you also can't wipe out S as it might
> > > > point at WORKDIR.
> > > > 
> > > > SPDX users have requested a json file of file and checksums after the
> > > > unpack and before do_patch. Such a manifest could also be useful for
> > > > attempting cleanup of an existing WORKDIR so I suspect the solution
> > > > probably lies in that direction, probably unpacking into a subdir,
> > > > indexing it, then moving into position.
> > > 
> > > By "moving it into position" do you mean moving the files from the
> > > clean subdirectory to the locations they would occupy today?
> > > 
> > > If so I don't understand why that's strictly necessary. It seems
> > > like almost all of the complexity of this will be to support a
> > > use-case we don't really like anyway (S = "${WORKDIR}"). Manifests are
> > > great and all, but it causes a lot of problems if they get out of sync
> > > and I suspect that would happen more often than we would like, e.g.
> > > with devtool, make config, manual editing, etc. If we can keep it
> > > simple and not rely on external state (e.g. a manifest) I think it
> > > will be a lot easier to maintain in the long run.
> > 
> > Dropping S = "

Re: [Openembedded-architecture] WORKDIR fetcher interaction issue

[Richard Purdie]

On Thu, 2022-12-29 at 08:50 -0600, Joshua Watt wrote:
> On Thu, Dec 29, 2022 at 7:56 AM Richard Purdie
>  wrote:
> > 
> > I was asked about a WORKDIR/fetcher interaction problem and the bugs it
> > results in. I've tried to write down my thoughts.
> > 
> > The unpack task writes it's output to WORKDIR as base.bbclass says:
> > 
> > fetcher = bb.fetch2.Fetch(src_uri, d)
> > fetcher.unpack(d.getVar('WORKDIR')
> > 
> > We historically dealt with tarballs which usually have a NAME-VERSION
> > directory within them, so when you extract them, they go into a sub
> > directory which tar creates. We usually call that subdirectory "S".
> > 
> > When we wrote the git fetcher, we emulated this by using a "git"
> > directory to extract into rather than WORKDIR.
> > 
> > For local files, there is no sub directory so they go into WORKDIR.
> > This includes patches, which do_patch looks for in WORKDIR and applies
> > them from there.
> > 
> > What issues does this cause? If you have an existing WORKDIR and run a
> > build with:
> > 
> > SRC_URI = "file://a file://b"
> > 
> > then change it to:
> > 
> > SRC_URI = "file://a"
> > 
> > and rebuild the recipe, the fetch and unpack tasks will rerun and their
> > hashes will change but the file "b" is still in WORKDIR. Nothing in the
> > codebase knows that it should delete "b" from there. If you have code
> > which does "if exists(b)", which is common, it will break.
> > 
> > There are variations on this, such as a conditional append on some
> > override to SRC_URI but the fundamental problem is one of cleanup when
> > unpack is to rerun.
> > 
> > The naive approach is then to think "lets just delete WORKDIR" when
> > running do_unpack. There is the small problem of WORKDIR/temp with logs
> > in. There is also the pseudo database and other things tasks could have
> > done. Basically, whilst tempting, it doesn't work out well in practise
> > particularly as that whilst unpack might rerun, not all other tasks
> > might.
> > 
> > I did also try a couple of other ideas. We could fetch into a
> > subdirectory, then either copy or symlink into place depending on which
> > set of performance/usabiity challenges you want to deal with. You could
> > involve a manifest of the files and then move into position so later
> > you'd know which ones to delete.
> > 
> > Part of the problem is that in some cases recipes do:
> > 
> > S = "${WORKDIR}"
> > 
> > for simplicity. This means that you also can't wipe out S as it might
> > point at WORKDIR.
> > 
> > SPDX users have requested a json file of file and checksums after the
> > unpack and before do_patch. Such a manifest could also be useful for
> > attempting cleanup of an existing WORKDIR so I suspect the solution
> > probably lies in that direction, probably unpacking into a subdir,
> > indexing it, then moving into position.
> 
> By "moving it into position" do you mean moving the files from the
> clean subdirectory to the locations they would occupy today?
> 
> If so I don't understand why that's strictly necessary. It seems
> like almost all of the complexity of this will be to support a
> use-case we don't really like anyway (S = "${WORKDIR}"). Manifests are
> great and all, but it causes a lot of problems if they get out of sync
> and I suspect that would happen more often than we would like, e.g.
> with devtool, make config, manual editing, etc. If we can keep it
> simple and not rely on external state (e.g. a manifest) I think it
> will be a lot easier to maintain in the long run.

Dropping S = "${WORKDIR}" doesn't solve the problem being described
here, it just removes something which complicates current code and
makes that problem harder to solve. Even not supporting S =
"${WORKDIR}", do_unpack still unpacks to WORKDIR with the S directory
created by the tarball.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1694): 
https://lists.openembedded.org/g/openembedded-architecture/message/1694
Mute This Topic: https://lists.openembedded.org/mt/95936561/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] WORKDIR fetcher interaction issue

[Richard Purdie]

I was asked about a WORKDIR/fetcher interaction problem and the bugs it
results in. I've tried to write down my thoughts.

The unpack task writes it's output to WORKDIR as base.bbclass says:

fetcher = bb.fetch2.Fetch(src_uri, d)
fetcher.unpack(d.getVar('WORKDIR')

We historically dealt with tarballs which usually have a NAME-VERSION
directory within them, so when you extract them, they go into a sub
directory which tar creates. We usually call that subdirectory "S".

When we wrote the git fetcher, we emulated this by using a "git"
directory to extract into rather than WORKDIR.

For local files, there is no sub directory so they go into WORKDIR.
This includes patches, which do_patch looks for in WORKDIR and applies
them from there.

What issues does this cause? If you have an existing WORKDIR and run a
build with:

SRC_URI = "file://a file://b"

then change it to:

SRC_URI = "file://a"

and rebuild the recipe, the fetch and unpack tasks will rerun and their
hashes will change but the file "b" is still in WORKDIR. Nothing in the
codebase knows that it should delete "b" from there. If you have code
which does "if exists(b)", which is common, it will break.

There are variations on this, such as a conditional append on some
override to SRC_URI but the fundamental problem is one of cleanup when
unpack is to rerun.

The naive approach is then to think "lets just delete WORKDIR" when
running do_unpack. There is the small problem of WORKDIR/temp with logs
in. There is also the pseudo database and other things tasks could have
done. Basically, whilst tempting, it doesn't work out well in practise
particularly as that whilst unpack might rerun, not all other tasks
might.

I did also try a couple of other ideas. We could fetch into a
subdirectory, then either copy or symlink into place depending on which
set of performance/usabiity challenges you want to deal with. You could
involve a manifest of the files and then move into position so later
you'd know which ones to delete.

Part of the problem is that in some cases recipes do:

S = "${WORKDIR}"

for simplicity. This means that you also can't wipe out S as it might
point at WORKDIR.

SPDX users have requested a json file of file and checksums after the
unpack and before do_patch. Such a manifest could also be useful for
attempting cleanup of an existing WORKDIR so I suspect the solution
probably lies in that direction, probably unpacking into a subdir,
indexing it, then moving into position.

Personally, I'd also like to see S = "${WORKDIR}" deprecated and
dropped so that a subdir is always used, just to stop our code getting
too full of corner cases which are hard to maintain.

I've had a few experiments with variations on both issues on various
branches at different times, I just haven't had enough time to
socialise the changes, migrate code and handle the inevitable fallout.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1690): 
https://lists.openembedded.org/g/openembedded-architecture/message/1690
Mute This Topic: https://lists.openembedded.org/mt/95936561/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OE-core] Y2038 proposal

[Richard Purdie]

On Fri, 2022-12-02 at 16:54 +0800, Matt Johnston wrote:
> Your email prompted me to check my own software (Dropbear) and it showed a
> few y2038 issues to fix. Those bugs wouldn't be noticed from a quick test -
> it "only" prevented auth and idle timeouts from occurring.
> 
> gcc and clang are able to flag truncated conversions for 64-bit time_t with 
> -Wconversion, but that's very noisy. Comparing that against a 32-bit time_t
> build, however, gives a pretty clean list of code that needs attention.
> 
> As an experiment I've built OpenBMC with and without 64-bit time_t,
> https://github.com/mkj/yocto-y2038 has the results and a description. There
> are a mix of false positives (particularly tv_usec/tv_nsec), but also some
> real-looking things. As an example, busybox using a uint32_t to copy a dhcpd
> lease expiry.
> 
> I'm not sure the best way to use these logs - they need manual review.
> Expanding the list of packages should be easy, but there will be more that
> need manual intervention to get rid of -Werror.

That is really interesting data as it confirmed there are real world
issues which changing the compiler flags is going to break. Thanks for
sharing.

What you describe is relatively easy for a maintainer to do as a one
off check but not really something we can do at scale for all the
software we build. It worries me :/. I guess the one upside is that
whilst it did break some functionality, it didn't actually crash the
runtime if I understand what happened correctly.

I'm not sure this should stop our plan to switch the flags but it is
certainly something to think about and be aware of.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1689): 
https://lists.openembedded.org/g/openembedded-architecture/message/1689
Mute This Topic: https://lists.openembedded.org/mt/95404085/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] [OE-core] [Openembedded-architecture] Y2038 proposal

[Richard Purdie]

On Mon, 2022-12-05 at 11:00 +0100, Ola x Nilsson wrote:
> On Wed, Nov 30 2022, Richard Purdie wrote:
> 
> > On Wed, 2022-11-30 at 17:56 +0100, Alexandre Belloni wrote:
> > > On 30/11/2022 16:46:17+, Ross Burton wrote:
> > > > On 30 Nov 2022, at 14:20, Richard Purdie via
> > > > lists.yoctoproject.org
> > > >  wrote:
> > > > > > > * Could we optionally disable some of the glibc 32 bit function 
> > > > > > > calls
> > > > > > > to ensure they're not being used? 
> > > > > > 
> > > > > > Could you be more specific here? Would you like to disable some
> > > > > > syscalls?
> > > > > 
> > > > > I'm meaning disabling the 32 bit glibc time functions.
> > > > 
> > > > Some time ago I filed
> > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=6803 as Debian
> > > > has a nice sanity check where it warns if non-LFS glibc functions
> > > > are used.  I imagine the same logic could be used to check for 32-
> > > > bit time_t use.
> > 
> > That sounds interesting and something we should probably look into for
> > both issues...
> 
> I have a working sanity checker that checks for any glibc functions
> affected by -D_FILE_OFFSET_BITS=64 or -D_TIME_BITS=64.
> The INSANE_SKIP functionality needs some more polish but I'd be happy to
> contribute it.
> 
> Some libraries use both 32 and 64 bit APIs to glibc and needs exceptions
> in the checker.
> 
> I have not run any world builds with this checker, I've focused on the
> recipes we actually use so far so we could get to a testable system.  My
> biggest worry at the moment is rust, I know to little to know if it is
> an actual problem and how to fix it.
> 
> I would like to be part of any "y2038 team" for Yocto.

That does sound useful, perhaps sharing it as an RFC patch might be a
good place to start? We might be able to run one of the autobuilder
world targets against it, see how it looks for our core recipes?

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1687): 
https://lists.openembedded.org/g/openembedded-architecture/message/1687
Mute This Topic: https://lists.openembedded.org/mt/95361985/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Y2038 proposal

[Richard Purdie]

On Thu, 2022-12-01 at 11:27 +0100, Alexander Kanavin wrote:
> On Wed, 30 Nov 2022 at 14:15, Richard Purdie
>  wrote:
> > * We need to have a 32 bit ptest run on the autobuilder (qemux86 should
> > work, not sure we can make qemuarm fast). Whether this is manually
> > triggered, not sure. We could have a smaller set of ptests to run for
> > it?
> 
> I just ran qemux86 full ptest locally. It took 4h:10m (same as
> qemuarm64 ptest on an arm worker). The fails were:
> 
> {'python3': ['test_deterministic_sets'],
>  'valgrind': ['gdbserver_tests/hgtls',
>   'gdbserver_tests/mcblocklistsearch',
>   'gdbserver_tests/mcbreak',
>   'gdbserver_tests/mcclean_after_fork',
>   'gdbserver_tests/mchelp',
>   'gdbserver_tests/mcinfcallRU',
>   'gdbserver_tests/mcinfcallWSRU',
>   'gdbserver_tests/mcinvokeRU',
>   'gdbserver_tests/mcinvokeWS',
>   'gdbserver_tests/mcleak',
>   'gdbserver_tests/mcmain_pic',
>   'gdbserver_tests/mcsignopass',
>   'gdbserver_tests/mcsigpass',
>   'gdbserver_tests/mcvabits',
>   'gdbserver_tests/mcwatchpoints',
>   'gdbserver_tests/mssnapshot',
>   'gdbserver_tests/nlcontrolc',
>   'gdbserver_tests/nlgone_abrt',
>   'gdbserver_tests/nlgone_exit',
>   'gdbserver_tests/nlgone_return',
>   'gdbserver_tests/nlpasssigalrm',
>   'gdbserver_tests/nlsigvgdb',
>   'gdbserver_tests/nlvgdbsigqueue',
>   'memcheck/tests/linux/memfd_create',
>   'memcheck/tests/linux/timerfd-syscall',
>   'memcheck/tests/origin5-bz2',
>   'massif/tests/mmapunmap']}
> 
> So I think we could as well fix these, and add full qemux86 ptest to
> a-full? It is not heavy on the builder machine (mostly just runs a
> single qemu thread), it's just long.

I think we should fix those and we should add the target to the
autobuilder but I'm reluctant to add a long test to a-full. The fact it
is relatively clean suggests it doesn't regress that often. We could do
something like a once a month trigger for it?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1683): 
https://lists.openembedded.org/g/openembedded-architecture/message/1683
Mute This Topic: https://lists.openembedded.org/mt/95354042/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] [OE-core] [Openembedded-architecture] Y2038 proposal

[Richard Purdie]

On Wed, 2022-11-30 at 17:56 +0100, Alexandre Belloni wrote:
> On 30/11/2022 16:46:17+, Ross Burton wrote:
> > On 30 Nov 2022, at 14:20, Richard Purdie via lists.yoctoproject.org 
> >  wrote:
> > > > > * Could we optionally disable some of the glibc 32 bit function calls
> > > > > to ensure they're not being used? 
> > > > 
> > > > Could you be more specific here? Would you like to disable some
> > > > syscalls?
> > > 
> > > I'm meaning disabling the 32 bit glibc time functions.
> > 
> > Some time ago I filed
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=6803 as Debian
> > has a nice sanity check where it warns if non-LFS glibc functions
> > are used.  I imagine the same logic could be used to check for 32-
> > bit time_t use.

That sounds interesting and something we should probably look into for
both issues...

> > 
> 
> We can simply disable COMPAT_32BIT_TIME in the kernel config.

That would cause runtime issues but not build time linking ones?

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1679): 
https://lists.openembedded.org/g/openembedded-architecture/message/1679
Mute This Topic: https://lists.openembedded.org/mt/95361985/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [Openembedded-architecture] Y2038 proposal

[Richard Purdie]

On Wed, 2022-11-30 at 14:36 +0100, Lukasz Majewski wrote:
> > On Wed, 2022-11-30 at 09:07 +0100, Alexander Kanavin wrote:
> > > On Tue, 29 Nov 2022 at 16:45, Stephen Jolley
> > >  wrote:  
> > > > We’d welcome a proposal/series on how to move forward with the
> > > > Y2038 work for 32 bit platforms.  
> > > 
> > > I have the following proposal:
> > > 
> > > 1. A branch is made where:
> > > a. "-D_TIME_BITS=64 -D_FILE_OFFSET_BITS=64" is enabled globally.
> > > b. qemu is always started with "-rtc base=2040-01-01", simulating
> > > Y2038 actually occurring.
> > > c. an additional runtime test verifies that both RTC clock and
> > > system clock report 2040.
> > > 
> > > 2. This branch is run through a-full on the autobuilder. Any
> > > uncovered issues are filed as bugs.
> > > 
> > > 3. Once *all* of the bugs are addressed, repeat point 2.
> > > 
> > > 4. Once there are no more open bugs, 1a is merged into master.
> > > 
> > > Any fatal flaws in the plan?  
> > 
> > Others have made some good comments. My thoughts:
> > 
> > * We need to add some runtime tests to oeqa for this (in addition to
> > the ptests)
> > 
> > * We need to have a 32 bit ptest run on the autobuilder (qemux86
> > should work, not sure we can make qemuarm fast). Whether this is
> > manually triggered, not sure. We could have a smaller set of ptests
> > to run for it?
> 
> Y2038 ptests maybe?
> 
> Here is the list of integrated tests to ptests:
> https://github.com/lmajewski/y2038-tests

Perhaps, yes.

> > * Could we optionally disable some of the glibc 32 bit function calls
> > to ensure they're not being used? 
> 
> Could you be more specific here? Would you like to disable some
> syscalls?

I'm meaning disabling the 32 bit glibc time functions.

> > We don't really want to diverge from
> > upstream glibc much though.
> 
> Could you be more specific here? The glibc now supports the whole set
> of syscalls as of 2.34 version?
> 
> To enable them one needs to pass -D_TIME_BITS=64 flag when compiling
> programs.
> 
> This is now the official glibc ABI.

Right, but the 32 bit time functions/symbols are still available for
older binaries. My point is that anything using those older functions
is likely in need of attention so for Yocto Project/OE usage,
identifying those would be helpful. If we were to disable them, that
would make such usage very obvious.

> 
> > The reason for that is that if someone has
> > existing binaries, there could be problems using them after the
> > change.
> 
> The binary shall work without issues on glibc 2.34+ and 5.10+ kernel
> without issues.

Not necessarily. If it were a binary library, compiled with 32 bit
time_t, new binaries using it would use a different sized field.

> The only problem happens when new binaries with 64 bit time support are
> run on glibc or kernel not supporting 64 bit time. 

That is definitely not the only problem. Some of the problems are
unlikely but we do need to consider them.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1675): 
https://lists.openembedded.org/g/openembedded-architecture/message/1675
Mute This Topic: https://lists.openembedded.org/mt/95357623/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Y2038 proposal

[Richard Purdie]

On Wed, 2022-11-30 at 09:07 +0100, Alexander Kanavin wrote:
> On Tue, 29 Nov 2022 at 16:45, Stephen Jolley  wrote:
> > We’d welcome a proposal/series on how to move forward with the Y2038 work 
> > for 32 bit platforms.
> 
> I have the following proposal:
> 
> 1. A branch is made where:
> a. "-D_TIME_BITS=64 -D_FILE_OFFSET_BITS=64" is enabled globally.
> b. qemu is always started with "-rtc base=2040-01-01", simulating
> Y2038 actually occurring.
> c. an additional runtime test verifies that both RTC clock and system
> clock report 2040.
> 
> 2. This branch is run through a-full on the autobuilder. Any uncovered
> issues are filed as bugs.
> 
> 3. Once *all* of the bugs are addressed, repeat point 2.
> 
> 4. Once there are no more open bugs, 1a is merged into master.
> 
> Any fatal flaws in the plan?

Others have made some good comments. My thoughts:

* We need to add some runtime tests to oeqa for this (in addition to
the ptests)

* We need to have a 32 bit ptest run on the autobuilder (qemux86 should
work, not sure we can make qemuarm fast). Whether this is manually
triggered, not sure. We could have a smaller set of ptests to run for
it?

* Could we optionally disable some of the glibc 32 bit function calls
to ensure they're not being used? We don't really want to diverge from
upstream glibc much though.

* We need to work out how to communicate this change happened and have
people "buy in" to it. The reason for that is that if someone has
existing binaries, there could be problems using them after the change.
We therefore need to be sure they are aware of it.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1673): 
https://lists.openembedded.org/g/openembedded-architecture/message/1673
Mute This Topic: https://lists.openembedded.org/mt/95354042/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OE-core] [yocto] Y2038 proposal

[Richard Purdie]

On Wed, 2022-11-30 at 13:07 +0100, Alexander Kanavin wrote:
> On Wed, 30 Nov 2022 at 12:40, Richard Purdie
>  wrote:
> 
> > To be clear, we don't run ptests on 32 bit targets, only on qemux86-64
> > and qemuarm64 where we have KVM available. We do run image, sdk and
> > eSDK tests on our supported qemu targets, 32 and 64 bit.
> 
> I think kvm does allow 32 bit guest on a 64 bit host.

For x86, yes. For arm, it varies and I know at least one of our arm
hosts doesn't support it, I don't know about the newer ones.

>  But I can imagine making full ptests work on 32 bit guests would be a 
> struggle
> for reasons unrelated to 2038, specifically lack of users outside of
> embedded world.

In general I think it shouldn't be too bad but we really do need to
test it and see.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1672): 
https://lists.openembedded.org/g/openembedded-architecture/message/1672
Mute This Topic: https://lists.openembedded.org/mt/95355886/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [OE-core] [yocto] Y2038 proposal

[Richard Purdie]

On Wed, 2022-11-30 at 12:02 +0100, Alexandre Belloni via
lists.openembedded.org wrote:
> On 30/11/2022 09:07:50+0100, Alexander Kanavin wrote:
> > On Tue, 29 Nov 2022 at 16:45, Stephen Jolley  
> > wrote:
> > > We’d welcome a proposal/series on how to move forward with the Y2038 work 
> > > for 32 bit platforms.
> > 
> > I have the following proposal:
> > 
> > 1. A branch is made where:
> > a. "-D_TIME_BITS=64 -D_FILE_OFFSET_BITS=64" is enabled globally.
> > b. qemu is always started with "-rtc base=2040-01-01", simulating
> > Y2038 actually occurring.
> > c. an additional runtime test verifies that both RTC clock and system
> > clock report 2040.
> > 
> > 2. This branch is run through a-full on the autobuilder. Any uncovered
> > issues are filed as bugs.
> > 
> 
> I ran a-full with "-D_TIME_BITS=64 -D_FILE_OFFSET_BITS=64" last week, it
> didn't go too well gcc-sanitizer and pulseaudio being the main offenders
> but buildtools needs to be investigated.

What is the potential issue with builtools?

> > 3. Once *all* of the bugs are addressed, repeat point 2.
> > 
> > 4. Once there are no more open bugs, 1a is merged into master.
> > 
> > Any fatal flaws in the plan?
> > 
> > It's not hard to see that Y2038 problem is real and serious, e.g. on
> > qemux86 core-image-full-cmdline built from master:
> > 
> > root@qemux86:~# ls /
> > bin  boot  devetc  home  liblost+found  media  mntproc
> > run  sbin  sys  tmp  usrvar
> > root@qemux86:~# date -s "2040-01-01"
> > Sun Jan  1 00:00:00 UTC 2040
> > root@qemux86:~# ls /
> > bin  boot  devetc  home  liblost+found  media  mntproc
> > run  sbin  sys  tmp  usrvar
> > root@qemux86:~# ls /
> > -sh: ls: command not found
> > 
> > On qemux86_64 the same sequence works as expected, of course.
> > 
> 
> The main issue with the plan is that we are not running tests on 32
> qemu anymore.

To be clear, we don't run ptests on 32 bit targets, only on qemux86-64
and qemuarm64 where we have KVM available. We do run image, sdk and
eSDK tests on our supported qemu targets, 32 and 64 bit.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1669): 
https://lists.openembedded.org/g/openembedded-architecture/message/1669
Mute This Topic: https://lists.openembedded.org/mt/95355886/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Bitbake changes for the next release?

[Richard Purdie]

Hi,

I've a few things I've been pondering for a while and given the summit
and OEDVM this week, I thought I'd sent out this email and some RFC
patches.

Export variable flag (and unexport/network)
---

The patch sent as an RFC to the bitbake list has the details on this
one, do we make the export and network flags more useful/useable at the
cost of a 4% parsing speed hit? I'm torn, I just don't know.

addpylib directive
--

The easier one is probably the "addpylib" directive. This would move
the python module import code into a usable ABI within conf files such
as layer.conf. There are patches to OE-Core and Bitbake on the mailing
list as RFCs.

The reasons for doing this are:

* establish a clean API for adding python modules
* drop what was originally a workaround from base.bbclass
* allow potential to extend existing functionality to cover such 
  modules accesses to the datastore in task hashes

I chose to make it append to sys.path in the new implementation and
stack on a first come, first in the queue approach so it will follow
bblayers.conf only. We have too many ways for things to break and don't
need that level of potential craziness IMO.

I added BBIMPORT as a way to list the namespaces to import into
bitbake. I did allow other namespaces to be imported as this is how
time, os and sys are accessed by our anonymous code. I suspect we
should perhaps only allow that from a different API but we can discuss
that or improve in due course.

There is a second patch, admittedly a bit thrown together which
implements variable dependency information from the python functions.
It basically runs the imported functions through our codeparser and
then merges that dependency information with the datastore task
signature/hash generation code. There are a number of things that need
to be improved with it but it does prove the concept can work. I don't
like the global variable, some of the functionality needs exporting to
a function and there are probably data lifecycle issues there but it is
good enough to start a discussion of the idea.

The big win if we do this is allowing large chunks of package.bbclass
and friends to be moved into lib/oe without having to worry about
variable accesses. This should then lead to better memory usage and
lower cache sizes.

Right now it doesn't invalidate the cache on changes to the functions
themselves, only the variable dependencies changing. That can be
changed easily enough if desired.

cache data changes
--

We now have the potential to add all of the data that makes up basehash
(and hence taskhash) to the bitbake cache file. The upside would be
much nicer information about why basehashes mismatch. The downsides
are:

* increased cache file size
* slower parsing speed
* slower load of the cache file back into memory

I have patches which pickle the cache all in one go at the expense of
the cache load progress bar. I did also develop a patch to support
efficient incremental pickling of the data and patches which compress
the cache file and add the basehash data. There is a tradeoff against
various speeds to be made and I'm torn on what to do. We could make the
data an option you could enable for debugging but often the situations
are hard to reproduce so that may not be as useful as it first seems.
These patches are in my poky-contrib branches.


There are further ideas I have floating around but these ones are the
ones there are patches out there for and that are probably ready/worthy
of discussion.

Cheers,

Richard






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1662): 
https://lists.openembedded.org/g/openembedded-architecture/message/1662
Mute This Topic: https://lists.openembedded.org/mt/95298341/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Relocatable binaries - better kernel support?

[Richard Purdie]

On Thu, 2022-11-03 at 10:20 -0400, Paul Gortmaker wrote:
> [Relocatable binaries - better kernel support?] On 03/11/2022 (Thu 13:56) 
> Richard Purdie wrote:
> 
> [...]
> 
> > The code that handles the interpreter is in the kernel, in
> > fs/binfmt_elf.c:load_elf_binary(). The idea would be to add support for
> > $ORIGIN there so that $ORIGIN is replaced with the location of the
> > binary.
> > 
> > Does anyone have an idea if such a change might be acceptable to the
> > upstream kernel? Has anyone ever proposed anything like that?
> 
> All the "normal" rules apply.  You'd need to demonstrate that there is a
> significant user base out there that wants this, that it doesn't disrupt
> any existing users, that it doesn't open the door for security issues
> and root-kits, and finally that the change doesn't make you want to go
> shower with steel wool and bleach after looking at the code.

That makes sense, I know we'd need to be careful about setuid (see my
other email). If I can find a few minutes I guess I'll experiment a bit
and see if I can come up with a patch to test the water with...

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1659): 
https://lists.openembedded.org/g/openembedded-architecture/message/1659
Mute This Topic: https://lists.openembedded.org/mt/94756656/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Relocatable binaries - better kernel support?

[Richard Purdie]

On Thu, 2022-11-03 at 15:24 +0100, Alexander Kanavin wrote:
> On Thu, 3 Nov 2022 at 14:56, Richard Purdie
>  wrote:
> 
> > The code that handles the interpreter is in the kernel, in
> > fs/binfmt_elf.c:load_elf_binary(). The idea would be to add support for
> > $ORIGIN there so that $ORIGIN is replaced with the location of the
> > binary.
> > 
> > Does anyone have an idea if such a change might be acceptable to the
> > upstream kernel? Has anyone ever proposed anything like that?
> 
> I quickly skimmed the ELF spec:
> https://refspecs.linuxfoundation.org/elf/elf.pdf
> and $ORIGIN is not there. It seems to be an extension specific to the
> ld.so implementation (man ld.so).
> 
> Not sure if this would help or hinder the case, just pointing out that
> the ELF spec neither stands in the way nor helps.

Yes, indeed. I also did notice:

https://engineering.backtrace.io/2016-06-29-exploiting-elf-expansion-variables/

which mentions Solaris does support $ORIGIN in PT_INTERP and that if
you're not careful about setuid it could create a security hole so that
would be something to be mindful of. Interesting that other systems
have done it though.

I do think we should explore this a bit. Whilst it wouldn't immediately
change much for us today, once this did get into distros in a couple of
years, we'd be in a much better shape.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1658): 
https://lists.openembedded.org/g/openembedded-architecture/message/1658
Mute This Topic: https://lists.openembedded.org/mt/94756656/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Relocatable binaries - better kernel support?

[Richard Purdie]

I thought I'd quickly write down some notes on the relocatable binary
"problem" we have.

We have a need to be able to run our binaries with a different dynamic
loader in a few cases which basically come down to:

* uninative relocation of "native" binaries so they work on different 
  distros via sstate
* creating the relocatable SDK without needing an old "lowest 
  common denominator" glibc
* used in our tools tarballs which are effectively SDKs with 
  different usage and without the target toolchain pieces

To make this work we end up running patchelf on binaries a lot to
change the interpreter.

The interpreter is set in the PT_INTERP section of the ELF binary and 
contains a full path. A relative path also works but that is relative
to CWD which isn't so useful as CWD can be anything.

What I've wondered is whether we could have linux support relative
paths in there with reference to the binary's own location on the
filesystem? This would be similar to $ORIGIN in RPATHs and similar.

If you have a build you can play around a bit with this:

$ cp /usr/bin/true .
$ tmp/sysroots-uninative/x86_64-linux/usr/bin/patchelf-uninative 
--print-interpreter ./true
/lib64/ld-linux-x86-64.so.2
$ cp /lib64/ld-linux-x86-64.so.2 ../
$ tmp/sysroots-uninative/x86_64-linux/usr/bin/patchelf-uninative 
--set-interpreter ../ld-linux-x86-64.so.2 ./true
$ ./true
$ cd ..
$ .//true 
.//true: No such file or directory

The code that handles the interpreter is in the kernel, in
fs/binfmt_elf.c:load_elf_binary(). The idea would be to add support for
$ORIGIN there so that $ORIGIN is replaced with the location of the
binary.

Does anyone have an idea if such a change might be acceptable to the
upstream kernel? Has anyone ever proposed anything like that?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1656): 
https://lists.openembedded.org/g/openembedded-architecture/message/1656
Mute This Topic: https://lists.openembedded.org/mt/94756656/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Template handling in OE-Core

[Richard Purdie]

On Wed, 2022-09-28 at 20:02 +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From: Alexander Kanavin 
> > Sent: den 28 september 2022 16:41
> > To: Peter Kjellerstedt 
> > Cc: Richard Purdie ;
> > openembedded-
> > architecture 
> > Subject: Re: Template handling in OE-Core
> > 
> > On Wed, 28 Sept 2022 at 16:25, Peter Kjellerstedt
> >  wrote:
> > 
> > > Personally, I would expect the file to contain the latest value
> > > used
> > > for TEMPLATECONF. I.e., if I remove the local.conf file to
> > > regenerate
> > > it because there has been upstream changes, I would expect it to
> > > use
> > > the TEMPLATECONF I specify at that time, and for that value to be
> > > recorded for posterity.
> > > 
> > > That said, I doubt the value of TEMPLATECONF will actually change
> > > for
> > > an existing build environment, so this is probably more on the
> > > theoretical side.
> > 
> > This would complicate the logic (update only if some config was
> > updated), and if you remove local.conf but not bblayers.conf, so
> > that
> > one is updated, but not the other, it is not anymore possible to
> > tell
> > what came from where. Creating it once at the start is more
> > straightforward, even if origins of further updates are not
> > recorded.
> > 
> > Alex
> 
> Yeah, I guess it's a case of damned if you do, damned if you don't. 
> However, as this patch solves our problem, I support applying this
> patch.

Ultimately, I think we'll end up removing this file. Whilst I
understand we might want to track where things came from, I think the
current way breaks for two files and doesn't give version information
anyway.

There may be a better way with a copy alongside of the template,
something involving symlinks or something else.

Meanwhile though I'll take this patch as it solves the problem enough
for release and as far as I can tell, shouldn't make anything worse.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1655): 
https://lists.openembedded.org/g/openembedded-architecture/message/1655
Mute This Topic: https://lists.openembedded.org/mt/93968540/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Template handling in OE-Core

[Richard Purdie]

On Wed, 2022-09-28 at 10:48 +0200, Alexander Kanavin wrote:
> For future enhancements, I would want to keep some kind of record
> inside build/conf of where the original template was taken from. I
> agree that build init scripts should not use it at all, but that
> record can be utilized to either update the build config from an
> updated template (with an explicit, separate command), or update the
> template itself without having to specify where it is.

I wondered about that. The thing is if you're planning to decide to do
some kind of automated update or user warning message, you need a
version string in the file itself to know what version you're dealing
with so whilst I can see why you might be interested in that
information, I'm not convinced it is as useful as you'd first think as
you'll need other information anyway...

Regardless, it would suggest we should be ignoring the file (whether
create it or not).

Taking another example to suggest it is broken, what if I delete one of
the files and let the setup scripts recreate it after using a different
TEMPLATECONF setting? Which value should templateconf.cfg have?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1650): 
https://lists.openembedded.org/g/openembedded-architecture/message/1650
Mute This Topic: https://lists.openembedded.org/mt/93968540/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Template handling in OE-Core

[Richard Purdie]

I'm starting a new thread/discussion on this with a new perspective.

I did a little research myself on how these files are being used. My
conclusion is that templateconf.cfg isn't really used by much at all
other than to decide whether to rerun setup pieces.

I'm going to make the assumption that once a build directory is setup,
the user does not want the env setup scripts messing with it again. If
they did, they'd start a new build directory.

As such:

a) showing the user a warning about the template locations changing is
probably not necessary
b) reading the value back in from templateconf.cfg isn't then necessary
as either the files don't exist and we set then up or we don't care

Therefore, my thinking is that we should simply remove
templateconf.cfg?

What am I missing?

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1648): 
https://lists.openembedded.org/g/openembedded-architecture/message/1648
Mute This Topic: https://lists.openembedded.org/mt/93968540/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [yocto-announce] [ANNOUNCEMENT] Milestone 3 for Yocto Project 4.1 (yocto-4.1_M3) Now Available

[Richard Purdie]

On Sun, 2022-09-25 at 10:53 +0200, Konrad Weihmann wrote:
> Hi all,
> 
> On 24.09.22 04:02, Lee Chee Yang wrote:
> > Hi
> > 
> > We are pleased to announce the third milestone release for Yocto Project 
> > 4.1 (yocto-4.1_M3) is now available for download.
> > 
> > Download:
> > 
> > http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-4.1_M3 
> > 
> > 
> > bitbake: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551
> > 
> > meta-agl: e4ea839db9c26e78175d61388c5408a79f6927dc
> > 
> > meta-arm: 261263a6701ab3530ff997643d08da50ac20
> > 
> > meta-aws: a16f35a73bff26d9506f519451dc75034211d61b
> > 
> > meta-gplv2: 43bf0e8d5985945d19d01f94bfbbda420c4435f3
> 
> I thought the agreement was to stop promoting meta-gplv2?
> Still I find it listed here - it somehow implies that this is still 
> officially support, which everyone agreed it isn't
> 
> Can this be removed from the announcement template for the next 
> releases? - just to avoid any future misunderstandings

Someone needs to do the work involved and I guess that hasn't happened.
It isn't entirely simple as this does still exist in the other
releases.

Could you open a bug please so we don't forget this.

I still don't know who will do it but that way we at least remember we
need to.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1646): 
https://lists.openembedded.org/g/openembedded-architecture/message/1646
Mute This Topic: https://lists.openembedded.org/mt/93903796/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Adding more information to the SBOM

[Richard Purdie]

On Mon, 2022-09-19 at 16:20 +0200, Carlo Piana wrote:
> thank you for a well detailed and sensible answer. I certainly cannot
> speak on technical issues, although I can understand there are
> activities which could seriously impact the overall process and need
> to be minimized.
> 
> 
> > On Fri, 2022-09-16 at 17:18 +0200, Alberto Pianon wrote:
> > > Il 2022-09-15 14:16 Richard Purdie wrote:
> > > > 
> > > > For the source issues above it basically it comes down to how much
> > > > "pain" we want to push onto all users for the sake of adding in this
> > > > data. Unfortunately it is data which many won't need or use and
> > > > different legal departments do have different requirements.
> > > 
> > > We didn't paint the overall picture sufficiently well, therefore our
> > > requirements may come across as coming from a particularly pedantic
> > > legal department; my fault :)
> > > 
> > > Oniro is not "yet another commercial Yocto project", we are not a legal
> > > department (even if we are experienced FLOSS lawyers and auditors, the
> > > most prominent of whom is Carlo Piana -- cc'ed -- former general counsel
> > > of FSFE and member of OSI Board).
> > > 
> > > Our rather ambitious goal is not limited to Oniro, and consists in doing
> > > compliance in the open source way and both setting an example and
> > > providing guidance and material for others to benefit from our effort.
> > > Our work will therefore be shared (and possibly improved by others) not
> > > only with Oniro-based projects but also with any Yocto project. Among
> > > other things, the most relevant bit of work that we want to share is
> > > **fully reviewed license information** and other legal metadata about a
> > > whole bunch of open source components commonly used in Yocto projects.
> > 
> > I certainly love the goal. I presume you're going to share your review
> > criteria somehow? There must be some further set of steps,
> > documentation and results beyond what we're discussing here?
> 
> Our mandate (and our own attitude) is precisely to make everything as
> public as possible.
> 
> We have published already about it
> https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain/toolchain/docs/-/tree/main/audit_workflow
> 
> The entire review process is made using GitLab's issues and will be
> made public.

I need to read into the details but that looks like a great start and
I'm happy to see the process being documented!

Thanks for the link, I'll try and have a read.

> We have only one reservation concerning sensitive material
> in case we found something legally problematic (to comply with
> attorney/client privilege) or security-wise critic (in which case we
> adopt a responsible disclosure principle and embargo some details).

That makes sense, it is a tricky balancing act at times.

> > I think the challenge will be whether you can publish that review with
> > sufficient "proof" that other legal departments can leverage it. I
> > wouldn't underestimate how different the requirements and process can
> > be between different people/teams/companies.
> 
> Speaking from a legal perspective, this is precisely the point. It is
> true that we want to create a curated database of decisions, which as
> any human enterprise is prone to errors and correction and therefore
> we cannot have the last word. However, IF we can at least point to a
> unique artifact and give its exact hash, there will be no need to
> trust us, that would be open to inspection, because everybody else
> could look at the same source we have identified and make sure we
> have extracted all the information.

I do love the idea and I think it is quite possible. I do think this
does lead to one of the key details we need to think about though.

>From a legal perspective I'd imagine you like dealing with a set of
files that make up the source of some piece of software. I'm not going
to use the word "package" since I think the term is overloaded and
confusing. That set of files can all be identified by checksums. This
pushes us towards wanting checksums of every file.

Stepping over to the build world, we have bitbake's fetcher and it
actually requires something similar - any given "input" must be
uniquely identifiable from the SRC_URI and possibly a set of SRCREVs.

Why? We firstly need to embed this information into the task signature.
If it changes, we know we need to rerun the fetch and re-obtain the
data. We work on inputs to generate this hash, not outputs and we
require all fetcher modules to be able to identify sources like this.

In th

Re: [Openembedded-architecture] Adding more information to the SBOM

[Richard Purdie]

On Fri, 2022-09-16 at 17:18 +0200, Alberto Pianon wrote:
> Il 2022-09-15 14:16 Richard Purdie wrote:
> > 
> > For the source issues above it basically it comes down to how much
> > "pain" we want to push onto all users for the sake of adding in this
> > data. Unfortunately it is data which many won't need or use and
> > different legal departments do have different requirements.
> 
> We didn't paint the overall picture sufficiently well, therefore our
> requirements may come across as coming from a particularly pedantic
> legal department; my fault :)
> 
> Oniro is not "yet another commercial Yocto project", we are not a legal
> department (even if we are experienced FLOSS lawyers and auditors, the
> most prominent of whom is Carlo Piana -- cc'ed -- former general counsel
> of FSFE and member of OSI Board).
> 
> Our rather ambitious goal is not limited to Oniro, and consists in doing
> compliance in the open source way and both setting an example and
> providing guidance and material for others to benefit from our effort.
> Our work will therefore be shared (and possibly improved by others) not
> only with Oniro-based projects but also with any Yocto project. Among
> other things, the most relevant bit of work that we want to share is
> **fully reviewed license information** and other legal metadata about a
> whole bunch of open source components commonly used in Yocto projects.

I certainly love the goal. I presume you're going to share your review
criteria somehow? There must be some further set of steps,
documentation and results beyond what we're discussing here?

I think the challenge will be whether you can publish that review with
sufficient "proof" that other legal departments can leverage it. I
wouldn't underestimate how different the requirements and process can
be between different people/teams/companies.

> To do that in a **scalable and fully automated way**, we need that Yocto
> collects some information that is currently disposed of (or simply not
> collected) at build time.
> 
> Oniro Project Leader, Davide Ricci - cc'ed - strongly encouraged us to
> seek for feedback from you in order to find out the best way to do it.
> 
> Maybe organizing a call would be more convenient than discussing
> background and requirements here, if you (and others) are available.

I don't mind having a call but the discussion in this current form may
have an important element we shouldn't overlook, which is that it isn't
just me you need to convince on some of this.

If, for example, we should radically change the unpack/patch process,
we need to have a good explanation for why people need to take that
build time/space/resource hit. If we conclude that on a call, the case
to the wider community would still have to be made.

> > Experience
> > with archiver.bbclass shows that multiple codepaths doing these things
> > is a nightmare to keep working, particularly for corner cases which do
> > interesting things with the code (externalsrc, gcc shared workdir, the
> > kernel and more).
> > 
> > I had a look at this and was a bit puzzled by some of it.
> > 
> > I can see the issues you'd have if you want to separate the unpatched
> > source from the patches and know which files had patches applied as
> > that is hard to track. There would be significiant overhead in trying
> > to process and store that information in the unpack/patch steps and the
> > archiver class does some of that already. It is messy, hard and doens't
> > perform well. I'm reluctant to force everyone to do it as a result but
> > that can also result in multiple code paths and when you have that, the
> > result is that one breaks :(.
> > 
> > I also can see the issue with multiple sources in SRC_URI, although you
> > should be able to map those back if you assume subtrees are "owned" by
> > given SRC_URI entries. I suspect there may be a SPDX format limit in
> > documenting that piece?
> 
> I'm replying in reverse order:
> 
> - there is a SPDX format limit, but it is by design: a SPDX package
>entity is a single sw distribution unit, so it may have only one
>downloadLocation; if you have more than one downloadLocation, you must
>have more than one SPDX package, according to SPDX specs;

I think we may need to talk to the SPDX people about that as I'm not
convinced it always holds that you can divide software into such units.
Certainly you can construct a situation where there are two
repositories, each containing a source file which are only ever linked
together as one binary.

> - I understand that my solution is a bit hacky; but IMHO any other
>*post-mortem* solution would be far more hacky; the real solution
>would be co

Re: [Openembedded-architecture] Adding more information to the SBOM

[Richard Purdie]

On Wed, 2022-09-14 at 16:16 +0200, Marta Rybczynska wrote:
> The sources with a long README are available at
> https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain/toolchain/tinfoilhat/-/tree/srctracker/srctracker
> 
> What do you think of this work? Would it be of interest to integrate
> into YP at some point? Shall we discuss this?

I had a look at this and was a bit puzzled by some of it.

I can see the issues you'd have if you want to separate the unpatched
source from the patches and know which files had patches applied as
that is hard to track. There would be significiant overhead in trying
to process and store that information in the unpack/patch steps and the
archiver class does some of that already. It is messy, hard and doens't
perform well. I'm reluctant to force everyone to do it as a result but
that can also result in multiple code paths and when you have that, the
result is that one breaks :(.

I also can see the issue with multiple sources in SRC_URI, although you
should be able to map those back if you assume subtrees are "owned" by
given SRC_URI entries. I suspect there may be a SPDX format limit in
documenting that piece?

Where I became puzzled is where you say "Information about debug
sources for each actual binary file is then taken from
tmp/pkgdata//extended/*.json.zstd". This is the data we added
and use for the spdx class so you shouldn't need to reinvent that
piece. It should be the exact same data the spdx class uses.

I was also puzzled about the difference between rpm and the other
package backends. The exact same files are packaged by all the package
backends so the checksums from do_package should be fine.


For the source issues above it basically it comes down to how much
"pain" we want to push onto all users for the sake of adding in this
data. Unfortunately it is data which many won't need or use and
different legal departments do have different requirements. Experience
with archiver.bbclass shows that multiple codepaths doing these things
is a nightmare to keep working, particularly for corner cases which do
interesting things with the code (externalsrc, gcc shared workdir, the
kernel and more).

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1639): 
https://lists.openembedded.org/g/openembedded-architecture/message/1639
Mute This Topic: https://lists.openembedded.org/mt/93678489/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] toolchain selection in core

[Richard Purdie]

I was asked about getting official support for the TOOLCHAIN variable
in core. This is obviously late in the release cycle and after a bit of
thought, the answer is no. I wanted to document why since I was forced
into thinking this through.

As background, meta-clang uses TOOLCHAIN internally to select between
toolchains. In itself that is good but the variable names used in that
layer don't scale well to core where they could be confused with other
things. For RUNTIME, I think Khem and I agree TCCXX_RUNTIME is better.

Specifically with TOOLCHAIN, the issue is that it doesn't scale.

Imagine we have random-si-vendor-cc (rsvcc), clang and gcc.

I can set TOOLCHAIN = "gcc" since a recipe might not build with rsvcc
but that stops it building with clang too.

What we really need is COMPATIBLE_TOOLCHAIN or INCOMPATIBLE_TOOLCHAIN
markup for recipes and a way to then select a toolchain based on some
priority list (e.g. clang first, gcc as a fallback).

I appreciate the desire to add the markup but we need to do this in a
way which scales.

As such I'm not rushing into making TOOLCHAIN a "standard" as I don't
think it will work beyond clang. I notice we do have a couple of cases
that have crept into core which should really be removed.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1633): 
https://lists.openembedded.org/g/openembedded-architecture/message/1633
Mute This Topic: https://lists.openembedded.org/mt/93576595/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [Automated-testing] RFC Linter in meta-openembedded

[Richard Purdie]

Hi Marius,

On Mon, 2022-09-05 at 13:25 +0200, Marius Kriegerowski wrote:
> Picking up on this thread now, almost half a year later :) Sorry for
> the late reply but turned out (a little surprising) that I’m going to
> have a daughter soon which shifted my priorities a little…

Understandable, congratulations! :)

> So, I looked into patchtest a few months ago and found a couple of
> issues where I considered a rewrite. So, I gave it a shot last
> weekend but instead of python I used rust. You can find the demo
> here: https://github.com/HerrMuellerluedenscheid/patchtest-rs

This doesn't seem like a good direction to me. I have nothing against
rust, I've spent the past couple of months trying to sort out rust
support in OE-Core. The problem is most of our tools (bitbake,
autobuilder, buildbot and so on) are python based and most of our
developers know python. Most don't know rust (myself included). Making
a core tool hard to understand by our core developer base doesn't seem
like a wise move.

> The functionality is currently limited to loading a patch from the
> commandline, checking that the summary is in place and applying the
> patch to a repository given by a url. The repo will be cloned on the
> fly.
> 
> The next step would be an smtp client that checks for incoming
> messages every N seconds and runs patchtest against patches and sends
> back a short report of what worked and what didn’t.
> 
> I just wanted to check that revitalising patchtest is still an open
> issue and would like to ask for quick feedback.

We definitely do want to revitialise it. One change is that we do now
have publicinbox available for OE-Core and bitbake:

https://lore.kernel.org/openembedded-core/
https://lore.kernel.org/bitbake-devel/

so rather than fighting SMTP, we should be able to read patches from
there via the git representation.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1631): 
https://lists.openembedded.org/g/openembedded-architecture/message/1631
Mute This Topic: https://lists.openembedded.org/mt/88763351/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] OE-Core copyright notices

[Richard Purdie]

On Tue, 2022-05-17 at 10:13 +0100, Richard Purdie via
lists.openembedded.org wrote:
> I've been worrying about license and copyright information in our
> codebase for a while. We're slowly getting there with license info and
> scripts have good information now. We still need to work out license
> info for recipe metadata and patches but one step at a time.
> 
> For copyright, I think we need a proposal, so here goes!
> 
> I think we should add something, the question is what/where. There was
> an LF post about this:
> 
> https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
> 
> and I'd like to propose we follow the recommendation there and for OE-
> Core, go with:
> 
> "Copyright OpenEmbedded Contributors"
> 
> where we don't already have a copyright statement. Similarly, bitbake
> would be "Copyright BitBake Contributors" but I think much of bitbake
> does have copyright statements so is already in a better state.
> 
> If individuals want to mention themselves as authors or named copyright
> holders, or it is a company requirement, that is fine too as things are
> today, this is mainly to cover the case where there is no current
> copyright.
> 
> Where we have year information, we may want to phase that out as it
> doesn't buy us anything.
> 
> I'm mentioning this as we have OEDVM on Friday and this may (or may
> not) be something we need to discuss. I do want to ensure people not at
> the meeting also see this and have an opportunity to discuss. 

I've gone ahead and sent some patches for this for OE-Core and BitBake.
There is more work to do so if anyone has a few minutes and wants to
clean up a few more file headers, further improvements would be very
welcome.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1626): 
https://lists.openembedded.org/g/openembedded-architecture/message/1626
Mute This Topic: https://lists.openembedded.org/mt/92944780/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Removing meta-gplv2 from release notes?

[Richard Purdie]

On Wed, 2022-08-03 at 10:33 +0200, Michael Opdenacker via
lists.openembedded.org wrote:
> Hi Lee
> 
> Thanks for the milestone release!
> 
> Have you seen the discussion on the Openembedded-architecture mailing 
> list? It seems there's a consensus to drop meta-gplv2 from master, and 
> also from the release notes.
> See https://lists.openembedded.org/g/openembedded-architecture/message/1606
> 
> Would you agree to do this for 4.1_M3 ?

Some of this is autogenerated from the autobuilder and we can't simply
remove it as it needs to stay for the older releases. We'll likely need
to tweak the code to better handle this. Patches would be very welcome.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1623): 
https://lists.openembedded.org/g/openembedded-architecture/message/1623
Mute This Topic: https://lists.openembedded.org/mt/92788422/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] should oe-core issue a warning when it reaches EOL?

[Richard Purdie]

On Tue, 2022-08-02 at 10:08 +0200, Alexander Kanavin wrote:
> I tend to agree that warnings popping out of nowhere when the stable
> branch is close to (or beyond) EOL is 'too late' - if you got yourself
> into that situation, you're likely to have a broader maintainability
> problem.
> 
> However, I do not agree with the "how can we justify doing nothing at
> all" rhetoric I see from people here: I do want to raise awareness of
> the lifecycles, and as early as possible for anyone using the project.
> So if you say 'this is not going to work', please try to be
> constructive and offer alternatives.

Keep in mind that the LTS are still comparatively new in the timescales
people work within and that we have changed and defined a new policy
comparatively recently. A large portion of this is a social issue as
Philip mentioned and people do pick up and learn that over time. This
isn't the only aspect of the project people need to learn.

I'm not convinced we have to "solve" this problem with technical means.
We do have information out there explaining things and that information
is improving over time (e.g. the EOL notices on the older release
manuals) which was surprisingly difficult and only done embarrassingly
recently. We are also trying our best to have good new features which
offer people a reason to upgrade. This isn't doing nothing.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1618): 
https://lists.openembedded.org/g/openembedded-architecture/message/1618
Mute This Topic: https://lists.openembedded.org/mt/92611044/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] should oe-core issue a warning when it reaches EOL?

[Richard Purdie]

On Mon, 2022-08-01 at 19:30 +0200, Alexander Kanavin wrote:
> On Mon, 1 Aug 2022 at 13:22, Ross Burton  wrote:
> > Would this be something done in oe-core, or something that distros
> > (such as Poky) can opt in to?  We don’t want to start warning when
> > oe-core is EOL if the user is using a commercial Yocto release
> > which still has support for many years.
> > 
> > Would this be done with a variable (be it a EOL date, or a marker)
> > in the git repository itself, or something that if fetched
> > periodically?  A variable in the git repository would need to be
> > kept up to date, and there’s potentially a significant delay
> > between a change landing in oe-core and reaching a user (oe-core
> > release -> OSV release -> BSP release) which could result in
> > inappropriate warnings.   The same information being online and
> > fetched on builds would solve that problem but instead add phone-
> > home issues and the usual network complications (caching, no-
> > network use cases, etc).
> > 
> > Whilst we can see that there are definite value propositions in
> > alerting users that a release is approaching EOL, there are
> > considerable complications to be thought though.
> 
> I think this could work the following way:
> 
> - EOL date defined in a variable in meta/conf/distro/end-of-life.inc,
> for stable branches only, and with ?=, so it can be easily overriden
> by commercial distros or users who know what they are doing.
> - if the EOL date is less than (say) 1 month away, bitbake prints a
> note. If the EOL date is in the past, it becomes a warning.
> - the note or the warning uses guarded language ('may', 'possibly',
> etc.) and points to the Releases wiki page and LTS policy pages
> 
> Any particular problems with this approach?

Taking something like Dunfell where it was originally for two years,
then was extended you could end up with the date not agreeing with the
reality in some checkouts if they weren't updated. There is an issue if
a user isn't updating but even so it isn't great for the project.

There is also a concern that OSVs may forget to override it (or just
not realise they needed to) and the "fires" that would start when the
behaviour of a product changed like this showing warnings to the user
would be significant.

I do feel that whilst well intentioned, there are too many ways it can
go wrong in ways that will cause bad feelings towards the project.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1610): 
https://lists.openembedded.org/g/openembedded-architecture/message/1610
Mute This Topic: https://lists.openembedded.org/mt/92611044/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Let's drop x86-x32 (was Re: [OE-core] [PATCH 05/51] rpm: update 4.17.0 -> 4.17.1)

[Richard Purdie]

On Wed, 2022-07-20 at 10:49 +0200, Alexander Kanavin wrote:
> Note: this update fails on x32 with
> 
> > configure: error: unrecognized GNU build triplet linux-gnux32
> 
> This time I want to put my foot down and suggest that we just drop the
> whole x32 variant from the autobuilder (I had already sent a patch for
> this previously). In all likelihood it was never used to ship anything
> to customers, and was only devised to look better in benchmarks
> against competition.

That amounts to dropping x32 support because as soon as we remove these
tests, it will bitrot.

There is still some value in the project being able to support
different architectures and different type sizes so I do still lean
towards keeping this alive at a minimal level.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1585): 
https://lists.openembedded.org/g/openembedded-architecture/message/1585
Mute This Topic: https://lists.openembedded.org/mt/92500615/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] meta-gplv2 - plan to drop support in master

[Richard Purdie]

On Wed, 2022-07-06 at 18:50 +0200, Alexander Kanavin wrote:
> On Wed, 6 Jul 2022 at 18:46, Khem Raj  wrote:
> > I agree that we need to act upon this situation and once we have
> > relevant reference images in core then need for this should be less
> > with time.
> 
> But we do:
> https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/incompatible_lic.py#n128
> 
> Anything in particular missing there?

In particular, it doesn't work with ptests or anything needing bash :(

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1571): 
https://lists.openembedded.org/g/openembedded-architecture/message/1571
Mute This Topic: https://lists.openembedded.org/mt/92209067/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] meta-gplv2 - plan to drop support in master

[Richard Purdie]

meta-gplv2 makes the TSCs very nervous (certainly YP and I believe OE
TSC people too although it hasn't had an official discussion there). It
has a load of old "pre GPLv3 license" versions of software that haven't
seen security fixes for years. We've limped it along to our last LTS.
It has no active maintainer.

We currently test it on the autobuilder, hence it becomes part of the
YP release process. Things break in it occasionally as development
moves forward and it has often been easier for me to put bandaids on
than doing something about it since I can't move anything forward
unless it passes testing.

We now find it has reproducibility issues which I don't intend to spend
my time fixing so I think it is time to:

a) Drop master from the autobuilder and from our release process.

b) merge a commit deleting layer.conf on master with a commit message
explaining why the layer is a really bad idea and why we're no longer
supporting it.

In parallel Khem has been working on alternative GPLv3 license
avoidance strategies so we will suggest people can use them as they
become available. We need to encourage people to help develop those in
my view which this move would help support.

I appreciate some people do use it but they really shouldn't.
Personally, I don't think we should continue supporting meta-gplv2 even
if someone volunteers as I think we've passed the point where this
makes sense, we need to find a different way.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1566): 
https://lists.openembedded.org/g/openembedded-architecture/message/1566
Mute This Topic: https://lists.openembedded.org/mt/92209067/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Yocto Project SState Mirror discussion

[Richard Purdie]

On Tue, 2022-06-14 at 07:08 -1000, Steve Sakoman wrote:
> At today's Yocto Project Technical Team Meeting Armin raised a
> question about the proper way to set up local.conf to use the Yocto
> Project state mirror for dunfell. This email is intended to bring
> those not attending up to speed and trigger further discussion.
> 
> Armin pointed out that the sstate mirror setup section in
> local.conf.sample seemed to be in error, and a subsequent web search
> didn't yield any useful information on how to proceed. So at a minimum
> we have a documentation issue!
> 
> Further discussion ensued, and from memory the following points were made:
> 
> 1. A number of issues have been addressed in the master and kirkstone
> branches which significantly improve the usability of the published
> project sstate mirror.  These improvements are not present in dunfell.
> 
> 2. There are also improvements in progress for the infrastructure
> exporting the sstate mirror.
> 
> 3. On the autobuilder there are separate development sstate caches for
> master, dunfell, and kirkstone.  While not strictly necessary, this
> was done to isolate potential sstate corruption to single branches.
> 
> 4. The published sstate mirror is generated at release time, i.e. the
> sstate cache from development builds is not used for the release
> build.
> 
> There was also some discussion about next steps, and Richard suggested
> this email to trigger a wider discussion.  Potential steps:
> 
> 1. Improve/fix the sstate cache section of local.conf.sample
> 
> 2. Improve documentation of sstate cache setup
> 
> 3. Consider a single sstate cache for all branches on the autobuilder
> 
> 4. Consider a single sstate mirror for all branches
> 
> Hopefully others who attended the meeting can jump in if I missed or
> misrepresented anything!

I've been thinking about this and also talking with Michael Halstead
about what makes sense from a mirror/server perspective. Going forward
we going to rename sstate.yoctoproject.org/dev/ to
sstate.yoctoproject.org/all/ and place the sstate output from all
builds into there.

We'd drop the release specific sstate directories and the "all"
directory would age artefacts that hadn't been used in a build for X
time such that older ones are deleted.

This makes things much easier for a user to configure as there is one
directory. It makes things easier to mirror too, even if the mirror
does become a bit more changable.

This does mean that over time, release minimal eSDKs would stop working
as their sstate would age out and disappear. I'm not sure I'm going to
worry too much about that.

We'll leave a compatibility link in place for /dev/. We'll need to
update the autobuilder configurations to match these changes and drop
the sstate part of the release process.

The docs changes Armin/Steve mention should then be straightforward.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1561): 
https://lists.openembedded.org/g/openembedded-architecture/message/1561
Mute This Topic: https://lists.openembedded.org/mt/91753971/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] The unpack dilemma

[Richard Purdie]

On Tue, 2022-05-31 at 07:11 -0700, Khem Raj wrote:
> 
> 
> On Mon, May 30, 2022 at 3:51 AM Richard Purdie
>  wrote:
> > I've gone around in circles on this for a long time. The background
> > isn't obvious at first glance.
> > 
> > "S" represents where our source is. There is code like do_unpack
> > which
> > wants to clean S before rerunning the task. It can't just clean
> > everything in WORKDIR as there is T, recipe-sysroot* and more that
> > need
> > to be left alone (e.g. recipe-sysroot-native may have
> > do_fetch/do_unpack dependencies).
> > 
> > S is usually defined as ${BPN}-${PV} via ${BP}.
> > 
> 
> 
> Maybe disallow workdir to be used as S it’s a rare case anyway. Add a
> QA check and that
> Should help recipes to migrate away 

It isn't that rare, there are 42 in OE-Core alone.

I do think we should probably migrate away from those though.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1557): 
https://lists.openembedded.org/g/openembedded-architecture/message/1557
Mute This Topic: https://lists.openembedded.org/mt/91428199/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] The unpack dilemma

[Richard Purdie]

On Mon, 2022-05-30 at 13:26 +0200, Alexander Kanavin wrote:
> Is it possible to transition from dumping source artifacts and patches
> directly into $WORKDIR to having a clean set of sub-directories for
> SRC_URI items? Not having to set S separately for git recipes would be
> nice too.

The question is what level of changes are we prepared to accept to
recipes? If we don't want to set S for git, where would we accept
setting things? Or do we want to make it all "magic"?

The one thing in the favour of the current approach is that it is
simple. We could put all kinds of logic in there to make it all more
magic but I'm reluctant to do so. The git recipe S thing has bothered
me for a long time but the question is what it should do instead?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1555): 
https://lists.openembedded.org/g/openembedded-architecture/message/1555
Mute This Topic: https://lists.openembedded.org/mt/91428199/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] The unpack dilemma

[Richard Purdie]

I've gone around in circles on this for a long time. The background
isn't obvious at first glance.

"S" represents where our source is. There is code like do_unpack which
wants to clean S before rerunning the task. It can't just clean
everything in WORKDIR as there is T, recipe-sysroot* and more that need
to be left alone (e.g. recipe-sysroot-native may have
do_fetch/do_unpack dependencies).

S is usually defined as ${BPN}-${PV} via ${BP}.

Many SRC_URIs are tarballs and tarballs usually extract into a
directory, which often happens to be BPN-PV. The tarball itself
therefore creates the ${S} directory.

For git, the git fetcher extracts into git/ so git recipes end up
setting:

S = "${WORKDIR}/git"

This was done historically since using ${PV} for the source dir for a
git recipe was likely going to complicate the paths needlessly.

The problem comes with recipes like keymaps which only have local files
referenced in SRC_URI. These do:

S = "${WORKDIR}"

so if do_unpack cleans S, it cleans WORKDIR which is very bad. We have
horrible code to guard against that. This is the first thing it would
be nice to fix.

We also have a second problem which is that if do_fetch/do_unpack
rerun, a file installed by a SRC_URI in the first run but removed in
the second, will remain in ${WORKDIR}. This means you can change a
recipe and the rebuild might not be deterministic. Again, this is
rather horrible.

It is very hard for the code to know which files it should be removing
when it reruns do_unpack. This is the second issue I'd ideally like to
fix.

Ideas:

The first issue can be improved with code like:

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index bdb3ac33c67..0da22b34e8a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -161,8 +161,10 @@ python base_do_fetch() {
 bb.fatal("Bitbake Fetcher Error: " + repr(e))
 }
 
+UNPACK_DIR = "${WORKDIR}"
+
 addtask unpack after do_fetch
-do_unpack[dirs] = "${WORKDIR}"
+do_unpack[dirs] = "${UNPACK_DIR}"
 
 do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != 
os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}"
 
@@ -173,7 +175,7 @@ python base_do_unpack() {
 
 try:
 fetcher = bb.fetch2.Fetch(src_uri, d)
-fetcher.unpack(d.getVar('WORKDIR'))
+fetcher.unpack(d.getVar('UNPACK_DIR'))
 except bb.fetch2.BBFetchException as e:
 bb.fatal("Bitbake Fetcher Error: " + repr(e))
 }


i.e. we make UNPACK_DIR configurable, then a recipe like keymaps can
do:

diff --git a/meta/recipes-bsp/keymaps/keymaps_1.0.bb 
b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
index 84d09cb965d..2927e7f1392 100644
--- a/meta/recipes-bsp/keymaps/keymaps_1.0.bb
+++ b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
@@ -25,14 +25,14 @@ SRC_URI = "file://keymap.sh"
 INITSCRIPT_NAME = "keymap.sh"
 INITSCRIPT_PARAMS = "start 01 S ."
 
-S = "${WORKDIR}"
+UNPACK_DIR = "${S}"
 
 do_install () {
 # Only install the script if 'sysvinit' is in DISTRO_FEATURES
 # THe ulitity this script provides could be achieved by 
systemd-vconsole-setup.service
 if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; 
then
install -d ${D}${sysconfdir}/init.d/
-   install -m 0755 ${WORKDIR}/keymap.sh ${D}${sysconfdir}/init.d/
+   install -m 0755 ${S}/keymap.sh ${D}${sysconfdir}/init.d/
 fi
 }
 

which I think then has the recipe make more logical sense.

For the second issue, the best idea I could come up with was to make
do_unpack extract to a subdirectory or WORKDIR, then symlink the files
into place in WORKDIR. do_unpack can then remove any symlink it finds
in WORKDIR when it reruns.

Whether either or both of these improvements are worthwhile and whether
there is the communuity demand and support to implement them and
migrate metadata to match, I don't know but I wanted to put the ideas
down here.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1553): 
https://lists.openembedded.org/g/openembedded-architecture/message/1553
Mute This Topic: https://lists.openembedded.org/mt/91428199/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] OE-Core copyright notices

[Richard Purdie]

I've been worrying about license and copyright information in our
codebase for a while. We're slowly getting there with license info and
scripts have good information now. We still need to work out license
info for recipe metadata and patches but one step at a time.

For copyright, I think we need a proposal, so here goes!

I think we should add something, the question is what/where. There was
an LF post about this:

https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/

and I'd like to propose we follow the recommendation there and for OE-
Core, go with:

"Copyright OpenEmbedded Contributors"

where we don't already have a copyright statement. Similarly, bitbake
would be "Copyright BitBake Contributors" but I think much of bitbake
does have copyright statements so is already in a better state.

If individuals want to mention themselves as authors or named copyright
holders, or it is a company requirement, that is fine too as things are
today, this is mainly to cover the case where there is no current
copyright.

Where we have year information, we may want to phase that out as it
doesn't buy us anything.

I'm mentioning this as we have OEDVM on Friday and this may (or may
not) be something we need to discuss. I do want to ensure people not at
the meeting also see this and have an opportunity to discuss. 

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1546): 
https://lists.openembedded.org/g/openembedded-architecture/message/1546
Mute This Topic: https://lists.openembedded.org/mt/91159593/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Openembedded-architecture] Thoughts on the eSDK

[Richard Purdie]

I've been asked about my thoughts on the eSDK. I'm going to try and
write down some of my ideas and thinking about it. I don't have a fully
thought out plan to pull off the shelf but can at least share the
ideas.

In summary, I think many of the concepts in the eSDK are good but the
implementation isn't great, it is effectively a beta where we needed to
go back and learn from the implementation to improve things.

To understand why, keep in mind it was put together as an experiment.
The idea was to have an environment which rather than being an SDK or a
full build system, could be configured to be different steps in a
spectrum between those two positions. The goals included:

* using prebuild binary objects rather than building from source
* a prebuilt toolchain
* being able to generate customised images quickly
* could be updated to new versions of the underlying environment
* making commands like devtool available to the "SDK" user
* hiding the build system complexity from the "SDK" user where possible
* allowing usage as per the current "standard SDK".

The design decisions this resulted in were:

* being able to lock the sstate hashes to a known version
* using sstate as the binary object mechanism (so not package 
  management based)
* needing to capture a locked down version of the metadata (i.e. the 
  layers in use)

The eSDK as implemented did do many of the things it was designed to
do. As with most engineering, there were challenges that were
encountered, things that didn't work as expected and also unanticipated
issues. It really needed someone to go back over it armed with that
knowledge for a v2 and that is still where we stand today as
development stopped.

The things which bother me about our current situation:

* there is no way to unlock the sstate hash of a specific component
  like a bblock/bbunlock command.
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=13425
* we need to work out the layer setup and capture it into the eSDK. If 
  we had a standard mechanism for layer setup and config this would be 
  much easier. The current approach is very sensitive to different 
  layouts and lacks any tests for anything that isn't poky
* where sstate checksum mismatches occur, they are fiendishly 
  difficult to debug (a wider problem with bitbake but they affect 
  eSDK particularly badly)
* no mechanism to turn an existing build into an eSDK
* no mechanism to turn an eSDK install into a full build
* can't build eSDK within eSDK

You can see the list of open bugs against eSDK:

https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=esdk_id=649733

In particular there are the following issues:

* it only works targeting a system of the same as it is being built 
  on (an aarch64 eSDK needs to be built on an aarch64 system)
* multilib may or may not work (and has no tests)
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=10693
* devtool commands may need to be disabled depending on context
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=8789
* sdk-update has issues, e.g. 
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=10798
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=14462
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=14462
* several things don't build within the eSDK, e.g. ovmf, multilib, 
  lttng, gstreamer-plugins-good


I've put off fixing some elements as I've hoped that if we fixed the
layer setup problem, it would then help elements of the eSDK. I think
we probably do need to go ahead and try and fix other elements without
that piece for now though (layer setup is a different problem in it's
own right).

I'm sharing this in the hopes it gets people thinking a little more and
can at least start the discussion.

Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1544): 
https://lists.openembedded.org/g/openembedded-architecture/message/1544
Mute This Topic: https://lists.openembedded.org/mt/90990557/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Dependent task hashes in depsig.*

[Richard Purdie]

On Wed, 2022-04-27 at 08:26 -0500, Joshua Watt wrote:
> On Wed, Apr 27, 2022 at 6:04 AM Richard Purdie
>  wrote:
> > 
> > On Wed, 2022-04-27 at 12:39 +0200, Jacob Kroon wrote:
> > > On 4/27/22 12:12, Richard Purdie wrote:
> > > > On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
> > > > > Hi Richard and Joshua,
> > > > > 
> > > > > When using hash equivalency, since commits
> > > > > 
> > > > > https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
> > > > > https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba
> > > > > 
> > > > > scrambling a header in one of the gcc patches causes all target 
> > > > > packages
> > > > > to rebuild.
> > > > 
> > > > That is probably unfortunately inevitable. If the output has changed 
> > > > (i.e. the
> > > > headers are different), it shouldn't be matching a previous build as we 
> > > > can't
> > > > know what has changed.
> > > > 
> > > 
> > > I don't think I was being clear enough. With "scrambling a header in a
> > > gcc patch" I mean that I change something in the section before the
> > > "---" line in a .patch file, like modifying the "Upstream-Status" line.
> > > To me, hash equivalence should be able to optimize that scenario, since
> > > the output from building gcc-cross is not changed.
> > 
> > That definitely wasn't clear! I now understand better what you mean and yes,
> > we're supposed to be optimising that scenario.
> > 
> > > 
> > > > >  This is because the depsig.do_populate_sysroot in "libgcc"
> > > > > changes:
> > > > > 
> > > > > > [jkroon@fedora work]$ diff -u 
> > > > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
> > > > > > --- 
> > > > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812
> > > > > >   2022-04-27 10:14:22.403251775 +0200
> > > > > > +++ 
> > > > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014
> > > > > >   2022-04-27 10:26:45.329365448 +0200
> > > > > > @@ -1,7 +1,7 @@
> > > > > >  OEOuthashBasic
> > > > > >  12
> > > > > >  glibc: 
> > > > > > 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
> > > > > > -libgcc: 
> > > > > > dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
> > > > > > +libgcc: 
> > > > > > 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
> > > > > >  linux-libc-headers: 
> > > > > > 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
> > > > > >  SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
> > > > > >  task=populate_sysroot
> > > > > 
> > > > > Is it the case that it is the dependent task hashes that are added
> > > > > above, and that the checksum of patches are included in the those task
> > > > > hashes ?
> > > > 
> > > > The dependent resolved hashes are used, as resolved by hashequiv which 
> > > > is a key
> > > > difference.
> > > > 
> > > 
> > > So it is the outhashes that are listed above?
> > 
> > No, they are hash equiv resolved hashes which would have a one to one 
> > mapping
> > with an outhash.
> > 
> > >  Then I don't understand
> > > the diff above. libgcc depends on itself ? But apparently no files in
> > > the sysroot changed, since the above is the only diff I get.
> > 
> > To be honest, I don't remember/understand offhand either. I'd need to go and
> > spend time trying to page in all the information.
> > 
> > We have too few people with the knowledge in these areas and I'm rapidly 
> > burning
> > out. I don't like this reply but I just don't have the time to dive into it 
> > and
> > debug it right now and I can't really give much more of a helpful 
> > comment/reply
> > without doing so.
> > 
> > I agree there is some issue here which does need investigation. At least do 
> > file
> > a bug so it doesn't get forgotten but we don't have many people taking on 
> > bugs
> > either. This one woul

Re: [Openembedded-architecture] Dependent task hashes in depsig.*

[Richard Purdie]

On Wed, 2022-04-27 at 12:39 +0200, Jacob Kroon wrote:
> On 4/27/22 12:12, Richard Purdie wrote:
> > On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
> > > Hi Richard and Joshua,
> > > 
> > > When using hash equivalency, since commits
> > > 
> > > https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
> > > https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba
> > > 
> > > scrambling a header in one of the gcc patches causes all target packages
> > > to rebuild.
> > 
> > That is probably unfortunately inevitable. If the output has changed (i.e. 
> > the
> > headers are different), it shouldn't be matching a previous build as we 
> > can't
> > know what has changed.
> > 
> 
> I don't think I was being clear enough. With "scrambling a header in a
> gcc patch" I mean that I change something in the section before the
> "---" line in a .patch file, like modifying the "Upstream-Status" line.
> To me, hash equivalence should be able to optimize that scenario, since
> the output from building gcc-cross is not changed.

That definitely wasn't clear! I now understand better what you mean and yes,
we're supposed to be optimising that scenario.

> 
> > >  This is because the depsig.do_populate_sysroot in "libgcc"
> > > changes:
> > > 
> > > > [jkroon@fedora work]$ diff -u 
> > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*   
> > > > 
> > > > 
> > > >
> > > > --- 
> > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812  
> > > > 2022-04-27 10:14:22.403251775 +0200 
> > > > 
> > > > 
> > > >  
> > > > +++ 
> > > > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014  
> > > > 2022-04-27 10:26:45.329365448 +0200 
> > > > 
> > > > 
> > > >  
> > > > @@ -1,7 +1,7 @@ 
> > > > 
> > > > 
> > > > 
> > > >  
> > > >  OEOuthashBasic 
> > > > 
> > > > 
> > > > 
> > > >  
> > > >  12 
> > > > 
> > > > 
> > > > 
> > > >  
> > > >  glibc: 
> > > > 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
> > > > 
> > > > 
> > > > 
> > > >  
> > > > -libgcc: 
> > > > dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
> > > > 
> > > > 
> > > > 
> > > > 
> > > > +libgcc: 
> >

Re: [Openembedded-architecture] Dependent task hashes in depsig.*

[Richard Purdie]

On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
> Hi Richard and Joshua,
> 
> When using hash equivalency, since commits
> 
> https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
> https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba
> 
> scrambling a header in one of the gcc patches causes all target packages
> to rebuild.

That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.

>  This is because the depsig.do_populate_sysroot in "libgcc"
> changes:
> 
> > [jkroon@fedora work]$ diff -u 
> > i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*   
> > 
> > 
> >
> > --- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812  
> > 2022-04-27 10:14:22.403251775 +0200 
> > 
> > 
> >  
> > +++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014  
> > 2022-04-27 10:26:45.329365448 +0200 
> > 
> > 
> >  
> > @@ -1,7 +1,7 @@ 
> > 
> > 
> > 
> >  
> >  OEOuthashBasic 
> > 
> > 
> > 
> >  
> >  12 
> > 
> > 
> > 
> >  
> >  glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
> > 
> > 
> > 
> >  
> > -libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226   
> > 
> > 
> > 
> >  
> > +libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5   
> > 
> > 
> > 
> >  
> >  linux-libc-headers: 
> > 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
> > 
> > 
> > 
> >  SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:   
> > 
> > 
> > 
> >  
> >  task=populate_sysroot
> 
> Is it the case that it is the dependent task hashes that are added
> above, and that the checksum of patches are included in the those task
> hashes ?

The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.

> In order to solve the original problem that those patches were fixing,
> would it not be possible to instead include the *outhashes* of the
> dependent recipes ?

Since the resolved hashes should map to a single outhash, I don't think it would
change anything?

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all 

Re: [Openembedded-architecture] [Automated-testing] RFC Linter in meta-openembedded

[Richard Purdie]

On Sat, 2022-04-16 at 17:15 +0200, Marius Kriegerowski wrote:
> Dear Richard, 
> 
> Thanks for the comprehensive answer! That helped me to get a better overview 
> of
> what is there and how to proceed.
> 
> I'll leave cosmetics for a later phase, stick to unittest and fix the
> integration so that we have a baseline from where to decide how to proceed.

FYI I did notice some docs here:

https://git.yoctoproject.org/patchtest/tree/usage.adoc

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1526): 
https://lists.openembedded.org/g/openembedded-architecture/message/1526
Mute This Topic: https://lists.openembedded.org/mt/88763351/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-