Re: [Openembedded-architecture] New mailing list for layer patches

[Peter Kjellerstedt]

> -Original Message-
> From: yo...@lists.yoctoproject.org  On
> Behalf Of Ross Burton
> Sent: den 28 mars 2024 15:37
> To: openembedded-architecture  architect...@lists.openembedded.org>; Yocto-mailing-list
> ; OE-core  c...@lists.openembedded.org>
> Subject: [yocto] New mailing list for layer patches
> 
> Hi,
> 
> At the moment if a layer doesn’t have enough volume to justify a dedicated
> mailing list for patches (see, for example, meta-
> a...@lists.yoctoproject.org ) then
> the convention is that the patches can be sent to
> yo...@lists.yoctoproject.org .
> 
> However, that’s also the list that we encourage users of Yocto to use for
> help, so the list has a mix of patches for various layers and people
> asking for help, which isn’t ideal: users who want help don’t want to be
> flooded with patches for layers they’re not using, layer maintainers may
> not want to see the requests for help, and people who want the status
> reports have to receive patches and support.
> 
> The Yocto TSC has been discussing this, and as a solution to this we’ve
> just created a new mailing list: yocto-patc...@lists.yoctoproject.org
> .  This list is specifically
> for Yocto-related projects which don’t have their own mailing list or
> other patch submission process.   We ask that all layers currently using
> yocto@ for patches move to yocto-patches@, so that yocto@ can be used
> purely for discussion and other non-patch related subjects.
> 
> Many thanks,
> Ross
> Yocto TSC member

To whoever is responsible for updating 
https://www.yoctoproject.org/community/mailing-lists/: the links for the 
two new lists both refer to yocto-announce rather than yocto-status and 
yocto-patches.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1985): 
https://lists.openembedded.org/g/openembedded-architecture/message/1985
Mute This Topic: https://lists.openembedded.org/mt/105197686/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Weak PACKAGECONFIG removal

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Joshua Watt
> Sent: den 22 december 2023 15:25
> To: Adrian Freihofer 
> Cc: openembedded-architecture@lists.openembedded.org
> Subject: Re: [Openembedded-architecture] Weak PACKAGECONFIG removal
> 
> On Fri, Dec 22, 2023 at 3:46 AM Adrian Freihofer
>  wrote:
> >
> > On Fri, 2023-12-22 at 10:25 +, Ross Burton via
> > lists.openembedded.org wrote:
> > > On 22 Dec 2023, at 04:59, Chen Qi via lists.openembedded.org
> > >  wrote:
> > > > > One of the major things that comes up frequently is that the
> > > > > PACKAGECONFIG for a recipe needs to be modified to remove some
> > > > > incompatible flag from the default options. This particular
> > > > > option is
> > > > > not very simple though, because most recipes set PACKAGECONFIG
> > > > > using
> > > > > the "deferred weak" assignment, as in `PACKAGECONFIG ??= "foo
> > > > > bar"`.
> > > > > Because of this, any operation that attempts to modify this
> > > > > variable
> > > > > will overwrite the default; commonly one might like to do
> > > > > something
> > > > > like `PACKAGECONFIG:remove = "foo"`, but since this happens
> > > > > before
> > > > > weak assignment, the result is an empty string instead of just
> > > > > the
> > > > > value "bar".
> > > >
> > > > Checking the master branch, and I get a different result.
> > >
> > > Indeed. With this in local.conf, or split across a recipe/bbappend:
> > >
> > > FOOBAR ??= "foo bar"
> > > FOOBAR:remove = “foo"
> > >
> > > $ bitbake-getvar  FOOBAR
> > > #
> > > # $FOOBAR [2 operations]
> > > #   set /yocto/ross/build/conf/local.conf:495
> > > # [_defaultval] "foo bar"
> > > #   :remove /yocto/ross/build/conf/local.conf:496
> > > # "foo"
> > > # pre-expansion value:
> > > #   "foo bar"
> > > FOOBAR=" bar”
> > >
> > > $ bitbake-getvar  FOOBAR -r psplash
> > > #
> > > # $FOOBAR [2 operations]
> > > #   set /home/ross/Yocto/poky/meta/recipes-
> > > core/psplash/psplash_git.bb:133
> > > # [_defaultval] "foo bar"
> > > #   :remove /home/ross/Yocto/poky/meta-poky/recipes-
> > > core/psplash/psplash_git.bbappend:3
> > > # "foo"
> > > # pre-expansion value:
> > > #   "foo bar"
> > > FOOBAR=" bar”
> > >
> > > Josh: can you provide a minimal reproducer, or did this quietly
> > > change behaviour at some point?
> >
> > When I read this discussion a few days ago, I also wondered. For
> > example, we have systemd on kirkstone which does:
> >
> > PACKAGECONFIG ??= " \
> > ${@bb.utils.filter('DISTRO_FEATURES', 'acl audit efi...
> > ...
> > "
> >
> > and in the distro there is for example:
> >
> > PACKAGECONFIG:append:pn-systemd = " cgroupv2"
> > PACKAGECONFIG:remove:pn-systemd = "sysvinit"
> >
> > The append and the remove operators do not reset the PACKAGECONFIG to
> > "".
> >
> > I see that e.g. += can be misleading. But append and remove seam to
> > work as expected (by me) on kirkstone.
> 
> Yes,
> 
> You and Ross are correct. For some reason, I was mentally hung up on
> PACKAGECONFIG:pn-systemd:remove being the correct way to do this,
> which is very obviously wrong after you think about it.
> 
> AFAICT, there is no reason to make any changes, and I sincerely
> apologize for wasting everyone's time and sanity on this e-mail chain.

Actually, I do not think it was a waste of time, because even if :append 
and :remove do work with ??=, you are forced to use :append if you want to 
add to PACKAGECONFIG in a bbappend. You cannot use +=. Changing the recipes 
to use = instead of ??= would allow this, without changing anything else 
(except for the case that Ross noted, which was wrong to begin with).

> 
> Joshua Watt
> 
> > Adrian
> >
> > > Ross

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1901): 
https://lists.openembedded.org/g/openembedded-architecture/message/1901
Mute This Topic: https://lists.openembedded.org/mt/103269425/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Weak PACKAGECONFIG removal

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Joshua Watt
> Sent: den 21 december 2023 16:57
> To: Richard Purdie 
> Cc: OpenEmbedded Architecture  architect...@lists.openembedded.org>
> Subject: Re: [Openembedded-architecture] Weak PACKAGECONFIG removal
> 
> On Thu, Dec 21, 2023 at 6:13 AM Richard Purdie
>  wrote:
> >
> > On Tue, 2023-12-19 at 13:19 -0700, Joshua Watt wrote:
> > > This message is to start a discussion about removal of undesired
> > > recipe PACKAGECONFIG options.
> > >
> > > Background:
> > > Since the dawn of using OpenEmbedded as part of my day job, we've used
> > > meta-gplv2 to avoid GPLv3 code in our products. However, I recently
> > > decided that it was past time to excise this particular wart from our
> > > code base, so I've been looking at how to do this. Ideally, I would
> > > like to make it some sort of shared .inc file that can be included and
> > > we can all collaborate on to make it easier for this fairly common use
> > > case.
> > >
> > >
> > > One of the major things that comes up frequently is that the
> > > PACKAGECONFIG for a recipe needs to be modified to remove some
> > > incompatible flag from the default options. This particular option is
> > > not very simple though, because most recipes set PACKAGECONFIG using
> > > the "deferred weak" assignment, as in `PACKAGECONFIG ??= "foo bar"`.
> > > Because of this, any operation that attempts to modify this variable
> > > will overwrite the default; commonly one might like to do something
> > > like `PACKAGECONFIG:remove = "foo"`, but since this happens before
> > > weak assignment, the result is an empty string instead of just the
> > > value "bar".
> > >
> > > I've come up with a number of solutions for this, but I'm unsure which
> > > ones make sense to explore as long term options for the project:
> > >
> > > 1) Do nothing and respecify the complete set of PACKAGECONFIG options
> > > when overriding. This basically means instead of PACKAGECONFIG:remove
> > > = "foo", you would explicitly set to the remaining items like
> > > PACKAGECONFIG = "bar". If no other solution is appealing, this is
> > > probably fine, and what I'll do. The main reason I don't like this is
> > > because package configs are frequently added to recipes and it's
> > > annoying to keep them in sync (mostly, because I'm lazy :) ).
> > >
> > > 2) Add a PACKAGECONFIG_REMOVE variable that is evaluated by the
> > > package config code and causes it to ignore any matching items. This
> > > one is pretty easy in the packageconfig code, and I think users would
> > > find it pretty easy to understand. The downside is that it's very
> > > specific to packageconfig; there are other variables that probably
> > > would want similar treatment, but it may not make as much intuitive
> > > sense to do for them also (e.g. IMAGE_FEATURES, DISTRO_FEATURES).
> > > Also, much like :remove, you wouldn't want this used in actual
> > > recipes; it should only be set by end users.
> > >
> > > 3) Change the order of weak assignment vs. remove. This one is a
> > > little trickier but the idea is that if ??= and :remove don't make
> > > sense in their current order, flip around the order of evaluation so
> > > that it does make sense (e.g. apply :remove after ??=). I'm not sure
> > > how easy this would be and if it could be done all the time, only when
> > > a variable was a "list", or some other criteria
> > >
> > > 4) Stop using ??= for default PACKAGECONFIG. I'm not sure the history
> > > as to why it's ??=, but switching it to ?= (or maybe even =) would
> > > solve the problem because then :append, :remove, +=, et. al. would
> > > work as expected. The only case I can think of where this wouldn't
> > > work is if a bbappend was doing a ?= to set a completely new set of
> > > default PACKAGECONFIG flags; I'm not sure why that would be necessary
> > > though. Are there any other cases I've not thought of?
> > >
> > > 5) Add "filters" to variables. The basic idea behind this one is that
> > > variables would get a new flag called "filter" where pipelines of
> > > limited expressions could be constructed. These filter expressions
> > > would be applied after the final value of the variable (and thus,
> > > after weak assignment). For this specific case, this might look
> > > something like: `PACKAGECONFIG[filter] = "remove(v, 'foo')"`. This is,
> > > admittedly, a _very_ general solution for this particular problem.
> > > However, this concept has been discussed in other contexts such as
> > > multilib and BBCLASSEXTEND, so if it is something that would help
> > > there, this problem could leverage that solution also. If you want a
> > > peek at what this might look like, there is a jpew/bb-filters branch
> > > in poky-contrib
> > >
> > >
> > > If you have any thoughts or other possible solutions, please let me
> know.
> >
> > I've some other 'random' thoughts.
> >
> > I've wondered 

Re: [Openembedded-architecture] [bitbake-devel] [PATCH] server/process: Disable the flush() call in server logging

[Peter Kjellerstedt]

> -Original Message-
> From: bitbake-de...@lists.openembedded.org 
>  On Behalf Of Richard Purdie
> Sent: den 15 september 2023 10:41
> To: bitbake-de...@lists.openembedded.org; openembedded-architecture 
> 
> Cc: Philip Balister 
> Subject: Re: [bitbake-devel] [PATCH] server/process: Disable the flush() call 
> in server logging
> 
> On Thu, 2023-09-14 at 17:28 +0100, Richard Purdie via lists.openembedded.org 
> wrote:
> > We've been chasing bitbake timeouts for a while and it was unclear where 
> > things
> > were blocking on IO. It appears the flush() call in server logging can cause
> > pauses up to minutes long on systems with slow (spinning) disks that are 
> > heavily
> > loaded with IO.
> >
> > Since the flush() was added to aid debugging of other timing issues, we 
> > shouldn't
> > need it now and it can be disabled. Leave a comment as a reminder of the 
> > pain this
> > can cause.
> >
> > Signed-off-by: Richard Purdie 
> > ---
> >  lib/bb/server/process.py | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
> > index 40cb99bc97..94e93bc3a6 100644
> > --- a/lib/bb/server/process.py
> > +++ b/lib/bb/server/process.py
> > @@ -43,7 +43,8 @@ def currenttime():
> >
> >  def serverlog(msg):
> >  print(str(os.getpid()) + " " +  currenttime() + " " + msg)
> > -sys.stdout.flush()
> > +#Seems a flush here triggers filesytem sync like behaviour and long 
> > hangs in the server
> > +#sys.stdout.flush()
> >
> >  #
> >  # When we have lockfile issues, try and find infomation about which 
> > process is
> 
> Whilst this patch looks simple, it has all the signs of turning into
> weeks of pain after looking at the results from testing. It looks like
> we have a race in the system which this flush is masking.
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio
> 
> Looking at just one of the failures there (of the 6), we see this test
> pass:
> 
> 1: 20/57 355/537 (0.12s) (0 failed)
> (recipetool.RecipetoolTests.test_recipetool_split_pkg_licenses)
> 
> then the next test to run on that build directory is:
> 
> 1: 21/57 359/537 (137.64s) (0 failed)
> (signing.Signing.test_signing_packages)
> 
> with an error which boils down to:
> 
> FileNotFoundError: [Errno 2] No such file or directory:
> '/tmp/recipetoolqaizvjzf8h'
> 
> from
> 
> run(cmd='git rev-parse --abbrev-ref HEAD', input=None, log=None,
> extrafiles=[], **options={'cwd': '/tmp/recipetoolqaizvjzf8h', 'shell':
> True})
> 
> i.e. it is trying to run a command in a directory which doesn't exist.
> 
> Without looking, I know "recipetoolqa" tmpfiles will come from
> RecipetoolTests so the previous build directory is being removed and
> yet still is cwd for the server.
> 
> I've not looked much into any of the other failures yet.
> 
> Looking at the 5 different selftest failures:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2082/steps/14/logs/stdio
> 
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_srpm_mode: FAILED (237.00s)
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: FAILED
> (13.25s)
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_deb_arch_dep: FAILED (126.30s)
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> signing.Signing.test_signing_packages: FAILED (137.64s)
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateHashSameSigs.test_sstate_32_64_same_hash: FAILED
> (541.51s)
> 2023-09-15 01:56:12,685 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateHashSameSigs3.test_sstate_multilib_or_not_native_samesig
> s: FAILED (224.05s)
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5710/steps/14/logs/stdio
> 
> 2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_srpm_mode: FAILED (59.78s)
> 2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS -
> bblogging.BitBakeLogging.test_shell_loggingA: FAILED (15.71s)
> 2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS -
> multiconfig.MultiConfig.test_multiconfig_reparse: FAILED (6.39s)
> 2023-09-14 16:00:20,642 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_rpm_arch_indep: FAILED (0.84s)
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5746/steps/14/logs/stdio
> 
> 2023-09-14 23:33:02,720 - oe-selftest - INFO - RESULTS -
> package.VersionOrdering.test_dpkg: ERROR
> 2023-09-14 23:33:02,720 - oe-selftest - INFO - RESULTS -
> package.VersionOrdering.test_opkg: ERROR
> 2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS -
> package.VersionOrdering.test_rpm: ERROR
> 2023-09-14 23:33:02,721 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_srpm_mode: FAILED (129.04s)
> 2023-09-14 23:33:02,721 - 

Re: [Openembedded-architecture] Changes needing documentation - a new procedure

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 2 september 2023 09:20
> To: openembedded-core ;
> openembedded-architecture  architect...@lists.openembedded.org>
> Cc: Michael Opdenacker 
> Subject: [Openembedded-architecture] Changes needing documentation - a new
> procedure
> 
> I've been concerned for a while about the process of adding
> documentation influencing changes. Our docs are in a separate repo and
> this does have advantages. They do span multiple areas of the project
> and in general this does work well for us. It does mean the docs are
> consistent and have a common style/voice.
> 
> Equally, it is hard to ask users to add documentation changes in their
> commits and this leads to the docs having to play catchup and pushes a
> lot of work to the docs maintainers.
> 
> We're therefore going to try something new where I'll ask people to add
> notes to a file in OE-Core, DOCS-WIP when they make a change which
> needs docs updates. This will include changes which need a release
> notes or migration guide entry. The docs maintainers will then know
> what needs to be done and hopefully have at least the basic information
> they need to make the docs changes needed.
> 
> This is an experiment but hopefully one that will be successful and let
> our docs continue to grow.
> 
> Cheers,
> 
> Richard

I would recommend to make DOCS-WIP a directory where you drop one file 
per change. That way you avoid all problems with conflicts during 
testing, integration and cherry-picking. You also automatically get a 
list of the pending changes by listing the files in that directory.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1744): 
https://lists.openembedded.org/g/openembedded-architecture/message/1744
Mute This Topic: https://lists.openembedded.org/mt/101109973/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Peter Kjellerstedt]

> -Original Message-
> From: Richard Purdie 
> Sent: den 16 augusti 2023 13:13
> To: Peter Kjellerstedt ; 
> openembedded-architecture 
> Subject: Re: [Openembedded-architecture] Workflow improvements - incremental 
> source updates?
> 
> On Tue, 2023-08-15 at 21:27 +, Peter Kjellerstedt wrote:
> >
> > I decided to give the related changes you have in master-next a try. I
> > cherry-picked the following three commits:
> >
> >  bitbake: fetch2: Add new srcrev fetcher API
> >  recipes/classes/scripts: Drop SRCPV usage in OE-Core
> >  base/package: Move source revision information from PV to PKGV

[cut]

> > One thing that I would prefer is if there is a variable, similar to
> > the old SRCPV, that contains "${@bb.fetch.get_pkgv_string(d)}" by
> > default and is used from package_convert_pr_autoinc() instead of
> > calling the method directly. The reason for that is to make it
> > possible to use a different method to determine the value. E.g., in
> > our case we have a variable called MAINTPV, which is similar to
> > SRCPV. The reason for it is that it handles Gerrit references and
> > expands them in a more suitable way than just a SHA-1. Thus I would
> > like to be able to use its underlying function instead of
> > bb.fetch.get_pkgv_string(d) to generate the string for PKGV.

[cut]

> I was thinking more about the "in a variable" request and there is a
> good reason for not doing it.

I think we can live without it. If I am not mistaken we can always 
inject our extra information by doing something like 
PKGV:append = "${MAINTPV}" in recipes where we need it and then rely on 
your code to add the SHA-1 if the output from ${MAINTPV} includes a "+".

> I'm deliberately trying to avoid putting
> the function call into a variable as it then becomes part of the task
> hash computations. This means there are calls into it during parsing
> rather than at runtime. That was part of the pain in the previous
> approach and something I'm keen to avoid this time around to simplify
> things if we can. It does get included in the taskhashes after the
> changes but at points we control (vardepvalue) rather than 'random'
> expansion points of a common variable like PV during parsing.

I am a bit unsure about the trying to avoid calls to the get_pkgv_string() 
function during parsing, because it seems to happen with the current 
code. When I first added your changes, I got errors for a number of 
recipes in meta-oe that failed to set SRCREV_FORMAT (Khem has since 
corrected those recipes). E.g., if you add meta-oe to the build and 
revert commit 4228656bae, then you will get the following error if 
you run `bitbake -p`:

WARNING: .../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb: Exception during 
build_dependencies for fetcher_hashes_dummyfunc
WARNING: .../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb: Error during 
finalise of 
/home/pkj/dists/poky-master/meta-oe/recipes-extended/sysdig/sysig_0.28.0.bb
ERROR: ExpansionError during parsing 
.../meta-oe/recipes-extended/sysdig/sysdig_0.28.0.bb

This is causing problems for me in our code base because we have a 
number of recipes that fetch from Git repositories with restricted 
access that I cannot fetch (we have alternative recipes that install 
prebuilt binaries for those of us without access to the source code). 
Now all those recipes fail during parsing because bitbake cannot run 
`git ls-remote`, even if they will never be built in my builds.

> If we did need control the use of it, we'd have to use some other
> variable to control it rather than the older approach. We then come
> back to my other feeling of wanting to get it right so people don't
> need to change it.
> 
> Cheers,
> 
> Richard

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1738): 
https://lists.openembedded.org/g/openembedded-architecture/message/1738
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org 
>  On Behalf Of Richard Purdie
> Sent: den 10 augusti 2023 18:53
> To: openembedded-architecture 
> 
> Subject: Re: [Openembedded-architecture] Workflow improvements - incremental 
> source updates?
> 
> On Tue, 2023-08-08 at 10:10 +0100, Richard Purdie via lists.openembedded.org 
> wrote:
> > There is a recent discussion about AUTOREV which has got me thinking a
> > bit about the big picture and incremental development.
> >
> > Currently AUTOREV injects into PV. The reasons for that are historical,
> > when OE was not taskhash based we had to do that to trigger updates.
> > There are now different ways it could be done with the revision just
> > needing injection into PKGPV for packaging purposes.
> >
> > I've also been looking at externalsrc bugs.
> >
> > This got me thinking, if we did make AUTOREV work without messing with
> > PV, we could probably go a step further and allow the fetcher to
> > incrementally update an existing git checkout rather than remove and
> > start again from scratch. Patches may or may not be an issue, we could
> > mandate a clean single git:// srcuri to start with although the patch
> > code can remove the patch stack already. This would need to hook in to
> > disable some of the cleanup/removal code.
> >
> > The usecase would be to allow the developer to pull updates from a
> > remote repo, or even commit changes locally and then have the system
> > "flow" the changes through from there. This would remove the annoying
> > clean and re-checkout process which drives some developers crazy. It
> > might also allow externalsrc to change to a new model with less
> > deviation from the normal workflow.
> 
> I tried some experiments and this doesn't actually turn out to be so
> difficult to do. I've included a patch below which basically drops the
> SRCPV pieces into PKGV, the conditional can probably be simplified
> further.
> 
> This patch isn't quite right in that we need to split get_srcrev into
> two versions, one which returns the SRCREV_FORMAT string and one which
> returns the full hashes for taskhash purposes.
> 
> The nice thing about this approach is that SRCPV can simply be removed
> over time, it no longer does anything. That means compatibility is
> retained with existing recipes.
> 
> I've not tested this extensively yet with AUTOREV but normal builds
> appear to work ok with it and the WORKDIR directory naming
> simplification is probably a win.
> 
> Cheers,
> 
> Richard

I decided to give the related changes you have in master-next a try. I 
cherry-picked the following three commits:

 bitbake: fetch2: Add new srcrev fetcher API
 recipes/classes/scripts: Drop SRCPV usage in OE-Core
 base/package: Move source revision information from PV to PKGV

I am of course aware that the above changes are not final, but I have 
some concerns.

I tested building "iso-codes" and "xcursor-transparent-theme". They are 
both allarch recipes with few dependencies. The former has a SRCREV that 
matches PV, and the latter does not (and thus uses PV = "0.1.1+git${SRCPV}").

Here are the names of the resulting RPM packages before the changes are 
applied:

  iso-codes-4.15.0-r0.4.noarch.rpm
  xcursor-transparent-theme-0.1.1+git0+23c8af5ba4-r0.0.noarch.rpm

And here they are afterwards:

  iso-codes-4.15.0AUTOINC+69ba16daef-r0.0.noarch.rpm
  xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4-r0.0.noarch.rpm

The first problem is of course that AUTOINC is not replaced as intended 
when a PR server is active. This is due to the code in package_get_auto_pr() 
that should update PRSERV_PV_AUTOINC not being triggered and thus it 
retains its default value "AUTOINC". The code isn't triggered because it 
looks for "AUTOINC" in PKGV, but it isn't there (yet) since ${SRCPV} is 
no longer used. The following patch solves it:

diff --git a/meta/classes-global/package.bbclass 
b/meta/classes-global/package.bbclass
index 7f55b123c4..bf824b0b83 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -266,6 +266,9 @@ python package_get_auto_pr() {
 d.setVar("PRSERV_HOST", host)

 pkgv = d.getVar("PKGV")
+srcpv = bb.fetch.get_pkgv_string(d)
+if srcpv:
+pkgv += srcpv

 # PR Server not active, handle AUTOINC
 if not d.getVar('PRSERV_HOST'):

The second problem is that the value from bb.fetch.get_pkgv_string() is 
concatenated to the version without any separator. So if the AUTOINC 
part had been replaced as expected, the RPM name for the iso-codes 
package would have become iso-codes-4.15.00+69ba16daef-r0.0.noarch.rpm, 
which I doubt was intended... The easiest solution should be to make 
get_pkgv_string() return "+AUTOINC+" instead of the current 
"AUTOINC+".

Third, I am not sure if it is a good idea to always include the SHA-1 
in the package name. Before it was typically only done when the SRCREV 
did not match PV. For most packages it is 

Re: [Openembedded-architecture] Workflow improvements - incremental source updates?

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 8 augusti 2023 11:11
> To: openembedded-architecture  architect...@lists.openembedded.org>
> Subject: [Openembedded-architecture] Workflow improvements - incremental
> source updates?
> 
> There is a recent discussion about AUTOREV which has got me thinking a
> bit about the big picture and incremental development.
> 
> Currently AUTOREV injects into PV. The reasons for that are historical,
> when OE was not taskhash based we had to do that to trigger updates.
> There are now different ways it could be done with the revision just
> needing injection into PKGPV for packaging purposes.
> 
> I've also been looking at externalsrc bugs.
> 
> This got me thinking, if we did make AUTOREV work without messing with
> PV, we could probably go a step further and allow the fetcher to
> incrementally update an existing git checkout rather than remove and
> start again from scratch. Patches may or may not be an issue, we could
> mandate a clean single git:// srcuri to start with although the patch
> code can remove the patch stack already. This would need to hook in to
> disable some of the cleanup/removal code.
> 
> The usecase would be to allow the developer to pull updates from a
> remote repo, or even commit changes locally and then have the system
> "flow" the changes through from there. This would remove the annoying
> clean and re-checkout process which drives some developers crazy. It
> might also allow externalsrc to change to a new model with less
> deviation from the normal workflow.
> 
> Whilst on the subject, I have also been wondering about a new image
> target, effectively one that would build a new image but then allow an
> rsync over to a running target. Obviously this may or may not be
> advisable depending on what was actually running but for some
> development, it could be useful.
> 
> Anyway, just thinking out loud but wanted to document the idea. I might
> look into at least improving the way AUTOREV works...
> 
> Cheers,
> 
> Richard

It sounds like there are a lot of good possibilities here.

One question though: would any of this make it possible to use 
${AUTOREV} together with BB_SRCREV_POLICY = "cache"?

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1728): 
https://lists.openembedded.org/g/openembedded-architecture/message/1728
Mute This Topic: https://lists.openembedded.org/mt/100618420/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] WORKDIR fetcher interaction issue

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org 
>  On Behalf Of Richard Purdie
> Sent: den 31 december 2022 18:03
> To: Mark Hatle ; Joshua Watt 
> 
> Cc: openembedded-architecture 
> ; Trevor Woerner 
> 
> Subject: Re: [Openembedded-architecture] WORKDIR fetcher interaction issue
> 
> On Sat, 2022-12-31 at 10:47 -0600, Mark Hatle wrote:
> >
> > On 12/29/22 9:06 AM, Richard Purdie wrote:
> > > On Thu, 2022-12-29 at 08:50 -0600, Joshua Watt wrote:
> > > > On Thu, Dec 29, 2022 at 7:56 AM Richard Purdie
> > > >  wrote:
> > > > >
> > > > > I was asked about a WORKDIR/fetcher interaction problem and the bugs 
> > > > > it
> > > > > results in. I've tried to write down my thoughts.
> > > > >
> > > > > The unpack task writes it's output to WORKDIR as base.bbclass says:
> > > > >
> > > > >  fetcher = bb.fetch2.Fetch(src_uri, d)
> > > > >  fetcher.unpack(d.getVar('WORKDIR')
> > > > >
> > > > > We historically dealt with tarballs which usually have a NAME-VERSION
> > > > > directory within them, so when you extract them, they go into a sub
> > > > > directory which tar creates. We usually call that subdirectory "S".
> > > > >
> > > > > When we wrote the git fetcher, we emulated this by using a "git"
> > > > > directory to extract into rather than WORKDIR.
> > > > >
> > > > > For local files, there is no sub directory so they go into WORKDIR.
> > > > > This includes patches, which do_patch looks for in WORKDIR and applies
> > > > > them from there.
> > > > >
> > > > > What issues does this cause? If you have an existing WORKDIR and run a
> > > > > build with:
> > > > >
> > > > > SRC_URI = "file://a file://b"
> > > > >
> > > > > then change it to:
> > > > >
> > > > > SRC_URI = "file://a"
> > > > >
> > > > > and rebuild the recipe, the fetch and unpack tasks will rerun and 
> > > > > their
> > > > > hashes will change but the file "b" is still in WORKDIR. Nothing in 
> > > > > the
> > > > > codebase knows that it should delete "b" from there. If you have code
> > > > > which does "if exists(b)", which is common, it will break.
> > > > >
> > > > > There are variations on this, such as a conditional append on some
> > > > > override to SRC_URI but the fundamental problem is one of cleanup when
> > > > > unpack is to rerun.
> > > > >
> > > > > The naive approach is then to think "lets just delete WORKDIR" when
> > > > > running do_unpack. There is the small problem of WORKDIR/temp with 
> > > > > logs
> > > > > in. There is also the pseudo database and other things tasks could 
> > > > > have
> > > > > done. Basically, whilst tempting, it doesn't work out well in practise
> > > > > particularly as that whilst unpack might rerun, not all other tasks
> > > > > might.
> > > > >
> > > > > I did also try a couple of other ideas. We could fetch into a
> > > > > subdirectory, then either copy or symlink into place depending on 
> > > > > which
> > > > > set of performance/usabiity challenges you want to deal with. You 
> > > > > could
> > > > > involve a manifest of the files and then move into position so later
> > > > > you'd know which ones to delete.
> > > > >
> > > > > Part of the problem is that in some cases recipes do:
> > > > >
> > > > > S = "${WORKDIR}"
> > > > >
> > > > > for simplicity. This means that you also can't wipe out S as it might
> > > > > point at WORKDIR.
> > > > >
> > > > > SPDX users have requested a json file of file and checksums after the
> > > > > unpack and before do_patch. Such a manifest could also be useful for
> > > > > attempting cleanup of an existing WORKDIR so I suspect the solution
> > > > > probably lies in that direction, probably unpacking into a subdir,
> > > > > indexing it, then moving into position.
> > > >
> > > > By "moving it into position" do you mean moving the files from the
> > > > clean subdirectory to the locations they would occupy today?
> > > >
> > > > If so I don't understand why that's strictly necessary. It seems
> > > > like almost all of the complexity of this will be to support a
> > > > use-case we don't really like anyway (S = "${WORKDIR}"). Manifests are
> > > > great and all, but it causes a lot of problems if they get out of sync
> > > > and I suspect that would happen more often than we would like, e.g.
> > > > with devtool, make config, manual editing, etc. If we can keep it
> > > > simple and not rely on external state (e.g. a manifest) I think it
> > > > will be a lot easier to maintain in the long run.
> > >
> > > Dropping S = "${WORKDIR}" doesn't solve the problem being described
> > > here, it just removes something which complicates current code and
> > > makes that problem harder to solve. Even not supporting S =
> > > "${WORKDIR}", do_unpack still unpacks to WORKDIR with the S directory
> > > created by the tarball.
> >
> > In this particular piece, it's always bugged me that I don't have control 
> > over
> > the place it unpacks (whatever it is), where it patches and the S directory.
> > 

Re: [Openembedded-architecture] Template handling in OE-Core

[Peter Kjellerstedt]

> -Original Message-
> From: Alexander Kanavin 
> Sent: den 28 september 2022 16:41
> To: Peter Kjellerstedt 
> Cc: Richard Purdie ; openembedded-
> architecture 
> Subject: Re: Template handling in OE-Core
> 
> On Wed, 28 Sept 2022 at 16:25, Peter Kjellerstedt
>  wrote:
> 
> > Personally, I would expect the file to contain the latest value used
> > for TEMPLATECONF. I.e., if I remove the local.conf file to regenerate
> > it because there has been upstream changes, I would expect it to use
> > the TEMPLATECONF I specify at that time, and for that value to be
> > recorded for posterity.
> >
> > That said, I doubt the value of TEMPLATECONF will actually change for
> > an existing build environment, so this is probably more on the
> > theoretical side.
> 
> This would complicate the logic (update only if some config was
> updated), and if you remove local.conf but not bblayers.conf, so that
> one is updated, but not the other, it is not anymore possible to tell
> what came from where. Creating it once at the start is more
> straightforward, even if origins of further updates are not recorded.
> 
> Alex

Yeah, I guess it's a case of damned if you do, damned if you don't. 
However, as this patch solves our problem, I support applying this patch.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1654): 
https://lists.openembedded.org/g/openembedded-architecture/message/1654
Mute This Topic: https://lists.openembedded.org/mt/93968540/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Template handling in OE-Core

[Peter Kjellerstedt]

> -Original Message-
> From: Alexander Kanavin 
> Sent: den 28 september 2022 11:07
> To: Richard Purdie 
> Cc: openembedded-architecture  architect...@lists.openembedded.org>; Peter Kjellerstedt
> 
> Subject: Re: Template handling in OE-Core
> 
> On Wed, 28 Sept 2022 at 10:55, Richard Purdie
>  wrote:
> > I wondered about that. The thing is if you're planning to decide to do
> > some kind of automated update or user warning message, you need a
> > version string in the file itself to know what version you're dealing
> > with so whilst I can see why you might be interested in that
> > information, I'm not convinced it is as useful as you'd first think as
> > you'll need other information anyway...
> >
> > Regardless, it would suggest we should be ignoring the file (whether
> > create it or not).
> >
> > Taking another example to suggest it is broken, what if I delete one of
> > the files and let the setup scripts recreate it after using a different
> > TEMPLATECONF setting? Which value should templateconf.cfg have?
> 
> I have already written a patch :) It creates templateconf.cfg if it
> does not yet exist,
> but otherwise doesn't read or write to it.
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/package-version-updates=0644e4cb1415b44833d57baf9d57845a331bb70b
> 
> Alex

Personally, I would expect the file to contain the latest value used 
for TEMPLATECONF. I.e., if I remove the local.conf file to regenerate 
it because there has been upstream changes, I would expect it to use 
the TEMPLATECONF I specify at that time, and for that value to be 
recorded for posterity.

That said, I doubt the value of TEMPLATECONF will actually change for 
an existing build environment, so this is probably more on the 
theoretical side.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1652): 
https://lists.openembedded.org/g/openembedded-architecture/message/1652
Mute This Topic: https://lists.openembedded.org/mt/93968540/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] meta-gplv2 - plan to drop support in master

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Ross Burton
> Sent: den 6 juli 2022 18:40
> To: Richard Purdie 
> Cc: openembedded-architecture  architect...@lists.openembedded.org>; nd 
> Subject: Re: [Openembedded-architecture] meta-gplv2 - plan to drop support
> in master
> 
> On 6 Jul 2022, at 16:34, Richard Purdie via lists.openembedded.org
>  wrote:
> >
> > meta-gplv2 makes the TSCs very nervous (certainly YP and I believe OE
> > TSC people too although it hasn't had an official discussion there). It
> > has a load of old "pre GPLv3 license" versions of software that haven't
> > seen security fixes for years. We've limped it along to our last LTS.
> > It has no active maintainer.
> 
> I for one agree with this proposal.  For several years we’ve said that
> this layer is basically on life support, someone who uses it should step
> up and maintain it, and yet at no point does this happen.  It’s actively
> accumulating security fixes so can’t seriously be used in any real
> deployments as-is, so it must be redundant.  There’s no point in redundant
> code taking up the not-maintainers time, so we can remove it.
> 
> Ross

For what it is worth, we stopped using meta-gplv2 earlier this year. What made 
it possible was that we stopped defining INCOMPATIBLE_LICENSE globally, and 
instead only define it for the image recipes. We also have never allowed bash 
in our images, which otherwise seems to be a blocker for many wanting to avoid 
GPL-3.0. In the end, we had to copy two recipes from meta-gplv2 (gnupg and 
dosfstools) to one of our own layers, but that at least indicates that we are 
now responsible for maintaining them ourselves.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1576): 
https://lists.openembedded.org/g/openembedded-architecture/message/1576
Mute This Topic: https://lists.openembedded.org/mt/92209067/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] [oe] INCOMPATIBLE_LICENSES and WHITELIST_ usage

[Peter Kjellerstedt]

> -Original Message-
> From: Richard Purdie 
> Sent: den 19 februari 2022 01:45
> To: Peter Kjellerstedt ; Saul Wold
> ; openembedded-
> architect...@lists.openembedded.org; OE-core  c...@lists.openembedded.org>; OpenEmbedded Devel List  de...@lists.openembedded.org>
> Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_ usage
> 
> On Fri, 2022-02-18 at 23:34 +, Peter Kjellerstedt wrote:
> > Warning: wall of text ahead. Sorry about that, but I believe it is
> > important that we get this right if we are redesigning it.
> >
> > TL;DR: see the end for the new suggested licensing variables.
> >
> > > -Original Message-
> > > From: openembedded-de...@lists.openembedded.org  > > de...@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 18 februari 2022 15:14
> > > To: Saul Wold ; openembedded-
> > > architect...@lists.openembedded.org; OE-core  > > c...@lists.openembedded.org>; OpenEmbedded Devel List  > > de...@lists.openembedded.org>
> > > Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_ usage
> > >
> > > On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > > > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is
> > > > used and processed to possibly include a COMPATIBLE_LICENSES variable
> > > > as well, see PeterK's email [0]
> > > >
> > > > I am trying to determine the usage of WHITELIST_ which
> > > > would be used to override a license that might be listed in
> > > > INCOMPATIBLE_LICENSES variable.
> > > >
> > > > Randy and I have done a quick and dirty survey of a 100 or so layers
> > > > (thanks Randy) and could not find any real usage other than what's
> > > > currently in OE-Core for WHITELIST_GPL-3.0.
> > > >
> > > > If you are using WHITELIST_, please let me reply with your
> > > > usage.
> > > >
> > > > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> > >
> > > We need to be mindful that we need to resolve this to unblock the
> > > other language changes and feature creep here is potentially 
> > > problematic. I do think it is worth trying to improve things rather 
> > > than blindly allowing the horrible syntax in this variable to 
> > > continue though.
> > >
> > > The test case we have for this currently is:
> > >
> > > WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> > >
> > > so I'd wondered about an alternative of:
> > >
> > > INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
> >
> > You do not really need the license here (more than possibly as
> > information to anyone reading the code). Specifying that a package is
> > allowed in an image regardless of its licenses can just as well be
> > done by allowing the variable to take a list of packages:
> >
> > INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"
> 
> I'm not sure I agree with that. I get the feeling some people want to
> allow these packages as long as they're with license X. Licenses can 
> change over time and this allows a safety net that if it changes, the 
> exception has to be updated too.

Well, to complicate matters then, if you really want to be explicit 
with the license exceptions you make for a recipe or a package, I will 
argue that you need to do it for all incompatible licenses a recipe 
uses. With today's code, as soon as you allow one incompatible license 
for a recipe, the recipe is allowed to be built, even if it has 
specified multiple incompatible licenses. E.g., if we take the gnupg 
recipe, which specifies LICENSE = "GPLv3 & LGPLv3", and we have an 
INCOMPATIBLE_LICENSE = "GPL-3.0-only LGPL-3.0-only", with your example 
I would then expect the following:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg:GPL-3.0-only 
gnupg:LGPL-3.0-only"

to allow the gnupg package to be installed in the image. However, I 
guess you could combine our suggestions by making the license part 
of the value optional, i.e., the following would then also allow the 
package to be included under the same conditions:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg"

> > An alternative (and in my opinion better) variable name would be:
> >
> > IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"
> >
> > focusing on that this is a list of _packages_ allowed for a given
> > _image_. This does not explicitly mention l

Re: [Openembedded-architecture] [oe] INCOMPATIBLE_LICENSES and WHITELIST_ usage

[Peter Kjellerstedt]

Warning: wall of text ahead. Sorry about that, but I believe it is 
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

> -Original Message-
> From: openembedded-de...@lists.openembedded.org  de...@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 18 februari 2022 15:14
> To: Saul Wold ; openembedded-
> architect...@lists.openembedded.org; OE-core  c...@lists.openembedded.org>; OpenEmbedded Devel List  de...@lists.openembedded.org>
> Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_ usage
> 
> On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> > and processed to possibly include a COMPATIBLE_LICENSES variable as
> > well, see PeterK's email [0]
> >
> > I am trying to determine the usage of WHITELIST_ which would be
> > used to override a license that might be listed in INCOMPATIBLE_LICENSES
> > variable.
> >
> > Randy and I have done a quick and dirty survey of a 100 or so layers
> > (thanks Randy) and could not find any real usage other than what's
> > currently in OE-Core for WHITELIST_GPL-3.0.
> >
> > If you are using WHITELIST_, please let me reply with your
> > usage.
> >
> >
> > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> 
> We need to be mindful that we need to resolve this to unblock the other
> language changes and feature creep here is potentially problematic. I do 
> think it is worth trying to improve things rather than blindly allowing 
> the horrible syntax in this variable to continue though.
> 
> The test case we have for this currently is:
> 
> WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> 
> so I'd wondered about an alternative of:
> 
> INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"

You do not really need the license here (more than possibly as 
information to anyone reading the code). Specifying that a package is 
allowed in an image regardless of its licenses can just as well be 
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given 
_image_. This does not explicitly mention licenses, but I do not 
believe that is needed. After all there could be multiple reasons a 
package is not allowed in an image and this variable would allow to 
ignore them all because that is typically what you want to do: 
specify that you really want that package in that image. I guess 
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess 
like :append vs :remove, if someone for some reason specifies a 
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the 
later should win (to err on the side of caution).

For the case where you want to allow a recipe to be built despite 
it using licenses that are otherwise not allowed you can simply use 

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.

> which matches the current functionality, removes the issue that the name
> of the variable is unknown without iterating every possible license name 
> and makes it clear where it is applying to.
> 
> I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:
> 
> a) it is long
> b) it refers to recipes when it works against packages
> 
> (INCOMPATIBLE_LICENSE_PACKAGE_EXCEPTIONS is more correct but still long)
> 
> I do like it mentioning INCOMPATIBLE_LICENSE in full since it works in
> conjunction with that variable and that is definitely not clear from the
> current WHITELIST_XXX until you look at the code.
> 
> I'm still of the opinion the AVAILABLE_LICENSES variable is something we
> should be aiming to remove ultimately too, it has horrible issues with 
> layers changing hashes for all recipes.

Since I was the one to introduce it, I will answer to that. It was 
introduced so that it is possible to specify compatible licenses 
instead of incompatible licenses by basically calculating the 
incompatible licenses by taking the set of available licenses minus 
the set of compatible licenses. If a mechanism to do that is 
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in 
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we 
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs 
disallowed licenses because the current code really cannot handled a 
list of many hundreds of incompatible licenses, which is what we got 
after all SPDX licenses were added to OE-Core. The code is extremely 
inefficient and evaluates the list of incompatible licenses over and 
over again. In our case that meant the recipe parsing time tripled.

> Cheers,
> 
> Richard

That said, we really need two sets of 

Re: [Openembedded-architecture] Disable bbappend file from another meta layer

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Andrea Adami
> Sent: den 16 september 2021 22:34
> To: Andrea Adami 
> Cc: Khem Raj ; Mayank Agarwal
> ; bitbake-devel  de...@lists.openembedded.org>; openembedded-architecture  architect...@lists.openembedded.org>
> Subject: Re: [Openembedded-architecture] Disable bbappend file from
> another meta layer
> 
> On Thu, Sep 16, 2021 at 10:30 PM Andrea Adami via
> lists.openembedded.org 
> wrote:
> >
> > On Thu, Sep 16, 2021 at 7:28 PM Khem Raj  wrote:
> > >
> > > On Thu, Sep 16, 2021 at 9:45 AM Mayank Agarwal
> > >  wrote:
> > > >
> > > > Dear Experts,
> > > >
> > > > Is there any way we can disable bbappend file from another meta-
> layer in the current
> > > > meta-layer or recipe file.It will be of great help.If we can use
> PNBLACKLIST or BBMASK
> > > > to disable bbappend file.
> > > >
> > >
> > > bbappends can not be undone sadly, maybe you can try some BBMASKing it
> > > if its in unique location perhaps.
> > >
> >
> > See here one example:
> > https://github.com/vu-plus/meta-vuplus/blob/master/conf/layer.conf#L4
> > vs one allowing them
> > https://git.openembedded.org/meta-handheld/tree/conf/layer.conf#n8
> > cheers
> > A.A.
> >
> 
> but maybe I misread 'in another meta-layer' out of your control. sorry...
> A.A.
> > > > Regards
> > > > Mayank

It is perfectly fine to add, e.g.:

BBMASK += "/meta-foo/recipes-foo/foo/foo_1.2.3.bbappend"

to, e.g., meta-bar/conf/layer.conf to disable the use of 
foo_1.2.3.bbappend from the meta-foo layer when the meta-bar layer is 
used.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1315): 
https://lists.openembedded.org/g/openembedded-architecture/message/1315
Mute This Topic: https://lists.openembedded.org/mt/85656329/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Overrides conversion plan

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 30 juli 2021 15:40
> To: openembedded-architecture  architect...@lists.openembedded.org>
> Subject: Re: [Openembedded-architecture] Overrides conversion plan
> 
> On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > I think the challenge is going to be the flag day issue for master
> branches.
> > For example, there is code in devtool and other places which knows about
> the
> > override character. If we allow mixing the different syntax for master
> then
> > those tools need to complicate things by referencing both characters. To
> try
> > and preserve what is left of my sanity, I'm starting to think we just
> require
> > layers to migrate to the new syntax to continue to work with master. The
> good
> > news is that those converted layers should work with dunfell and older
> releases
> > where the layer already does that with the backported bitbake syntax
> update.
> >
> > If we accept that we need to have a flag day for master use, the
> question is
> > when. We could pick some data well in the future or even post 3.4
> however I'm
> > not sure this buys much and we probably may as well get on and do it.
> >
> > Given these things, I therefore propose that we should start these
> changes
> > and require it for master, probably relatively quickly within a couple
> of
> > weeks?
> 
> I have continued to work on this and I now think we're as ready as we'll
> ever
> be with the core. I have:
> 
> * submitted a section for the migration guide documenting the conversion
> process
> * increased the minimum bitbake version for OE-Core
> * bumped the local.conf version to require new versions of the config file
> * added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in
>   variable names which would suggest an unconverted layer. If you use
> those
>   in function names in the datastore that was never a good idea and is no
>   longer supported.
> * merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
> * merged the conversion script to OE-Core
> * merged submitted tweaks to the conversion script (thanks Martin!)
> * made OE-Core honister only, no longer supporting hardknott
> * updated converted layers to be honister compatible
> * converted autobuilder-helper to use the new syntax
> * ensured all of a-quick builds on the autobuilder
> * submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake
> 
> I plan to merge these various things on Monday (2nd August). After that
> time, unconverted layers will no longer work with master.
> 
> Cheers,
> 
> Richard

[ I am on summer vacation so I just happened to see a tweet about this. ]

I am sorry to say, but I think you are going too fast. AFAICT, the support 
for using : instead of _ in overrides is not yet in any Dunfell or Hardknott 
release, only on the branches. I can't speak for others of course, but at 
least we will not pick it up until it is included in an actual release.

//Peter


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1300): 
https://lists.openembedded.org/g/openembedded-architecture/message/1300
Mute This Topic: https://lists.openembedded.org/mt/84508148/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Should we change variable override formatting?

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture@lists.openembedded.org  architect...@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 16 juli 2021 12:09
> To: Nicolas Dechesne 
> Cc: openembedded-architecture  architect...@lists.openembedded.org>
> Subject: Re: [Openembedded-architecture] Should we change variable
> override formatting?
> 
> On Fri, 2021-07-16 at 10:12 +0200, Nicolas Dechesne wrote:
> > hey!
> >
> > On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie
>  wrote:
> > > b) Some of the packaging code (at the very least) needs rewriting as
> it accesses
> > >    both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses
> RDEPENDS. I'm not
> > >    sure what else may be assuming this works.
> > >
> >
> >
> > Ouch.. I never realized that! I like Mark's suggestion to convert that
> to variables. RDEPENDS is typically
> > used in users' layers, so that  way they wouldn't be impacted by an
> override syntax change.
> >
> 
> I have to admit I'm leaning the other way, make this explicit and hope
> that
> it actually helps users understand overrides a little better by making it
> clear.
> 
> 
> > > This change does buy us cleaner looking metadata and ultimately, a
> faster
> > > and cleaner internal bitbake that at least would use less memory. It
> could set
> > > the stage to allow the defval idea I mentioned in a previous mail to
> happen.
> > >
> > > It is a huge change and would need a lot of work to make it happen. Is
> it worth
> > > doing? Not sure. I'm putting it out there for discussion.
> > >
> >
> >
> > There is no doubt that the change is a good change and the project would
> benefit
> > from it. However I think we must worry about our users, and even look
> beyond the
> > layer maintainers that we know (e.g. the one on this list). There are
> *way* more
> > layer maintainers that we don't know about in all the many companies who
> are
> > successfully using YP to build their products. I am not worried about
> all the main
> > layers at all, since we know all their maintainers, and we know they
> will understand
> > why we make this change, and will make the effort to support it. But I
> am very
> > worried about the hundreds (thousands??) of layer maintainers in all the
> companies
> > using Yocto.
> >
> > So if we do anything, I would really prefer if we find a way to do it in
> a good way
> > *for them* , not *for us*. which means finding a way to support a soft
> transition.
> > And of course we need to align this change with our LTS release cycle.
> So perhaps
> > we can figure out how to support both syntaxes for some time (from one
> LTS to the
> > next?), even if supporting both is impacting the performance. Or we can
> have a layer
> > flag that indicates if it uses the old or the new syntax?
> 
> If we support both forms, firstly, few of the layers you're concerned
> about will
> change as it will break older compatibility and they have no incentive to
> do so.
> 
> As I mentioned in another reply, there is a way some backwards
> compatibility could
> be added however I do worry about the horrible corner cases that may
> result in.
> 
> Secondly, supporting both means we can't take advantage of any benefits
> the change
> brings to further improve for a period of time. Do we want to commit to
> making this
> change so that we can benefit from it in say four years time (a couple of
> LTS cycles
> you mention)?
> 
> I'm worried about the pain of transitioning, but I'm also very worried
> that unless
> we figure out how we can change something, we're not going to develop
> beyond where
> we are now.
> 
> I'm trying hard to find any way we can move forward with some of the
> issues people
> are raising and I'm not seeing many options. I'm not seeing any proposals
> from anyone
> else either, probably as there is so much inertia to overcome to make any
> change :(.
> 
> Cheers,
> 
> Richard

This is a long answer, and responding to multiple subjects that have been 
brought up in the different threads, so please bear with me.

I would love to see improvements to the bitbake syntax when it comes to 
overrides. Unfortunately, I do not see how I would be able to ever update 
to said version if it is done as a flag day release. The way we work is 
to have our layers compatible with the version of bitbake we are using 
(currently Hardknott). Then I (who basically do all work to build our 
layers with master of Poky) have a special adaptation layer where I add 
bbappends and some occasional backported bbclass to make our layers work 
with Poky master. This work runs in parallel with the normal recipe updates 
that the developers do so that when a new major release of Poky is ready, 
so are we. However, if we would have to use a whole different version of 
all our recipes, there is no way this process would work, and I can't stop 
the normal development for a flag day upgrade of Poky. :(

The way I see it can be done would involve supporting both syntaxes in 
one 

Re: [Openembedded-architecture] Syntax for multiple overrides?

[Peter Kjellerstedt]

> -Original Message-
> From: openembedded-architecture-boun...@lists.openembedded.org
>  On Behalf Of
> Joshua Watt
> Sent: den 12 juni 2019 15:56
> To: richard.pur...@linuxfoundation.org; openembedded-architecture
> 
> Subject: Re: [Openembedded-architecture] Syntax for multiple overrides?
> 
> On 6/12/19 8:51 AM, richard.pur...@linuxfoundation.org wrote:
> > On Wed, 2019-06-12 at 08:44 -0500, Joshua Watt wrote:
> >> On 6/12/19 7:35 AM, Richard Purdie wrote:
> >>> I'm not 100% sure whether I like this idea or not but wanted to
> >>> share it and see what others think.
> >>>
> >>> We're looking at a recipe which would do:
> >>>
> >>> BBCLASSEXTEND = "devupstream:target"
> >>> LIC_FILES_CHKSUM_class-devupstream = 
> >>> "file://LICENSE;md5=3f882d431dc0f32f1f44c0707aa41128"
> >>> DEFAULT_PREFERENCE_class-devupstream = "-1"
> >>> SRC_URI_class-devupstream = 
> >>> "git://git.lttng.org/lttng-modules;branch=master \
> >>>  
> >>> file://Makefile-Do-not-fail-if-CONFIG_TRACEPOINTS-is-not-en.patch \
> >>>  file://BUILD_RUNTIME_BUG_ON-vs-gcc7.patch \
> >>>  "
> >>> SRCREV_class-devupstream = "${AUTOREV}"
> >>> PV_class-devupstream = "2.11.0-rc+git${SRCPV}"
> >>> S_class-devupstream = "${WORKDIR}/git"
> >>>
> >>> which is a bit ugly. I wondered whether we'd want to support some
> >>> kind of collective override syntax, e.g.:
> >>>
> >>> BBCLASSEXTEND = "devupstream:target"
> >>> override class-devupstream {
> >>> LIC_FILES_CHKSUM = "file://LICENSE;md5=3f882d431dc0f32f1f44c0707aa41128"
> >>> DEFAULT_PREFERENCE = "-1"
> >>> SRC_URI = "git://git.lttng.org/lttng-modules;branch=master \
> >>>  
> >>> file://Makefile-Do-not-fail-if-CONFIG_TRACEPOINTS-is-not-en.patch \
> >>>  file://BUILD_RUNTIME_BUG_ON-vs-gcc7.patch \
> >>>  "
> >>> SRCREV = "${AUTOREV}"
> >>> PV = "2.11.0-rc+git${SRCPV}"
> >>> S = "${WORKDIR}/git"
> >>> }
> >>>
> >>> It would be a pain to implement in bitbake but should be possible,
> >>> it would literally just append _ to the variable names when
> >>> storing them.
> >> An alternative syntax proposal:
> >> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-May/001411.html
> >>
> >> In that one, at least bitbake wouldn't have to deal with more braces
> >> :).
> > The braces bit is actually easier for the parser!
> 
> Ah, fair enough

The linked suggestion above shows another problem:

| push override "foo"
| DEPENDS += "libbar"
| SRC_URI_append = "file://foo"
| pop override
| 
| # The above is syntactically equivalent to:
| # DEPENDS_foo += "libbar"

That may be syntactically equivalent, but typically not want you 
want and not what you expect from reading the push/pop section 
above. What I would expect to get with the above code more 
corresponds to:

DEPENDS_append_foo = " libbar"

but this too is not equivalent as the _append is not evaluated at 
the same time as the += operator was. To actually get a correct 
equivalent code, you would need it to expand to something like:

DEPENDS-foo_foo = "libbar"
DEPENDS += "${DEPENDS-foo}"

| # SRC_URI_append_foo += "file://foo"

So this raises the question what do you want the override block to 
generate, code that is syntactically equivalent or functionally 
equivalent? Because if we choose the former, then the users will 
probably be even more confused about bitbake's already weird syntax. 

> >> I'm not partial to one or the other really, but the compulsive side
> >> of  me is *really* going to want to indent the variables (which I
> >> know would break all sorts of bitbake parsing).
> > I wanted to indent it too. The parser could probably be taught about
> > it relatively easily, its just regexs. I'm not sure that is a good 
> > idea though.
> >
> > In the link, Patrick makes a good point, we can't handle all syntax
> > within the braces as some don't have overrides conditionals (like
> > addtask).
> 
> Ya. I suspect you'd want to make anything other than variable
> assignments in a block syntax errors (at least until a sane mechanism
> is determined to handle them).
> 
> > Cheers,
> >
> > Richard

//Peter

___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture