Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 13:52, Alexander Kanavin  wrote:
>
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> > Alexander Kanavin will be working on the core workflow topic

I thought I'd write a summary of where we are with these subjects, and
make a plan for what needs to be done still:

1. Tools for figuring out what needs to be rebuilt and tests for them.

There was a large amount of corner cases to chase here, and fixing all
of that took a bit of time. I'm glad to report that all known issues
have been addressed, all patches have landed in master, and hopefully
this topic doesn't need further attention for now.

2. CDN sstate cache mirror and tests for it.

The tests for the mirror have landed as well, and there is an ongoing
issue that they expose: the CDN server isn't 100% robust. Sometimes it
returns a 5xx error, sometimes simply times out without responding.
This happens particularly with large objects, e.g. 50-60 Mb in size.
The problems are tracked in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15335 - hopefully
Michael Halstead can take a look soon-ish.

3. Build replicator tool.

I've written a proof of concept:
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all

There was positive feedback for the idea, so next I'll add tests and
submit it as a proper patch aiming for inclusion.

4. Any other business.

At this point I don't have any other subjects in mind, but if you do,
please write them here. Ideas, or maybe I missed something super
important.

Appendix.

There's also the oe-setup-build tool in development, which isn't
strictly a part of the core workflow topic, but is related, and I'd
like to get it to completion as well:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/setup-layers-tweaks
I'll check if I had missed any feedback, and re-submit.

Alex


Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1912): 
https://lists.openembedded.org/g/openembedded-architecture/message/1912
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 30 Oct 2023 at 15:07, Richard Purdie
 wrote:
> > a) esdk sets a number of variables from its initialization script that
> > aid with cross-compiling components directly (e.g. the core use case
> > of SDKs). Normal mode doesn't do that, but recently added
> > meta-ide-support will generate a similar initialization script that
> > will set up the same environment from the normal mode. There are tests
> > and documentation for it.
>
> In that case, this one is something we can document as how to make the
> functionality available in the normal build.

This has been documented in the sdk manual:
https://docs.yoctoproject.org/sdk-manual/extensible.html#two-ways-to-install-the-extensible-sdk
https://docs.yoctoproject.org/sdk-manual/extensible.html#installing-additional-items-into-the-extensible-sdk

Actually, I have on purpose presented both options in the manual as
'esdk', the standalone installer option is simply locked down more
(task signatures and tooling wise) than the direct yocto build.

So here's what could be done:

- esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
environment script puts that in PATH, rather than some custom
esdk-specific location (the code to generate that can then be
dropped).

- esdk tweaks to local.conf move into a dedicated include file, which
can be static and under version control, except for perhaps
METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
I need to check why is it there and how that is used.

Something else perhaps?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1816): 
https://lists.openembedded.org/g/openembedded-architecture/message/1816
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > There are design elements to this work. We need to work out how we can
> > make eSDK and "normal" builds more similar and less of an overhead to
> > switch between one and the other. A "bblock all" command does partly
> > get you to an eSDK effectively so part of this may be switching eSDK to
> > use the new lock command. What other differences are there? What other
> > differences are necessary or make sense for the use cases eSDK was
> > designed for? How would you turn an existing build into an eSDK like
> > one? Could you provide a copy of a local build to someone else easily
> > using something like eSDK's tooling? What does the eSDK look like at
> > the end of this. One section we don't have good answers to yet is setup
> > and configuration although I know you've started on some of that.
> 
> So I see the following differences between esdk and normal modes:
> 
> 1. Environment and tooling availability.
> 
> a) esdk sets a number of variables from its initialization script that
> aid with cross-compiling components directly (e.g. the core use case
> of SDKs). Normal mode doesn't do that, but recently added
> meta-ide-support will generate a similar initialization script that
> will set up the same environment from the normal mode. There are tests
> and documentation for it.

In that case, this one is something we can document as how to make the
functionality available in the normal build.

> b) PATH. eSDK has a number of items in PATH that point to various
> locations inside tmp/sysroots/, collectively they provide the
> cross-toolchain.
> 
> eSDK also puts a selection of yocto tools into path - wic, devtool but
> not bitbake:
> 
> 
> alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
> total 48
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
> ../../../../layers/poky/scripts/devtool
> lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
> ../../../../layers/poky/scripts/oe-find-native-sysroot
> lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
> ../../../../layers/poky/scripts/recipetool
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
> ../../../../layers/poky/scripts/runqemu
> lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
> ../../../../layers/poky/scripts/runqemu-addptable2image
> lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
> ../../../../layers/poky/scripts/runqemu-export-rootfs
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
> ../../../../layers/poky/scripts/runqemu-extract-sdk
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
> ../../../../layers/poky/scripts/runqemu-gen-tapdevs
> lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
> ../../../../layers/poky/scripts/runqemu-ifdown
> lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
> ../../../../layers/poky/scripts/runqemu-ifup
> lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
> lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
> ../../../../layers/poky/scripts/wic
> ==
> 
> 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
> Cross-toolchain can be added by the same environment script made by
> meta-ide-support as mentioned in 1a.

Right, so in theory we can change PATH and change this which can also
easily be documented.

> 2. Configuration (e.g. local.conf).
> 
> eSDK local.conf is local.conf from the normal mode that was used to
> produce eSDK, stripped of all comments, and with a bunch of extra
> settings:
> 
> 
> INHERIT:remove = "buildhistory icecc"
> CONNECTIVITY_CHECK_URIS = ""
> 
> SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
> 
> SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
> 
> BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
> 
> BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
> *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
> 
> BUILDCFG_HEADER = ""
> 
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
> 
> # Provide a flag to indicate we are in the EXT_SDK Context
> WITHIN_EXT_SDK = "1"
> 
> SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
> file://universal-4.9/(.*) file://universal-4.8/\1"
> 

Perhaps some of this should become a generic include file which is
included which would then make it easy to document as how to lock
things down to match it? That would take the config out of the class
file too which is probably nicer.

> require conf/locked-sigs.inc
> require conf/unlocked-sigs.inc
> ===
> 
> Devtool also has a special configuration:
> =
> alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
> [General]
> bitbake_subdir = layers/poky/bitbake
> init_path = layers/poky/oe-init-build-env
> 

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
 wrote:
> There are design elements to this work. We need to work out how we can
> make eSDK and "normal" builds more similar and less of an overhead to
> switch between one and the other. A "bblock all" command does partly
> get you to an eSDK effectively so part of this may be switching eSDK to
> use the new lock command. What other differences are there? What other
> differences are necessary or make sense for the use cases eSDK was
> designed for? How would you turn an existing build into an eSDK like
> one? Could you provide a copy of a local build to someone else easily
> using something like eSDK's tooling? What does the eSDK look like at
> the end of this. One section we don't have good answers to yet is setup
> and configuration although I know you've started on some of that.

So I see the following differences between esdk and normal modes:

1. Environment and tooling availability.

a) esdk sets a number of variables from its initialization script that
aid with cross-compiling components directly (e.g. the core use case
of SDKs). Normal mode doesn't do that, but recently added
meta-ide-support will generate a similar initialization script that
will set up the same environment from the normal mode. There are tests
and documentation for it.

b) PATH. eSDK has a number of items in PATH that point to various
locations inside tmp/sysroots/, collectively they provide the
cross-toolchain.

eSDK also puts a selection of yocto tools into path - wic, devtool but
not bitbake:


alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
total 48
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
../../../../layers/poky/scripts/devtool
lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
../../../../layers/poky/scripts/oe-find-native-sysroot
lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
../../../../layers/poky/scripts/recipetool
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
../../../../layers/poky/scripts/runqemu
lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
../../../../layers/poky/scripts/runqemu-addptable2image
lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
../../../../layers/poky/scripts/runqemu-export-rootfs
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
../../../../layers/poky/scripts/runqemu-extract-sdk
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
../../../../layers/poky/scripts/runqemu-gen-tapdevs
lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
../../../../layers/poky/scripts/runqemu-ifdown
lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
../../../../layers/poky/scripts/runqemu-ifup
lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
../../../../layers/poky/scripts/wic
==

'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
Cross-toolchain can be added by the same environment script made by
meta-ide-support as mentioned in 1a.

2. Configuration (e.g. local.conf).

eSDK local.conf is local.conf from the normal mode that was used to
produce eSDK, stripped of all comments, and with a bunch of extra
settings:


INHERIT:remove = "buildhistory icecc"
CONNECTIVITY_CHECK_URIS = ""

SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"

SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"

BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"

BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
*:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"

BUILDCFG_HEADER = ""

METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"

# Provide a flag to indicate we are in the EXT_SDK Context
WITHIN_EXT_SDK = "1"

SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
file://universal-4.9/(.*) file://universal-4.8/\1"

require conf/locked-sigs.inc
require conf/unlocked-sigs.inc
===

Devtool also has a special configuration:
=
alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
[General]
bitbake_subdir = layers/poky/bitbake
init_path = layers/poky/oe-init-build-env
core_meta_subdir = layers/poky/meta

[SDK]
sdk_targets = core-image-minimal
==
There is currently no tooling to add/remove these extras in either
esdk mode or normal mode as far as I understand. Their individual
purposes and effects are also not exactly clear to me, and need to be
investigated one by one.

3. Setting up a normal mode in a eSDK installation.

This is actually pretty easy: rather than sourcing the sdk environment
script, source the poky/oe-init-build-env:
==
alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .

### Shell environment set up for builds. ###
...
===
Bitbake will then simply run, albeit with all those extra
esdk-specific items in 

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Fri, 29 Sept 2023 at 14:27, Richard Purdie
 wrote:

> I'd prefer to see some dedicated bitbake API used even if we need to
> create/add it. tinfoil and some of the bblock/unlock work shows we can
> get stamp data, the question would be how to get it without
> "disturbing" the existing build.
>
> By using dedicated API, we'd be able to control the console output.

I just noticed that compare_sigfiles() has a 'collapsed' argument,
which, when set to True, omits most or all of the task hash printing,
and with that its output should approximate what bitbake-whatchanged
is aiming to do. This is currently used only by buildhistory, but it
could be used by -S printdiff too, e.g. by having verbose and concise
modes.

I'll run some experiments, and let's see what the overall output looks
like in real scenarios (e.g. 4.3_M3 vs current master).

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1785): 
https://lists.openembedded.org/g/openembedded-architecture/message/1785
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-29 at 14:06 +0200, Alexander Kanavin wrote:
> On Thu, 28 Sept 2023 at 18:49, Richard Purdie
>  wrote:
> 
> > I'm curious to see what you find with analysis of bitbake-whatchanged.
> 
> I've taken a look a the script. It obtains the current location of
> STAMPS_DIR, then runs this:
> 
> # Generate the new stamps dir
> print("Generating the new stamps ... (need several minutes)")
> cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
> args.recipe)
> 
> Then it walks both trees, matching up file names with a regex:
> 
> # Match the stamp's filename
> # group(1): PE_PV (may no PE)
> # group(2): PR
> # group(3): TASK
> # group(4): HASH
> stamp_re = 
> re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)")
> 
> Then there's some code that finds out what changed in the above
> between the two sets.
> 
> I don't see a way to make it work: messing about with STAMPS_DIR like
> that isn't supported, and will either do nothing, or remove the
> original stamps. Also stamp filenames aren't really a 'public API',
> are they?
> 
> Should the script simply be removed, or is there some better way to
> re-implement answering the 'what has changed' question in a way that
> doesn't flood the console with task hashes? I'd be glad to get
> suggestions for this.

I'd prefer to see some dedicated bitbake API used even if we need to
create/add it. tinfoil and some of the bblock/unlock work shows we can
get stamp data, the question would be how to get it without
"disturbing" the existing build.

By using dedicated API, we'd be able to control the console output.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1784): 
https://lists.openembedded.org/g/openembedded-architecture/message/1784
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 28 Sept 2023 at 18:49, Richard Purdie
 wrote:

> I'm curious to see what you find with analysis of bitbake-whatchanged.

I've taken a look a the script. It obtains the current location of
STAMPS_DIR, then runs this:

# Generate the new stamps dir
print("Generating the new stamps ... (need several minutes)")
cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
args.recipe)

Then it walks both trees, matching up file names with a regex:

# Match the stamp's filename
# group(1): PE_PV (may no PE)
# group(2): PR
# group(3): TASK
# group(4): HASH
stamp_re = 
re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)")

Then there's some code that finds out what changed in the above
between the two sets.

I don't see a way to make it work: messing about with STAMPS_DIR like
that isn't supported, and will either do nothing, or remove the
original stamps. Also stamp filenames aren't really a 'public API',
are they?

Should the script simply be removed, or is there some better way to
re-implement answering the 'what has changed' question in a way that
doesn't flood the console with task hashes? I'd be glad to get
suggestions for this.


Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1783): 
https://lists.openembedded.org/g/openembedded-architecture/message/1783
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 28 Sept 2023 at 18:49, Richard Purdie
 wrote:
> I've wondered if we should split bitbake -S printdiff into a separate
> utility? It exists from a time before we had bitbake command APIs.

It can also be a simple shell wrapper.

I've separated -S lockedsigs into it's own option as well, so it can
be explicitly requested (just need to fix up a couple of selftests
that relied on -S none to get locked-sigs.inc):
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all=41e8a8a79bac4e3b9eb74a99e32e0d26ac8af0c5

> I'm curious to see what you find with analysis of bitbake-whatchanged.
> I'm also somewhat surprised the scenarios you're testing all work!

It's not 100% perfect. One out of 2*3=6 scenarios isn't fully
functional (gcc-source:do_preconfigure signature isn't found from
sstate even though it exists there - but it is found if it's in
tmp/stamps/). I left a FIXME in the test.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1782): 
https://lists.openembedded.org/g/openembedded-architecture/message/1782
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-28 at 18:43 +0200, Alexander Kanavin wrote:
> On Fri, 22 Sept 2023 at 12:42, Richard Purdie
>  wrote:
> 
> > Things which used to be problematic:
> > 
> > a) changes involving changes to gcc-source since it uses a shared
> > sources stamps which confused the tools (at least used to). That may
> > have been before gcc-source became a recipe?
> > b) changes to a very common component (e.g. autoconf-native's
> > do_configure) which make it hard to understand where the root cause of
> > the changes came from
> > c) changes which affect many recipes at once, e.g. the do_configure
> > function in base.bbclass
> > 
> > It might be helpful to write test cases for the scenario you showed as
> > working above and some of the ones I mention above, then we can
> > document they work and have an easier way to add tests for issues
> > if/as/when we identify the problematic scenarios in future.
> 
> I've now written down the tests for these three scenarios and got them
> to pass (in oe-selftest too \0/):
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
> (check the commit message too)
> 
> I am going to look closer at bitbake-whatchanged, what it aims to do
> and why it doesn't work. I have a hunch it can produce useful high
> level reports, and so shouldn't be simply thrown away. 'bitbake -S
> printdiff' is too techy and verbose for some use cases. Maybe we can
> fold that functionality into 'bitbake -S whatchanged'.

I've wondered if we should split bitbake -S printdiff into a separate
utility? It exists from a time before we had bitbake command APIs.

I'm curious to see what you find with analysis of bitbake-whatchanged.
I'm also somewhat surprised the scenarios you're testing all work!

I'm guess one of the commits I pointed to must have fixed them (the
removal of paths from the sig files)?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1781): 
https://lists.openembedded.org/g/openembedded-architecture/message/1781
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Fri, 22 Sept 2023 at 12:42, Richard Purdie
 wrote:

> Things which used to be problematic:
>
> a) changes involving changes to gcc-source since it uses a shared
> sources stamps which confused the tools (at least used to). That may
> have been before gcc-source became a recipe?
> b) changes to a very common component (e.g. autoconf-native's
> do_configure) which make it hard to understand where the root cause of
> the changes came from
> c) changes which affect many recipes at once, e.g. the do_configure
> function in base.bbclass
>
> It might be helpful to write test cases for the scenario you showed as
> working above and some of the ones I mention above, then we can
> document they work and have an easier way to add tests for issues
> if/as/when we identify the problematic scenarios in future.

I've now written down the tests for these three scenarios and got them
to pass (in oe-selftest too \0/):
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
(check the commit message too)

I am going to look closer at bitbake-whatchanged, what it aims to do
and why it doesn't work. I have a hunch it can produce useful high
level reports, and so shouldn't be simply thrown away. 'bitbake -S
printdiff' is too techy and verbose for some use cases. Maybe we can
fold that functionality into 'bitbake -S whatchanged'.

I'm on holiday next week.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1780): 
https://lists.openembedded.org/g/openembedded-architecture/message/1780
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-22 at 11:17 +0200, Alexander Kanavin wrote:
> On Thu, 21 Sept 2023 at 16:39, chris.lapla...@agilent.com
>  wrote:
> 
> > That is very impressive and I'd also love to hear about what heuristics it 
> > uses.
> 
> It's actually rather simple. It uses glob.glob on stamps in tmp/, then
> on local sstate to find possible matches, then sorts them by mtime and
> takes the most recent. It's what would work most of the time, but we
> could add printdiff-all (print difference with all sstate matches) or
> printdiff-N (N most recent). It also could abstain from dumping
> locked-sigs.inc into cwd with both -S none and -S printdiff, unless
> explicitly asked
> 
> I just discovered there's also scripts/bitbake-whatchanged (that
> hasn't seen activity in years and is neither documented nor tested).
> Unsurprisingly then, it doesn't work in the same scenario:
> 
> 
> alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
> libsolv-native
> Figuring out the STAMPS_DIR ...
> Generating the new stamps ... (need several minutes)
> 
> === Summary: (0 changed, 0 unchanged)
> Newly added: 0
> PV changed: 0
> PR changed: 0
> Dependencies changed: 0
> 
> Removing the newly generated stamps dir ...
> 
> 
> Maybe this is what RP was referring to when he said the tools don't
> work properly?

No, I've believed that should probably be removed. I think there was a
recent change to it.

I think we had a major step change in this functionality working when
this was fixed:

https://git.yoctoproject.org/poky/commit/?id=84a7485025dd4473403b8da36a0c979a3afd5e93

and this test case was added:

https://git.yoctoproject.org/poky/commit/?id=1bdcd76d2968c3cc6ec2815afceba1cf98efd6d5

Things which used to be problematic:

a) changes involving changes to gcc-source since it uses a shared
sources stamps which confused the tools (at least used to). That may
have been before gcc-source became a recipe?
b) changes to a very common component (e.g. autoconf-native's
do_configure) which make it hard to understand where the root cause of
the changes came from
c) changes which affect many recipes at once, e.g. the do_configure
function in base.bbclass

It might be helpful to write test cases for the scenario you showed as
working above and some of the ones I mention above, then we can
document they work and have an easier way to add tests for issues
if/as/when we identify the problematic scenarios in future.

As you mention, it also uses mtime so perhaps issues happen if you run
a different build, then try and go back to the other config? I suspect
once you understand the algorithm the code uses, you can pick holes in
it.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1773): 
https://lists.openembedded.org/g/openembedded-architecture/message/1773
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 21 Sept 2023 at 16:39, chris.lapla...@agilent.com
 wrote:

> That is very impressive and I'd also love to hear about what heuristics it 
> uses.

It's actually rather simple. It uses glob.glob on stamps in tmp/, then
on local sstate to find possible matches, then sorts them by mtime and
takes the most recent. It's what would work most of the time, but we
could add printdiff-all (print difference with all sstate matches) or
printdiff-N (N most recent). It also could abstain from dumping
locked-sigs.inc into cwd with both -S none and -S printdiff, unless
explicitly asked

I just discovered there's also scripts/bitbake-whatchanged (that
hasn't seen activity in years and is neither documented nor tested).
Unsurprisingly then, it doesn't work in the same scenario:


alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
libsolv-native
Figuring out the STAMPS_DIR ...
Generating the new stamps ... (need several minutes)

=== Summary: (0 changed, 0 unchanged)
Newly added: 0
PV changed: 0
PR changed: 0
Dependencies changed: 0

Removing the newly generated stamps dir ...


Maybe this is what RP was referring to when he said the tools don't
work properly?


Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1772): 
https://lists.openembedded.org/g/openembedded-architecture/message/1772
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Chris Laplante via lists.openembedded.org]

> On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>  wrote:
> > The tools are already supposed to support doing this with local file
> > sstate sources, they just do a bad job at getting the diffs right. One
> > intent of this work item was to try and understand why they don't work
> > and address that so at least for filesystem sstate mirrors, you can
> > get better results. I don't know how we solve the remote http issue as yet.
> 
> I ran a few experiments with bitbake -S printdiff, and so far I haven't yet 
> seen
> an unhelpful result. Here's the most impressive one:
> 
> $ bitbake libsolv-native (uses cmake-native) $ git revert  version
> update> $ rm -rf build/tmp/ $ bitbake -S printdiff libsolv-native ...
> The differences between the current build and any cached tasks start at the
> following tasks:
> /srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-
> native_3.27.4.bb:do_recipe_qa
> NOTE: Writing task signature files
> Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc
> 
> Task cmake-native:do_recipe_qa couldn't be used from the cache because:
>   We need hash
> 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb,
> closest matching task was
> 85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7
>   basehash changed from
> 7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to
> 7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09
>   Variable PV value changed from '3.27.5' to '3.27.4'
>   Variable SRC_URI[sha256sum] value changed from
> '5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9'
> to
> '0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af'
> 
> That's pretty good, isn't it? It does print both what needs to be re-run, and
> *why* as well.
> 
> I have no idea yet what kind of magic it does to find the 'closest matching 
> task'
> in sstate, but if this breaks down in some other scenarios, we need to find 
> them
> to get a starting point for making the tools better. Ideas? I'm ready to try 
> them
> :)

That is very impressive and I'd also love to hear about what heuristics it 
uses. 

We use Artifactory to host our sstate. (Artifactory doesn’t have specific 
support for it, it's just basically acting as a generic HTTP server. But it 
makes things like LDAP easy). I have for a while been thinking of building a 
tool that tries to find the "closest" sstate on the server and then recursively 
runs bitbake-diffsig on it. The tool was going to be called 'why-not-sstate'.

Thanks,
Chris

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1771): 
https://lists.openembedded.org/g/openembedded-architecture/message/1771
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
 wrote:
> The tools are already supposed to support doing this with local file
> sstate sources, they just do a bad job at getting the diffs right. One
> intent of this work item was to try and understand why they don't work
> and address that so at least for filesystem sstate mirrors, you can get
> better results. I don't know how we solve the remote http issue as yet.

I ran a few experiments with bitbake -S printdiff, and so far I
haven't yet seen an unhelpful result. Here's the most impressive one:

$ bitbake libsolv-native (uses cmake-native)
$ git revert 
$ rm -rf build/tmp/
$ bitbake -S printdiff libsolv-native
...
The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-native_3.27.4.bb:do_recipe_qa
NOTE: Writing task signature files
Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc

Task cmake-native:do_recipe_qa couldn't be used from the cache because:
  We need hash 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb,
closest matching task was
85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7
  basehash changed from
7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to
7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09
  Variable PV value changed from '3.27.5' to '3.27.4'
  Variable SRC_URI[sha256sum] value changed from
'5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' to
'0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af'

That's pretty good, isn't it? It does print both what needs to be
re-run, and *why* as well.

I have no idea yet what kind of magic it does to find the 'closest
matching task' in sstate, but if this breaks down in some other
scenarios, we need to find them to get a starting point for making the
tools better. Ideas? I'm ready to try them :)

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1770): 
https://lists.openembedded.org/g/openembedded-architecture/message/1770
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Unless I missed something, I think I fixed all comments from RP on the
branch I force pushed :)

Cheers
Julien

Le mer. 20 sept. 2023 à 16:31, Alexander Kanavin
 a écrit :
>
> That’s great to hear! When you have a new patch set please post it here. I 
> think there’s also a new reply from RP that you need to consider.
>
> Alex
>
> On Wed 20. Sep 2023 at 16.25, Julien Stephan  wrote:
>>
>> Hi Alexander,
>> About bblock tool: I just force pushed my branch on poky-contrib
>> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
>> trying to fix an autobuilder issue reported by Alexandre Belloni.
>> I am still working on this, and I would be very happy to get this merged :)
>>
>> Cheers
>> Julien
>>
>> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
>>  a écrit :
>> >
>> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>> >  wrote:
>> >
>> > > Effectively those "configs" are what the eSDK is, you're just proposing
>> > > a server side set of them rather than a built copy. In many ways that
>> > > could be useful way of possibly rethinking the way the eSDK works.
>> > >
>> > > So in that sense I like the idea, we should just be clear the problem
>> > > it solves as it doesn't solve the "why is my build not using the cache"
>> > > question specifically but it potentially solves other issues.
>> >
>> > Right, I didn't make it clear that this idea is not about 'why sstate
>> > is not being reused', but rather something I thought of to avoid
>> > placing people in that situation to begin with - offering them
>> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
>> > hunch there'll be plenty, for whom this is just fine, and eventually
>> > this could be the basis of that oe-setup holy grail :)
>> >
>> > I'll start with the simplest enhancement of printing what would be
>> > rebuilt. There's a lot of code I need to understand, so this needs to
>> > be in manageable chunks :)
>> >
>> > Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1769): 
https://lists.openembedded.org/g/openembedded-architecture/message/1769
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Hi Alexander,
About bblock tool: I just force pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
trying to fix an autobuilder issue reported by Alexandre Belloni.
I am still working on this, and I would be very happy to get this merged :)

Cheers
Julien

Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
 a écrit :
>
> On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>  wrote:
>
> > Effectively those "configs" are what the eSDK is, you're just proposing
> > a server side set of them rather than a built copy. In many ways that
> > could be useful way of possibly rethinking the way the eSDK works.
> >
> > So in that sense I like the idea, we should just be clear the problem
> > it solves as it doesn't solve the "why is my build not using the cache"
> > question specifically but it potentially solves other issues.
>
> Right, I didn't make it clear that this idea is not about 'why sstate
> is not being reused', but rather something I thought of to avoid
> placing people in that situation to begin with - offering them
> 'pre-fabricated builds' with guaranteed sstate availability. I have a
> hunch there'll be plenty, for whom this is just fine, and eventually
> this could be the basis of that oe-setup holy grail :)
>
> I'll start with the simplest enhancement of printing what would be
> rebuilt. There's a lot of code I need to understand, so this needs to
> be in manageable chunks :)
>
> Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1768): 
https://lists.openembedded.org/g/openembedded-architecture/message/1768
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

That’s great to hear! When you have a new patch set please post it here. I
think there’s also a new reply from RP that you need to consider.

Alex

On Wed 20. Sep 2023 at 16.25, Julien Stephan  wrote:

> Hi Alexander,
> About bblock tool: I just force pushed my branch on poky-contrib
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
> trying to fix an autobuilder issue reported by Alexandre Belloni.
> I am still working on this, and I would be very happy to get this merged :)
>
> Cheers
> Julien
>
> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
>  a écrit :
> >
> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
> >  wrote:
> >
> > > Effectively those "configs" are what the eSDK is, you're just proposing
> > > a server side set of them rather than a built copy. In many ways that
> > > could be useful way of possibly rethinking the way the eSDK works.
> > >
> > > So in that sense I like the idea, we should just be clear the problem
> > > it solves as it doesn't solve the "why is my build not using the cache"
> > > question specifically but it potentially solves other issues.
> >
> > Right, I didn't make it clear that this idea is not about 'why sstate
> > is not being reused', but rather something I thought of to avoid
> > placing people in that situation to begin with - offering them
> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
> > hunch there'll be plenty, for whom this is just fine, and eventually
> > this could be the basis of that oe-setup holy grail :)
> >
> > I'll start with the simplest enhancement of printing what would be
> > rebuilt. There's a lot of code I need to understand, so this needs to
> > be in manageable chunks :)
> >
> > Alex
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1767): 
https://lists.openembedded.org/g/openembedded-architecture/message/1767
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
 wrote:

> Effectively those "configs" are what the eSDK is, you're just proposing
> a server side set of them rather than a built copy. In many ways that
> could be useful way of possibly rethinking the way the eSDK works.
>
> So in that sense I like the idea, we should just be clear the problem
> it solves as it doesn't solve the "why is my build not using the cache"
> question specifically but it potentially solves other issues.

Right, I didn't make it clear that this idea is not about 'why sstate
is not being reused', but rather something I thought of to avoid
placing people in that situation to begin with - offering them
'pre-fabricated builds' with guaranteed sstate availability. I have a
hunch there'll be plenty, for whom this is just fine, and eventually
this could be the basis of that oe-setup holy grail :)

I'll start with the simplest enhancement of printing what would be
rebuilt. There's a lot of code I need to understand, so this needs to
be in manageable chunks :)

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1758): 
https://lists.openembedded.org/g/openembedded-architecture/message/1758
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > For the task signatures, we need to think about some questions. If I
> > make a change locally, can I query how much will rebuild and how much
> > will be reused? There is bitbake --dry-run but perhaps it is time for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps should
> > define some existing scenarios where it is currently very difficult to
> > work this out and then work out how we can report that information to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time.

Right, what we lack is a way for the user to ask this and see the
result easily. As you say, bitbake can do this already.

>  Is there something else that's easily done and useful to print?

I think there is also the scenario of:
"I've run a build and have an existing TMPDIR and stamp info. I've now
pulled in a change. How much is going to rebuild and more importantly
*why*?"

This is different to a remote sstate situation as you have the stamp
info of the previous build already there to compare against.

> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 
> Existing cache objects are not suitable because:
>  was built on  and has a mismatching SRCREV
>  was built on  and has a different do_compile()

In theory you can do an:

ls 
sstate/*/*/sstate:curl:corei7-64-poky-linux:::corei7-64:10:*_package.tar.zst

and get a list of possible objects. Some key information was put into
the cache file names for this reason. Obviously this becomes much
trickier when the sstate is remote over http though.

"newest" is problematic in this context as you really want the closest
match. There is no concept of build date in the cache as it often isn't
relevant (building an old release for example). The only point a
date/time is used is for cleaning out the cache for files which haven't
been accessed in a long time.

The tools are already supposed to support doing this with local file
sstate sources, they just do a bad job at getting the diffs right. One
intent of this work item was to try and understand why they don't work
and address that so at least for filesystem sstate mirrors, you can get
better results. I don't know how we solve the remote http issue as yet.

> > One of the big problems in the past was that we lost much of the hash
> > information after parsing completed. This meant that if the hashes then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain more of the information
> > so that we didn't have to recompute it every time to be able to do
> > processing with it. I have to admit I've totally lost track of where I
> > got to with that.
> 
> Here's an idea I can't get out of my head. Right now, the cache is
> simply an amorphous mass of objects, with no information regarding how
> they were created. How about storing complete build confgurations as
> well into the same directory? There would be a dedicated, separate
> area for each configuration that placed objects into the cache,
> containing:
> - list of layers and revisions
> - config template used
> - complete content of build/conf
> - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
> - complete list of sstate objects that were produced as a result, so
> they can be checked for existence
> 
> This would be written into the cache dir at the very end of the build
> when everything else is already there.

I'm not sure this helps as much as you'd like. For example I build
core-image-sato-sdk on the autobuilder and populate this but you want
to build core-image-sato locally. There would be no info here that
would help with information about what core-image-sato would need,
despite the fact we know one is a derivative of the other and there
would be a near full cache for it.

Taking your example above and curl:do_package, how would the list of
information above tell us 

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
 wrote:
> For the task signatures, we need to think about some questions. If I
> make a change locally, can I query how much will rebuild and how much
> will be reused? There is bitbake --dry-run but perhaps it is time for a
> an option (or dedicated separate command?) to give some statistics
> about what bitbake would do? How much sstate would be reused?
>
> That then logically leads into the questions, can we tell what has
> changed? Why isn't my sstate being reused? For that we perhaps should
> define some existing scenarios where it is currently very difficult to
> work this out and then work out how we can report that information to
> the user. These could become test cases?

So I think there are two questions here that the tools should answer:

1. If I would run a build, what would be missing in the cache and need
to be built? The missing cache objects are in a dependency hierarchy,
so only those missing objects with no dependecies on other missing
objects would be printed. That should be comparatively easy to add as
bitbake already does those checks all the time. Is there something
else that's easily done and useful to print?

2. Then there's the question of *why* they are missing, which is
harder to answer. If, say, curl:do_package is not in the cache, then
the tool would have to walk the cache tree (I/O heavy operation as
there is no index), make a list of all curl:do_package objects that
are there, and do a recursive bitbake-diffsig (going up the task tree)
on them vs the one we want. Then print them starting with the newest.
Something like:

Existing cache objects are not suitable because:
 was built on  and has a mismatching SRCREV
 was built on  and has a different do_compile()

> One of the big problems in the past was that we lost much of the hash
> information after parsing completed. This meant that if the hashes then
> didn't match, we couldn't tell why as the original computation was
> lost. I did some work on allowing us to retain more of the information
> so that we didn't have to recompute it every time to be able to do
> processing with it. I have to admit I've totally lost track of where I
> got to with that.

Here's an idea I can't get out of my head. Right now, the cache is
simply an amorphous mass of objects, with no information regarding how
they were created. How about storing complete build confgurations as
well into the same directory? There would be a dedicated, separate
area for each configuration that placed objects into the cache,
containing:
- list of layers and revisions
- config template used
- complete content of build/conf
- bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
- complete list of sstate objects that were produced as a result, so
they can be checked for existence

This would be written into the cache dir at the very end of the build
when everything else is already there.

Right now, everyone sets up their own builds first, then points
local.conf or site.conf to the cache, and hopes for the best regarding
hit rates. Having stored build configs would allow inverting the
workflow, so that you first ask from the cache what it can provide
(e.g. it can provide mickledore or kirkstone core-image-minimal for
qemux86, and that's exactly what you want as a starting point), then
you use the build config stored in the cache to set up a build, and
run it - and that would guarantee complete sstate reuse and getting to
a functional image as soon as possible. Kind of like binary distro,
but implemented with sstate.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1753): 
https://lists.openembedded.org/g/openembedded-architecture/message/1753
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote:
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> > Alexander Kanavin will be working on the core workflow topic
> 
> I am now ready to start doing this, but before I do, I'd like to
> decompose the subject into manageable tasks with a bit of help from RP
> and the community:
> 
> https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/
> 
> 
> Core Workflow – Process Improvements
> 
> Background
> 
> The project builds everything from source by default. This means it
> has a reputation for being slow and heavy. There are ways the project
> can accelerate this which means faster workflows and improved
> developer experience but these are currently cumbersome to use.
> 
> Rationale
> 
> The project aims to defragment customised embedded Linux. This is
> important as if we succeed at this, it gives benefits to the wider
> ecosystem through making it easier to inject security fixes and an
> ability to share and collaborate without re-inventing the wheel.
> 
> To do this, we need to provide best-in-class support and compete
> against binary distributions for usability and speed. One way we can
> do this is provide better support for binary artifacts via our sstate
> mechanism. We do already have some of this functionality in our
> “extensible SDK” or “eSDK”.
> 
> Deliverables
> 
> Enable a public sstate mirror via a content delivery network (CDN) and
> populate using the autobuilder
> Ensure CDN sstate is reused under normal use case scenarios,
> particularly for slow components like rust-native. Identify any common
> sstate mismatch causes. Ensure test cases are added to cover the use
> cases and prevent regressions.
> Add lock and unlock commands to allow specific components to be locked
> down to specific sstate checksums or allow them to vary
> Allow switching between eSDK and non-eSDK modes
> Add tooling so we can understand why something is rebuilding when it
> isn’t expected to.
> ===
> 
> So: where to start? Do we need to 'design' something, or maybe time
> should go directly into addressing specific sore points? All feedback
> welcome.
> 
> There's also an unfinished patchset for adding bblock/bbunlock, which
> I would prefer to *not* 'take over and finish' but rather see the
> author get it merged:
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

To start with I'll try and write down and give a random walk through my
thoughts.

Certainly we need to get something like that patchset over the line. I
think it was blocked on a reply to this email:

https://lists.openembedded.org/g/openembedded-core/message/186497

which has suffered whilst I was pulled into the qemuppc mess. I feel
bad for not getting to a reply to that.

There are design elements to this work. We need to work out how we can
make eSDK and "normal" builds more similar and less of an overhead to
switch between one and the other. A "bblock all" command does partly
get you to an eSDK effectively so part of this may be switching eSDK to
use the new lock command. What other differences are there? What other
differences are necessary or make sense for the use cases eSDK was
designed for? How would you turn an existing build into an eSDK like
one? Could you provide a copy of a local build to someone else easily
using something like eSDK's tooling? What does the eSDK look like at
the end of this. One section we don't have good answers to yet is setup
and configuration although I know you've started on some of that.

For the task signatures, we need to think about some questions. If I
make a change locally, can I query how much will rebuild and how much
will be reused? There is bitbake --dry-run but perhaps it is time for a
an option (or dedicated separate command?) to give some statistics
about what bitbake would do? How much sstate would be reused?

That then logically leads into the questions, can we tell what has
changed? Why isn't my sstate being reused? For that we perhaps should
define some existing scenarios where it is currently very difficult to
work this out and then work out how we can report that information to
the user. These could become test cases?

One of the big problems in the past was that we lost much of the hash
information after parsing completed. This meant that if the hashes then
didn't match, we couldn't tell why as the original computation was
lost. I did some work on allowing us to retain more of the information
so that we didn't have to recompute it every time to be able to do
processing with it. I have to admit I've totally lost track of where I
got to with that.

Michael Halstead will be working on setting the CDN up so I'll let him
comment on when we'll have things ready for testing with that. We do
already have sstate shared from the autobuilder so some basic tests to
make sure our "base" shared files do work as expected is something
which can happen there already.

Another interesting 

[Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> Alexander Kanavin will be working on the core workflow topic

I am now ready to start doing this, but before I do, I'd like to
decompose the subject into manageable tasks with a bit of help from RP
and the community:

https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/


Core Workflow – Process Improvements

Background

The project builds everything from source by default. This means it
has a reputation for being slow and heavy. There are ways the project
can accelerate this which means faster workflows and improved
developer experience but these are currently cumbersome to use.

Rationale

The project aims to defragment customised embedded Linux. This is
important as if we succeed at this, it gives benefits to the wider
ecosystem through making it easier to inject security fixes and an
ability to share and collaborate without re-inventing the wheel.

To do this, we need to provide best-in-class support and compete
against binary distributions for usability and speed. One way we can
do this is provide better support for binary artifacts via our sstate
mechanism. We do already have some of this functionality in our
“extensible SDK” or “eSDK”.

Deliverables

Enable a public sstate mirror via a content delivery network (CDN) and
populate using the autobuilder
Ensure CDN sstate is reused under normal use case scenarios,
particularly for slow components like rust-native. Identify any common
sstate mismatch causes. Ensure test cases are added to cover the use
cases and prevent regressions.
Add lock and unlock commands to allow specific components to be locked
down to specific sstate checksums or allow them to vary
Allow switching between eSDK and non-eSDK modes
Add tooling so we can understand why something is rebuilding when it
isn’t expected to.
===

So: where to start? Do we need to 'design' something, or maybe time
should go directly into addressing specific sore points? All feedback
welcome.

There's also an unfinished patchset for adding bblock/bbunlock, which
I would prefer to *not* 'take over and finish' but rather see the
author get it merged:
https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

Thanks,
Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1751): 
https://lists.openembedded.org/g/openembedded-architecture/message/1751
Mute This Topic: https://lists.openembedded.org/mt/101356419/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-