Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 13:52, Alexander Kanavin  wrote:
>
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> > Alexander Kanavin will be working on the core workflow topic

I thought I'd write a summary of where we are with these subjects, and
make a plan for what needs to be done still:

1. Tools for figuring out what needs to be rebuilt and tests for them.

There was a large amount of corner cases to chase here, and fixing all
of that took a bit of time. I'm glad to report that all known issues
have been addressed, all patches have landed in master, and hopefully
this topic doesn't need further attention for now.

2. CDN sstate cache mirror and tests for it.

The tests for the mirror have landed as well, and there is an ongoing
issue that they expose: the CDN server isn't 100% robust. Sometimes it
returns a 5xx error, sometimes simply times out without responding.
This happens particularly with large objects, e.g. 50-60 Mb in size.
The problems are tracked in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15335 - hopefully
Michael Halstead can take a look soon-ish.

3. Build replicator tool.

I've written a proof of concept:
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all

There was positive feedback for the idea, so next I'll add tests and
submit it as a proper patch aiming for inclusion.

4. Any other business.

At this point I don't have any other subjects in mind, but if you do,
please write them here. Ideas, or maybe I missed something super
important.

Appendix.

There's also the oe-setup-build tool in development, which isn't
strictly a part of the core workflow topic, but is related, and I'd
like to get it to completion as well:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/setup-layers-tweaks
I'll check if I had missed any feedback, and re-submit.

Alex


Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194140): 
https://lists.openembedded.org/g/openembedded-core/message/194140
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Sun, 5 Nov 2023 at 20:43,  wrote:
> Another topic where additional meta data about the sstate-cache seams
> to be beneficial is sstate-mirror retention. Knowing which artifact was
> compiled for which tag or commit of the bitbake layer could help to
> wipe out some artifacts which are not needed anymore.

There should be progress on this particular point soon when I get
oe-replicate-build prototype to function. It also records the list of
needed sstate objects into the replica bundle (in the form of
locked-sigs.inc, which is in itself not included into the build conf),
and if you place that bundle next to sstate, and make a superset of
all the objects in all bundles registered that way, you have a list of
things that can be pruned from the cache without loss of build times.
There could also be more interesting use cases, like querying the
sstate server for available builds with guaranteed sstate coverage
etc.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190206): 
https://lists.openembedded.org/g/openembedded-core/message/190206
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Adrian Freihofer]

On Sat, 2023-11-04 at 11:09 +, Richard Purdie wrote:
> On Sat, 2023-11-04 at 11:29 +0100, adrian.freiho...@gmail.com wrote:
> > Hi Alex, hi Richard
> > 
> > After some internal discussions, I would like to clarify my
> > previous
> > answers on this topic.
> > 
> >  * Usually there are two different workflows
> >     - application developers: could use an SDK with a locked
> > sstate-cache.
> >     - Yocto/BSP developers: need an unlocked SDK. They change the
> > recipes.
> >  * A locked SDK
> >     - can work with setscene from SSTATE_MIRRORS
> >     - setscene does caching in the SSTATE_DIR (no issue about that)
> >     - But network problems can occur during the initial build
> > because
> >   bitbake executes many independent setscene tasks. Opening so
> > many
> >   independent connections slows down the build, especially if
> > the
> >   server treats them as a denial of service attack.
> >     - The denial of service problem is difficult to solve because
> > each
> >   setscene task runs in its own bibtake task. Reusing a
> > connection to
> >   download multiple sstate artifacts seems almost impossible.
> >   This is much easier to solve with separate sstate download
> > script.
> 
> FWIW, we did have a similar issue with do_fetch overloading
> servers/proxies/ISPs and added:
> 
> do_fetch[number_threads] = "4"
> 
> Finding the right place to put a thread limit on overall setscene
> tasks
> is harder but in theory possible. Or perhaps a "network capable
> tasks"
> thread limit?
> 
> Is the overload caused by the initial query of sstate presence, or,
> does it happen when the setscene tasks themselves run?

The most extreme situation is probably bitbake --setscene-only with an
empty TMPDIR. Each of the setscene tasks establishes a new connection.
A server receives so many connections that it treats them as a denial
of service attack by throttling. A separate script would allow the same
connection to be reused to download all the required artifacts.
Limiting the number of threads does not really solve the issue because
there are still the same amount of connections which get quickly
opened.

> 
> 
> >  * An unlocked SDK
> >     - Tries to download the sstate cache for changed recipes and
> > their
> >   dependencies, which obviously can't work.
> >     - The useless download requests slow down the build
> > considerably and
> >   cause a high load on the servers without any benefit.
> 
> Is this sstate over http(s) or something else? I seem to remember you
> mentioning sftp. If this were using sftp, it would be horribly slow
> as
> it was designed for a light overhead "does this exist?" check which
> http(s) can manage well.

Yes, we are evaluating sftp. You are right, it is not optimal from a
performance point of view. For example S3 is much faster. A compromise
is to set up a limited number of parallel sftp connections. This has
worked very well so far.

The question of why we use sftp brings us to a larger topic that is
probably relevant for almost all Yocto users, but not for the Yocto
project itself: Security.

There is usually a git server infrastructure that makes it possible to
protect Git repositories with finely graded access policies. As the
sstate-cache contains the same source code, the protection concept for
the Git repositories must also be applied to the sstate-cache
artifacts.

First of all a user authentication is required for the sstate-mirror.
An obvious idea is to use the same user authentication for the sstate-
cache server as for the Git server. In addition to https, ssh is also
often used for git repositories. SSH even offers some advantages in
terms of user-friendliness and security (if a ssh agent is used). This
consideration finally leads us to use the sftp protocol for the sstate
mirror. This is also relatively easy to administer: Simply copy the
user's public ssh keys from the git server to the sftp server.

If one then wants to scale an sstate-cache server for many different
projects and users, one quickly wishes for an option for authorization
at artifact level. Ideally, the access rights to the source code would
be completely transferred to the associated sstate artifacts. For such
an authorization the ssate mirror server would require the SRC_URI
which was used to compile the sstate artifact. With this information,
it could ask the Git server whether or not a user has access to all
source code repositories to grant or deny access to a particular sstate
artifact. It should not be forgotten that the access rights to the Git
repositories can change.

> 
> Recently we've been wondering about teaching the hashequiv server
> about
> "presence", which would then mean the build would only query things
> that stood a good chance of existing.
> 
Yes, that sound very interesting. There are probably even more such
kind of meta data which could be provided by the hashserver to improve
the management of a shared sstate mirror.

Would it make 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Sat, 2023-11-04 at 11:29 +0100, adrian.freiho...@gmail.com wrote:
> Hi Alex, hi Richard
> 
> After some internal discussions, I would like to clarify my previous
> answers on this topic.
> 
>  * Usually there are two different workflows
> - application developers: could use an SDK with a locked sstate-cache.
> - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
>  * A locked SDK
> - can work with setscene from SSTATE_MIRRORS
> - setscene does caching in the SSTATE_DIR (no issue about that)
> - But network problems can occur during the initial build because
>   bitbake executes many independent setscene tasks. Opening so many
>   independent connections slows down the build, especially if the
>   server treats them as a denial of service attack.
> - The denial of service problem is difficult to solve because each
>   setscene task runs in its own bibtake task. Reusing a connection to
>   download multiple sstate artifacts seems almost impossible.
>   This is much easier to solve with separate sstate download script.

FWIW, we did have a similar issue with do_fetch overloading
servers/proxies/ISPs and added:

do_fetch[number_threads] = "4"

Finding the right place to put a thread limit on overall setscene tasks
is harder but in theory possible. Or perhaps a "network capable tasks"
thread limit?

Is the overload caused by the initial query of sstate presence, or,
does it happen when the setscene tasks themselves run?


>  * An unlocked SDK
> - Tries to download the sstate cache for changed recipes and their
>   dependencies, which obviously can't work.
> - The useless download requests slow down the build considerably and
>   cause a high load on the servers without any benefit.

Is this sstate over http(s) or something else? I seem to remember you
mentioning sftp. If this were using sftp, it would be horribly slow as
it was designed for a light overhead "does this exist?" check which
http(s) can manage well.

Recently we've been wondering about teaching the hashequiv server about
"presence", which would then mean the build would only query things
that stood a good chance of existing.

> - A script which gets a list of sstate artifacts from bitbake and then
>   does a upfront download works much better
>+ The script runs only when the user calls it or the SDK gets boot-
>  strapped
>+ The script uses a reasonable amount of parallel connections which
>  are re-used for more then one artifact download

Explaining to users they need to do X before Y quickly gets tiring,
both for people explaining it and the people doing it trying to
remember. I'd really like to get to a point where the system "does the
right thing" if we can.

I don't believe the problems you describe are insurmountable. If you
are using sftp, that is going to be a big chunk of the problem as the
system assumes something faster is available. Yes, I've taken patches
to make sftp work but it isn't recommended at all. I appreciate there
would be reasons why you use sftp but if it is possible to get a list
of "available sstate" via other means, it would improve things.

>  * Idea for a smart lock/unlock implementation
> - Form a user's perspective a locked vs. an unlocked SDK does not make
>   much sense. It makes more sense if the SDK would automatically
>   download the sstate-cache if it is expected to be available.
>   Lets think about an implementation (which allows to override the
>   logic) to switch from automatic to manual mode:
>   
>   SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"

What determines this availability? I worry that is something very
fragile and specific to your use case. It is also not an all or nothing
binary thing.

>   In our case the sstate mirror is expected to provide all artifacts
>   for tagged commits and for some git branches of the layer
>   repositories.
>   The sstate is obviousely not usable for a "dirty" git layer
>   repository.

That isn't correct and isn't going to work. If I make a single change
locally, there is a good chance that 99.9% of the sstate could still be
valid in some cases. Forcing the user through 10 hours of rebuild when
potentially that much was available is a really really bad user
experience.

>  That's what the is_sstate_mirror_available function
>   could check to automatically enable and disable lazy downloads.
>   
> - If is_sstate_mirror_available() returns false, it should still be
>   possible to initiate a sstate-cache download manually.
>   
>  * Terminology
> - Older Yocto Releases:
>+ eSDK means an installer which provides a different environment with
>  different tools
>+ The eSDK was static, with a locked sstate cache
>+ Was for one MACHINE, for one image...
> - Newer Yocto Releases:
>+ The bitbake environment offers all 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Adrian Freihofer]

Hi Alex, hi Richard

After some internal discussions, I would like to clarify my previous
answers on this topic.

 * Usually there are two different workflows
- application developers: could use an SDK with a locked sstate-cache.
- Yocto/BSP developers: need an unlocked SDK. They change the recipes.
 * A locked SDK
- can work with setscene from SSTATE_MIRRORS
- setscene does caching in the SSTATE_DIR (no issue about that)
- But network problems can occur during the initial build because
  bitbake executes many independent setscene tasks. Opening so many
  independent connections slows down the build, especially if the
  server treats them as a denial of service attack.
- The denial of service problem is difficult to solve because each
  setscene task runs in its own bibtake task. Reusing a connection to
  download multiple sstate artifacts seems almost impossible.
  This is much easier to solve with separate sstate download script.
 * An unlocked SDK
- Tries to download the sstate cache for changed recipes and their
  dependencies, which obviously can't work.
- The useless download requests slow down the build considerably and
  cause a high load on the servers without any benefit.
- A script which gets a list of sstate artifacts from bitbake and then
  does a upfront download works much better
   + The script runs only when the user calls it or the SDK gets boot-
 strapped
   + The script uses a reasonable amount of parallel connections which
 are re-used for more then one artifact download
 * Idea for a smart lock/unlock implementation
- Form a user's perspective a locked vs. an unlocked SDK does not make
  much sense. It makes more sense if the SDK would automatically
  download the sstate-cache if it is expected to be available.
  Lets think about an implementation (which allows to override the
  logic) to switch from automatic to manual mode:
  
  SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
  
  In our case the sstate mirror is expected to provide all artifacts
  for tagged commits and for some git branches of the layer
  repositories.
  The sstate is obviousely not usable for a "dirty" git layer
  repository. That's what the is_sstate_mirror_available function
  could check to automatically enable and disable lazy downloads.
  
- If is_sstate_mirror_available() returns false, it should still be
  possible to initiate a sstate-cache download manually.
  
 * Terminology
- Older Yocto Releases:
   + eSDK means an installer which provides a different environment with
 different tools
   + The eSDK was static, with a locked sstate cache
   + Was for one MACHINE, for one image...
- Newer Yocto Releases:
   + The bitbake environment offers all features of the eSDK installer. I
 consider this as already implemented with meta-ide-support and
 build-sysroots.
   + The term eSDK means a replicable bitbake environment. (The
 documentation was recently changed in that sense)
   + The new SDK installer can be generated in different variants similar
 to what was already supported by the eSDK
 installer: https://docs.yoctoproject.org/sdk-manual/appendix-
 customizing.html#customizing-the-extensible-sdk-standalone-
 installer.
  * The lightest variant is just a script that sets up the layers (git
clone or git checkout) and provides the build config. If
SSTATE_MIRRORS are configured lazy downloads will just work
otherwise bitbake will compile everything from scratch.
  * The heaviest variant of the SDK installer includes the layers and
the sstate-cache. After installing it is_sstate_mirror_available()
evaluates to True.
* devtool ide
- Tries to bring this ideas further towards IDE configuration
- It supports two modes: No recipe mode is like the old eSDK
environment. The recipe mode goes beyond that. It is based on
devtool modify.
- Naming: I was thinking about "devtool ide" versus "devtool esdk"
   + Why ide? I want to set up my IDE to work with the Yocto SDK. And
 that means, of course, that I have to bootstrap the SDK.
   + Could also be esdk: I want to bootstrap the eSDK and that should
 also configure my IDE.
 * Latest patch series is here: 
- https://lists.openembedded.org/g/openembedded-core/message/189899
- docs: https://lists.yoctoproject.org/g/docs/message/4578

Adrian


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190170): 
https://lists.openembedded.org/g/openembedded-core/message/190170
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Wed, 1 Nov 2023 at 15:18,  wrote:
> We are currently experimenting with replacing the eSDK installer with
> the bitbake build environment for our users. Part of this
> transformation is, of course, the shared sstate-cache, for which this
> discussion seems quite relevant. The workflow we are aiming for is as
> follows:
>
>1. Setup the layers and build config (out of scope here)
>2. Download the sstate for a particular recipe (usually an image
>   recipe).

I'm not sure I understand the case for downloading the complete set of
cache objects up front. What's wrong with 'lazy' downloading, e.g.
only when the objects are actually needed in a build?

Even then, does --setscene-only do just that, or am I completely confused?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190041): 
https://lists.openembedded.org/g/openembedded-core/message/190041
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Adrian Freihofer]

Hi Alex, hi Richard

The discussion looks really interesting. I would like to contribute
some comments from the point of view of a rather naive user and try to
understand the workflows for which these improvements would be
beneficial also on a bigger picture.


On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > For the task signatures, we need to think about some questions. If
> > I
> > make a change locally, can I query how much will rebuild and how
> > much
> > will be reused? There is bitbake --dry-run but perhaps it is time
> > for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps
> > should
> > define some existing scenarios where it is currently very difficult
> > to
> > work this out and then work out how we can report that information
> > to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and
> need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time. Is there something
> else that's easily done and useful to print?
> 
> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task
> tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 

We are currently experimenting with replacing the eSDK installer with
the bitbake build environment for our users. Part of this
transformation is, of course, the shared sstate-cache, for which this
discussion seems quite relevant. The workflow we are aiming for is as
follows:

   1. Setup the layers and build config (out of scope here)
   2. Download the sstate for a particular recipe (usually an image
  recipe).
  
  Note: Working with SSTATE_MIRRORS does not work very well because
  bitbake connects way too often to the sstate server. So we
  started to develop a script which downloads the sstate artifacts
  into SSTATE_DIR to get the SDK set up.
  
  Your point 1. is basically what our (a bit hacky) download script
  does in the --dry-run mode: Printing all the required sstate
  artifacts. I think as a first step, it would be very valuable to
  have a function or a tinfoil API in the core that returns a list
  of sstate artifacts for a given recipe.
  
  As a second step a tool or a new feature of an existing tool
  could download the artifacts into SSTATE_DIR. This would also be
  a great successor for the not so much maintained devtool sdk-
  update command.
   3. At this point, the user is basically in the same situation as
  after installing the eSDK installer. devtool and now also bitbake
  are available, the sstate-cache is fully populated.

Maintaining such an SDK and sstate mirror infrastructure brings us to
your point 2. Tooling for maintaining the sstate cache becomes even
more important than it is now. Also, locking the sstate cache and
treating missing artifacts as errors seems to be an important feature. 

But I would consider the locking/unlocking more relevant for testing
rather than for deploying locked SDKs. Unlocked SDKs allow the user to
switch the branches of the layers to commits where the sstate is not
available. If the user is in a full featured bitbake environment rather
than the constrained eSDK installer environment this is perfectly fine.
Bitbake can just compile the missing recipes.
Such an SDK would combine all the advantages of the current eSDK
installer and the much more flexible Bitbake environment. This would
imply that the sstate download script should just try to download
what's available on the mirror and maybe print a warning for artifacts
which cannot be downloaded. But it should not abort with an error for
missing artifacts.

Does that make sense?

Regrads,
Adrian

> Existing cache objects are not suitable because:
>  was built on  and has a mismatching SRCREV
>  was built on  and has a different
> do_compile()
> 
> > One of the big problems in the past was that we lost much of the
> > hash
> > information after parsing completed. This meant that if the hashes
> > then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 31 Oct 2023 at 13:28, Richard Purdie
 wrote:

> > Then we can pull all of it together into 'devtool esdk '
> > command (or similar), which would enter the esdk environment directly
> > via:
> > - running 'bitbake  meta-ide-support'
> > - running the above mentioned bitbake local.conf task to generate the
> > esdk-specific local.conf
> > - sourcing the environment script produced by meta-ide-support
> > - rewriting PATH to provide only the curated esdk tools and not
> > everything plus bitbake.
> > - writing a custom devtool.conf similar to that of standalone esdk so
> > that devtool can find bitbake and bitbake can use the esdk-specific
> > local.conf
> >
> > And it would be tested in the same way standalone esdks are.
> >
> > Thoughts? Anything missing from the above list?
>
> That sounds like a good way to handle this to me!

A couple followup points:

- copy_buildsystem() in populate_sdk_ext class is overly long at 400
lines and does many different barely related things, writing
local.conf for esdk one of them:

https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_ext.bbclass#n189

The first step would be to structure it into separate functions each
doing one thing and hopefully fitting on a single screen. Then these
functions can be hand-picked into a task designed to provide esdk
things in a yocto build context, and the code would simply be more
readable, as right now I can barely understand all the various things
that function does, and the spaghetti of local variables etc.

- the tool that would set up the esdk environment in a plain yocto
build could be a shell script in scripts/oe-init-esdk-env perhaps,
similar to oe-init-build-env. No need to make it a devtool plugin; it
could use python helpers to program the more tricky bits such as PATH
manipulations etc. So one would first initialize the yocto
environment, then transition to an esdk environment from that (there
would be no way to go back, as one can simply start a new session).

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189843): 
https://lists.openembedded.org/g/openembedded-core/message/189843
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Tue, 2023-10-31 at 13:08 +0100, Alexander Kanavin wrote:
> On Mon, 30 Oct 2023 at 16:02, Alexander Kanavin via
> lists.openembedded.org 
> wrote:
> > So here's what could be done:
> > 
> > - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
> > environment script puts that in PATH, rather than some custom
> > esdk-specific location (the code to generate that can then be
> > dropped).
> 
> This is now implemented (needs to be tested on AB).
> 
> > - esdk tweaks to local.conf move into a dedicated include file, which
> > can be static and under version control, except for perhaps
> > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
> > I need to check why is it there and how that is used.
> 
> It's actually more complicated. The code to generate esdk-specific
> local.conf with all the tweaks has too much dynamic stuff in it which
> is subject to what various variables are set to. So I'm thinking of
> extracting that to a dedicated function, then attaching a bitbake task
> to that function.
> 
> Then we can pull all of it together into 'devtool esdk '
> command (or similar), which would enter the esdk environment directly
> via:
> - running 'bitbake  meta-ide-support'
> - running the above mentioned bitbake local.conf task to generate the
> esdk-specific local.conf
> - sourcing the environment script produced by meta-ide-support
> - rewriting PATH to provide only the curated esdk tools and not
> everything plus bitbake.
> - writing a custom devtool.conf similar to that of standalone esdk so
> that devtool can find bitbake and bitbake can use the esdk-specific
> local.conf
> 
> And it would be tested in the same way standalone esdks are.
> 
> Thoughts? Anything missing from the above list?

That sounds like a good way to handle this to me!

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189839): 
https://lists.openembedded.org/g/openembedded-core/message/189839
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 30 Oct 2023 at 16:02, Alexander Kanavin via
lists.openembedded.org 
wrote:
> So here's what could be done:
>
> - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
> environment script puts that in PATH, rather than some custom
> esdk-specific location (the code to generate that can then be
> dropped).

This is now implemented (needs to be tested on AB).

> - esdk tweaks to local.conf move into a dedicated include file, which
> can be static and under version control, except for perhaps
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
> I need to check why is it there and how that is used.

It's actually more complicated. The code to generate esdk-specific
local.conf with all the tweaks has too much dynamic stuff in it which
is subject to what various variables are set to. So I'm thinking of
extracting that to a dedicated function, then attaching a bitbake task
to that function.

Then we can pull all of it together into 'devtool esdk '
command (or similar), which would enter the esdk environment directly
via:
- running 'bitbake  meta-ide-support'
- running the above mentioned bitbake local.conf task to generate the
esdk-specific local.conf
- sourcing the environment script produced by meta-ide-support
- rewriting PATH to provide only the curated esdk tools and not
everything plus bitbake.
- writing a custom devtool.conf similar to that of standalone esdk so
that devtool can find bitbake and bitbake can use the esdk-specific
local.conf

And it would be tested in the same way standalone esdks are.

Thoughts? Anything missing from the above list?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189837): 
https://lists.openembedded.org/g/openembedded-core/message/189837
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 30 Oct 2023 at 15:07, Richard Purdie
 wrote:
> > a) esdk sets a number of variables from its initialization script that
> > aid with cross-compiling components directly (e.g. the core use case
> > of SDKs). Normal mode doesn't do that, but recently added
> > meta-ide-support will generate a similar initialization script that
> > will set up the same environment from the normal mode. There are tests
> > and documentation for it.
>
> In that case, this one is something we can document as how to make the
> functionality available in the normal build.

This has been documented in the sdk manual:
https://docs.yoctoproject.org/sdk-manual/extensible.html#two-ways-to-install-the-extensible-sdk
https://docs.yoctoproject.org/sdk-manual/extensible.html#installing-additional-items-into-the-extensible-sdk

Actually, I have on purpose presented both options in the manual as
'esdk', the standalone installer option is simply locked down more
(task signatures and tooling wise) than the direct yocto build.

So here's what could be done:

- esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
environment script puts that in PATH, rather than some custom
esdk-specific location (the code to generate that can then be
dropped).

- esdk tweaks to local.conf move into a dedicated include file, which
can be static and under version control, except for perhaps
METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
I need to check why is it there and how that is used.

Something else perhaps?

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189798): 
https://lists.openembedded.org/g/openembedded-core/message/189798
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > There are design elements to this work. We need to work out how we can
> > make eSDK and "normal" builds more similar and less of an overhead to
> > switch between one and the other. A "bblock all" command does partly
> > get you to an eSDK effectively so part of this may be switching eSDK to
> > use the new lock command. What other differences are there? What other
> > differences are necessary or make sense for the use cases eSDK was
> > designed for? How would you turn an existing build into an eSDK like
> > one? Could you provide a copy of a local build to someone else easily
> > using something like eSDK's tooling? What does the eSDK look like at
> > the end of this. One section we don't have good answers to yet is setup
> > and configuration although I know you've started on some of that.
> 
> So I see the following differences between esdk and normal modes:
> 
> 1. Environment and tooling availability.
> 
> a) esdk sets a number of variables from its initialization script that
> aid with cross-compiling components directly (e.g. the core use case
> of SDKs). Normal mode doesn't do that, but recently added
> meta-ide-support will generate a similar initialization script that
> will set up the same environment from the normal mode. There are tests
> and documentation for it.

In that case, this one is something we can document as how to make the
functionality available in the normal build.

> b) PATH. eSDK has a number of items in PATH that point to various
> locations inside tmp/sysroots/, collectively they provide the
> cross-toolchain.
> 
> eSDK also puts a selection of yocto tools into path - wic, devtool but
> not bitbake:
> 
> 
> alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
> total 48
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
> ../../../../layers/poky/scripts/devtool
> lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
> ../../../../layers/poky/scripts/oe-find-native-sysroot
> lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
> ../../../../layers/poky/scripts/recipetool
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
> ../../../../layers/poky/scripts/runqemu
> lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
> ../../../../layers/poky/scripts/runqemu-addptable2image
> lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
> ../../../../layers/poky/scripts/runqemu-export-rootfs
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
> ../../../../layers/poky/scripts/runqemu-extract-sdk
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
> ../../../../layers/poky/scripts/runqemu-gen-tapdevs
> lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
> ../../../../layers/poky/scripts/runqemu-ifdown
> lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
> ../../../../layers/poky/scripts/runqemu-ifup
> lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
> lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
> ../../../../layers/poky/scripts/wic
> ==
> 
> 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
> Cross-toolchain can be added by the same environment script made by
> meta-ide-support as mentioned in 1a.

Right, so in theory we can change PATH and change this which can also
easily be documented.

> 2. Configuration (e.g. local.conf).
> 
> eSDK local.conf is local.conf from the normal mode that was used to
> produce eSDK, stripped of all comments, and with a bunch of extra
> settings:
> 
> 
> INHERIT:remove = "buildhistory icecc"
> CONNECTIVITY_CHECK_URIS = ""
> 
> SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
> 
> SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
> 
> BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
> 
> BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
> *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
> 
> BUILDCFG_HEADER = ""
> 
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
> 
> # Provide a flag to indicate we are in the EXT_SDK Context
> WITHIN_EXT_SDK = "1"
> 
> SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
> file://universal-4.9/(.*) file://universal-4.8/\1"
> 

Perhaps some of this should become a generic include file which is
included which would then make it easy to document as how to lock
things down to match it? That would take the config out of the class
file too which is probably nicer.

> require conf/locked-sigs.inc
> require conf/unlocked-sigs.inc
> ===
> 
> Devtool also has a special configuration:
> =
> alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
> [General]
> bitbake_subdir = layers/poky/bitbake
> init_path = layers/poky/oe-init-build-env
> 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
 wrote:
> There are design elements to this work. We need to work out how we can
> make eSDK and "normal" builds more similar and less of an overhead to
> switch between one and the other. A "bblock all" command does partly
> get you to an eSDK effectively so part of this may be switching eSDK to
> use the new lock command. What other differences are there? What other
> differences are necessary or make sense for the use cases eSDK was
> designed for? How would you turn an existing build into an eSDK like
> one? Could you provide a copy of a local build to someone else easily
> using something like eSDK's tooling? What does the eSDK look like at
> the end of this. One section we don't have good answers to yet is setup
> and configuration although I know you've started on some of that.

So I see the following differences between esdk and normal modes:

1. Environment and tooling availability.

a) esdk sets a number of variables from its initialization script that
aid with cross-compiling components directly (e.g. the core use case
of SDKs). Normal mode doesn't do that, but recently added
meta-ide-support will generate a similar initialization script that
will set up the same environment from the normal mode. There are tests
and documentation for it.

b) PATH. eSDK has a number of items in PATH that point to various
locations inside tmp/sysroots/, collectively they provide the
cross-toolchain.

eSDK also puts a selection of yocto tools into path - wic, devtool but
not bitbake:


alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
total 48
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
../../../../layers/poky/scripts/devtool
lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
../../../../layers/poky/scripts/oe-find-native-sysroot
lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
../../../../layers/poky/scripts/recipetool
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
../../../../layers/poky/scripts/runqemu
lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
../../../../layers/poky/scripts/runqemu-addptable2image
lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
../../../../layers/poky/scripts/runqemu-export-rootfs
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
../../../../layers/poky/scripts/runqemu-extract-sdk
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
../../../../layers/poky/scripts/runqemu-gen-tapdevs
lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
../../../../layers/poky/scripts/runqemu-ifdown
lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
../../../../layers/poky/scripts/runqemu-ifup
lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
../../../../layers/poky/scripts/wic
==

'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
Cross-toolchain can be added by the same environment script made by
meta-ide-support as mentioned in 1a.

2. Configuration (e.g. local.conf).

eSDK local.conf is local.conf from the normal mode that was used to
produce eSDK, stripped of all comments, and with a bunch of extra
settings:


INHERIT:remove = "buildhistory icecc"
CONNECTIVITY_CHECK_URIS = ""

SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"

SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"

BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"

BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
*:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"

BUILDCFG_HEADER = ""

METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"

# Provide a flag to indicate we are in the EXT_SDK Context
WITHIN_EXT_SDK = "1"

SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
file://universal-4.9/(.*) file://universal-4.8/\1"

require conf/locked-sigs.inc
require conf/unlocked-sigs.inc
===

Devtool also has a special configuration:
=
alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
[General]
bitbake_subdir = layers/poky/bitbake
init_path = layers/poky/oe-init-build-env
core_meta_subdir = layers/poky/meta

[SDK]
sdk_targets = core-image-minimal
==
There is currently no tooling to add/remove these extras in either
esdk mode or normal mode as far as I understand. Their individual
purposes and effects are also not exactly clear to me, and need to be
investigated one by one.

3. Setting up a normal mode in a eSDK installation.

This is actually pretty easy: rather than sourcing the sdk environment
script, source the poky/oe-init-build-env:
==
alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .

### Shell environment set up for builds. ###
...
===
Bitbake will then simply run, albeit with all those extra
esdk-specific items in 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
 wrote:
> The tools are already supposed to support doing this with local file
> sstate sources, they just do a bad job at getting the diffs right. One
> intent of this work item was to try and understand why they don't work
> and address that so at least for filesystem sstate mirrors, you can get
> better results. I don't know how we solve the remote http issue as yet.

I ran a few experiments with bitbake -S printdiff, and so far I
haven't yet seen an unhelpful result. Here's the most impressive one:

$ bitbake libsolv-native (uses cmake-native)
$ git revert 
$ rm -rf build/tmp/
$ bitbake -S printdiff libsolv-native
...
The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-native_3.27.4.bb:do_recipe_qa
NOTE: Writing task signature files
Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc

Task cmake-native:do_recipe_qa couldn't be used from the cache because:
  We need hash 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb,
closest matching task was
85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7
  basehash changed from
7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to
7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09
  Variable PV value changed from '3.27.5' to '3.27.4'
  Variable SRC_URI[sha256sum] value changed from
'5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' to
'0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af'

That's pretty good, isn't it? It does print both what needs to be
re-run, and *why* as well.

I have no idea yet what kind of magic it does to find the 'closest
matching task' in sstate, but if this breaks down in some other
scenarios, we need to find them to get a starting point for making the
tools better. Ideas? I'm ready to try them :)

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187989): 
https://lists.openembedded.org/g/openembedded-core/message/187989
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Unless I missed something, I think I fixed all comments from RP on the
branch I force pushed :)

Cheers
Julien

Le mer. 20 sept. 2023 à 16:31, Alexander Kanavin
 a écrit :
>
> That’s great to hear! When you have a new patch set please post it here. I 
> think there’s also a new reply from RP that you need to consider.
>
> Alex
>
> On Wed 20. Sep 2023 at 16.25, Julien Stephan  wrote:
>>
>> Hi Alexander,
>> About bblock tool: I just force pushed my branch on poky-contrib
>> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
>> trying to fix an autobuilder issue reported by Alexandre Belloni.
>> I am still working on this, and I would be very happy to get this merged :)
>>
>> Cheers
>> Julien
>>
>> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
>>  a écrit :
>> >
>> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>> >  wrote:
>> >
>> > > Effectively those "configs" are what the eSDK is, you're just proposing
>> > > a server side set of them rather than a built copy. In many ways that
>> > > could be useful way of possibly rethinking the way the eSDK works.
>> > >
>> > > So in that sense I like the idea, we should just be clear the problem
>> > > it solves as it doesn't solve the "why is my build not using the cache"
>> > > question specifically but it potentially solves other issues.
>> >
>> > Right, I didn't make it clear that this idea is not about 'why sstate
>> > is not being reused', but rather something I thought of to avoid
>> > placing people in that situation to begin with - offering them
>> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
>> > hunch there'll be plenty, for whom this is just fine, and eventually
>> > this could be the basis of that oe-setup holy grail :)
>> >
>> > I'll start with the simplest enhancement of printing what would be
>> > rebuilt. There's a lot of code I need to understand, so this needs to
>> > be in manageable chunks :)
>> >
>> > Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187937): 
https://lists.openembedded.org/g/openembedded-core/message/187937
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

That’s great to hear! When you have a new patch set please post it here. I
think there’s also a new reply from RP that you need to consider.

Alex

On Wed 20. Sep 2023 at 16.25, Julien Stephan  wrote:

> Hi Alexander,
> About bblock tool: I just force pushed my branch on poky-contrib
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
> trying to fix an autobuilder issue reported by Alexandre Belloni.
> I am still working on this, and I would be very happy to get this merged :)
>
> Cheers
> Julien
>
> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
>  a écrit :
> >
> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
> >  wrote:
> >
> > > Effectively those "configs" are what the eSDK is, you're just proposing
> > > a server side set of them rather than a built copy. In many ways that
> > > could be useful way of possibly rethinking the way the eSDK works.
> > >
> > > So in that sense I like the idea, we should just be clear the problem
> > > it solves as it doesn't solve the "why is my build not using the cache"
> > > question specifically but it potentially solves other issues.
> >
> > Right, I didn't make it clear that this idea is not about 'why sstate
> > is not being reused', but rather something I thought of to avoid
> > placing people in that situation to begin with - offering them
> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
> > hunch there'll be plenty, for whom this is just fine, and eventually
> > this could be the basis of that oe-setup holy grail :)
> >
> > I'll start with the simplest enhancement of printing what would be
> > rebuilt. There's a lot of code I need to understand, so this needs to
> > be in manageable chunks :)
> >
> > Alex
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187927): 
https://lists.openembedded.org/g/openembedded-core/message/187927
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Hi Alexander,
About bblock tool: I just force pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
trying to fix an autobuilder issue reported by Alexandre Belloni.
I am still working on this, and I would be very happy to get this merged :)

Cheers
Julien

Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
 a écrit :
>
> On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>  wrote:
>
> > Effectively those "configs" are what the eSDK is, you're just proposing
> > a server side set of them rather than a built copy. In many ways that
> > could be useful way of possibly rethinking the way the eSDK works.
> >
> > So in that sense I like the idea, we should just be clear the problem
> > it solves as it doesn't solve the "why is my build not using the cache"
> > question specifically but it potentially solves other issues.
>
> Right, I didn't make it clear that this idea is not about 'why sstate
> is not being reused', but rather something I thought of to avoid
> placing people in that situation to begin with - offering them
> 'pre-fabricated builds' with guaranteed sstate availability. I have a
> hunch there'll be plenty, for whom this is just fine, and eventually
> this could be the basis of that oe-setup holy grail :)
>
> I'll start with the simplest enhancement of printing what would be
> rebuilt. There's a lot of code I need to understand, so this needs to
> be in manageable chunks :)
>
> Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187926): 
https://lists.openembedded.org/g/openembedded-core/message/187926
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
 wrote:

> Effectively those "configs" are what the eSDK is, you're just proposing
> a server side set of them rather than a built copy. In many ways that
> could be useful way of possibly rethinking the way the eSDK works.
>
> So in that sense I like the idea, we should just be clear the problem
> it solves as it doesn't solve the "why is my build not using the cache"
> question specifically but it potentially solves other issues.

Right, I didn't make it clear that this idea is not about 'why sstate
is not being reused', but rather something I thought of to avoid
placing people in that situation to begin with - offering them
'pre-fabricated builds' with guaranteed sstate availability. I have a
hunch there'll be plenty, for whom this is just fine, and eventually
this could be the basis of that oe-setup holy grail :)

I'll start with the simplest enhancement of printing what would be
rebuilt. There's a lot of code I need to understand, so this needs to
be in manageable chunks :)

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187661): 
https://lists.openembedded.org/g/openembedded-core/message/187661
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
>  wrote:
> > For the task signatures, we need to think about some questions. If I
> > make a change locally, can I query how much will rebuild and how much
> > will be reused? There is bitbake --dry-run but perhaps it is time for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps should
> > define some existing scenarios where it is currently very difficult to
> > work this out and then work out how we can report that information to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time.

Right, what we lack is a way for the user to ask this and see the
result easily. As you say, bitbake can do this already.

>  Is there something else that's easily done and useful to print?

I think there is also the scenario of:
"I've run a build and have an existing TMPDIR and stamp info. I've now
pulled in a change. How much is going to rebuild and more importantly
*why*?"

This is different to a remote sstate situation as you have the stamp
info of the previous build already there to compare against.

> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 
> Existing cache objects are not suitable because:
>  was built on  and has a mismatching SRCREV
>  was built on  and has a different do_compile()

In theory you can do an:

ls 
sstate/*/*/sstate:curl:corei7-64-poky-linux:::corei7-64:10:*_package.tar.zst

and get a list of possible objects. Some key information was put into
the cache file names for this reason. Obviously this becomes much
trickier when the sstate is remote over http though.

"newest" is problematic in this context as you really want the closest
match. There is no concept of build date in the cache as it often isn't
relevant (building an old release for example). The only point a
date/time is used is for cleaning out the cache for files which haven't
been accessed in a long time.

The tools are already supposed to support doing this with local file
sstate sources, they just do a bad job at getting the diffs right. One
intent of this work item was to try and understand why they don't work
and address that so at least for filesystem sstate mirrors, you can get
better results. I don't know how we solve the remote http issue as yet.

> > One of the big problems in the past was that we lost much of the hash
> > information after parsing completed. This meant that if the hashes then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain more of the information
> > so that we didn't have to recompute it every time to be able to do
> > processing with it. I have to admit I've totally lost track of where I
> > got to with that.
> 
> Here's an idea I can't get out of my head. Right now, the cache is
> simply an amorphous mass of objects, with no information regarding how
> they were created. How about storing complete build confgurations as
> well into the same directory? There would be a dedicated, separate
> area for each configuration that placed objects into the cache,
> containing:
> - list of layers and revisions
> - config template used
> - complete content of build/conf
> - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
> - complete list of sstate objects that were produced as a result, so
> they can be checked for existence
> 
> This would be written into the cache dir at the very end of the build
> when everything else is already there.

I'm not sure this helps as much as you'd like. For example I build
core-image-sato-sdk on the autobuilder and populate this but you want
to build core-image-sato locally. There would be no info here that
would help with information about what core-image-sato would need,
despite the fact we know one is a derivative of the other and there
would be a near full cache for it.

Taking your example above and curl:do_package, how would the list of
information above tell us 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
 wrote:
> For the task signatures, we need to think about some questions. If I
> make a change locally, can I query how much will rebuild and how much
> will be reused? There is bitbake --dry-run but perhaps it is time for a
> an option (or dedicated separate command?) to give some statistics
> about what bitbake would do? How much sstate would be reused?
>
> That then logically leads into the questions, can we tell what has
> changed? Why isn't my sstate being reused? For that we perhaps should
> define some existing scenarios where it is currently very difficult to
> work this out and then work out how we can report that information to
> the user. These could become test cases?

So I think there are two questions here that the tools should answer:

1. If I would run a build, what would be missing in the cache and need
to be built? The missing cache objects are in a dependency hierarchy,
so only those missing objects with no dependecies on other missing
objects would be printed. That should be comparatively easy to add as
bitbake already does those checks all the time. Is there something
else that's easily done and useful to print?

2. Then there's the question of *why* they are missing, which is
harder to answer. If, say, curl:do_package is not in the cache, then
the tool would have to walk the cache tree (I/O heavy operation as
there is no index), make a list of all curl:do_package objects that
are there, and do a recursive bitbake-diffsig (going up the task tree)
on them vs the one we want. Then print them starting with the newest.
Something like:

Existing cache objects are not suitable because:
 was built on  and has a mismatching SRCREV
 was built on  and has a different do_compile()

> One of the big problems in the past was that we lost much of the hash
> information after parsing completed. This meant that if the hashes then
> didn't match, we couldn't tell why as the original computation was
> lost. I did some work on allowing us to retain more of the information
> so that we didn't have to recompute it every time to be able to do
> processing with it. I have to admit I've totally lost track of where I
> got to with that.

Here's an idea I can't get out of my head. Right now, the cache is
simply an amorphous mass of objects, with no information regarding how
they were created. How about storing complete build confgurations as
well into the same directory? There would be a dedicated, separate
area for each configuration that placed objects into the cache,
containing:
- list of layers and revisions
- config template used
- complete content of build/conf
- bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
- complete list of sstate objects that were produced as a result, so
they can be checked for existence

This would be written into the cache dir at the very end of the build
when everything else is already there.

Right now, everyone sets up their own builds first, then points
local.conf or site.conf to the cache, and hopes for the best regarding
hit rates. Having stored build configs would allow inverting the
workflow, so that you first ask from the cache what it can provide
(e.g. it can provide mickledore or kirkstone core-image-minimal for
qemux86, and that's exactly what you want as a starting point), then
you use the build config stored in the cache to set up a build, and
run it - and that would guarantee complete sstate reuse and getting to
a functional image as soon as possible. Kind of like binary distro,
but implemented with sstate.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187647): 
https://lists.openembedded.org/g/openembedded-core/message/187647
Mute This Topic: https://lists.openembedded.org/mt/101356420/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote:
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley  wrote:
> > Alexander Kanavin will be working on the core workflow topic
> 
> I am now ready to start doing this, but before I do, I'd like to
> decompose the subject into manageable tasks with a bit of help from RP
> and the community:
> 
> https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/
> 
> 
> Core Workflow – Process Improvements
> 
> Background
> 
> The project builds everything from source by default. This means it
> has a reputation for being slow and heavy. There are ways the project
> can accelerate this which means faster workflows and improved
> developer experience but these are currently cumbersome to use.
> 
> Rationale
> 
> The project aims to defragment customised embedded Linux. This is
> important as if we succeed at this, it gives benefits to the wider
> ecosystem through making it easier to inject security fixes and an
> ability to share and collaborate without re-inventing the wheel.
> 
> To do this, we need to provide best-in-class support and compete
> against binary distributions for usability and speed. One way we can
> do this is provide better support for binary artifacts via our sstate
> mechanism. We do already have some of this functionality in our
> “extensible SDK” or “eSDK”.
> 
> Deliverables
> 
> Enable a public sstate mirror via a content delivery network (CDN) and
> populate using the autobuilder
> Ensure CDN sstate is reused under normal use case scenarios,
> particularly for slow components like rust-native. Identify any common
> sstate mismatch causes. Ensure test cases are added to cover the use
> cases and prevent regressions.
> Add lock and unlock commands to allow specific components to be locked
> down to specific sstate checksums or allow them to vary
> Allow switching between eSDK and non-eSDK modes
> Add tooling so we can understand why something is rebuilding when it
> isn’t expected to.
> ===
> 
> So: where to start? Do we need to 'design' something, or maybe time
> should go directly into addressing specific sore points? All feedback
> welcome.
> 
> There's also an unfinished patchset for adding bblock/bbunlock, which
> I would prefer to *not* 'take over and finish' but rather see the
> author get it merged:
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

To start with I'll try and write down and give a random walk through my
thoughts.

Certainly we need to get something like that patchset over the line. I
think it was blocked on a reply to this email:

https://lists.openembedded.org/g/openembedded-core/message/186497

which has suffered whilst I was pulled into the qemuppc mess. I feel
bad for not getting to a reply to that.

There are design elements to this work. We need to work out how we can
make eSDK and "normal" builds more similar and less of an overhead to
switch between one and the other. A "bblock all" command does partly
get you to an eSDK effectively so part of this may be switching eSDK to
use the new lock command. What other differences are there? What other
differences are necessary or make sense for the use cases eSDK was
designed for? How would you turn an existing build into an eSDK like
one? Could you provide a copy of a local build to someone else easily
using something like eSDK's tooling? What does the eSDK look like at
the end of this. One section we don't have good answers to yet is setup
and configuration although I know you've started on some of that.

For the task signatures, we need to think about some questions. If I
make a change locally, can I query how much will rebuild and how much
will be reused? There is bitbake --dry-run but perhaps it is time for a
an option (or dedicated separate command?) to give some statistics
about what bitbake would do? How much sstate would be reused?

That then logically leads into the questions, can we tell what has
changed? Why isn't my sstate being reused? For that we perhaps should
define some existing scenarios where it is currently very difficult to
work this out and then work out how we can report that information to
the user. These could become test cases?

One of the big problems in the past was that we lost much of the hash
information after parsing completed. This meant that if the hashes then
didn't match, we couldn't tell why as the original computation was
lost. I did some work on allowing us to retain more of the information
so that we didn't have to recompute it every time to be able to do
processing with it. I have to admit I've totally lost track of where I
got to with that.

Michael Halstead will be working on setting the CDN up so I'll let him
comment on when we'll have things ready for testing with that. We do
already have sstate shared from the autobuilder so some basic tests to
make sure our "base" shared files do work as expected is something
which can happen there already.

Another interesting