Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 2 Nov 2023 at 10:02, Alexander Kanavin via
lists.yoctoproject.org
wrote:
> So here's what I'd like to try:
>
> - write a new populate_build_replica task that writes a few things
> under ${WORKDIR}/replica
> -- setup-layers json and script
> (another option is to copy the layer trees themselves like esdk does,
> which is left for maybe later. Completely offline replication is not
> an initial goal, and it's good to get to a minimally viable
> implementation asap.)
> -- meta-build-config layer with the local.conf/bblayer.conf template.
> -- sstate cache needed to fulfil the bitbake target that the task is
> for (this would reuse code from esdk that does the same as much as
> possible)
>
> All of this is then packaged into a self-extracting shell archive that:
> - unpacks itself
> - fetches layers using the json/script from the unpacked tree
> - sets up a build directory using the template from meta-build-config
> in the unpacked tree
> - tweaks site.conf to point to the prepackaged sstate
>
> And voila! (in theory) This should be the same build as the one that
> was produced elsewhere.
I started writing this as a bitbake task, but then quickly hit an
obstacle: bitbake-layers deadlocks when executed in a bitbake task.
At which point it dawned on me: all of the required functions are
available in command-line utility form, and so this should be a script
as well, with as many options as people find useful (as opposed to
bitbake tasks which are more difficult to parametrize).
So:
- bitbake -S lockedsigs to get the list of sstate objects
- gen-lockedsig-cache to populate a newly made sstate cache directory
with them (optional)
- bitbake-layers to save the layer configuration and build configuration
Then either leave all that in an unpacked form, or package it up in a
self-extracting shell archive or plain tarball. It would also contain
a script that would set up a plain bitbake build from what is in the
archive.
This could be poky/scripts/oe-replicate-build ?
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61576): https://lists.yoctoproject.org/g/yocto/message/61576
Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 2 Nov 2023 at 09:32, wrote:
> Sorry for repeating some parts which we already had in other emails.
> But I tried to summarize the lengthy discussion a bit in one place.
So here's what I'd like to try:
- write a new populate_build_replica task that writes a few things
under ${WORKDIR}/replica
-- setup-layers json and script
(another option is to copy the layer trees themselves like esdk does,
which is left for maybe later. Completely offline replication is not
an initial goal, and it's good to get to a minimally viable
implementation asap.)
-- meta-build-config layer with the local.conf/bblayer.conf template.
-- sstate cache needed to fulfil the bitbake target that the task is
for (this would reuse code from esdk that does the same as much as
possible)
All of this is then packaged into a self-extracting shell archive that:
- unpacks itself
- fetches layers using the json/script from the unpacked tree
- sets up a build directory using the template from meta-build-config
in the unpacked tree
- tweaks site.conf to point to the prepackaged sstate
And voila! (in theory) This should be the same build as the one that
was produced elsewhere.
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61573): https://lists.yoctoproject.org/g/yocto/message/61573
Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Wed, 2023-11-01 at 18:28 +0100, Alexander Kanavin wrote:
> On Wed, 1 Nov 2023 at 16:45, wrote:
> > I think these differences between SDK and bitbake environment are
> > no
> > longer required and they have been problematic. I would try to make
> > the
> > bitbake environment usable like the eSDK environment was without
> > trying
> > to replicate all the details of the eSDK installer such as these
> > local.conf settings.
>
> I have now split up the populate_sdk_ext into separate functions [1]
> for better maintainability, and the more I think about what to do
> next, the more I agree with Adrian. I just don't see why (in a
> standard yocto build) would we want to manipulate PATH to provide a
> restricted set of tools, or to create a "local.conf+extra stuff"
> (locked signatures, esdk tweaks) environment, when existing
> local.conf
> by itself is already working fine, and full set of tools is better
> than a restricted one. If we want to add or remove locked signatures,
> this can be done with 'bitbake -s lockedsigs' or bblock for specific
> recipes only. And SDK's cross-toolchain is accessible via
> meta-ide-support/build-sysroots flow.
>
> [1]
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
Splitting allows to copy the new function with the + from the patch
into this email and comment it.
+python copy_buildsystem () {
+ import oe.copy_buildsystem
+
+ baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
+
+ # Determine if we're building a derivative extensible SDK (from
devtool build-sdk)
+ derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
What's the advantage of that? There are now two worlds: The bitbake
world and the SDK world which behave similar but not equal. Both need
to be maintained and tested.
We use only bitbake on our CI/CD infrastructure. For users who have
only the eSDK installed, it's hard to understand what the CI does and
even harder to reproduce errors happening on the CI.
We also had some setups with a containerized eSDK on the CI for the
integration of application source code. But this did not work at all.
The SDK container is basically outdated when the first component of the
firmware gets updated. If other components depend on that component an
SDK container update is required. Handling breaking changes easily is a
big advantage of bitbake which gets lost when using any kind of locked
or badly packaged variant of it.
+
+ conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers =
copy_bitbake_and_layers(d, baseoutpath, derivative)
Changing the directory layout leads to many pitfalls especially if more
layers than just poky are used. This should be replaced by the new
bitbake-layers tools. This means there is only one official way for
setting up bitbake layers and the folder structure gets exactly
replicated.
+
+ write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath,
core_meta_subdir)
Not sure there is much left when we have only the bitbake world. But
defining some defaults might be still useful.
+
+ write_unlocked_sigs(d, baseoutpath)
Lets turn this more towards a QA check. As a SDK maintainer I would
like to provide SDKs with 100% sstate included. But as a user, if I
have a choice between waiting a few minutes until bitbake compiled some
missing parts or getting an error message telling me I can't get an SDK
now, I'd probably choose compile.
If I remember correctly, with the old eSDK installer this is even
worse. This error happens during the installation which leads to an SDK
in an undefined state. The user must delete it again and fix the
generation of the SDK installer, which might be a very complicated and
time consuming task with the existing tools.
+
+ write_bblayers_conf(d, baseoutpath, sdkbblayers)
Also something which can be replaced by the new bitbake-layers utility.
+
+ uninative_checksum = copy_uninative(d, baseoutpath)
Not sure if this is still needed. With a bitbake environment this just
happens from sstate, I guess. So why doing it differently for the SDK?
+
+ write_local_conf(d, baseoutpath, derivative, core_meta_subdir,
uninative_checksum)
Also something which can be replaced by the new bitbake-layers utility.
+
+ prepare_locked_cache(d, baseoutpath, conf_initpath)
The sstate could be shipped in three different ways:
* Included in the installer and just extracted into $SSTATE_DIR. This
is simple but it does not scale at all. If you maintain multiple
distros and MACHINES and want to have fast update cycles,
distributing complete sstate archives quickly becomes practically
impossible, as the same data is packed into several huge archives.
That leads to issues on the infrastructure side. But also on the
user's machine having one sstate folder e.g. ~/sstate-cache instead
of several $TMPDIR/sstate-cache folders is beneficial.
* No sstate is included, the sstate gets "lazy" fetched from
SSTATE_MIRROR. Also that looks easy but does not scale very well for
the
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Wed, 1 Nov 2023 at 16:45, wrote: > I think these differences between SDK and bitbake environment are no > longer required and they have been problematic. I would try to make the > bitbake environment usable like the eSDK environment was without trying > to replicate all the details of the eSDK installer such as these > local.conf settings. I have now split up the populate_sdk_ext into separate functions [1] for better maintainability, and the more I think about what to do next, the more I agree with Adrian. I just don't see why (in a standard yocto build) would we want to manipulate PATH to provide a restricted set of tools, or to create a "local.conf+extra stuff" (locked signatures, esdk tweaks) environment, when existing local.conf by itself is already working fine, and full set of tools is better than a restricted one. If we want to add or remove locked signatures, this can be done with 'bitbake -s lockedsigs' or bblock for specific recipes only. And SDK's cross-toolchain is accessible via meta-ide-support/build-sysroots flow. [1] https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all THAT SAID. I also agree there should be a way to package the 'standard yocto build' that does not also do 'weird esdk stuff' on top. Just a tarball that has 'everything', and that you unpack and get a standard build exactly like the one that was used to produce the tarball. Something like what Adrian suggested: basically place all information needed to replicate a build into one place, then either make a tarball out of it (with sstate for fast builds), or publish it into the network. We might be able to reuse some of esdk packaging code for this, or we might not, but I think this is the direction that should be taken. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61562): https://lists.yoctoproject.org/g/yocto/message/61562 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
Hi Richard, hi Alex On Mon, 2023-10-30 at 14:07 +, Richard Purdie wrote: > On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote: > > On Thu, 14 Sept 2023 at 14:56, Richard Purdie > > wrote: > > > There are design elements to this work. We need to work out how > > > we can > > > make eSDK and "normal" builds more similar and less of an > > > overhead to > > > switch between one and the other. A "bblock all" command does > > > partly > > > get you to an eSDK effectively so part of this may be switching > > > eSDK to > > > use the new lock command. What other differences are there? What > > > other > > > differences are necessary or make sense for the use cases eSDK > > > was > > > designed for? How would you turn an existing build into an eSDK > > > like > > > one? Could you provide a copy of a local build to someone else > > > easily > > > using something like eSDK's tooling? What does the eSDK look like > > > at > > > the end of this. One section we don't have good answers to yet is > > > setup > > > and configuration although I know you've started on some of that. > > > > So I see the following differences between esdk and normal modes: > > > > 1. Environment and tooling availability. > > > > a) esdk sets a number of variables from its initialization script > > that > > aid with cross-compiling components directly (e.g. the core use > > case > > of SDKs). Normal mode doesn't do that, but recently added > > meta-ide-support will generate a similar initialization script that > > will set up the same environment from the normal mode. There are > > tests > > and documentation for it. > > In that case, this one is something we can document as how to make > the > functionality available in the normal build. > > > b) PATH. eSDK has a number of items in PATH that point to various > > locations inside tmp/sysroots/, collectively they provide the > > cross-toolchain. > > > > eSDK also puts a selection of yocto tools into path - wic, devtool > > but > > not bitbake: > > > > > > alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/ > > total 48 > > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 devtool -> > > ../../../../layers/poky/scripts/devtool > > lrwxrwxrwx 1 alex alex 54 Oct 30 12:52 oe-find-native-sysroot -> > > ../../../../layers/poky/scripts/oe-find-native-sysroot > > lrwxrwxrwx 1 alex alex 42 Oct 30 12:52 recipetool -> > > ../../../../layers/poky/scripts/recipetool > > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 runqemu -> > > ../../../../layers/poky/scripts/runqemu > > lrwxrwxrwx 1 alex alex 55 Oct 30 12:52 runqemu-addptable2image -> > > ../../../../layers/poky/scripts/runqemu-addptable2image > > lrwxrwxrwx 1 alex alex 53 Oct 30 12:52 runqemu-export-rootfs -> > > ../../../../layers/poky/scripts/runqemu-export-rootfs > > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-extract-sdk -> > > ../../../../layers/poky/scripts/runqemu-extract-sdk > > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-gen-tapdevs -> > > ../../../../layers/poky/scripts/runqemu-gen-tapdevs > > lrwxrwxrwx 1 alex alex 46 Oct 30 12:52 runqemu-ifdown -> > > ../../../../layers/poky/scripts/runqemu-ifdown > > lrwxrwxrwx 1 alex alex 44 Oct 30 12:52 runqemu-ifup -> > > ../../../../layers/poky/scripts/runqemu-ifup > > lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd -> > > ../../../../tmp/work/qemuarm64-poky-linux/core-image- > > minimal/1.0/recipe-sysroot-native/usr/bin/unfsd > > lrwxrwxrwx 1 alex alex 35 Oct 30 12:52 wic -> > > ../../../../layers/poky/scripts/wic > > == > > > > 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH. > > Cross-toolchain can be added by the same environment script made by > > meta-ide-support as mentioned in 1a. > > Right, so in theory we can change PATH and change this which can also > easily be documented. > > > 2. Configuration (e.g. local.conf). I think these differences between SDK and bitbake environment are no longer required and they have been problematic. I would try to make the bitbake environment usable like the eSDK environment was without trying to replicate all the details of the eSDK installer such as these local.conf settings. > > > > eSDK local.conf is local.conf from the normal mode that was used to > > produce eSDK, stripped of all comments, and with a bunch of extra > > settings: > > > > > > INHERIT:remove = "buildhistory icecc" That's not needed it there is a full bitbake environment. > > CONNECTIVITY_CHECK_URIS = "" > > > > SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none" > > > > SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" As already mentioned in my previous mail, a locked SDK is not a must if the full bitbake environment with shared sstate-cache is available. Locking might be added later as an optional feature. > > > > BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES" > > > > BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir > > *:do_rm_work
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Mon, 30 Oct 2023 at 15:07, Richard Purdie wrote: > > a) esdk sets a number of variables from its initialization script that > > aid with cross-compiling components directly (e.g. the core use case > > of SDKs). Normal mode doesn't do that, but recently added > > meta-ide-support will generate a similar initialization script that > > will set up the same environment from the normal mode. There are tests > > and documentation for it. > > In that case, this one is something we can document as how to make the > functionality available in the normal build. This has been documented in the sdk manual: https://docs.yoctoproject.org/sdk-manual/extensible.html#two-ways-to-install-the-extensible-sdk https://docs.yoctoproject.org/sdk-manual/extensible.html#installing-additional-items-into-the-extensible-sdk Actually, I have on purpose presented both options in the manual as 'esdk', the standalone installer option is simply locked down more (task signatures and tooling wise) than the direct yocto build. So here's what could be done: - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk environment script puts that in PATH, rather than some custom esdk-specific location (the code to generate that can then be dropped). - esdk tweaks to local.conf move into a dedicated include file, which can be static and under version control, except for perhaps METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" - I need to check why is it there and how that is used. Something else perhaps? Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61521): https://lists.yoctoproject.org/g/yocto/message/61521 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote: > On Thu, 14 Sept 2023 at 14:56, Richard Purdie > wrote: > > There are design elements to this work. We need to work out how we can > > make eSDK and "normal" builds more similar and less of an overhead to > > switch between one and the other. A "bblock all" command does partly > > get you to an eSDK effectively so part of this may be switching eSDK to > > use the new lock command. What other differences are there? What other > > differences are necessary or make sense for the use cases eSDK was > > designed for? How would you turn an existing build into an eSDK like > > one? Could you provide a copy of a local build to someone else easily > > using something like eSDK's tooling? What does the eSDK look like at > > the end of this. One section we don't have good answers to yet is setup > > and configuration although I know you've started on some of that. > > So I see the following differences between esdk and normal modes: > > 1. Environment and tooling availability. > > a) esdk sets a number of variables from its initialization script that > aid with cross-compiling components directly (e.g. the core use case > of SDKs). Normal mode doesn't do that, but recently added > meta-ide-support will generate a similar initialization script that > will set up the same environment from the normal mode. There are tests > and documentation for it. In that case, this one is something we can document as how to make the functionality available in the normal build. > b) PATH. eSDK has a number of items in PATH that point to various > locations inside tmp/sysroots/, collectively they provide the > cross-toolchain. > > eSDK also puts a selection of yocto tools into path - wic, devtool but > not bitbake: > > > alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/ > total 48 > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 devtool -> > ../../../../layers/poky/scripts/devtool > lrwxrwxrwx 1 alex alex 54 Oct 30 12:52 oe-find-native-sysroot -> > ../../../../layers/poky/scripts/oe-find-native-sysroot > lrwxrwxrwx 1 alex alex 42 Oct 30 12:52 recipetool -> > ../../../../layers/poky/scripts/recipetool > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 runqemu -> > ../../../../layers/poky/scripts/runqemu > lrwxrwxrwx 1 alex alex 55 Oct 30 12:52 runqemu-addptable2image -> > ../../../../layers/poky/scripts/runqemu-addptable2image > lrwxrwxrwx 1 alex alex 53 Oct 30 12:52 runqemu-export-rootfs -> > ../../../../layers/poky/scripts/runqemu-export-rootfs > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-extract-sdk -> > ../../../../layers/poky/scripts/runqemu-extract-sdk > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-gen-tapdevs -> > ../../../../layers/poky/scripts/runqemu-gen-tapdevs > lrwxrwxrwx 1 alex alex 46 Oct 30 12:52 runqemu-ifdown -> > ../../../../layers/poky/scripts/runqemu-ifdown > lrwxrwxrwx 1 alex alex 44 Oct 30 12:52 runqemu-ifup -> > ../../../../layers/poky/scripts/runqemu-ifup > lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd -> > ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd > lrwxrwxrwx 1 alex alex 35 Oct 30 12:52 wic -> > ../../../../layers/poky/scripts/wic > == > > 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH. > Cross-toolchain can be added by the same environment script made by > meta-ide-support as mentioned in 1a. Right, so in theory we can change PATH and change this which can also easily be documented. > 2. Configuration (e.g. local.conf). > > eSDK local.conf is local.conf from the normal mode that was used to > produce eSDK, stripped of all comments, and with a bunch of extra > settings: > > > INHERIT:remove = "buildhistory icecc" > CONNECTIVITY_CHECK_URIS = "" > > SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none" > > SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" > > BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES" > > BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir > *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot" > > BUILDCFG_HEADER = "" > > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" > > # Provide a flag to indicate we are in the EXT_SDK Context > WITHIN_EXT_SDK = "1" > > SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1 > file://universal-4.9/(.*) file://universal-4.8/\1" > Perhaps some of this should become a generic include file which is included which would then make it easy to document as how to lock things down to match it? That would take the config out of the class file too which is probably nicer. > require conf/locked-sigs.inc > require conf/unlocked-sigs.inc > === > > Devtool also has a special configuration: > = > alex@Zen2:~/poky_sdk$ cat conf/devtool.conf > [General] > bitbake_subdir = layers/poky/bitbake > init_path = layers/poky/oe-init-build-env >
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 14 Sept 2023 at 14:56, Richard Purdie wrote: > There are design elements to this work. We need to work out how we can > make eSDK and "normal" builds more similar and less of an overhead to > switch between one and the other. A "bblock all" command does partly > get you to an eSDK effectively so part of this may be switching eSDK to > use the new lock command. What other differences are there? What other > differences are necessary or make sense for the use cases eSDK was > designed for? How would you turn an existing build into an eSDK like > one? Could you provide a copy of a local build to someone else easily > using something like eSDK's tooling? What does the eSDK look like at > the end of this. One section we don't have good answers to yet is setup > and configuration although I know you've started on some of that. So I see the following differences between esdk and normal modes: 1. Environment and tooling availability. a) esdk sets a number of variables from its initialization script that aid with cross-compiling components directly (e.g. the core use case of SDKs). Normal mode doesn't do that, but recently added meta-ide-support will generate a similar initialization script that will set up the same environment from the normal mode. There are tests and documentation for it. b) PATH. eSDK has a number of items in PATH that point to various locations inside tmp/sysroots/, collectively they provide the cross-toolchain. eSDK also puts a selection of yocto tools into path - wic, devtool but not bitbake: alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/ total 48 lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 devtool -> ../../../../layers/poky/scripts/devtool lrwxrwxrwx 1 alex alex 54 Oct 30 12:52 oe-find-native-sysroot -> ../../../../layers/poky/scripts/oe-find-native-sysroot lrwxrwxrwx 1 alex alex 42 Oct 30 12:52 recipetool -> ../../../../layers/poky/scripts/recipetool lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 runqemu -> ../../../../layers/poky/scripts/runqemu lrwxrwxrwx 1 alex alex 55 Oct 30 12:52 runqemu-addptable2image -> ../../../../layers/poky/scripts/runqemu-addptable2image lrwxrwxrwx 1 alex alex 53 Oct 30 12:52 runqemu-export-rootfs -> ../../../../layers/poky/scripts/runqemu-export-rootfs lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-extract-sdk -> ../../../../layers/poky/scripts/runqemu-extract-sdk lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-gen-tapdevs -> ../../../../layers/poky/scripts/runqemu-gen-tapdevs lrwxrwxrwx 1 alex alex 46 Oct 30 12:52 runqemu-ifdown -> ../../../../layers/poky/scripts/runqemu-ifdown lrwxrwxrwx 1 alex alex 44 Oct 30 12:52 runqemu-ifup -> ../../../../layers/poky/scripts/runqemu-ifup lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd -> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd lrwxrwxrwx 1 alex alex 35 Oct 30 12:52 wic -> ../../../../layers/poky/scripts/wic == 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH. Cross-toolchain can be added by the same environment script made by meta-ide-support as mentioned in 1a. 2. Configuration (e.g. local.conf). eSDK local.conf is local.conf from the normal mode that was used to produce eSDK, stripped of all comments, and with a bunch of extra settings: INHERIT:remove = "buildhistory icecc" CONNECTIVITY_CHECK_URIS = "" SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none" SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES" BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot" BUILDCFG_HEADER = "" METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" # Provide a flag to indicate we are in the EXT_SDK Context WITHIN_EXT_SDK = "1" SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1 file://universal-4.9/(.*) file://universal-4.8/\1" require conf/locked-sigs.inc require conf/unlocked-sigs.inc === Devtool also has a special configuration: = alex@Zen2:~/poky_sdk$ cat conf/devtool.conf [General] bitbake_subdir = layers/poky/bitbake init_path = layers/poky/oe-init-build-env core_meta_subdir = layers/poky/meta [SDK] sdk_targets = core-image-minimal == There is currently no tooling to add/remove these extras in either esdk mode or normal mode as far as I understand. Their individual purposes and effects are also not exactly clear to me, and need to be investigated one by one. 3. Setting up a normal mode in a eSDK installation. This is actually pretty easy: rather than sourcing the sdk environment script, source the poky/oe-init-build-env: == alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env . ### Shell environment set up for builds. ### ... === Bitbake will then simply run, albeit with all those extra esdk-specific items in
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 14 Sept 2023 at 21:54, Richard Purdie wrote: > The tools are already supposed to support doing this with local file > sstate sources, they just do a bad job at getting the diffs right. One > intent of this work item was to try and understand why they don't work > and address that so at least for filesystem sstate mirrors, you can get > better results. I don't know how we solve the remote http issue as yet. I ran a few experiments with bitbake -S printdiff, and so far I haven't yet seen an unhelpful result. Here's the most impressive one: $ bitbake libsolv-native (uses cmake-native) $ git revert $ rm -rf build/tmp/ $ bitbake -S printdiff libsolv-native ... The differences between the current build and any cached tasks start at the following tasks: /srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-native_3.27.4.bb:do_recipe_qa NOTE: Writing task signature files Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc Task cmake-native:do_recipe_qa couldn't be used from the cache because: We need hash 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb, closest matching task was 85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7 basehash changed from 7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to 7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09 Variable PV value changed from '3.27.5' to '3.27.4' Variable SRC_URI[sha256sum] value changed from '5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' to '0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af' That's pretty good, isn't it? It does print both what needs to be re-run, and *why* as well. I have no idea yet what kind of magic it does to find the 'closest matching task' in sstate, but if this breaks down in some other scenarios, we need to find them to get a starting point for making the tools better. Ideas? I'm ready to try them :) Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61065): https://lists.yoctoproject.org/g/yocto/message/61065 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
Unless I missed something, I think I fixed all comments from RP on the branch I force pushed :) Cheers Julien Le mer. 20 sept. 2023 à 16:31, Alexander Kanavin a écrit : > > That’s great to hear! When you have a new patch set please post it here. I > think there’s also a new reply from RP that you need to consider. > > Alex > > On Wed 20. Sep 2023 at 16.25, Julien Stephan wrote: >> >> Hi Alexander, >> About bblock tool: I just force pushed my branch on poky-contrib >> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock), >> trying to fix an autobuilder issue reported by Alexandre Belloni. >> I am still working on this, and I would be very happy to get this merged :) >> >> Cheers >> Julien >> >> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin >> a écrit : >> > >> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie >> > wrote: >> > >> > > Effectively those "configs" are what the eSDK is, you're just proposing >> > > a server side set of them rather than a built copy. In many ways that >> > > could be useful way of possibly rethinking the way the eSDK works. >> > > >> > > So in that sense I like the idea, we should just be clear the problem >> > > it solves as it doesn't solve the "why is my build not using the cache" >> > > question specifically but it potentially solves other issues. >> > >> > Right, I didn't make it clear that this idea is not about 'why sstate >> > is not being reused', but rather something I thought of to avoid >> > placing people in that situation to begin with - offering them >> > 'pre-fabricated builds' with guaranteed sstate availability. I have a >> > hunch there'll be plenty, for whom this is just fine, and eventually >> > this could be the basis of that oe-setup holy grail :) >> > >> > I'll start with the simplest enhancement of printing what would be >> > rebuilt. There's a lot of code I need to understand, so this needs to >> > be in manageable chunks :) >> > >> > Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61054): https://lists.yoctoproject.org/g/yocto/message/61054 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
That’s great to hear! When you have a new patch set please post it here. I think there’s also a new reply from RP that you need to consider. Alex On Wed 20. Sep 2023 at 16.25, Julien Stephan wrote: > Hi Alexander, > About bblock tool: I just force pushed my branch on poky-contrib > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock), > trying to fix an autobuilder issue reported by Alexandre Belloni. > I am still working on this, and I would be very happy to get this merged :) > > Cheers > Julien > > Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin > a écrit : > > > > On Thu, 14 Sept 2023 at 21:54, Richard Purdie > > wrote: > > > > > Effectively those "configs" are what the eSDK is, you're just proposing > > > a server side set of them rather than a built copy. In many ways that > > > could be useful way of possibly rethinking the way the eSDK works. > > > > > > So in that sense I like the idea, we should just be clear the problem > > > it solves as it doesn't solve the "why is my build not using the cache" > > > question specifically but it potentially solves other issues. > > > > Right, I didn't make it clear that this idea is not about 'why sstate > > is not being reused', but rather something I thought of to avoid > > placing people in that situation to begin with - offering them > > 'pre-fabricated builds' with guaranteed sstate availability. I have a > > hunch there'll be plenty, for whom this is just fine, and eventually > > this could be the basis of that oe-setup holy grail :) > > > > I'll start with the simplest enhancement of printing what would be > > rebuilt. There's a lot of code I need to understand, so this needs to > > be in manageable chunks :) > > > > Alex > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61051): https://lists.yoctoproject.org/g/yocto/message/61051 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
Hi Alexander, About bblock tool: I just force pushed my branch on poky-contrib (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock), trying to fix an autobuilder issue reported by Alexandre Belloni. I am still working on this, and I would be very happy to get this merged :) Cheers Julien Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin a écrit : > > On Thu, 14 Sept 2023 at 21:54, Richard Purdie > wrote: > > > Effectively those "configs" are what the eSDK is, you're just proposing > > a server side set of them rather than a built copy. In many ways that > > could be useful way of possibly rethinking the way the eSDK works. > > > > So in that sense I like the idea, we should just be clear the problem > > it solves as it doesn't solve the "why is my build not using the cache" > > question specifically but it potentially solves other issues. > > Right, I didn't make it clear that this idea is not about 'why sstate > is not being reused', but rather something I thought of to avoid > placing people in that situation to begin with - offering them > 'pre-fabricated builds' with guaranteed sstate availability. I have a > hunch there'll be plenty, for whom this is just fine, and eventually > this could be the basis of that oe-setup holy grail :) > > I'll start with the simplest enhancement of printing what would be > rebuilt. There's a lot of code I need to understand, so this needs to > be in manageable chunks :) > > Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61050): https://lists.yoctoproject.org/g/yocto/message/61050 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 14 Sept 2023 at 21:54, Richard Purdie wrote: > Effectively those "configs" are what the eSDK is, you're just proposing > a server side set of them rather than a built copy. In many ways that > could be useful way of possibly rethinking the way the eSDK works. > > So in that sense I like the idea, we should just be clear the problem > it solves as it doesn't solve the "why is my build not using the cache" > question specifically but it potentially solves other issues. Right, I didn't make it clear that this idea is not about 'why sstate is not being reused', but rather something I thought of to avoid placing people in that situation to begin with - offering them 'pre-fabricated builds' with guaranteed sstate availability. I have a hunch there'll be plenty, for whom this is just fine, and eventually this could be the basis of that oe-setup holy grail :) I'll start with the simplest enhancement of printing what would be rebuilt. There's a lot of code I need to understand, so this needs to be in manageable chunks :) Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61016): https://lists.yoctoproject.org/g/yocto/message/61016 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote: > On Thu, 14 Sept 2023 at 14:56, Richard Purdie > wrote: > > For the task signatures, we need to think about some questions. If I > > make a change locally, can I query how much will rebuild and how much > > will be reused? There is bitbake --dry-run but perhaps it is time for a > > an option (or dedicated separate command?) to give some statistics > > about what bitbake would do? How much sstate would be reused? > > > > That then logically leads into the questions, can we tell what has > > changed? Why isn't my sstate being reused? For that we perhaps should > > define some existing scenarios where it is currently very difficult to > > work this out and then work out how we can report that information to > > the user. These could become test cases? > > So I think there are two questions here that the tools should answer: > > 1. If I would run a build, what would be missing in the cache and need > to be built? The missing cache objects are in a dependency hierarchy, > so only those missing objects with no dependecies on other missing > objects would be printed. That should be comparatively easy to add as > bitbake already does those checks all the time. Right, what we lack is a way for the user to ask this and see the result easily. As you say, bitbake can do this already. > Is there something else that's easily done and useful to print? I think there is also the scenario of: "I've run a build and have an existing TMPDIR and stamp info. I've now pulled in a change. How much is going to rebuild and more importantly *why*?" This is different to a remote sstate situation as you have the stamp info of the previous build already there to compare against. > 2. Then there's the question of *why* they are missing, which is > harder to answer. If, say, curl:do_package is not in the cache, then > the tool would have to walk the cache tree (I/O heavy operation as > there is no index), make a list of all curl:do_package objects that > are there, and do a recursive bitbake-diffsig (going up the task tree) > on them vs the one we want. Then print them starting with the newest. > Something like: > > Existing cache objects are not suitable because: > was built on and has a mismatching SRCREV > was built on and has a different do_compile() In theory you can do an: ls sstate/*/*/sstate:curl:corei7-64-poky-linux:::corei7-64:10:*_package.tar.zst and get a list of possible objects. Some key information was put into the cache file names for this reason. Obviously this becomes much trickier when the sstate is remote over http though. "newest" is problematic in this context as you really want the closest match. There is no concept of build date in the cache as it often isn't relevant (building an old release for example). The only point a date/time is used is for cleaning out the cache for files which haven't been accessed in a long time. The tools are already supposed to support doing this with local file sstate sources, they just do a bad job at getting the diffs right. One intent of this work item was to try and understand why they don't work and address that so at least for filesystem sstate mirrors, you can get better results. I don't know how we solve the remote http issue as yet. > > One of the big problems in the past was that we lost much of the hash > > information after parsing completed. This meant that if the hashes then > > didn't match, we couldn't tell why as the original computation was > > lost. I did some work on allowing us to retain more of the information > > so that we didn't have to recompute it every time to be able to do > > processing with it. I have to admit I've totally lost track of where I > > got to with that. > > Here's an idea I can't get out of my head. Right now, the cache is > simply an amorphous mass of objects, with no information regarding how > they were created. How about storing complete build confgurations as > well into the same directory? There would be a dedicated, separate > area for each configuration that placed objects into the cache, > containing: > - list of layers and revisions > - config template used > - complete content of build/conf > - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.) > - complete list of sstate objects that were produced as a result, so > they can be checked for existence > > This would be written into the cache dir at the very end of the build > when everything else is already there. I'm not sure this helps as much as you'd like. For example I build core-image-sato-sdk on the autobuilder and populate this but you want to build core-image-sato locally. There would be no info here that would help with information about what core-image-sato would need, despite the fact we know one is a derivative of the other and there would be a near full cache for it. Taking your example above and curl:do_package, how would the list of information above tell us
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 14 Sept 2023 at 14:56, Richard Purdie wrote: > For the task signatures, we need to think about some questions. If I > make a change locally, can I query how much will rebuild and how much > will be reused? There is bitbake --dry-run but perhaps it is time for a > an option (or dedicated separate command?) to give some statistics > about what bitbake would do? How much sstate would be reused? > > That then logically leads into the questions, can we tell what has > changed? Why isn't my sstate being reused? For that we perhaps should > define some existing scenarios where it is currently very difficult to > work this out and then work out how we can report that information to > the user. These could become test cases? So I think there are two questions here that the tools should answer: 1. If I would run a build, what would be missing in the cache and need to be built? The missing cache objects are in a dependency hierarchy, so only those missing objects with no dependecies on other missing objects would be printed. That should be comparatively easy to add as bitbake already does those checks all the time. Is there something else that's easily done and useful to print? 2. Then there's the question of *why* they are missing, which is harder to answer. If, say, curl:do_package is not in the cache, then the tool would have to walk the cache tree (I/O heavy operation as there is no index), make a list of all curl:do_package objects that are there, and do a recursive bitbake-diffsig (going up the task tree) on them vs the one we want. Then print them starting with the newest. Something like: Existing cache objects are not suitable because: was built on and has a mismatching SRCREV was built on and has a different do_compile() > One of the big problems in the past was that we lost much of the hash > information after parsing completed. This meant that if the hashes then > didn't match, we couldn't tell why as the original computation was > lost. I did some work on allowing us to retain more of the information > so that we didn't have to recompute it every time to be able to do > processing with it. I have to admit I've totally lost track of where I > got to with that. Here's an idea I can't get out of my head. Right now, the cache is simply an amorphous mass of objects, with no information regarding how they were created. How about storing complete build confgurations as well into the same directory? There would be a dedicated, separate area for each configuration that placed objects into the cache, containing: - list of layers and revisions - config template used - complete content of build/conf - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.) - complete list of sstate objects that were produced as a result, so they can be checked for existence This would be written into the cache dir at the very end of the build when everything else is already there. Right now, everyone sets up their own builds first, then points local.conf or site.conf to the cache, and hopes for the best regarding hit rates. Having stored build configs would allow inverting the workflow, so that you first ask from the cache what it can provide (e.g. it can provide mickledore or kirkstone core-image-minimal for qemux86, and that's exactly what you want as a starting point), then you use the build config stored in the cache to set up a build, and run it - and that would guarantee complete sstate reuse and getting to a functional image as soon as possible. Kind of like binary distro, but implemented with sstate. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61010): https://lists.yoctoproject.org/g/yocto/message/61010 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote: > On Tue, 12 Sept 2023 at 16:44, Stephen Jolley wrote: > > Alexander Kanavin will be working on the core workflow topic > > I am now ready to start doing this, but before I do, I'd like to > decompose the subject into manageable tasks with a bit of help from RP > and the community: > > https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/ > > > Core Workflow – Process Improvements > > Background > > The project builds everything from source by default. This means it > has a reputation for being slow and heavy. There are ways the project > can accelerate this which means faster workflows and improved > developer experience but these are currently cumbersome to use. > > Rationale > > The project aims to defragment customised embedded Linux. This is > important as if we succeed at this, it gives benefits to the wider > ecosystem through making it easier to inject security fixes and an > ability to share and collaborate without re-inventing the wheel. > > To do this, we need to provide best-in-class support and compete > against binary distributions for usability and speed. One way we can > do this is provide better support for binary artifacts via our sstate > mechanism. We do already have some of this functionality in our > “extensible SDK” or “eSDK”. > > Deliverables > > Enable a public sstate mirror via a content delivery network (CDN) and > populate using the autobuilder > Ensure CDN sstate is reused under normal use case scenarios, > particularly for slow components like rust-native. Identify any common > sstate mismatch causes. Ensure test cases are added to cover the use > cases and prevent regressions. > Add lock and unlock commands to allow specific components to be locked > down to specific sstate checksums or allow them to vary > Allow switching between eSDK and non-eSDK modes > Add tooling so we can understand why something is rebuilding when it > isn’t expected to. > === > > So: where to start? Do we need to 'design' something, or maybe time > should go directly into addressing specific sore points? All feedback > welcome. > > There's also an unfinished patchset for adding bblock/bbunlock, which > I would prefer to *not* 'take over and finish' but rather see the > author get it merged: > https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276 To start with I'll try and write down and give a random walk through my thoughts. Certainly we need to get something like that patchset over the line. I think it was blocked on a reply to this email: https://lists.openembedded.org/g/openembedded-core/message/186497 which has suffered whilst I was pulled into the qemuppc mess. I feel bad for not getting to a reply to that. There are design elements to this work. We need to work out how we can make eSDK and "normal" builds more similar and less of an overhead to switch between one and the other. A "bblock all" command does partly get you to an eSDK effectively so part of this may be switching eSDK to use the new lock command. What other differences are there? What other differences are necessary or make sense for the use cases eSDK was designed for? How would you turn an existing build into an eSDK like one? Could you provide a copy of a local build to someone else easily using something like eSDK's tooling? What does the eSDK look like at the end of this. One section we don't have good answers to yet is setup and configuration although I know you've started on some of that. For the task signatures, we need to think about some questions. If I make a change locally, can I query how much will rebuild and how much will be reused? There is bitbake --dry-run but perhaps it is time for a an option (or dedicated separate command?) to give some statistics about what bitbake would do? How much sstate would be reused? That then logically leads into the questions, can we tell what has changed? Why isn't my sstate being reused? For that we perhaps should define some existing scenarios where it is currently very difficult to work this out and then work out how we can report that information to the user. These could become test cases? One of the big problems in the past was that we lost much of the hash information after parsing completed. This meant that if the hashes then didn't match, we couldn't tell why as the original computation was lost. I did some work on allowing us to retain more of the information so that we didn't have to recompute it every time to be able to do processing with it. I have to admit I've totally lost track of where I got to with that. Michael Halstead will be working on setting the CDN up so I'll let him comment on when we'll have things ready for testing with that. We do already have sstate shared from the autobuilder so some basic tests to make sure our "base" shared files do work as expected is something which can happen there already. Another interesting
