On Wed, 2023-11-01 at 18:28 +0100, Alexander Kanavin wrote:
> On Wed, 1 Nov 2023 at 16:45, <adrian.freiho...@gmail.com> wrote:
> > I think these differences between SDK and bitbake environment are
> > no
> > longer required and they have been problematic. I would try to make
> > the
> > bitbake environment usable like the eSDK environment was without
> > trying
> > to replicate all the details of the eSDK installer such as these
> > local.conf settings.
>
> I have now split up the populate_sdk_ext into separate functions [1]
> for better maintainability, and the more I think about what to do
> next, the more I agree with Adrian. I just don't see why (in a
> standard yocto build) would we want to manipulate PATH to provide a
> restricted set of tools, or to create a "local.conf+extra stuff"
> (locked signatures, esdk tweaks) environment, when existing
> local.conf
> by itself is already working fine, and full set of tools is better
> than a restricted one. If we want to add or remove locked signatures,
> this can be done with 'bitbake -s lockedsigs' or bblock for specific
> recipes only. And SDK's cross-toolchain is accessible via
> meta-ide-support/build-sysroots flow.
>
> [1]
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
Splitting allows to copy the new function with the + from the patch
into this email and comment it.
+python copy_buildsystem () {
+ import oe.copy_buildsystem
+
+ baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
+
+ # Determine if we're building a derivative extensible SDK (from
devtool build-sdk)
+ derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
What's the advantage of that? There are now two worlds: The bitbake
world and the SDK world which behave similar but not equal. Both need
to be maintained and tested.
We use only bitbake on our CI/CD infrastructure. For users who have
only the eSDK installed, it's hard to understand what the CI does and
even harder to reproduce errors happening on the CI.
We also had some setups with a containerized eSDK on the CI for the
integration of application source code. But this did not work at all.
The SDK container is basically outdated when the first component of the
firmware gets updated. If other components depend on that component an
SDK container update is required. Handling breaking changes easily is a
big advantage of bitbake which gets lost when using any kind of locked
or badly packaged variant of it.
+
+ conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers =
copy_bitbake_and_layers(d, baseoutpath, derivative)
Changing the directory layout leads to many pitfalls especially if more
layers than just poky are used. This should be replaced by the new
bitbake-layers tools. This means there is only one official way for
setting up bitbake layers and the folder structure gets exactly
replicated.
+
+ write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath,
core_meta_subdir)
Not sure there is much left when we have only the bitbake world. But
defining some defaults might be still useful.
+
+ write_unlocked_sigs(d, baseoutpath)
Lets turn this more towards a QA check. As a SDK maintainer I would
like to provide SDKs with 100% sstate included. But as a user, if I
have a choice between waiting a few minutes until bitbake compiled some
missing parts or getting an error message telling me I can't get an SDK
now, I'd probably choose compile.
If I remember correctly, with the old eSDK installer this is even
worse. This error happens during the installation which leads to an SDK
in an undefined state. The user must delete it again and fix the
generation of the SDK installer, which might be a very complicated and
time consuming task with the existing tools.
+
+ write_bblayers_conf(d, baseoutpath, sdkbblayers)
Also something which can be replaced by the new bitbake-layers utility.
+
+ uninative_checksum = copy_uninative(d, baseoutpath)
Not sure if this is still needed. With a bitbake environment this just
happens from sstate, I guess. So why doing it differently for the SDK?
+
+ write_local_conf(d, baseoutpath, derivative, core_meta_subdir,
uninative_checksum)
Also something which can be replaced by the new bitbake-layers utility.
+
+ prepare_locked_cache(d, baseoutpath, conf_initpath)
The sstate could be shipped in three different ways:
* Included in the installer and just extracted into $SSTATE_DIR. This
is simple but it does not scale at all. If you maintain multiple
distros and MACHINES and want to have fast update cycles,
distributing complete sstate archives quickly becomes practically
impossible, as the same data is packed into several huge archives.
That leads to issues on the infrastructure side. But also on the
user's machine having one sstate folder e.g. ~/sstate-cache instead
of several $TMPDIR/sstate-cache folders is beneficial.
* No sstate is included, the sstate gets "lazy" fetched from
SSTATE_MIRROR. Also that looks easy but does not scale very well for
the SDK use case. bitbake opens a connection for every setscene task
which is usually handled as a denial of service attack by modern
infrastructures.
Fetching in the setscene tasks does not cache the sstate artifact
locally (not 100% sure here). So whenever the same setscene tasks
runs again a new download is needed. This is not efficient and also
not the quickest possible implementation. Local caching is important
for the sstate.
* An sstate-download script which downloads the sstate from the
SSTATE_MIRROR into SSTATE_DIR before bitbake gets started is working
very well for us. We share the sstate cache via sftp. This allows us
to use the public keys which we anyway need for git for the
authentication and authorization. Since the download script opens
more than one connection for downloading the speed is also very
nice. We are still evaluating this. The script is not really
upstreameable. It's very specific to our infrastructure.
Probably a bad idea, or something for later: A download script would
have all the logic for finding the required sstate files on the client.
Not sure if also a server side implementation could be interesting. I'm
thinking about a service with access to the shared sstate of a build
infrastructure. The client could ask for an sstate bundle for a given
recipe and the server would pack it on the fly. Similar to what git
does. Intelligence on the server side would also allow to have a fine
grained authorization concept e.g. per sstate artifact.
+
+ write_manifest(d, baseoutpath)
Not sure if this is still needed.
+
+}
>
> THAT SAID. I also agree there should be a way to package the
> 'standard
> yocto build' that does not also do 'weird esdk stuff' on top. Just a
> tarball that has 'everything', and that you unpack and get a standard
> build exactly like the one that was used to produce the tarball.
> Something like what Adrian suggested: basically place all information
> needed to replicate a build into one place, then either make a
> tarball
> out of it (with sstate for fast builds), or publish it into the
> network. We might be able to reuse some of esdk packaging code for
> this, or we might not, but I think this is the direction that should
> be taken.
Ideally the "new installer" can be extracted/installed over an existing
SDK=bitbake setup. I think the bitbake-layer tool supports this for the
layers. Extracting a full sstate archive into an already populated
folder should also just work, I assume.
Such a design would allow more advance users to just use git and an
editor to change their SDK. But it would also support less experienced
users with a process where they can just download a minimal installer
which brings the SDK=bitbake into a defined state.
Sorry for repeating some parts which we already had in other emails.
But I tried to summarize the lengthy discussion a bit in one place.
Thank you very much!
Adrian
>
> Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61572): https://lists.yoctoproject.org/g/yocto/message/61572
Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
