On 2026-05-05 06:17, Alexander Kanavin via lists.openembedded.org wrote:
Hello,
bitbake-setup currently doesn't support describing build pipelines: a
sequence of commands that actually produce something useful. For a
true end-to-end solution this gap should be filled.
Here's my proposal, which probably has obvious shortcomings, that I
would like to hear about :)
Hi,
Thanks for submitting this proposal. I will try to summarize some of my
thoughts from the public call and otherwise in-line.
0. Design goals:
- something suitable for most situations, including newcomers running
a local build, and CI doing a nightly job
- avoid custom scripts in most situations
- self-documented; everything must be described
- extensible; if there's something missing that no one thought about,
it can be added later in a way that supplements existing features
without replacing them.
1. Specification in a build configuration would look like this:
"build-targets": {
"steps" : [
{ "name": "build-apple", "description": "Build an apple for
the purpose of making apple pie", "command": "bitbake apple-image"},
{ "name": "build-orange", "description": "Build an orange for
the purpose of making orange juice", "command": "bitbake
orange-image"},
{ "name": "test-apple", "description": "Test an apple using
testimage class", "command": "bitbake -c testimage apple-image"},
{ "name": "test-orange", "description": "Test an orange using
testimage class", "command": "bitbake -c testimage orange-image"},
{ "name": "qemu-console", "description": "Start the last built
image in qemu with console UI", "command": "runqemu kvm snapshot
nographic"},
{ "name": "publish", "description":"Copy build artefacts to a
public download server", "command":
"path/to/publish-image-executable"}
],
"pipelines": [
{"name": "publish-apple", "description": "Build, test, and
publish apple (requires publishing rights on the download server)",
"steps": ["build-apple", "test-apple", "publish"]},
{"name": "publish-orange", "description": "Build, test, and
publish orange (requires publishing rights on the download server)",
"steps": ["build-orange", "test-orange", "publish"]},
{"name": "qemu-apple", "description": "Build and start apple
in qemu with console UI", "steps": ["build-apple", "qemu-console"]},
{"name": "qemu-orange", "description": "Build and start orange
in qemu with console UI", "steps": ["build-orange", "qemu-console"]}
]
}
Configurations would include a list of possible pipelines:
"configurations": [
{
"name": "fruitgarden",
"description": "Alex's fruit garden",
"bb-layers": ["meta-fruit"],
"oe-fragments": [ "distro/fruit-garden", "machine/earth" ],
"pipelines": ["qemu-apple", "qemu-orange",
"publish-apple", "publish-orange"]
}
I think it's worth considering for the nightly-job-in-CI case whether
people will want to use bitbake-setup to cover that entire scope (i.e.,
build-test-publish). There are a lot of existing CI systems out there
(Buildbot, Tekton, GitHub/Gitea/Forgejo Actions, GitLab CI, etc.) where
you already define pipelines as a series of stages/steps, and each one
is generally intended to cover either a single command or a handful of
closely-associated ones. I am not sure that users would want to invoke a
bitbake-setup pipeline as a single step in these systems, as that would
potentially obfuscate failures and configuration issues just by having
too many things happening per step. However, I can see it being useful
to people who want a fire-and-forget reproducer for typical build/test
configurations.
As a side note, it feels like stepping outside of the "build setup"
space and including things like runtime testing, artifact publishing,
etc. feels like a clash with bitbake-setup's name.
2. Nomenclature:
- "step" is a single command that is also given a short name and a
longer description. Steps are building blocks for pipelines and can't
be individually executed.
Why not? Why not have the ability to run something like 'bitbake-setup
run-step <step_name'? That would actually be an advantage this feature
could provide over many other CI systems, as it's usually difficult or
impossible to rerun individual steps (you have to re-run a whole job or
pipeline). I can see this being very useful for example when you need to
run 'bitbake -c testimage' in a loop many times to try and catch
intermittent failures.
- "pipeline" is a sequence of steps, that is also given a short name
and a longer description. Pipelines can mix and match defined steps as
they please.
- each configuration includes a list of possible targets, and once it
is set up, any of them can be executed (see below for the UI).
3. User Interface to pipelines
Once there is a setup, there are two ways to execute a pipeline:
a) For each pipeline there is a shell script that can be run directly:
/path/to/bitbake-builds/fruitgarden/qemu-apple
/path/to/bitbake-builds/fruitgarden/publish-apple
etc.
qemu-apple script would look something like:
#!/bin/sh
# Build and start apple in qemu with console UI
. init-build-env
# Build an apple for the purpose of making apple pie
bitbake apple-image
# Start the last built image in qemu with console UI
runqemu kvm snapshot nographic
e.g. assembled from pipeline steps and their descriptions.
Similarly here - maybe a 'bitbake-setup run-pipeline <pipeline_name>'
would be more effective?
b) Interactive UI :'bitbake-setup build'
This would present an interactive UI similar to 'bitbake-setup init', e.g:
$ bitbake-setup build
Available build pipelines:
1. qemu-apple Build and start apple in qemu with console UI
2. qemu-orange Build and start orange in qemu with console UI
... etc
Please choose one of the above: 1
The following commands will be executed:
# Build an apple for the purpose of making apple pie
bitbake apple-image
# Start the last built image in qemu with console UI
runqemu kvm snapshot nographic
Proceed? [y/N]
etc.
4. Implementation
This should actually be fairly straightforward. There's no need to add
new features to bitbake (e.g. 'standard target definitions' in layers
was considered at some point, but after more consideration I think it
can be postponed or perhaps altogether avoided), or modify oe-core.
My other main concern about this is around maintenance - depending on
how it is ultimately implemented, it could end up looking like a
miniature CI/CD system for Yocto builds - that is certainly how I find
myself seeing it, and I've written my responses in that mindset. I can
see the utility in having the feature available more generally to reduce
time spent reproducing a common build/test case, but I wonder if it'd
get used often enough under its currently proposed form.
Trevor
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2325):
https://lists.openembedded.org/g/openembedded-architecture/message/2325
Mute This Topic: https://lists.openembedded.org/mt/119158252/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
