Greetings,
The Yocto Project has received funding for developing the processes and
tooling
for enabling binary distributions. As part of that, the project would
need some
form of prototype test binary distro to develop and test this. The main
goal of
this would be to develop the tools, and documentation needed to build a
binary
distribution and allow application and system developers to progress
through a
hierarchy of different uses of the resulting environment:
- Beginner use case: begin by fetching a binary image, ready to boot on a
target machine. Extend the system by installing extra applications
through
package feeds.
- Intermediate use case: tweak and rebuild existing packages, compile new
applications and modify images by using an eSDK and other binary
artifacts.
- Advanced use case: completely reorganize and optimize the system for a
specific target, switching to building from source
The first step in this effort is to define the scope of the prototype we
will
develop. We are looking for feedback and comments about issues we may
not have
considered.
Bear in mind that a prototype has to be limited in scope. However, the
developed tools, procedures and documentation will open a pathway for
the Yocto
Project and OpenEmbedded community to experiment with more architectures,
features, recipes and images.
Also bear in mind that the effort is not planning to adopt, replace or
repeat
any existing distributions such as Angstrom or Yoe.
Target architectures and machines
---------------------------------
The first prototype would only target the below architectures:
- "genericx86-64" architecture through the "genericx86-64" machine
(https://git.yoctoproject.org/poky/tree/meta-yocto-bsp/conf/machine/genericx86-64.conf)
- "arm64" architecture through the "qemuarm64" (and later "genericarm64"
when
hready) machine
(https://git.yoctoproject.org/poky/tree/meta/conf/machine/qemuarm64.conf)
Testing would happen through QEMU system emulation. The emulated
architectures
will be sufficient to test the outputs of this effort, but the developed
tooling and documentation should allow the community to experiment with
other
target architectures and machines.
Target tunes
------------
The binary images and other artifacts should be generated for the below
tunes:
- armv8a
(https://git.yoctoproject.org/poky/tree/meta/conf/machine/include/arm/arch-armv8a.inc)
- core2-64
(https://git.yoctoproject.org/poky/tree/meta/conf/machine/include/x86/tune-core2.inc)
See https://docs.yoctoproject.org/ref-manual/variables.html#term-DEFAULTTUNE
for details.
Distribution, Init Manager, C library
-------------------------------------
Images and package feeds will be generated for the "poky-altcfg"
distribution
(https://git.yoctoproject.org/poky/tree/meta-poky/conf/distro/poky-altcfg.conf),
using the systemd init manager.
Choosing systemd makes the binary reference distribution as generic as
possible, enabling the widest set of features, and being more familiar
to more
users. Of course, switching to other init managers and more restricted
features
will be possible by later compiling the system from source (advanced use
case).
This also implies the choice of the GNU C library (glibc). Selecting
glibc also
makes sense to enable the widest possible selection of packages, and some
packages have limited or no support for the Musl C library. That's typically
the case of systemd, which upstream only supports glibc.
Last but not least, people who want to use the Musl C library probably
do that
to optimize the size of their system, which is a use case not well
adapted to
using ready-made binary images. Such an advanced use case will be
addressed by
re-compiling from source, after testing initial prototypes with the binary
image based on glibc.
Binary images
-------------
The prototype will provide as a reference, for each targeted architecture:
- The root filesystem binary image, generated by building the
"core-image-full-cmdline" target.
- An image containing on-target development tools (e.g compiler, debugger)
- The corresponding SD card images to be booted through QEMU. Other boot
methods will be possible, but will be outside of the scope of the
reference
distribution.
Package feeds
-------------
For each targeted architecture, several package feeds will be available to
allow for bugfix or security updates or to extend the image with additional
packages:
- Architecture independent feed
- Architecture dependent feed
- Machine dependent feed
See
https://docs.yoctoproject.org/dev-manual/packages.html#using-runtime-package-management
for details.
The feeds for the first prototype will contain all the packages which are
supported by the "poky-altcfg" distribution as produced by the "bitbake
world"
command for the set of layers supplied by Poky.
Available reference package formats
-----------------------------------
To keep the first prototype simple, we will only release images and package
feeds based on the "ipk" format supported by "opkg" package manager.
Of course, the OpenEmbedded build system also supports the "deb" and "rpm"
package formats, so users will be able to generate their own images and
feeds
based on these other formats.
Another reason for not providing "deb" and "rpm" packages is that we do not
want inexperienced users to expect to be able to grab and use such packages
produced for regular Linux distributions (Debian, Fedora, etc.). Being built
with different C library versions and toolchains, the corresponding binaries
are most likely to be broken anyway.
Available reference binary artifacts
------------------------------------
The first prototype will make the below binary artifacts publicly available:
- Root filesystem images (as specified above)
- Binary package feeds (as specified above)
- PR (https://docs.yoctoproject.org/ref-manual/variables.html#term-PR)
database, so that people can manage updates to packages, by importing the
database in a local PR server.
- Prebuilt object data ("Shared State cache) through a public server
(see
https://docs.yoctoproject.org/ref-manual/variables.html#term-SSTATE_MIRRORS),
to reuse binary output already built by the Yocto Project
autobuilders, to
reduce the compile time of additional packages.
- Hash Equivalence data through a public server
(see
https://docs.yoctoproject.org/bitbake/2.4/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_HASHSERVE_UPSTREAM),
to increase the reusability of prebuilt objects.
Share State and Hash Equivalence data would be only useful to "Intermediate"
and "Advanced" use cases, to rebuild parts or the whole system from source.
Such artifacts will be served through a Content Delivery Network (CDN),
to make
sure that they can be downloaded fast enough from any location in the
world, in
a way that remains much faster than re-compiling such artifacts from
sources.
Defining policies and processes
-------------------------------
This reference binary distribution prototype will also drive the
documentation
of policies and process to address future work such as:
- What criteria should a new CPU architecture or platform tuning meet to be
added / tested?
- What criteria should other layers meet to be tested, in addition to being
already tested by the Autobuilder? Should these criteria only apply to
specific recipes or to the entire layers containing them?
Testing plans
-------------
Here are the tests we plan to implement:
- Producing and testing binary images through the autobuilder
- Making nightly builds available to testers
- Testing that package updates apply successfully:
- Within the current stable branch (nanbield): from one release
update to the
next, and from the latest release update to the tip of the branch
- Between release branches: from the latest update to our latest LTS
(kirkstone), to the latest update to the current stable release
(nanbield).
- Testing that it's also possible to build and deploy packages from the
meta-openembedded and meta-virtualization layers, as a way to assess the
relevance of our policies and process for additional layers or
recipes. This
would also be a way to demonstrate the ability to build container images,
which would be a great way to ship binary distributions.
We will work with recipe maintainers to publish fixes for update issues that
testing will find in recipes. Each issue will be reported in Bugzilla.
Thank you in advance for your review and feedback.
Cheers
Michael.
--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1846):
https://lists.openembedded.org/g/openembedded-architecture/message/1846
Mute This Topic: https://lists.openembedded.org/mt/102755562/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
