On Wed, Mar 30, 2016 at 8:07 PM, Christopher Barry <
christopher.r.ba...@gmail.com> wrote:
> On Wed, 30 Mar 2016 19:03:59 +0200
> Baurzhan Ismagulov <i...@radix50.net> wrote:
>
> >Hello,
> >
> >we're developing an integration system for building Debian-based
> >products, isar [1]. It is essentially a set of commands for building
> >product packages and installing them into a Debian root filesystem to
> >produce a complete working image. It aims at supporting the whole
> >product development lifecycle:
> >
> >1. Create a base system:
> > * Define a list of binary Debian packages as a base system.
> > * Define base system customizations, such as inittab, fstab, etc.
> >
> >2. Use a clean environment (chroot / VM) for building the following
> >components:
> > * Product's modified Debian packages.
> > * Product's own driver, library, and application packages.
> >
> >3. Support multiple diverse targets in one project:
> > * Debug / release builds.
> > * Images for different products.
> > * Images for different hardware platforms.
> >
> >4. Good support for typical project use cases, such as:
> > * Create complete, bootable images automatically, with one command.
> > * Do that reproducibly, for any older version.
> >
> >
> >= Prior Art =
> >
> >The task is trivial enough that everyone develops his own scripts for
> >that [2]. After maintaining such scripts since 2004, we'd like to have
> >something better. Reasons:
> >
> >* The scripts have become a mess and are difficult to maintain.
> >
> >* They don't scale well for the following use cases:
> >
> > * Maintaining several products (sharing or not sharing components)
> > leads to exploding complexity.
> >
> > * The same applies for several levels of development (upstream
> > software, Debian packages, hardware vendors' BSP components, software
> > vendors' libraries, product's own components).
> >
> >* They don't offer structure, and developers happen to solve every
> >next problem
> > in a different way.
> >
> >* "Every tool is tailored for the developer's use case and personal
> >taste" [3].
> >
> >A build system should Just Work(tm), be easy to use, offer structure,
> >and be flexible. The best would be to combine individual efforts to
> >avoid solving the same problems many times and constantly improve a
> >single toolset that is suitable for various use cases.
> >
> >The closest to that vision is ELBE [4]. It solves the right problem
> >and has many features OOTB -- very important if you want to focus on
> >your end product and not develop another build system. However, its
> >current implementation doesn't scale well for our use cases, and we
> >expect maintainability issues in the projects using it.
> >
> >
> >= How Isar Works =
> >
> >Isar follows the one job, one tool approach. Integration is achieved
> >by using BitBake, a build tool similar to make. It is the tool behind
> >the source-based Yocto embedded Linux distribution. With BitBake, one
> >can define files with build targets (recipes like coreutils, busybox),
> >their dependencies (other recipes like libc), and rules how to build
> >them (shell or python statements). All recipes together are
> >collectively called "metadata" as opposed to the source code itself,
> >which may contain configure, Makefile.am, debian/, etc.
> >
> >Isar is implemented as a set of BitBake recipes. There is one recipe
> >per source package. Build target is the Debian package. Since download
> >("fetch") and build rules ("tasks") in Debian are uniform thanks to
> >apt and dpkg, they are defined once in a BitBake class and don't have
> >to be duplicated in every recipe. The source code of individual
> >packages may be a part of an isar-based project, or it may be in a
> >different (public or private) repo. The recipes check out the software
> >packages from their repos and call dpkg-buildpackage on them. The
> >packages are later installed into the root filesystem using
> >multistrap. There are separate recipes for creating the build chroot
> >and the final rootfs.
> >
> >Conceptually, the whole build system can be seen as a result of
> >splitting one monolithic script into chunks within a predefined
> >structure. Recipes, in turn, are grouped in directories ("layers") by
> >distro, vendor, product, or target platform.
> >
> >The structure is achieved as follows:
> >
> >* On file level, by BitBake recipes. Different packages are built in
> >different
> > recipes. Recipe execution order results from dependencies specified
> > in the recipes.
> >
> >* On directory level, layers:
> >
> > * meta: Base system recipes (e.g., coreutils).
> >
> > * meta-VENDOR-bsp: Hardware vendor's recipes (e.g., drivers, libs).
> >
> > * meta-PRODUCT: Product recipes and customizations (e.g., product
> > apps and libs, partitioning, splash screen).
> >
> > Layer directories may be a part of a project repo, or be external to
> > it. In this way, anyone can fork an upstream repo, add his own layers
> > (possibly extending or overriding the core recipes), and pass that to
> > his downstream without having to fork and modify the core recipes
> > (thus reducing maintenance effort). The desired layer mix is chosen
> > in the build config file by the end developer.
> >
> >* Different target platforms are chosen by specifying the MACHINE
> >variable
> > in the BitBake configuration file in the local build directory.
> >
> >* Component sharing between products is achieved by dependencies
> >specified in
> > BitBake recipes. If several products share 90% of components, their
> > top-level recipes depend on the recipe of a common base system,
> > similar to Debian metapackages.
> >
> > Component variation (kernel, U-Boot) is achieved by the Provides:
> > mechanism, like in Debian.
> >
> > Depending on the extent of differences, product variation may be
> > achieved by different build targets or co-existing meta-PRODUCT1,
> > meta-PRODUCT2 layers.
> >
> >With added structure, flexibility of "raw" scripts is also preserved:
> >BitBake recipes contain procedural shell or Python code. New tasks and
> >their order can be defined in any recipe; the development is not
> >blocked if the build system doesn't provide an appropriate hook.
> >
> >An important issue is readable SCM history. With isar, editing one big
> >build script or mixing changes to code and blobs (e.g., tarballs) can
> >be avoided. Patches tend to be very local and very readable.
> >
> >
> >= Workflow =
> >
> >The workflow is similar to that of Yocto.
> >
> >To create a project based on isar, its repository is cloned to become
> >the project repository. Then, recipes for project-specific packages
> >are added. Debian package recipes are almost one-liners with no
> >procedural code necessary by default.
> >
> >To build the images, a build directory must be created and configured.
> >This is done with a helper script: isar-init-build-env <dir>. The
> >config files in <dir>/conf/ can be customized.
> >
> >System images are built with one command: bitbake <image name>. A
> >default image recipe is supplied in the isar repo. Packages are built
> >natively using a chroot with qemu (cross-building is also possible).
> >Root filesystem customizations go into multistrap scripts.
> >Repeatability is achieved by versioning apt repos and referencing the
> >tags in project recipes.
> >
> >
> >= Advantages of Isar =
> >
> >* Modularity, flexibility and scalability through using BitBake.
> >
> >* Proven-in-use structure and workflows of the Yocto project.
> >
> >* Re-use of familiar, mature tools like BitBake, multistrap, dpkg,
> >apt, qemu.
> >
> >* Re-use of pre-built binary Debian packages rather than building them
> >from
> > scratch every time.
> >
> >
> >= What's next? =
> >
> >Siemens is using a predecessor of this system and would like to
> >develop it further. What we are looking for is a discussion. Do you
> >see a need for uniting forces to develop such a system as a community?
>
Absolutely. Hitachi has created meta-debian:
https://github.com/meta-debian/meta-debian While it is not identical to the
ISAR work it is similar in that it assembles bitbakeable meta-data. It also
shows companies are interested in building Debian this way.
> >Would you use it for your next project?
>
I will test it to build a GENIVI image.
>
Do you think this is the
> >optimal approach? Feedback and questions welcome. We'd like to combine
> >efforts under a shared vision rather than having everyone developing
> >his own tool.
>
I think that you ought to announce this more widely, not just in other
debian projects, like debian-arm, but more widely in the ecosystem of
embedded SoCs.
In addition, since you seem to have a connection with Siemens, it would
seem likely that you'd be interested in the OSADL / Linux Foundation work
regarding functional safety (ISO 262626) and PREEMPT_RT. While the real
time patches are apparently going to make their way into the kernel via the
work group[0], OSADL is looking to certify a Debian based build as
safety-critical. Using ISAR as a build tool might provide an important
piece of the puzzle for such a project, especially as you focus on
reproducibility, since process around the development of safety-critical
software is often as important as the code itself.
0. http://www.linuxfoundation.org/collaborate/workgroups/real-time
Regards,
Jeremiah
>
> >References:
> >
> >1. Proof-of-concept implementation: https://github.com/ilbers/isar
> >2.
> >
> http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/
> >3.
> >
> http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/24
> >4. https://lists.debian.org/debian-embedded/2016/01/msg00000.html
> >
> >
> >With kind regards,
>
>
> Looks very interesting. Per your TODO, it looks like you're
> already thinking about what I had questions about, e.g. dependencies
> (would there be specification duplication here?)
>
> Curious, was there a reason you did not submodule bitbake in your repo?
> Seems like you should keep bb current, but also be able to reproduce an
> image at the originally recorded release-time bb commitish too if
> desired/required.
>
> --
> Regards,
> Christopher
>
>
--
Jeremiah C. Foster
GENIVI COMMUNITY MANAGER
Pelagicore AB
Ekelundsgatan 4, 6tr, SE-411 18
Gothenburg, Sweden
M: +46 (0)73 093 0506
jeremiah.fos...@pelagicore.com
