On Mon, Dec 2, 2019 at 9:37 PM Mark Hatle <[email protected]> wrote: > > Problem Statement > ----------------- > In the current world there is an increasing number of heterogeneous > systems being developed. Currently these components can be built > independently of each other, and then combined later. For ease of use, > it would be nice to be able to build these systems with a single build, > including generating a final bootable image. A heterogeneous system may > simply be different configurations for different components, different > operating systems for different components, or a system made up of > diverse processor architectures. Recently the Yocto Project has added > multiconfig in order to enable these types of configurations, but > suggested workflows for various configurations are needed to avoid > confusion and to avoid developers implementing their own schemes. > > Proposal > -------- > During the course of discussing this with other people a number of terms > have been involved, and after numerous discussions it is clear that > people have a different definition for the various components. The > purpose of this document is to explain the various pieces of this > proposal, so that we can all be using the same items for the various > components necessary to build a heterogeneous system. > > For a homogeneous build, the traditional Open Embedded/Yocto Project > components that are used include: > > * Build Configuration (local.conf) > * MACHINE > * Specifies target device information, including hardware > capabilities, console settings, boot image configurations, etc. > These settings are used by MACHINE packages, as well as image > generation. > * Define MACHINE in the conf/local.conf file in the Build Directory. > * SOC_FAMILY (optional, but implied by the machine) > * Way to group together machines based upon common System On Chip > components. > * A SOC_FAMILY by itself is not a fully configured and bootable > machine, but may be used by a series of machines to specify the > common components. > * TUNE (typically specified implicitly by the machine) > * Specifies the CPU (instruction set), and ABI settings available to > the user > * DEFAULTTUNE then selects which of the available tunes is to be used. > This is usually set by the machine. > * DISTRO (Distribution Configuration) > * Specifies cross recipe configurations that together result in an > overall distribution configuration. > * Define DISTRO in the conf/local.conf file in the Build Directory. > If not specified, a default "nodistro" distribution is used. > * Recipes (.bb files) > * Specify how to generically build individual items. These recipes > will inherit the system wide settings from the distribution > configuration, and tune. For MACHINE specific packages they can > also inherit machine specific settings as well. > * Image Recipes > * Specify a series of dependencies that cause recipes to be built and > a list of resulting packages to be installed into a target image. > * Image recipes are responsible for constructing a filesystem image. > Further the system can extend these into a bootable disc image > format. > > In any sort of heterogeneous configuration we want to use and build upon > the existing homogeneous components. A heterogeneous solution is really > comprised of a number of homogeneous configurations that when deployed > together result in a fully functional device. In other words, each of > individual parts of the heterogeneous build are standalone and not tied > to the assembly of a specific system. > > Based on this, we want to avoid any changes that complicate the existing > homogeneous build components or even adding additional levels of > configuration as this will complicate existing and future uses. For > example, in the past there have been suggestions for SUBMACHINE or other > levels of hierarchy between SOC_FAMILY and MACHINE. Adding this level > of indirection can make it more difficult to combine different > configurations into new heterogeneous solutions. For example, if > someone has already defined a heterogeneous solution using a MACHINE, > SUBMACHINE, and SOC_FAMILY heirachy and you wish to extend it the > existing MACHINE / SUBMACHINE may interfear with your own systems unique > configuration. > > There are a few types of heterogenous systems that I have seen. Each of > them can be constructed by combining the output of homogeneous > configurations. The most basic heterogenous systems I have seen include > either a collection of containers or different OSes, a primary CPU + > co-processors, or all of the CPUs are independent of each other but > share resources. It is also possible to combine these heterogenous > configurations such as a multiple CPU system, with some CPUs having > co-processors with one or more CPU running containers. > > In the case of the container based system, you really want a master > homogeneous machine configuration along with a few additional > configurations that can be incorporated into that image. Using a > multilib configuration you would have something like: > > build (build directory) > conf > local.conf: > MACHINE = "genericx86_64" > DISTRO = "poky" > BBMULTICONFIG = "container1 containter2" > multiconfig > container1.conf: > MACHINE = "genericx86_64" > DISTRO = "mydistro1" > TMPDIR = "${TOPDIR}/tmp/multi/container1" > container2.conf: > MACHINE = "genericx86_64" > DISTRO = "mydistro2" > TMPDIR = "${TOPDIR}/tmp/multi/container2" > layers > meta-<custom_layer> > conf > distro > mydistro1.conf > mydistro2.conf > recipes-images > microservices > service-image-1.bb > service-image-2.bb > other > my-custom-image-recipe.bb: > do_image[mcdepends] = "mc:container1:service-image1:do_rootfs \ > mc:container2:service-image2:do_rootfs" > do_image() { ... instructions for combining stuff ... } > > > This configuration allows for individual configurations for each > container and changes to the multiconfigs, but general re-use of much of > the system. Especially if the distros in each one are same or similar. > > Similar to the above, you could use a multiconfig system to combine > different operating systems. For instance, instead of building a > standard-alone bare-metal style bootloader as part of the OS > configuration, you could think of it as an external non-OS application. > A configuration might look like: > > build > conf > local.conf: > MACHINE = "genericx86_64" > DISTRO = "poky" > BBMULTICONFIG = "bootloader" > multiconfig > bootloader.conf: > MACHINE = "genericx86_64" > DISTRO = "baremetal" > TMPDIR = "${TOPDIR}/tmp/multi/bootloader" > layers > meta-mybootloader > conf > distro > baremetal.conf > recipes > newlib > newlib_ver.bb > first_stage > first_stage.bb > second_stage > second_stage.bb > bootloader > bootloader.bb > DEPENDS = "newlib first_stage second_stage" > meta-<custom_layer> > recipes > bootloader > mybootloader.bb > depend on mc:bootloader:second_stage:.... > > > For a simple heterogeneous solution, where the main CPU may need to load > software for co-processors, the configuration would be similar to the > above bare-metal example. The difference would be that the MACHINE and > DISTRO settings for a DSP would need to have the necessary > configurations to properly build applications for the DSP. Then the > Linux side of things can take this baremetal software and package it up > independently or in conjunction with software to actually configure and > load that software: > > build > conf > local.conf: > MACHINE = "myarmccpu" > DISTRO = "poky" > BBMULTICONFIG = "dsp" > multiconfig > dsp.conf: > MACHINE = "magic-dsp" > DISTRO = "baremetal" > TMPDIR = "${TOPDIR}/tmp/multi/dsp" > layers > meta-dsp > conf > distro > baremetal.conf > machine > dsp.conf > recipes-dsp > newlib > newlib_ver.bb > library > library.bb > application > application.bb > DEPENDS = "newlib library application" > recipes-linux > dsp-application > ...depends on application... > > Note: in the above, separating the distro, machine and application > components into individual layers may be needed for Yocto Project > compliance. > > > In all of the above examples, the user has manually configured the > multiconfig within their project. There is a simple way to move that > configuration to a layer, simply place it in a conf/multiconfig > directory within that layer. > > This ability suggests to be that there should be a standard way to > specify a layer above that of machine, which defines the overall > characteristics of the system. > > I'm proposing calling this new layer type as a system layer and it's > configuration variable "SYSTEM". It will be used instead of MACHINE > when there is no single machine to quantify the contents of the produced > system image. When implementing a system, we do not want to make major > changes to any other components. Due to the existing implementation > requiring MACHINE and certain TUNE parameters, this will require us to > provide a special MACHINE value that can be used for a heterogeneous > system. I suggest we create a new 'nomachine' system that only > defines/uses an equivalent noarch style tune. This will instruct the > system that this configuration can be used to noarch software and create > images (including with wic), but it is not able to compile specific > applications. Each of these applications or images must come from a > defined MACHINE. > > The SYSTEM level multiconfig could be used to combine any homogeneous or > heterogeneous configuration. For example: > > build > conf > local.conf: > SYSTEM = "mysystem" > layers > meta-<system> > conf > system > mysystem.conf > MACHINE = "nomachine" > BBMULTICONFIG = "bootloader fpga linux" > mysystem.wks > multiconfig > bootloader.conf: > MACHINE = "zcu102_microblaze" > DISTRO = "baremetal" > TMPDIR = "${TOPDIR}/tmp/multi/bootloader" > fpga.conf: > MACHINE = "zcu_fpga" > DISTRO = "baremetal" > TMPDIR = "${TOPDIR}/tmp/multi/fpga" > linux.conf: > MACHINE = "zcu_cortex-a72" > DISTRO = "poky" > TMPDIR = "${TOPDIR}/tmp/multi/linux" > recipes > images > system-images.bb > do_image[mcdepends] = "mc:bootloader:application:do_deploy \ > mc:fpga:application:do_deploy \ > mc:linux:core-image-minimal:do_rootfs" > do_image() { ... instructions for combining stuff ... } > meta-<machine> > conf > machine > zcu_microblaze.conf > zcu_fpga.conf > zcu_cortext-a72.conf > meta-mybootloader > conf > distro > baremetal.conf > recipes > newlib > newlib_ver.bb > first_stage > first_stage.bb > second_stage > second_stage.bb > bootloader > bootloader.bb > DEPENDS = "newlib first_stage second_stage" > meta-fpga > conf > distro > baremetal.conf > recipes-baremetal > newlib > newlib_ver.bb > library > library.bb > application > application.bb > DEPENDS = "newlib library application"
A lot of what you're talking about here we've been doing for a while in meta-oryx with SYSTEM_PROFILE and APPLICATION_PROFILE. https://gitlab.com/oryx/meta-oryx/tree/master ├── conf │ ├── application-profiles │ │ ├── full-cmdline.conf │ │ ├── host.conf │ │ ├── host-mender-update-modules.conf │ │ ├── host-test.conf │ │ └── minimal.conf │ ├── distro │ │ └── oryx.conf │ ├── layer.conf │ └── system-profiles │ ├── guest-mender-update-module.conf │ ├── native.conf │ └── native-mender.conf In what we're doing, we're mainly focusing on native hosts and containerised guests, but there is no reason this couldn't be expanded to fit your use case. -b > > So proposal: > > Introduce a new OPTIONAL "SYSTEM" variable. Which, like machine, would > automatically "include conf/${SYSTEM}.conf", if defined. > > Introduce nomachine and notune, and adjust system components to know how > to deal with these properly. I.e. recipes that are NOT noarch would be > be rejected in this configuration. > > Introduce system layers to the yocto check script, and layer index. > > Add documentation explaining the new SYSTEM variable and it's usage for > building complex heterogeneous systems. > > Option: > > If SYSTEM (as described above) does not make sense, then we could re-use > MACHINE instead, but I would still like a notune option for a 'machine' > that shouldn't have software compiled for it. > > --Mark > This email and any attachments are intended for the sole use of the named > recipient(s) and contain(s) confidential information that may be proprietary, > privileged or copyrighted under applicable law. If you are not the intended > recipient, do not read, copy, or forward this email message or any > attachments. > Delete this email message and any attachments immediately. > _______________________________________________ > Openembedded-architecture mailing list > [email protected] > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture _______________________________________________ Openembedded-architecture mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
