Since the subject of how to manage product platforms based on OE interests me very much, I took the liberty of giving this discussion its own subject since it really hasn't got much to do with the original discussion about splitting meta-oe.
> -----Original Message----- > From: openembedded-devel-boun...@lists.openembedded.org > [mailto:openembedded-devel-boun...@lists.openembedded.org] On Behalf Of > Otavio Salvador > Sent: den 22 februari 2018 10:41 > To: Patrick Ohly <patrick.o...@intel.com> > Cc: OpenEmbedded Devel List <firstname.lastname@example.org> > Subject: Re: [oe] Splitting meta-oe? > > On Thu, Feb 22, 2018 at 6:27 AM, Patrick Ohly <patrick.o...@intel.com> wrote: > > On Thu, 2018-02-22 at 07:53 +0100, Jonas Bonn wrote: > >> On 21 February 2018 at 15:09, Martin Hundebøll <m...@prevas.dk> wrote: > >> > >> > Now that the discussion branched out a bit... > >> > > >> > We would like better support for this too. Our setup uses a "manifest" > >> > repository with git submodules to setup the layers: > >> > > >> > > yocto/ > >> > > meta-poky/ > >> > > meta-qt5/ > >> > > meta-foo/ > >> > > meta-bar/ > >> > > conf/ > >> > > bblayers.conf > >> > > local.conf > >> > > .gitmodules > >> > > >> > With this setup, customers simply need to clone our yocto repo > >> > recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then > >> > `bitbake image-recipe`. > >> > > >> > But this is rather inflexible, as it requires the "yocto" folder > >> > to be the build folder to activate the config files... > >> > > >> > We looked into putting the configs in "meta-foo/conf/*.conf.sample" and > >> > using TEMPLATECONF, but the "oe-init-build-env" script is rather picky > >> > about poky being the "top" directory. > >> > > >> > I guess the oe-init-build-env script can be changed to look for > >> > .templateconf in any parent folder? > >> > >> > >> Putting together a deliverable setup that's easy for the customer to get > >> started with is a bit tricky. Here's the approach that's worked well for > >> me: > >> > >> /myproject > >> /env <-- script > >> /build > >> /meta-myproject > >> /bitbake > >> /oe-core > >> /meta-layer1 > >> /meta-layer2 > >> > >> env, build, meta-myproject are part of the myproject repo, everything > >> else is a submodule. > > > > refkit used the same approach. One thing that I would prefer to do > > differently is the location of the submodule: having them in their own > > directory would make it more transparent which code is "external" and > > which is "internal". > > > >> "env" is a script containing just the following: > >> . ./oe-core/oe-init-build-env build/ bitbake/ > > > > We ended up with a top-level "oe-init-build-env" wrapper script around > > the actual oe-core/oe-init-build-env. That way the repo could be used > > the same way as poky. The script sets TEMPLATECONF, so the usual local > > build setup happens based on refkit sample files. > > We have a script set and a document which describes how we do it: > > http://doc.ossystems.com.br/managing-platforms.html > > The script sources can be seen at: > > https://code.ossystems.com.br/gitweb?p=ossystems-yocto-base-scripts.git > > -- > Otavio Salvador O.S. Systems > http://www.ossystems.com.br http://code.ossystems.com.br > Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750 As I see it there are two major use cases for how you work with layers. One is the ad hoc method, where you start with some base set of layers, e.g., Poky and then add layers as you need using, e.g., bitbake-layers. When you want to build something else you start over. This method is what BitBake and OE Core currently provides. The second method is the case where you have a well-defined set of layers for your project. Typically you want to build a specific product, and what layers are needed to do that is specified by someone else. Anyone who builds the same product uses the same set of layers. The set of layers are typically defined in some kind of manifest (e.g., as a Git repository with submodules as Martin mentions above or as a repo manifest as Otavio uses). Since I work for a company where we have some 80+ products that we can build based on OE, I will concern myself with the second method. We too base our platform on repo manifests. We store all our manifests in a single Git repository. Whenever a new project starts, they add a new manifest that specifies the layers they need. Typically this is done by referencing the manifest for our base platform and then add the project specific layer(s). This makes it very easy to add a new project, as the manifest file typically just looks something like: <?xml version="1.0" encoding="UTF-8"?> <manifest> <include name="cvp/cvp.xml"/> <project path="meta-foobar" name="layers/projects/meta-foobar" revision="master"/> </manifest> cvp.xml is the platform manifest for our Core Video Platform and is used by all our video products. It is actually a symbolic link to the CVP manifest for the current version of OE that we currently use, so right now it is cvp-pyro.xml. This means that when we want to upgrade to a new version of OE, all we do is redirect that link to, e.g., cvp-rocko.xml instead, and all products will from then on be based on Rocko instead. Once repo has checked out all layers specified in the manifest, we end up with something like: . |-- axis-oe-init | |-- add_paths | |-- create_bblayers_conf_sample | |-- create_sample_file | |-- find_all_layers | `-- oe-init-build-env |-- bitbake -> poky/bitbake |-- meta -> poky/meta |-- meta-axis | `-- conf | |-- conf-notes.txt.40 | |-- local.conf.sample.40 | `-- local.conf.sample.60 |-- meta-axis-bsp | `-- conf | |-- local.conf.sample.10 | `-- local.conf.sample.55 |-- meta-comptools | `-- conf | `-- local.conf.sample.65 |-- meta-gplv2 |-- meta-oe -> openembedded/meta-oe |-- meta-poky -> poky/meta-poky |-- meta-python -> openembedded/meta-python |-- meta-webserver -> openembedded/meta-webserver |-- oe-init-build-env -> axis-oe-init/oe-init-build-env |-- oe-init-build-env-memres -> axis-oe-init/oe-init-build-env |-- openembedded |-- poky |-- scripts -> poky/scripts |-- templateconf | |-- bblayers.conf.sample | |-- conf-notes.txt | `-- local.conf.sample `-- zzz-repo-workaround |-- openembedded `-- poky There are a couple of interesting observations that can be made from looking at the above. First, we too use a wrapper script to set up the environment. It is available to the user as a symbolic link from oe-init-build-env, so for anyone used to OE Core, it will look and work exactly as usual. What this script does is basically to gather the set of layers from the top directory that match "meta*" and create a bblayers.conf.sample file in a templateconf directory. Our own layers are typically Git repositories, and the layers from Poky and OpenEmbedded are symbolic links into those repositories. The other thing the wrapper script does is to create local.conf.sample and conf-notes.txt. This is done by looking in all the layers for files matching conf/local.conf.sample.XX and conf/conf-notest.txt.XX where XX is a number from 00 to 99. We treat the files from meta as if they had XX == 50. These files are then concatenated together in numeric order to form the final files. We came up with this solution to handle that layers may be combined in different combinations and it is not possible to have static bblayers.conf.sample and local.conf.sample files in the layers as it is impossible to know what other layers may be in use. I think this is similar to the hooks that Otavio's solution use. The last thing the wrapper script does is to add paths for scripts directories found in the various layers, in addition to the scripts directory provided by OE Core. The above should be typical examples of what support is needed from Bitbake to be able to extend the pretty static setup that OE Core currently provides to something that is flexible and adapts to the layers that are used, both when it comes to what layers go into bblayers.conf and the contents of local.conf. Finally, if anyone wonders about the zzz-repo-workaround directory above, the rest of this mail is a description of a repo "hack" I came up with to make "repo init" create links without having to bind them to a specific repository (normally links created by repo are listed in the manifest together with the respective repository, which makes it impossible to make them conditional). However, if one writes them like this (this example matches the links we set up by default for Poky): <project path="zzz-repo-workaround/poky" name="layers/repo-workaround" revision="master"> <linkfile src="../../poky/bitbake" dest="bitbake"/> <linkfile src="../../poky/meta" dest="meta"/> <linkfile src="../../poky/meta-poky" dest="meta-poky"/> <linkfile src="../../poky/scripts" dest="scripts"/> </project> we can then easily change the repository definition without having to also change the links (something that is needed if, e.g., a project manifest wants to override the revision of a layer that is specified in the platform manifests). Conditional links can then be specified like: <project groups="notdefault,meta-selftest" path="zzz-repo-workaround/meta-selftest" name="layers/repo-workaround" revision="master"> <linkfile src="../../poky/meta-selftest" dest="meta-selftest"/> </project> <project groups="notdefault,meta-skeleton" path="zzz-repo-workaround/meta-skeleton" name="layers/repo-workaround" revision="master"> <linkfile src="../../poky/meta-skeleton" dest="meta-skeleton"/> </project> <project groups="notdefault,meta-yocto-bsp" path="zzz-repo-workaround/meta-yocto-bsp" name="layers/repo-workaround" revision="master"> <linkfile src="../../poky/meta-yocto-bsp" dest="meta-yocto-bsp"/> </project> They can then be enabled via the --groups option to repo init, or if a project needs one of the conditional layers (e.g., meta-filesystems), it can add something like this in its manifest: <extend-project groups="default" path="zzz-repo-workaround/meta-filesystems" name="layers/repo-workaround"/> It is not very pretty, but it works. :) //Peter -- _______________________________________________ Openembedded-devel mailing list Openembeddedemail@example.com http://lists.openembedded.org/mailman/listinfo/openembedded-devel