Since the subject of how to manage product platforms based on OE 
interests me very much, I took the liberty of giving this discussion 
its own subject since it really hasn't got much to do with the 
original discussion about splitting meta-oe.

> -----Original Message-----
> From:
> [] On Behalf Of
> Otavio Salvador
> Sent: den 22 februari 2018 10:41
> To: Patrick Ohly <>
> Cc: OpenEmbedded Devel List <>
> Subject: Re: [oe] Splitting meta-oe?
> On Thu, Feb 22, 2018 at 6:27 AM, Patrick Ohly <> wrote:
> > On Thu, 2018-02-22 at 07:53 +0100, Jonas Bonn wrote:
> >> On 21 February 2018 at 15:09, Martin Hundebøll <> wrote:
> >>
> >> > Now that the discussion branched out a bit...
> >> >
> >> > We would like better support for this too. Our setup uses a "manifest"
> >> > repository with git submodules to setup the layers:
> >> >
> >> > > yocto/
> >> > >       meta-poky/
> >> > >       meta-qt5/
> >> > >       meta-foo/
> >> > >       meta-bar/
> >> > >       conf/
> >> > >            bblayers.conf
> >> > >            local.conf
> >> > >       .gitmodules
> >> >
> >> > With this setup, customers simply need to clone our yocto repo
> >> > recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then
> >> > `bitbake image-recipe`.
> >> >
> >> > But this is rather inflexible, as it requires the "yocto" folder 
> >> > to be the  build folder to activate the config files...
> >> >
> >> > We looked into putting the configs in "meta-foo/conf/*.conf.sample" and
> >> > using TEMPLATECONF, but the "oe-init-build-env" script is rather picky
> >> > about poky being the "top" directory.
> >> >
> >> > I guess the oe-init-build-env script can be changed to look for
> >> > .templateconf in any parent folder?
> >>
> >>
> >> Putting together a deliverable setup that's easy for the customer to get
> >> started with is a bit tricky.  Here's the approach that's worked well for
> >> me:
> >>
> >> /myproject
> >>     /env        <-- script
> >>     /build
> >>     /meta-myproject
> >>     /bitbake
> >>     /oe-core
> >>     /meta-layer1
> >>     /meta-layer2
> >>
> >> env, build, meta-myproject are part of the myproject repo, everything
> >> else is a submodule.
> >
> > refkit used the same approach. One thing that I would prefer to do
> > differently is the location of the submodule: having them in their own
> > directory would make it more transparent which code is "external" and
> > which is "internal".
> >
> >> "env" is a script containing just the following:
> >> . ./oe-core/oe-init-build-env build/ bitbake/
> >
> > We ended up with a top-level "oe-init-build-env" wrapper script around
> > the actual oe-core/oe-init-build-env. That way the repo could be used
> > the same way as poky. The script sets TEMPLATECONF, so the usual local
> > build setup happens based on refkit sample files.
> We have a script set and a document which describes how we do it:
> The script sources can be seen at:
> --
> Otavio Salvador                             O.S. Systems
> Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

As I see it there are two major use cases for how you work with layers. 

One is the ad hoc method, where you start with some base set of layers, 
e.g., Poky and then add layers as you need using, e.g., bitbake-layers.
When you want to build something else you start over. This method is 
what BitBake and OE Core currently provides.

The second method is the case where you have a well-defined set of 
layers for your project. Typically you want to build a specific product, 
and what layers are needed to do that is specified by someone else. 
Anyone who builds the same product uses the same set of layers. The 
set of layers are typically defined in some kind of manifest (e.g., as 
a Git repository with submodules as Martin mentions above or as a 
repo manifest as Otavio uses).

Since I work for a company where we have some 80+ products that we can 
build based on OE, I will concern myself with the second method.

We too base our platform on repo manifests. We store all our manifests 
in a single Git repository. Whenever a new project starts, they add a 
new manifest that specifies the layers they need. Typically this is done 
by referencing the manifest for our base platform and then add the 
project specific layer(s). This makes it very easy to add a new project, 
as the manifest file typically just looks something like:

<?xml version="1.0" encoding="UTF-8"?>
  <include name="cvp/cvp.xml"/>

  <project path="meta-foobar" name="layers/projects/meta-foobar" 

cvp.xml is the platform manifest for our Core Video Platform and is 
used by all our video products. It is actually a symbolic link to 
the CVP manifest for the current version of OE that we currently use, 
so right now it is cvp-pyro.xml. This means that when we want to 
upgrade to a new version of OE, all we do is redirect that link to, 
e.g., cvp-rocko.xml instead, and all products will from then on be 
based on Rocko instead.

Once repo has checked out all layers specified in the manifest, we 
end up with something like:

|-- axis-oe-init
|   |-- add_paths
|   |-- create_bblayers_conf_sample
|   |-- create_sample_file
|   |-- find_all_layers
|   `-- oe-init-build-env
|-- bitbake -> poky/bitbake
|-- meta -> poky/meta
|-- meta-axis
|   `-- conf
|       |-- conf-notes.txt.40
|       |-- local.conf.sample.40
|       `-- local.conf.sample.60
|-- meta-axis-bsp
|   `-- conf
|       |-- local.conf.sample.10
|       `-- local.conf.sample.55
|-- meta-comptools
|   `-- conf
|       `-- local.conf.sample.65
|-- meta-gplv2
|-- meta-oe -> openembedded/meta-oe
|-- meta-poky -> poky/meta-poky
|-- meta-python -> openembedded/meta-python
|-- meta-webserver -> openembedded/meta-webserver
|-- oe-init-build-env -> axis-oe-init/oe-init-build-env
|-- oe-init-build-env-memres -> axis-oe-init/oe-init-build-env
|-- openembedded
|-- poky
|-- scripts -> poky/scripts
|-- templateconf
|   |-- bblayers.conf.sample
|   |-- conf-notes.txt
|   `-- local.conf.sample
`-- zzz-repo-workaround
    |-- openembedded
    `-- poky

There are a couple of interesting observations that can be made 
from looking at the above.

First, we too use a wrapper script to set up the environment. It is 
available to the user as a symbolic link from oe-init-build-env, so 
for anyone used to OE Core, it will look and work exactly as usual. 
What this script does is basically to gather the set of layers from 
the top directory that match "meta*" and create a bblayers.conf.sample 
file in a templateconf directory. Our own layers are typically Git 
repositories, and the layers from Poky and OpenEmbedded are symbolic 
links into those repositories.

The other thing the wrapper script does is to create local.conf.sample 
and conf-notes.txt. This is done by looking in all the layers for 
files matching conf/local.conf.sample.XX and conf/conf-notest.txt.XX 
where XX is a number from 00 to 99. We treat the files from meta as 
if they had XX == 50. These files are then concatenated together in 
numeric order to form the final files. We came up with this solution 
to handle that layers may be combined in different combinations and 
it is not possible to have static bblayers.conf.sample and 
local.conf.sample files in the layers as it is impossible to know 
what other layers may be in use. I think this is similar to the hooks 
that Otavio's solution use.

The last thing the wrapper script does is to add paths for scripts 
directories found in the various layers, in addition to the scripts 
directory provided by OE Core.

The above should be typical examples of what support is needed from 
Bitbake to be able to extend the pretty static setup that OE Core 
currently provides to something that is flexible and adapts to the 
layers that are used, both when it comes to what layers go into 
bblayers.conf and the contents of local.conf.

Finally, if anyone wonders about the zzz-repo-workaround directory 
above, the rest of this mail is a description of a repo "hack" I came 
up with to make "repo init" create links without having to bind them to 
a specific repository (normally links created by repo are listed in the 
manifest together with the respective repository, which makes it 
impossible to make them conditional). However, if one writes them like 
this (this example matches the links we set up by default for Poky):

  <project path="zzz-repo-workaround/poky" name="layers/repo-workaround" 
    <linkfile src="../../poky/bitbake" dest="bitbake"/>
    <linkfile src="../../poky/meta" dest="meta"/>
    <linkfile src="../../poky/meta-poky" dest="meta-poky"/>
    <linkfile src="../../poky/scripts" dest="scripts"/>

we can then easily change the repository definition without having to also 
change the links (something that is needed if, e.g., a project manifest 
wants to override the revision of a layer that is specified in the platform 

Conditional links can then be specified like:

  <project groups="notdefault,meta-selftest" 
path="zzz-repo-workaround/meta-selftest" name="layers/repo-workaround" 
    <linkfile src="../../poky/meta-selftest" dest="meta-selftest"/>

  <project groups="notdefault,meta-skeleton" 
path="zzz-repo-workaround/meta-skeleton" name="layers/repo-workaround" 
    <linkfile src="../../poky/meta-skeleton" dest="meta-skeleton"/>

  <project groups="notdefault,meta-yocto-bsp" 
path="zzz-repo-workaround/meta-yocto-bsp" name="layers/repo-workaround" 
    <linkfile src="../../poky/meta-yocto-bsp" dest="meta-yocto-bsp"/>

They can then be enabled via the --groups option to repo init, or if 
a project needs one of the conditional layers (e.g., meta-filesystems), 
it can add something like this in its manifest:

  <extend-project groups="default" path="zzz-repo-workaround/meta-filesystems" 

It is not very pretty, but it works. :)


Openembedded-devel mailing list

Reply via email to