On Fri, Jul 1, 2022 at 3:29 PM Alexander Kanavin <[email protected]> wrote:
>
> First of all this provides a single command that does all of the work
> to capture the layers and their revisions from currently enabled set
> in a bitbake build. No other tool does this because they do not know
> anything about bitbake or tinfoil, and so you have to write your
> config by hand, with all of them.
>
> Here's how the captured metadata loks like [1]. I find it more useful
> for understanding what are we actually checking out than submodules'
> bare, unhelpful set of git revisions. It also doesn't require a
> super-repo containing all of the submodules: you can just toss the
> script into one of the layers and it will do the right thing for
> bootstrapping. There was also a comment from one of our customers: "
> In practice we learned that recursive submodules are not always
> correctly handled by GitLab CI. "
>
> I also find git submodules command line UI supremely confusing.
> Actually wait, that's what git as a whole is like: confusing.
> https://xkcd.com/1597/
>
> But if you would like, we can add a fourth output format to the
> bblayers plugin added in this patch that sets up git submodules
> instead of writing a standalone python script with embedded json
> (which is what I prefer).
Thanks for the write up. I don't particularly care if we go down this
route; it seems fine as long as it doesn't become "the one true way"
to do layer setup (because I'm still going to use submodules :) )
>
> Alex
>
> [1]
> {
> "meta-alex": {
> "branch": "master",
> "describe": "",
> "is_bootstrap": true,
> "layers": [
> {
> "name": "meta-alex",
> "path": ""
> }
> ],
> "remotes": [
> {
> "name": "remote-alex",
> "uri": "https://github.com/kanavin/meta-alex";
> }
> ],
> "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
> },
> "meta-intel": {
> "branch": "master",
> "describe": "15.0-hardknott-3.3-310-g0a96edae",
> "layers": [
> {
> "name": "meta-intel",
> "path": ""
> }
> ],
> "remotes": [
> {
> "name": "origin",
> "uri": "git://git.yoctoproject.org/meta-intel"
> }
> ],
> "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a"
> },
> "poky": {
> "branch": "akanavin/setup-layers",
> "describe": "4.1_M1-295-gfd524e4984",
> "layers": [
> {
> "name": "meta",
> "path": "meta"
> },
> {
> "name": "meta-poky",
> "path": "meta-poky"
> },
> {
> "name": "meta-yocto-bsp",
> "path": "meta-yocto-bsp"
> }
>
> ],
> "remotes": [
> {
> "name": "origin",
> "uri": "git://git.yoctoproject.org/poky"
> },
> {
> "name": "poky-contrib",
> "uri": "ssh://[email protected]/poky-contrib"
> }
> ],
> "rev": "fd524e4984eb535f16a14dd7d79fde661b44aa78"
> }
> }
>
> On Fri, 1 Jul 2022 at 21:46, Joshua Watt <[email protected]> wrote:
> >
> > Can you outline what this provides over `git submodules`? It seems
> > pretty similar and I'm struggling to see what this provides that isn't
> > already provided by that tool.
> >
> > On 7/1/22 14:24, Alexander Kanavin wrote:
> > > This addresses a long standing gap in the core offering:
> > > there is no tooling to capture the currently configured layers
> > > with their revisions, or restore the layers from a configuration
> > > file (without using external tools, some of which aren't particularly
> > > suitable for the task). This plugin addresses the gap.
> > >
> > > How to use:
> > >
> > > 1. Saving a layer configuration:
> > >
> > > a) Command line options:
> > >
> > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers
> > > create-layers-setup -h
> > > NOTE: Starting bitbake server...
> > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT]
> > > [--format {python,json,kas}] destdir
> > >
> > > Writes out a python script/kas config/json config that replicates the
> > > directory structure and revisions of the layers in a current build.
> > >
> > > positional arguments:
> > > destdir Directory where to write the output
> > > (if it is inside one of the layers, the layer
> > > becomes a bootstrap repository and thus will be excluded from fetching by
> > > the script).
> > >
> > > optional arguments:
> > > -h, --help show this help message and exit
> > > --output OUTPUT, -o OUTPUT
> > > File name where to write the output, if the
> > > default (setup-layers.py/.json/.yml) is undesirable.
> > > --format {python,json,kas}, -f {python,json,kas}
> > > Format of the output. The options are:
> > > python - a self contained python script
> > > that fetches all the needed layers and sets them to correct revisions
> > > (default, recommended)
> > > kas - a configuration file for the kas tool
> > > that allows the tool to do the same
> > > json - a json formatted file containing all
> > > the needed metadata to do the same by any external or custom tool.
> > >
> > > b) Running with default choices:
> > >
> > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers
> > > create-layers-setup ../../meta-alex/
> > > NOTE: Starting bitbake server...
> > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py
> > >
> > > 2. Restoring the layers from the saved configuration:
> > >
> > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should
> > > already contain setup-layers.py created in the previous step.
> > >
> > > b) Command line options:
> > >
> > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h
> > > usage: setup-layers.py [-h] [--force-meta-alex-checkout]
> > > [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR]
> > >
> > > A self contained python script that fetches all the needed layers and
> > > sets them to correct revisions
> > >
> > > optional arguments:
> > > -h, --help show this help message and exit
> > > --force-meta-alex-checkout
> > > Force the checkout of the bootstrap layer
> > > meta-alex (by default it is presumed that this script is in it, and so
> > > the layer is already in place).
> > > --choose-poky-remote {origin,poky-contrib}
> > > Choose a remote server for layer poky (default:
> > > origin)
> > > --destdir DESTDIR Where to check out the layers (default is
> > > /srv/work/alex/layers-test).
> > >
> > > c) Running with default options:
> > >
> > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py
> > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to
> > > override.
> > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae,
> > > branch master from remote origin at git://git.yoctoproject.org/meta-intel
> > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel'
> > > in /srv/work/alex/layers-test
> > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in
> > > /srv/work/alex/layers-test/meta-intel
> > > Note: multiple remotes defined for layer poky, using origin (run with -h
> > > to see others).
> > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch
> > > akanavin/setup-layers from remote origin at
> > > git://git.yoctoproject.org/poky
> > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in
> > > /srv/work/alex/layers-test
> > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in
> > > /srv/work/alex/layers-test/poky
> > >
> > > And that's it!
> > >
> > > FIXMEs:
> > > - kas config writer not yet implemented
> > > - oe-selftest test cases not yet written
> > >
> > > Signed-off-by: Alexander Kanavin <[email protected]>
> > > ---
> > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++
> > > .../templates/setup-layers.py.template | 77 ++++++++++++
> > > 2 files changed, 194 insertions(+)
> > > create mode 100644 meta/lib/bblayers/makesetup.py
> > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template
> > >
> > > diff --git a/meta/lib/bblayers/makesetup.py
> > > b/meta/lib/bblayers/makesetup.py
> > > new file mode 100644
> > > index 0000000000..3c86eea3c4
> > > --- /dev/null
> > > +++ b/meta/lib/bblayers/makesetup.py
> > > @@ -0,0 +1,117 @@
> > > +#
> > > +# SPDX-License-Identifier: GPL-2.0-only
> > > +#
> > > +
> > > +import logging
> > > +import os
> > > +import stat
> > > +import sys
> > > +import shutil
> > > +import json
> > > +
> > > +import bb.utils
> > > +import bb.process
> > > +
> > > +from bblayers.common import LayerPlugin
> > > +
> > > +logger = logging.getLogger('bitbake-layers')
> > > +
> > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
> > > +
> > > +import oe.buildcfg
> > > +
> > > +def plugin_init(plugins):
> > > + return MakeSetupPlugin()
> > > +
> > > +class MakeSetupPlugin(LayerPlugin):
> > > +
> > > + def _write_python(self, repos, output):
> > > + with open(os.path.join(os.path.dirname(__file__), "templates",
> > > "setup-layers.py.template")) as f:
> > > + template = f.read()
> > > + args = sys.argv
> > > + args[0] = os.path.basename(args[0])
> > > + script = template.replace('{cmdline}', "
> > > ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True,
> > > indent=4))
> > > + with open(output, 'w') as f:
> > > + f.write(script)
> > > + st = os.stat(output)
> > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP |
> > > stat.S_IXOTH)
> > > +
> > > + def _write_json(self, repos, output):
> > > + with open(output, 'w') as f:
> > > + json.dump(repos, f, sort_keys=True, indent=4)
> > > +
> > > + def _write_kas(self, repos, output):
> > > + raise NotImplementedError('Kas config writer not implemented
> > > yet')
> > > +
> > > + _write_config = {"python":_write_python, "json":_write_json,
> > > "kas":_write_kas}
> > > + _output_filename =
> > > {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"}
> > > +
> > > + def _get_repo_path(self, layer_path):
> > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel',
> > > cwd=layer_path)
> > > + return repo_path.strip()
> > > +
> > > + def _get_remotes(self, repo_path):
> > > + remotes = []
> > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path)
> > > + for r in remotes_list.split():
> > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r),
> > > cwd=repo_path)
> > > + remotes.append({'name':r,'uri':uri.strip()})
> > > + return remotes
> > > +
> > > + def _get_describe(self, repo_path):
> > > + try:
> > > + describe,_ = bb.process.run('git describe --tags',
> > > cwd=repo_path)
> > > + except bb.process.ExecutionError:
> > > + return ""
> > > + return describe.strip()
> > > +
> > > + def _make_repo_config(self, destdir):
> > > + repos = {}
> > > + layers =
> > > oe.buildcfg.get_layer_revisions(self.tinfoil.config_data)
> > > + for l in layers:
> > > + if l[1] == 'workspace':
> > > + continue
> > > + if l[4]:
> > > + logger.error("Layer {name} in {path} has uncommitted
> > > modifications or is not in a git repository.".format(name=l[1],path=l[0]))
> > > + return
> > > + repo_path = self._get_repo_path(l[0])
> > > + if repo_path not in repos.keys():
> > > + repos[repo_path] = {'rev':l[3], 'branch':l[2],
> > > 'remotes':self._get_remotes(repo_path), 'layers':[],
> > > 'describe':self._get_describe(repo_path)}
> > > + if not repos[repo_path]['remotes']:
> > > + logger.error("Layer repository in {path} does not
> > > have any remotes configured. Please add at least one with 'git remote
> > > add'.".format(path=repo_path))
> > > + return
> > > + if repo_path in os.path.abspath(destdir):
> > > + repos[repo_path]['is_bootstrap'] = True
> > > +
> > > repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]})
> > > +
> > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()])
> > > + if len(repo_dirs) > 1:
> > > + logger.error("Layer repositories are not all in the same
> > > parent directory: {repo_dirs}. They need to be relocated into the same
> > > directory.".format(repo_dirs=repo_dirs))
> > > + return
> > > +
> > > + repos_nopaths = {}
> > > + for r in repos.keys():
> > > + r_nopath = os.path.basename(r)
> > > + repos_nopaths[r_nopath] = repos[r]
> > > + return repos_nopaths
> > > +
> > > + def do_make_setup(self, args):
> > > + """ Writes out a python script/kas config/json config that
> > > replicates the directory structure and revisions of the layers in a
> > > current build. """
> > > + repos = self._make_repo_config(args.destdir)
> > > + if not repos:
> > > + return
> > > + output = args.output
> > > + if not output:
> > > + output = self._output_filename[args.format]
> > > + output = os.path.join(os.path.abspath(args.destdir),output)
> > > + self._write_config[args.format](self, repos, output)
> > > + logger.info('Created {}'.format(output))
> > > +
> > > + def register_commands(self, sp):
> > > + parser_setup_layers = self.add_command(sp,
> > > 'create-layers-setup', self.do_make_setup, parserecipes=False)
> > > + parser_setup_layers.add_argument('destdir',
> > > + help='Directory where to write the output\n(if it is inside
> > > one of the layers, the layer becomes a bootstrap repository and thus will
> > > be excluded from fetching by the script).')
> > > + parser_setup_layers.add_argument('--output', '-o',
> > > + help='File name where to write the output, if the default
> > > (setup-layers.py/.json/.yml) is undesirable.')
> > > + parser_setup_layers.add_argument('--format', '-f',
> > > choices=['python', 'json', 'kas'], default='python',
> > > + help='Format of the output. The options are:\n\tpython - a
> > > self contained python script that fetches all the needed layers and sets
> > > them to correct revisions (default, recommended)\n\tkas - a configuration
> > > file for the kas tool that allows the tool to do the same\n\tjson - a
> > > json formatted file containing all the needed metadata to do the same by
> > > any external or custom tool.')
> > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template
> > > b/meta/lib/bblayers/templates/setup-layers.py.template
> > > new file mode 100644
> > > index 0000000000..a704ad3d70
> > > --- /dev/null
> > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template
> > > @@ -0,0 +1,77 @@
> > > +#!/usr/bin/env python3
> > > +#
> > > +# This file was generated by running
> > > +#
> > > +# {cmdline}
> > > +#
> > > +# It is recommended that you do not modify it directly, but rather
> > > re-run the above command.
> > > +#
> > > +
> > > +layerdata = """
> > > +{layerdata}
> > > +"""
> > > +
> > > +import argparse
> > > +import json
> > > +import os
> > > +import subprocess
> > > +
> > > +def _do_checkout(args):
> > > + for l_name in layers:
> > > + l_data = layers[l_name]
> > > + if 'is_bootstrap' in l_data.keys():
> > > + force_arg =
> > > 'force_{}_checkout'.format(l_name.replace('-','_'))
> > > + if not args[force_arg]:
> > > + print('Note: not checking out layer {layer}, use
> > > {layerflag} to override.'.format(layer=l_name,
> > > layerflag='--force-{}-checkout'.format(l_name)))
> > > + continue
> > > + rev = l_data['rev']
> > > + desc = l_data['describe']
> > > + if not desc:
> > > + desc = rev[:10]
> > > + branch = l_data['branch']
> > > + remotes = l_data['remotes']
> > > + remote = remotes[0]
> > > + if len(remotes) > 1:
> > > + remotechoice =
> > > args['choose_{}_remote'.format(l_name.replace('-','_'))]
> > > + for r in remotes:
> > > + if r['name'] == remotechoice:
> > > + remote = r
> > > + print('Note: multiple remotes defined for layer {},
> > > using {} (run with -h to see others).'.format(l_name, r['name']))
> > > + print('Checking out layer {}, revision {}, branch {} from remote
> > > {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri']))
> > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name)
> > > + cwd = args['destdir']
> > > + print("Running '{}' in {}".format(cmd, cwd))
> > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd)
> > > + cmd = 'git checkout -q {}'.format(rev)
> > > + cwd = os.path.join(args['destdir'], l_name)
> > > + print("Running '{}' in {}".format(cmd, cwd))
> > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd)
> > > +
> > > +layers = json.loads(layerdata)
> > > +parser = argparse.ArgumentParser(description='A self contained python
> > > script that fetches all the needed layers and sets them to correct
> > > revisions')
> > > +
> > > +bootstraplayer = None
> > > +for l in layers:
> > > + if 'is_bootstrap' in layers[l]:
> > > + bootstraplayer = l
> > > +
> > > +if bootstraplayer:
> > > +
> > > parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer),
> > > action='store_true',
> > > + help='Force the checkout of the bootstrap layer {bootstraplayer}
> > > (by default it is presumed that this script is in it, and so the layer is
> > > already in place).'.format(bootstraplayer=bootstraplayer))
> > > +
> > > +for l in layers:
> > > + remotes = layers[l]['remotes']
> > > + if len(remotes) > 1:
> > > +
> > > parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name']
> > > for r in remotes], default=remotes[0]['name'],
> > > + help='Choose a remote server for layer
> > > {multipleremoteslayer} (default:
> > > {defaultremote})'.format(multipleremoteslayer=l,
> > > defaultremote=remotes[0]['name']))
> > > +
> > > +try:
> > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse
> > > --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__)))
> > > +except subprocess.CalledProcessError as e:
> > > + defaultdest = os.path.abspath(".")
> > > +
> > > +parser.add_argument('--destdir', default=defaultdest, help='Where to
> > > check out the layers (default is
> > > {defaultdest}).'.format(defaultdest=defaultdest))
> > > +
> > > +args = parser.parse_args()
> > > +
> > > +_do_checkout(vars(args))
> > >
> > >
> > >
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#167543):
https://lists.openembedded.org/g/openembedded-core/message/167543
Mute This Topic: https://lists.openembedded.org/mt/92117681/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
