On Fri, Jul 1, 2022 at 1:44 PM Joshua Watt <[email protected]> wrote:
> On Fri, Jul 1, 2022 at 3:29 PM Alexander Kanavin <[email protected]> > wrote: > > > > First of all this provides a single command that does all of the work > > to capture the layers and their revisions from currently enabled set > > in a bitbake build. No other tool does this because they do not know > > anything about bitbake or tinfoil, and so you have to write your > > config by hand, with all of them. > > > > Here's how the captured metadata loks like [1]. I find it more useful > > for understanding what are we actually checking out than submodules' > > bare, unhelpful set of git revisions. It also doesn't require a > > super-repo containing all of the submodules: you can just toss the > > script into one of the layers and it will do the right thing for > > bootstrapping. There was also a comment from one of our customers: " > > In practice we learned that recursive submodules are not always > > correctly handled by GitLab CI. " > > > > I also find git submodules command line UI supremely confusing. > > Actually wait, that's what git as a whole is like: confusing. > > https://xkcd.com/1597/ > > > > But if you would like, we can add a fourth output format to the > > bblayers plugin added in this patch that sets up git submodules > > instead of writing a standalone python script with embedded json > > (which is what I prefer). > > Thanks for the write up. I don't particularly care if we go down this > route; it seems fine as long as it doesn't become "the one true way" > to do layer setup (because I'm still going to use submodules :) ) > This is exactly the problem. Everyone bike shedding; beautiful paint btw ;). We MUST find some solution that is a common reference. Otherwise we can just spin wheels forever. I will state for the record that git-repo phoning home on every invocation (and failing in very non helpful ways without proper ssh credentials or corporate proxies) is a non-starter. Yes it works. No, I will die on this mountain fighting against it being the way. No. Bad design. I like git sub modules. Yoe distro is a very good example of how to make this work. I did an ELCE presentation based on a git sub module system. Our entire team of three was git savvy. This is not a guarantee. Downstream from us broke very rapidly because folks didn’t understand how to contribute to the sub modules and keep it in sync. In the long run kas was drastically simpler to roll out and sustain. One remaining question is how do we handle configuration? This seems to address manifest? Git sub modules do not address configuration. Git-repo scripts are the first thing I have to fix/throw away. Do not assume where source and builds happen. My builds are on a different drive than my source. One-line command to fetch the layers and configure a build environment that will JustWork(TM) is the need. Kas is the only tool that does this as far as I have seen. In an opinionated way that overwrites local.conf (bad). For the record, I wholeheartedly endorse this direction, Alex. > > > > Alex > > > > [1] > > { > > "meta-alex": { > > "branch": "master", > > "describe": "", > > "is_bootstrap": true, > > "layers": [ > > { > > "name": "meta-alex", > > "path": "" > > } > > ], > > "remotes": [ > > { > > "name": "remote-alex", > > "uri": "https://github.com/kanavin/meta-alex"; > > } > > ], > > "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" > > }, > > "meta-intel": { > > "branch": "master", > > "describe": "15.0-hardknott-3.3-310-g0a96edae", > > "layers": [ > > { > > "name": "meta-intel", > > "path": "" > > } > > ], > > "remotes": [ > > { > > "name": "origin", > > "uri": "git://git.yoctoproject.org/meta-intel" > > } > > ], > > "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" > > }, > > "poky": { > > "branch": "akanavin/setup-layers", > > "describe": "4.1_M1-295-gfd524e4984", > > "layers": [ > > { > > "name": "meta", > > "path": "meta" > > }, > > { > > "name": "meta-poky", > > "path": "meta-poky" > > }, > > { > > "name": "meta-yocto-bsp", > > "path": "meta-yocto-bsp" > > } > > > > ], > > "remotes": [ > > { > > "name": "origin", > > "uri": "git://git.yoctoproject.org/poky" > > }, > > { > > "name": "poky-contrib", > > "uri": "ssh://[email protected]/poky-contrib" > > } > > ], > > "rev": "fd524e4984eb535f16a14dd7d79fde661b44aa78" > > } > > } > > > > On Fri, 1 Jul 2022 at 21:46, Joshua Watt <[email protected]> wrote: > > > > > > Can you outline what this provides over `git submodules`? It seems > > > pretty similar and I'm struggling to see what this provides that isn't > > > already provided by that tool. > > > > > > On 7/1/22 14:24, Alexander Kanavin wrote: > > > > This addresses a long standing gap in the core offering: > > > > there is no tooling to capture the currently configured layers > > > > with their revisions, or restore the layers from a configuration > > > > file (without using external tools, some of which aren't particularly > > > > suitable for the task). This plugin addresses the gap. > > > > > > > > How to use: > > > > > > > > 1. Saving a layer configuration: > > > > > > > > a) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers > create-layers-setup -h > > > > NOTE: Starting bitbake server... > > > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] > [--format {python,json,kas}] destdir > > > > > > > > Writes out a python script/kas config/json config that replicates > the directory structure and revisions of the layers in a current build. > > > > > > > > positional arguments: > > > > destdir Directory where to write the output > > > > (if it is inside one of the layers, the > layer becomes a bootstrap repository and thus will be excluded from > fetching by the script). > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --output OUTPUT, -o OUTPUT > > > > File name where to write the output, if the > default (setup-layers.py/.json/.yml) is undesirable. > > > > --format {python,json,kas}, -f {python,json,kas} > > > > Format of the output. The options are: > > > > python - a self contained python > script that fetches all the needed layers and sets them to correct > revisions (default, recommended) > > > > kas - a configuration file for the kas > tool that allows the tool to do the same > > > > json - a json formatted file > containing all the needed metadata to do the same by any external or custom > tool. > > > > > > > > b) Running with default choices: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers > create-layers-setup ../../meta-alex/ > > > > NOTE: Starting bitbake server... > > > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > > > > > 2. Restoring the layers from the saved configuration: > > > > > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It > should already contain setup-layers.py created in the previous step. > > > > > > > > b) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] > [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > > > > > A self contained python script that fetches all the needed layers > and sets them to correct revisions > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --force-meta-alex-checkout > > > > Force the checkout of the bootstrap layer > meta-alex (by default it is presumed that this script is in it, and so the > layer is already in place). > > > > --choose-poky-remote {origin,poky-contrib} > > > > Choose a remote server for layer poky > (default: origin) > > > > --destdir DESTDIR Where to check out the layers (default is > /srv/work/alex/layers-test). > > > > > > > > c) Running with default options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > > > Note: not checking out layer meta-alex, use > --force-meta-alex-checkout to override. > > > > Checking out layer meta-intel, revision > 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git:// > git.yoctoproject.org/meta-intel > > > > Running 'git clone -q git://git.yoctoproject.org/meta-intel > meta-intel' in /srv/work/alex/layers-test > > > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' > in /srv/work/alex/layers-test/meta-intel > > > > Note: multiple remotes defined for layer poky, using origin (run > with -h to see others). > > > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch > akanavin/setup-layers from remote origin at git:// > git.yoctoproject.org/poky > > > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in > /srv/work/alex/layers-test > > > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' > in /srv/work/alex/layers-test/poky > > > > > > > > And that's it! > > > > > > > > FIXMEs: > > > > - kas config writer not yet implemented > > > > - oe-selftest test cases not yet written > > > > > > > > Signed-off-by: Alexander Kanavin <[email protected]> > > > > --- > > > > meta/lib/bblayers/makesetup.py | 117 > ++++++++++++++++++ > > > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > > > 2 files changed, 194 insertions(+) > > > > create mode 100644 meta/lib/bblayers/makesetup.py > > > > create mode 100644 > meta/lib/bblayers/templates/setup-layers.py.template > > > > > > > > diff --git a/meta/lib/bblayers/makesetup.py > b/meta/lib/bblayers/makesetup.py > > > > new file mode 100644 > > > > index 0000000000..3c86eea3c4 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/makesetup.py > > > > @@ -0,0 +1,117 @@ > > > > +# > > > > +# SPDX-License-Identifier: GPL-2.0-only > > > > +# > > > > + > > > > +import logging > > > > +import os > > > > +import stat > > > > +import sys > > > > +import shutil > > > > +import json > > > > + > > > > +import bb.utils > > > > +import bb.process > > > > + > > > > +from bblayers.common import LayerPlugin > > > > + > > > > +logger = logging.getLogger('bitbake-layers') > > > > + > > > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > > > + > > > > +import oe.buildcfg > > > > + > > > > +def plugin_init(plugins): > > > > + return MakeSetupPlugin() > > > > + > > > > +class MakeSetupPlugin(LayerPlugin): > > > > + > > > > + def _write_python(self, repos, output): > > > > + with open(os.path.join(os.path.dirname(__file__), > "templates", "setup-layers.py.template")) as f: > > > > + template = f.read() > > > > + args = sys.argv > > > > + args[0] = os.path.basename(args[0]) > > > > + script = template.replace('{cmdline}', " > ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, > indent=4)) > > > > + with open(output, 'w') as f: > > > > + f.write(script) > > > > + st = os.stat(output) > > > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | > stat.S_IXOTH) > > > > + > > > > + def _write_json(self, repos, output): > > > > + with open(output, 'w') as f: > > > > + json.dump(repos, f, sort_keys=True, indent=4) > > > > + > > > > + def _write_kas(self, repos, output): > > > > + raise NotImplementedError('Kas config writer not > implemented yet') > > > > + > > > > + _write_config = {"python":_write_python, "json":_write_json, > "kas":_write_kas} > > > > + _output_filename = > {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > > > + > > > > + def _get_repo_path(self, layer_path): > > > > + repo_path, _ = bb.process.run('git rev-parse > --show-toplevel', cwd=layer_path) > > > > + return repo_path.strip() > > > > + > > > > + def _get_remotes(self, repo_path): > > > > + remotes = [] > > > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > > > + for r in remotes_list.split(): > > > > + uri,_ = bb.process.run('git remote get-url > {r}'.format(r=r), cwd=repo_path) > > > > + remotes.append({'name':r,'uri':uri.strip()}) > > > > + return remotes > > > > + > > > > + def _get_describe(self, repo_path): > > > > + try: > > > > + describe,_ = bb.process.run('git describe --tags', > cwd=repo_path) > > > > + except bb.process.ExecutionError: > > > > + return "" > > > > + return describe.strip() > > > > + > > > > + def _make_repo_config(self, destdir): > > > > + repos = {} > > > > + layers = > oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > > > + for l in layers: > > > > + if l[1] == 'workspace': > > > > + continue > > > > + if l[4]: > > > > + logger.error("Layer {name} in {path} has > uncommitted modifications or is not in a git > repository.".format(name=l[1],path=l[0])) > > > > + return > > > > + repo_path = self._get_repo_path(l[0]) > > > > + if repo_path not in repos.keys(): > > > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], > 'remotes':self._get_remotes(repo_path), 'layers':[], > 'describe':self._get_describe(repo_path)} > > > > + if not repos[repo_path]['remotes']: > > > > + logger.error("Layer repository in {path} does > not have any remotes configured. Please add at least one with 'git remote > add'.".format(path=repo_path)) > > > > + return > > > > + if repo_path in os.path.abspath(destdir): > > > > + repos[repo_path]['is_bootstrap'] = True > > > > + > repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > > > + > > > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > > > + if len(repo_dirs) > 1: > > > > + logger.error("Layer repositories are not all in the > same parent directory: {repo_dirs}. They need to be relocated into the same > directory.".format(repo_dirs=repo_dirs)) > > > > + return > > > > + > > > > + repos_nopaths = {} > > > > + for r in repos.keys(): > > > > + r_nopath = os.path.basename(r) > > > > + repos_nopaths[r_nopath] = repos[r] > > > > + return repos_nopaths > > > > + > > > > + def do_make_setup(self, args): > > > > + """ Writes out a python script/kas config/json config that > replicates the directory structure and revisions of the layers in a current > build. """ > > > > + repos = self._make_repo_config(args.destdir) > > > > + if not repos: > > > > + return > > > > + output = args.output > > > > + if not output: > > > > + output = self._output_filename[args.format] > > > > + output = os.path.join(os.path.abspath(args.destdir),output) > > > > + self._write_config[args.format](self, repos, output) > > > > + logger.info('Created {}'.format(output)) > > > > + > > > > + def register_commands(self, sp): > > > > + parser_setup_layers = self.add_command(sp, > 'create-layers-setup', self.do_make_setup, parserecipes=False) > > > > + parser_setup_layers.add_argument('destdir', > > > > + help='Directory where to write the output\n(if it is > inside one of the layers, the layer becomes a bootstrap repository and thus > will be excluded from fetching by the script).') > > > > + parser_setup_layers.add_argument('--output', '-o', > > > > + help='File name where to write the output, if the > default (setup-layers.py/.json/.yml) is undesirable.') > > > > + parser_setup_layers.add_argument('--format', '-f', > choices=['python', 'json', 'kas'], default='python', > > > > + help='Format of the output. The options are:\n\tpython > - a self contained python script that fetches all the needed layers and > sets them to correct revisions (default, recommended)\n\tkas - a > configuration file for the kas tool that allows the tool to do the > same\n\tjson - a json formatted file containing all the needed metadata to > do the same by any external or custom tool.') > > > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template > b/meta/lib/bblayers/templates/setup-layers.py.template > > > > new file mode 100644 > > > > index 0000000000..a704ad3d70 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > > > @@ -0,0 +1,77 @@ > > > > +#!/usr/bin/env python3 > > > > +# > > > > +# This file was generated by running > > > > +# > > > > +# {cmdline} > > > > +# > > > > +# It is recommended that you do not modify it directly, but rather > re-run the above command. > > > > +# > > > > + > > > > +layerdata = """ > > > > +{layerdata} > > > > +""" > > > > + > > > > +import argparse > > > > +import json > > > > +import os > > > > +import subprocess > > > > + > > > > +def _do_checkout(args): > > > > + for l_name in layers: > > > > + l_data = layers[l_name] > > > > + if 'is_bootstrap' in l_data.keys(): > > > > + force_arg = > 'force_{}_checkout'.format(l_name.replace('-','_')) > > > > + if not args[force_arg]: > > > > + print('Note: not checking out layer {layer}, use > {layerflag} to override.'.format(layer=l_name, > layerflag='--force-{}-checkout'.format(l_name))) > > > > + continue > > > > + rev = l_data['rev'] > > > > + desc = l_data['describe'] > > > > + if not desc: > > > > + desc = rev[:10] > > > > + branch = l_data['branch'] > > > > + remotes = l_data['remotes'] > > > > + remote = remotes[0] > > > > + if len(remotes) > 1: > > > > + remotechoice = > args['choose_{}_remote'.format(l_name.replace('-','_'))] > > > > + for r in remotes: > > > > + if r['name'] == remotechoice: > > > > + remote = r > > > > + print('Note: multiple remotes defined for layer > {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > > > + print('Checking out layer {}, revision {}, branch {} from > remote {} at {}'.format(l_name, desc, branch, remote['name'], > remote['uri'])) > > > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > > > + cwd = args['destdir'] > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + cmd = 'git checkout -q {}'.format(rev) > > > > + cwd = os.path.join(args['destdir'], l_name) > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + > > > > +layers = json.loads(layerdata) > > > > +parser = argparse.ArgumentParser(description='A self contained > python script that fetches all the needed layers and sets them to correct > revisions') > > > > + > > > > +bootstraplayer = None > > > > +for l in layers: > > > > + if 'is_bootstrap' in layers[l]: > > > > + bootstraplayer = l > > > > + > > > > +if bootstraplayer: > > > > + > parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), > action='store_true', > > > > + help='Force the checkout of the bootstrap layer > {bootstraplayer} (by default it is presumed that this script is in it, and > so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > > > + > > > > +for l in layers: > > > > + remotes = layers[l]['remotes'] > > > > + if len(remotes) > 1: > > > > + > parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] > for r in remotes], default=remotes[0]['name'], > > > > + help='Choose a remote server for layer > {multipleremoteslayer} (default: > {defaultremote})'.format(multipleremoteslayer=l, > defaultremote=remotes[0]['name'])) > > > > + > > > > +try: > > > > + defaultdest = os.path.dirname(subprocess.check_output('git > rev-parse --show-toplevel', text=True, shell=True, > cwd=os.path.dirname(__file__))) > > > > +except subprocess.CalledProcessError as e: > > > > + defaultdest = os.path.abspath(".") > > > > + > > > > +parser.add_argument('--destdir', default=defaultdest, help='Where > to check out the layers (default is > {defaultdest}).'.format(defaultdest=defaultdest)) > > > > + > > > > +args = parser.parse_args() > > > > + > > > > +_do_checkout(vars(args)) > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#167544): https://lists.openembedded.org/g/openembedded-core/message/167544 Mute This Topic: https://lists.openembedded.org/mt/92117681/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
