Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
Hi Alex, On Thu, 25 Aug 2022 22:34:50 +0200 "Alexander Kanavin" wrote: > This script can be used directly from poky or oe-core, or can be copied > directly > into alayer or any other repository - it is self-suffucient and requires only > python3 alayer -> a layer suffucient -> sufficient Added to my testing branch with those typos fixed. -- Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169909): https://lists.openembedded.org/g/openembedded-core/message/169909 Mute This Topic: https://lists.openembedded.org/mt/93257025/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
This script can be used directly from poky or oe-core, or can be copied directly into alayer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://g...@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --helpshow this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 88 + 1 file changed, 88 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 00..cbd2efb5c7 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): +layers = json['sources'] +buildconfs = [] +oecorepath = "" +for l_name in layers: +l_data = layers[l_name] +layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + +for ll_name in l_data['layers']: +if ll_name == 'meta': +oecorepath = layerdir +ll_data = l_data['layers'][ll_name] +if 'buildconfigs' in ll_data: +for c in ll_data['buildconfigs']: +buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + +if 'contains_this_file' in l_data.keys(): +force_arg = 'force_bootstraplayer_checkout' +if not args[force_arg]: +print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) +continue +l_remote = l_data['git-remote'] +rev = l_remote['rev'] +desc = l_remote['describe'] +if not desc: +desc = rev[:10] +branch = l_remote['branch'] +remotes = l_remote['remotes'] + +print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) +cmd = 'git init -q {}'.format(layerdir) +print("Running '{}'".format(cmd)) +subprocess.check_output(cmd,
Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
On Mon, 22 Aug 2022 at 03:20, Chuck Wolber wrote: > One feedback item I might suggest is to check if the layer is already pulled > and to avoid pulling > again. Same goes for the commit hash - no reason to checkout the hash if HEAD > is already pointing > at it. The script does 'git fetch' which is an incremental operation, and becomes cheap and quick after the initial fetch. Same for the checkout, which is trivial. I'd rather not complicate the code with additional conditions, unless there's some scenario I am not aware of. The script needs to be robust on a bulletproof level. > Adding those checks would enable your script to be added as a pre-build step > in a developer > workflow wrapper script. Our environment takes a different approach to > repository pinning, but if > setup-layers had those checks, we could probably switch to this approach. Please explain the details, I'd like to understand where the problem is specifically. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169662): https://lists.openembedded.org/g/openembedded-core/message/169662 Mute This Topic: https://lists.openembedded.org/mt/93080238/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
On Mon, 22 Aug 2022 at 00:04, Philip Balister wrote: > > On 8/17/22 09:10, Alexander Kanavin wrote: > > This script can be used directly from poky, or can be copied directly into a > > Does this work if you are not using poky? Yes. It's a case of 'poky chauvinism' on my part. I'll edit the commit messages to reassure everyone :) https://en.wikipedia.org/wiki/Carbon_chauvinism Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169661): https://lists.openembedded.org/g/openembedded-core/message/169661 Mute This Topic: https://lists.openembedded.org/mt/93080238/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
On Wed, Aug 17, 2022 at 6:10 AM Alexander Kanavin wrote: > This script can be used directly from poky, or can be copied directly into > a > layer or any other repository - it is self-suffucient and requires only > python3 > and git on the host where it will run. It is also copied by the > bitbake-layers > layers-setup plugin together with the json, unless requested otherwise. > I need to review the code a bit closer, but on the face of it, this appears to solve the repository pinning problem quite elegantly. This matters a great deal in tightly configuration controlled environments, such as where regulatory oversight is involved. One feedback item I might suggest is to check if the layer is already pulled and to avoid pulling again. Same goes for the commit hash - no reason to checkout the hash if HEAD is already pointing at it. Adding those checks would enable your script to be added as a pre-build step in a developer workflow wrapper script. Our environment takes a different approach to repository pinning, but if setup-layers had those checks, we could probably switch to this approach. ..Ch:W.. -- *"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire* -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169657): https://lists.openembedded.org/g/openembedded-core/message/169657 Mute This Topic: https://lists.openembedded.org/mt/93080238/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
On 8/17/22 09:10, Alexander Kanavin wrote: This script can be used directly from poky, or can be copied directly into a Does this work if you are not using poky? Philip layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://g...@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --helpshow this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 89 + 1 file changed, 89 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 00..fe948898b0 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): +layers = json['sources'] +buildconfs = [] +oecorepath = "" +for l_name in layers: +l_data = layers[l_name] +layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + +for ll_name in l_data['layers']: +if ll_name == 'meta': +oecorepath = layerdir +ll_data = l_data['layers'][ll_name] +if 'buildconfigs' in ll_data: +for c in ll_data['buildconfigs']: +buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + +if 'contains_this_file' in l_data.keys(): +force_arg = 'force_bootstraplayer_checkout' +if not args[force_arg]: +print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) +continue +l_remote = l_data['git-remote'] +rev = l_remote['rev'] +desc = l_remote['describe'] +
[OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://g...@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --helpshow this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 89 + 1 file changed, 89 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 00..fe948898b0 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): +layers = json['sources'] +buildconfs = [] +oecorepath = "" +for l_name in layers: +l_data = layers[l_name] +layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + +for ll_name in l_data['layers']: +if ll_name == 'meta': +oecorepath = layerdir +ll_data = l_data['layers'][ll_name] +if 'buildconfigs' in ll_data: +for c in ll_data['buildconfigs']: +buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + +if 'contains_this_file' in l_data.keys(): +force_arg = 'force_bootstraplayer_checkout' +if not args[force_arg]: +print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) +continue +l_remote = l_data['git-remote'] +rev = l_remote['rev'] +desc = l_remote['describe'] +if not desc: +desc = rev[:10] +branch = l_remote['branch'] +remotes =
[OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://g...@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --helpshow this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 85 + 1 file changed, 85 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 00..39cc16a30d --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): +layers = json['sources'] +buildconfs = [] +oecorepath = "" +for l_name in layers: +l_data = layers[l_name] +layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + +for ll_name in l_data['layers']: +if ll_name == 'meta': +oecorepath = layerdir +ll_data = l_data['layers'][ll_name] +if 'buildconfigs' in ll_data: +for c in ll_data['buildconfigs']: +buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + +if 'contains_this_file' in l_data.keys(): +force_arg = 'force_bootstraplayer_checkout' +if not args[force_arg]: +print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) +continue +l_remote = l_data['git-remote'] +rev = l_remote['rev'] +desc = l_remote['describe'] +if not desc: +desc = rev[:10] +branch = l_remote['branch'] +remotes =