[gentoo-commits] proj/catalyst:master commit in: doc/, targets/support/
commit: 90e1cbf32bf1de2854e8ed21dbd644b10b4e22ad Author: Matt Turner gentoo org> AuthorDate: Mon Jul 26 23:34:19 2021 + Commit: Matt Turner gentoo org> CommitDate: Thu Jul 29 00:32:08 2021 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=90e1cbf3 targets: Remove some ISOLINUX leftovers We don't use ISOLINUX since the commit listed below. Fixes: 0a27a7a3 ("targets: Use GRUB for BIOS boot") Signed-off-by: Matt Turner gentoo.org> doc/HOWTO.txt | 7 +-- targets/support/create-iso.sh | 2 +- targets/support/functions.sh | 2 +- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt index 7b759121..865e348d 100644 --- a/doc/HOWTO.txt +++ b/doc/HOWTO.txt @@ -216,16 +216,11 @@ the kernel, bootloader, filesystem, and other details. See Live USBs - -The easiest way to create a live USB is currently to install a live CD -ISO using -http://www.syslinux.org/wiki/index.php/Doc/isolinux#HYBRID_CD-ROM.2FHARD_DISK_MODE[isohybrid] -and `dd`: +The easiest way to create a live USB is to install a live CD ISO using -# isohybrid filename.iso # dd if=filename.iso of=/dev/sdX replacing `X` with the appropriate drive letter for your USB disk. -See https://bugs.gentoo.org/251719[bug 251719] for details. Running catalyst from a Git checkout diff --git a/targets/support/create-iso.sh b/targets/support/create-iso.sh index 70684f78..9b2cbb60 100755 --- a/targets/support/create-iso.sh +++ b/targets/support/create-iso.sh @@ -102,7 +102,7 @@ isoroot_checksum() { echo ">> Creating checksums for all files included in the ISO" pushd "${clst_target_path}" - find -type f ! -name 'isolinux.bin' -exec b2sum {} + > /tmp/isoroot_b2sums + find -type f -exec b2sum {} + > /tmp/isoroot_b2sums popd mv /tmp/isoroot_b2sums "${clst_target_path}"/ diff --git a/targets/support/functions.sh b/targets/support/functions.sh index d1252369..4340a3c7 100755 --- a/targets/support/functions.sh +++ b/targets/support/functions.sh @@ -49,7 +49,7 @@ extract_kernels() { # ${clst_target_path}/kernel is often a good choice for ${1} # Takes the relative desination dir for the kernel as an arguement - # i.e boot or isolinux + # i.e boot [ -z "$clst_boot_kernel" ] && \ die "Required key boot/kernel not defined, exiting" # install the kernels built in kmerge.sh
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/targets/, catalyst/
commit: d5f85bc8d71d89920332ce760e5da3dca257dc36 Author: Felix Bier rohde-schwarz com> AuthorDate: Thu Feb 4 00:34:14 2021 + Commit: Matt Turner gentoo org> CommitDate: Sat Feb 20 02:27:06 2021 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d5f85bc8 Rename config option 'repos' -> 'repos_storedir' This commit renames the config option 'repos' to 'repos_storedir'. This is done as preparation for renaming 'portage_overlay' to 'repos'. Signed-off-by: Felix Bier rohde-schwarz.com> Signed-off-by: Matt Turner gentoo.org> catalyst/defaults.py | 4 ++-- catalyst/targets/snapshot.py | 2 +- doc/catalyst-config.5.txt| 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/catalyst/defaults.py b/catalyst/defaults.py index 3f12b8d5..3d5c0a7f 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -20,7 +20,7 @@ valid_config_file_values = frozenset([ "port_logdir", "repo_basedir", "repo_name", -"repos", +"repos_storedir", "sharedir", "storedir", "target_distdir", @@ -46,7 +46,7 @@ confdefaults = { "port_tmpdir": "/var/tmp/portage", "repo_basedir": "/var/db/repos", "repo_name": "gentoo", -"repos": "%(storedir)s/repos", +"repos_storedir": "%(storedir)s/repos", "sharedir": "/usr/share/catalyst", "shdir": "%(sharedir)s/targets", "storedir": "/var/tmp/catalyst", diff --git a/catalyst/targets/snapshot.py b/catalyst/targets/snapshot.py index 6b727600..b494575a 100644 --- a/catalyst/targets/snapshot.py +++ b/catalyst/targets/snapshot.py @@ -27,7 +27,7 @@ class snapshot(TargetBase): TargetBase.__init__(self, myspec, addlargs) self.git = command('git') -self.ebuild_repo = Path(self.settings['repos'], +self.ebuild_repo = Path(self.settings['repos_storedir'], self.settings['repo_name']).with_suffix('.git') self.gitdir = str(self.ebuild_repo) diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index 570d42c2..ebd99b55 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -109,7 +109,7 @@ Repository settings Defines the location of your local source file repository. Defaults to the host's DISTDIR. -*repos*:: +*repos_storedir*:: The directory in which git repositories exist for use by the snapshot target. Defaults to `${storedir}/repos`. @@ -120,7 +120,7 @@ gentoo repo) and any overlays. The default location is *repo_name*:: The name of the main repository (e.g. gentoo). The git repository at -`${repos}/${repo_name}.git` will be used to produce the portdir sqfs +`${repos_storedir}/${repo_name}.git` will be used to produce the portdir sqfs snapshot. *target_distdir*::
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/
commit: 87b0588ab4f77e413c250a4a3e357624ec41c374 Author: Matt Turner gentoo org> AuthorDate: Mon Jan 18 16:52:20 2021 + Commit: Matt Turner gentoo org> CommitDate: Sat Jan 23 16:22:22 2021 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=87b0588a catalyst: Add option to enter the chroot before building With --enter-chroot, after the mounts and environment are set up, catalyst will drop you into a shell inside the chroot. Useful for hacking or debugging. Signed-off-by: Matt Turner gentoo.org> catalyst/base/stagebase.py | 16 +++- catalyst/main.py | 4 doc/catalyst.1.txt | 3 +++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index 40b60af3..676206ff 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -20,7 +20,7 @@ from catalyst import log from catalyst.context import namespace from catalyst.defaults import (confdefaults, MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN) from catalyst.support import (CatalystError, file_locate, normpath, - cmd, read_makeconf, get_repo_name, ismount, + cmd, command, read_makeconf, get_repo_name, file_check, sanitize_name) from catalyst.base.targetbase import TargetBase from catalyst.base.clearbase import ClearBase @@ -95,6 +95,9 @@ class StageBase(TargetBase, ClearBase, GenBase): self.chroot_setup, self.setup_environment, ] +if 'enter-chroot' in self.settings['options']: +self.build_sequence.append(self.enter_chroot) + self.finish_sequence = [] self.set_valid_build_kernel_vars(addlargs) @@ -1326,6 +1329,17 @@ class StageBase(TargetBase, ClearBase, GenBase): log.debug('setup_environment(); env = %r', self.env) +def enter_chroot(self): +chroot = command('chroot') +bash = command('bash') + +log.notice("Entering chroot") +try: +cmd([chroot, self.settings['chroot_path'], bash, '-l'], +env=self.env) +except CatalystError: +pass + def run(self): self.chroot_lock.write_lock() diff --git a/catalyst/main.py b/catalyst/main.py index 48daf004..b0d9015f 100644 --- a/catalyst/main.py +++ b/catalyst/main.py @@ -120,6 +120,8 @@ def get_parser(): parser.add_argument('-V', '--version', action='version', version=get_version(), help='display version information') +parser.add_argument('--enter-chroot', default=False, action='store_true', +help='Enter chroot before starting the build') group = parser.add_argument_group('Program output options') group.add_argument('-d', '--debug', @@ -293,6 +295,8 @@ def _main(parser, opts): options.append('purgetmponly') if opts.clear_autoresume: options.append('clear-autoresume') +if opts.enter_chroot: +options.append('enter-chroot') # Make sure we have some work before moving further. if not myspecfile and not mycmdline: diff --git a/doc/catalyst.1.txt b/doc/catalyst.1.txt index 90d5a24b..217fc86a 100644 --- a/doc/catalyst.1.txt +++ b/doc/catalyst.1.txt @@ -39,6 +39,9 @@ configuration file is installed at '/etc/catalyst/catalyst.conf'. *-d*:: Enable debugging mode +*--enter-chroot*:: +Enter the chroot before starting the build. + *--fetchonly*:: *-F*:: This tells *catalyst* to only fetch distfiles for the given packages without
[gentoo-commits] proj/catalyst:master commit in: doc/, examples/
commit: 2861038ae8c3c67f3000c976ed8177f63169712c Author: Felix Bier rohde-schwarz com> AuthorDate: Tue Nov 10 00:56:19 2020 + Commit: Matt Turner gentoo org> CommitDate: Sat Nov 14 16:34:57 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=2861038a doc: Make existing multi-overlay support more explicit The code already supports multiple repositories. This commit adapts the documentation to make this more explicit. Signed-off-by: Felix Bier rohde-schwarz.com> Signed-off-by: Matt Turner gentoo.org> doc/catalyst-spec.5.txt | 5 +++-- examples/generic_stage_template.spec | 5 +++-- examples/livecd-stage1_template.spec | 5 +++-- examples/livecd-stage2_template.spec | 5 +++-- examples/netboot_template.spec | 5 +++-- examples/stage4_template.spec| 5 +++-- 6 files changed, 18 insertions(+), 12 deletions(-) diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index 31e69820..4c1df857 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -81,8 +81,9 @@ This is an optional directory containing portage configuration files problems. *portage_overlay*:: -This option specifies the location to a portage overlay that you would -like to use when building this target (example: `/usr/local/portage`). +This option specifies the location of the ebuild repositories that you would +like to have used when building this target. It takes a space-separated list +of directory names. (example: `/usr/local/portage`). *pkgcache_path*:: This allows the optional directory containing the output packages for diff --git a/examples/generic_stage_template.spec b/examples/generic_stage_template.spec index 8f0375c4..8c8889bf 100644 --- a/examples/generic_stage_template.spec +++ b/examples/generic_stage_template.spec @@ -89,8 +89,9 @@ decompressor_search_order: lbzip2 bzip2 tar pixz xz gzip squashfs # portage_confdir: /etc/portage portage_confdir: -# This option specifies the location to a portage overlay that you would like to -# have used when building this target. +# This option specifies the location of the ebuild repositories that you would +# like to have used when building this target. It takes a space-separated list +# of directory names. # example: # portage_overlay: /usr/local/portage portage_overlay: diff --git a/examples/livecd-stage1_template.spec b/examples/livecd-stage1_template.spec index b921372a..269a1e3d 100644 --- a/examples/livecd-stage1_template.spec +++ b/examples/livecd-stage1_template.spec @@ -52,8 +52,9 @@ source_subpath: # portage_confdir: /etc/portage portage_confdir: -# This option specifies the location to a portage overlay that you would like to -# have used when building this target. +# This option specifies the location of the ebuild repositories that you would +# like to have used when building this target. It takes a space-separated list +# of directory names. # example: # portage_overlay: /usr/local/portage portage_overlay: diff --git a/examples/livecd-stage2_template.spec b/examples/livecd-stage2_template.spec index ac1cd3e4..4f50919f 100644 --- a/examples/livecd-stage2_template.spec +++ b/examples/livecd-stage2_template.spec @@ -52,8 +52,9 @@ source_subpath: # portage_confdir: /etc/portage portage_confdir: -# This option specifies the location to a portage overlay that you would like to -# have used when building this target. +# This option specifies the location of the ebuild repositories that you would +# like to have used when building this target. It takes a space-separated list +# of directory names. # example: # portage_overlay: /usr/local/portage portage_overlay: diff --git a/examples/netboot_template.spec b/examples/netboot_template.spec index 4f76b8ff..ad9d993c 100644 --- a/examples/netboot_template.spec +++ b/examples/netboot_template.spec @@ -6,8 +6,9 @@ profile: uclibc/mips snapshot: 20060107 source_subpath: default/stage3-mips-uclibc-mips3-2006.126 -# This option specifies the location to a portage overlay that you would like to -# have used when building this target. +# This option specifies the location of the ebuild repositories that you would +# like to have used when building this target. It takes a space-separated list +# of directory names. # example: # portage_overlay: /usr/local/portage portage_overlay: diff --git a/examples/stage4_template.spec b/examples/stage4_template.spec index c901eabc..aea70e86 100644 --- a/examples/stage4_template.spec +++ b/examples/stage4_template.spec @@ -52,8 +52,9 @@ source_subpath: # portage_confdir: /etc/portage portage_confdir: -# This option specifies the location to a portage overlay that you would like to -# have used when building this target. +# This option specifies the location of the ebuild repositories that you would +# like to have used when building this target. It takes a space-separated list +# of directory names. # example: # portage_overlay: /usr/local/portage
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/, examples/, etc/, catalyst/base/
commit: f9ec6eb347542d35208a94a07af155dcd473e4e1 Author: Matt Turner gentoo org> AuthorDate: Tue May 19 00:59:10 2020 + Commit: Matt Turner gentoo org> CommitDate: Wed Oct 21 01:07:32 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f9ec6eb3 catalyst: Configure distcc_hosts in the config file distcc_hosts are independent of the build itself, and therefore should be configured system-wide in catalyst.conf and not in each spec file. Signed-off-by: Matt Turner gentoo.org> catalyst/base/stagebase.py | 1 - catalyst/defaults.py | 1 + doc/catalyst-config.5.txt| 7 ++- doc/catalyst-spec.5.txt | 6 -- etc/catalyst.conf| 3 +-- examples/generic_stage_template.spec | 7 --- examples/livecd-stage1_template.spec | 7 --- examples/livecd-stage2_template.spec | 7 --- examples/stage4_template.spec| 7 --- 9 files changed, 8 insertions(+), 38 deletions(-) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index afea5776..be7b96c8 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -51,7 +51,6 @@ class StageBase(TargetBase, ClearBase, GenBase): "compression_mode", "cxxflags", "decompressor_search_order", -"distcc_hosts", "fcflags", "fflags", "hostuse", diff --git a/catalyst/defaults.py b/catalyst/defaults.py index 7a2fe3f3..0f399b56 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -11,6 +11,7 @@ from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, LIST_XATTRS_OPTIONS valid_config_file_values = frozenset([ "compression_mode", "digests", +"distcc_hosts", "distdir", "envscript", "jobs", diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index cbef6092..570d42c2 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -78,7 +78,7 @@ be closed invalid. distcc:: Enable distcc support for building. You have to set distcc_hosts in -your spec file. +your config file. icecream:: Enable icecream compiler cluster support for building. @@ -136,6 +136,11 @@ written to the target's make.conf if it is not the default value of Other settings ~~ +*distcc_hosts*:: +These are the hosts used as distcc slaves when distcc is enabled in +your `catalyst.conf` (example: `127.0.0.1 192.168.0.1`). It follows +the same syntax as `distcc-config --set-hosts`. + *jobs*:: Integral value passed to *emerge(1)* as the parameter to --jobs and is used to define *MAKEOPTS* during the target build. diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index 43cd5d2f..31e69820 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -74,12 +74,6 @@ This specifies where the seed stage for this target comes from `$storedir/builds`. The `rel_type` is also used as a path prefix for the seed. -*distcc_hosts*:: -These are the hosts used as distcc slaves when distcc is enabled in -your `catalyst.conf` (example: `127.0.0.1 192.168.0.1`). It follows -the same syntax as `distcc-config --set-hosts` and is entirely -optional. - *portage_confdir*:: This is an optional directory containing portage configuration files (example: `/etc/portage`). It follows the same syntax as diff --git a/etc/catalyst.conf b/etc/catalyst.conf index 81693c25..b0b284fa 100644 --- a/etc/catalyst.conf +++ b/etc/catalyst.conf @@ -35,8 +35,7 @@ options = [ # Enable FEATURES=ccache # "ccache", - # Enable FEATURES=distcc. You have to set distcc_hosts in your spec - # file. + # Enable FEATURES=distcc. Make sure to set distcc_hosts too. # "distcc", # Enable FEATURES=icecream diff --git a/examples/generic_stage_template.spec b/examples/generic_stage_template.spec index 01c37789..8f0375c4 100644 --- a/examples/generic_stage_template.spec +++ b/examples/generic_stage_template.spec @@ -82,13 +82,6 @@ compressor_arch": # decompressor_search_order: lbzip2 bzip2 tar pixz xz gzip squashfs -# These are the hosts used as distcc slaves when distcc is enabled in your -# catalyst.conf. It follows the same syntax as distcc-config --set-hosts and -# is entirely optional. -# example: -# distcc_hosts: 127.0.0.1 192.168.0.1 -distcc_hosts: - # This is an optional directory containing portage configuration files. It # follows the same syntax as /etc/portage and should be consistent across all # targets to minimize problems. diff --git a/examples/livecd-stage1_template.spec b/examples/livecd-stage1_template.spec index c7086c91..b921372a 100644 --- a/examples/livecd-stage1_template.spec +++ b/examples/livecd-stage1_template.spec @@ -45,13 +45,6 @@ snapshot: # default/stage3-x86-2006.1 source_subpath: -# These are the hosts used as distcc slaves when distcc is enabled in your -# catalyst.conf. It follows the same syntax
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: 1fa044467282982563c2cf42c9918b8f4d03d58b Author: Matt Turner gentoo org> AuthorDate: Thu May 14 02:10:40 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri May 15 00:53:50 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=1fa04446 doc: Drop pre-EAPI-5 text There are very few EAPI < 5 ebuilds left. Signed-off-by: Matt Turner gentoo.org> doc/catalyst-config.5.txt | 30 +- 1 file changed, 1 insertion(+), 29 deletions(-) diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index 4fe1af8c..f17944b2 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -168,22 +168,7 @@ This section is only important if you are using binary packages to build your stages (by enabling the `pkgcache` option and restarting incomplete builds). -Before EAPI-5 introduced ABI sub-slots, the build-time compatibility -of packages was not recorded. This leads to problems such as binary -GCC packages built against mpc-0.8.2 (which installs libmpc.so.2) -being installed on systems that only have mpc-1.0.1 (which installs -libmpc.so.3), resulting in: - -- -/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1: - error while loading shared libraries: libmpc.so.2: - cannot open shared object file: No such file or directory -- - -As long as there are packages in your stage that don't use ABI -sub-slots, you may experience errors like this due to untracked ABI -missmatches in binary packages. Packages generated by catalyst builds -are currently namespaced: +Packages generated by catalyst builds are namespaced: If versioned_cache is set: - @@ -194,19 +179,6 @@ Otherwise: .../packages//-/Packages - -so running into these out-of-date packages is unlikely. You may run -into problems if: - -* you enable `update_seed` in your stage1 spec after a previous run - which generated packages linking against out-of-date seed libraries - or -* you update your snapshot and an untracked ABI dependency is bumped - without a similar bump in the dependent package. - -without also bumping any of the package namespace variables in your -spec. If you do make such a change, it's a good idea to clear the -package cache in question and rebuild the packages from scratch. - FILES -
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: 9c4b7f79232b772404cab909b89b73bb8dc2d9c5 Author: Matt Turner gentoo org> AuthorDate: Fri May 15 00:46:56 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri May 15 00:53:50 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9c4b7f79 doc: Remove extra asterisk Fixes: c7688c666567 (doc/catalyst-config.5.txt: Add man page for catalyst.conf) Signed-off-by: Matt Turner gentoo.org> doc/catalyst-config.5.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index d5444b38..4fe1af8c 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -51,7 +51,7 @@ variables using POSIX shell notation: export FOO="bar" - -**options*:: +*options*:: Set different build-time options (example: `autoresume bindist kerncache pkgcache seedcache`). Supported values: +
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: b0d2777092252988a72f30dc915db93eb4d993d0 Author: Matt Turner gentoo org> AuthorDate: Thu May 14 02:16:07 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri May 15 00:57:16 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b0d27770 doc: Merge SUPPORTED HASHES section into digests section It made sense to have this as a separate section when there were multiple configuration options (digests and hash_function) but hash_function is now gone. Signed-off-by: Matt Turner gentoo.org> doc/catalyst-config.5.txt | 17 +++-- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index f17944b2..7ac9a2a3 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -38,8 +38,13 @@ Basic configuration *digests*:: Create a `.DIGESTS` file containing the hash output from any of the supported options below. Adding them all may take a long time. -(example: `md5 sha1 sha512 whirlpool`). See the *SUPPORTED HASHES* -section for a list of supported hashes. +(example: `md5 sha1 sha512 whirlpool`). The set of supported hashes +is dependent on the version of Python. To see the set of hashes +supported by the version of Python in use, run ++ +- +$ python3 -c 'import hashlib; print(hashlib.algorithms_available)' +- *envscript*:: Environment script location, which allows users to set options such as @@ -154,14 +159,6 @@ takes place in RAM. This feature requires a pretty large tmpfs much RAM everything will fail horribly and it is not our fault. -SUPPORTED HASHES - -The list of supported hashes is dependent on the version of Python. To -see the list of hashes supported by the version of Python in use, run - -$ python3 -c 'import hashlib; print(hashlib.algorithms_available)' - - BINARY PACKAGE DEPENDENCIES --- This section is only important if you are using binary packages to
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/targets/, catalyst/, /
commit: 5b29d4a88f492f6890d0574d0addefb9e6a13271 Author: Matt Turner gentoo org> AuthorDate: Fri Apr 17 23:31:52 2020 + Commit: Matt Turner gentoo org> CommitDate: Thu Apr 30 23:04:34 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=5b29d4a8 catalyst: Make and use squashfs snapshots There were a number of problems with catalyst's snapshot system. It was built around using the build system's portdir and had no control over what was in that portdir or when it was updated. As a result, when a stage build failed, it was difficult to tell what the snapshot consistet of precisely or whether it contained a particular recent fix. With snapcache disabled, ebuild repo snapshots were tar'd and compressed and then unpacked into the stage chroot which is an unnecessarily expensive process. Moreover, a porttree has more than 100k small files, which are stored extremely inefficiently on most file systems—a whole porttree is usually around 700M on disk. Just removing all of those files during the cleaning stage is an expensive operation. Instead, we make a compressed squashfs image and mount it in the build chroot. The porttree has many duplicate files, and squashfs deduplicates the files and then compresses, so the result is very efficiently packed: ~38M with gzip -9 compression. The snapshot target has been modified to generate a squashfs image from a bare ebuild git repo. Piping git-archive to tar2sqfs generates the squashfs image in less than 10 seconds on a modern system. The git repo is fetched with --depth=1 to minize bandwidth and disk usage, and git gc is run after fetch to minimize disk usage. Storage requirements for the git ebuild repo with metadata are ~70M. The squashfs snapshot is stored in /var/tmp/catalyst/snapshots/ by default with a name -.sqfs. With this convention, we know the exact point in history that the snapshot was taken. The catalyst-auto script can use the sha1 to get a deterministic timestamp, so that it is independent on when `catalyst -s` was run, but is instead the timestamp of the commit date of the repo's git SHA1. Signed-off-by: Matt Turner gentoo.org> README| 3 +- catalyst/base/stagebase.py| 88 catalyst/base/targetbase.py | 16 +++- catalyst/defaults.py | 2 +- catalyst/main.py | 28 +++ catalyst/targets/embedded.py | 1 - catalyst/targets/livecd_stage1.py | 1 - catalyst/targets/livecd_stage2.py | 1 - catalyst/targets/netboot.py | 1 - catalyst/targets/snapshot.py | 165 +++--- catalyst/targets/stage4.py| 1 - doc/catalyst-config.5.txt | 13 +-- doc/catalyst-spec.5.txt | 4 +- 13 files changed, 139 insertions(+), 185 deletions(-) diff --git a/README b/README index 1a039fca..1cceb63e 100644 --- a/README +++ b/README @@ -18,8 +18,9 @@ Requirements === - Python 3.6 or greater -- An ebuild repository snapshot (or an ebuild tree to create one) - A generic stage3 tarball for your architecture +- A squashfs ebuild repository snapshot + - Or an ebuild git repo with sys-fs/squashfs-tools-ng and dev-vcs/git What is catalyst? diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index 9aecf013..41da97b3 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -35,7 +35,7 @@ class StageBase(TargetBase, ClearBase, GenBase): self.required_values |= frozenset([ "profile", "rel_type", -"snapshot", +"snapshot_treeish", "source_subpath", "subarch", "target", @@ -149,7 +149,7 @@ class StageBase(TargetBase, ClearBase, GenBase): self.set_source_subpath() # Set paths -self.set_snapshot_path() +self.set_snapshot() self.set_root_path() self.set_source_path() self.set_chroot_path() @@ -191,9 +191,8 @@ class StageBase(TargetBase, ClearBase, GenBase): # Setup our mount points. self.mount = MOUNT_DEFAULTS.copy() -# Always unpack snapshot tarball -self.mount['portdir']['enable'] = False - +self.mount['portdir']['source'] = self.snapshot +self.mount['portdir']['target'] = self.settings['repo_basedir'] + '/' + self.settings['repo_name'] self.mount['distdir']['source'] = self.settings['distdir'] self.mount["distdir"]['target'] = self.settings['target_distdir'] @@ -435,21 +434,11 @@ class StageBase(TargetBase, ClearBase, GenBase): self.settings["destpath"] = normpath(self.settings["chroot_path"]) def set_cleanables(self): -self.settings["cleanables"] = ["/etc/resolv.conf", "/var/tmp/*", "/tmp/*", - self.settings["repo_basedir"] + "/" + -
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/
commit: c98bd2b223fab1a2784427d8b09fe8a682647bf4 Author: Matt Turner gentoo org> AuthorDate: Wed Apr 22 00:57:34 2020 + Commit: Matt Turner gentoo org> CommitDate: Thu Apr 30 23:04:34 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=c98bd2b2 catalyst: Only write out non-default paths to make.conf Signed-off-by: Matt Turner gentoo.org> catalyst/base/stagebase.py | 11 +++ doc/catalyst-config.5.txt | 19 +-- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index 1f091829..affdabbe 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -14,7 +14,7 @@ from snakeoil.osutils import pjoin from DeComp.compress import CompressMap from catalyst import log -from catalyst.defaults import (MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN) +from catalyst.defaults import (confdefaults, MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN) from catalyst.support import (CatalystError, file_locate, normpath, cmd, read_makeconf, ismount, file_check) from catalyst.base.targetbase import TargetBase @@ -1055,9 +1055,12 @@ class StageBase(TargetBase, ClearBase, GenBase): myf.write(hostuseexpand + '="' + ' '.join(myuseexpandvars[hostuseexpand]) + '"\n') -myf.write('PORTDIR="%s"\n' % self.settings['target_portdir']) -myf.write('DISTDIR="%s"\n' % self.settings['target_distdir']) -myf.write('PKGDIR="%s"\n' % self.settings['target_pkgdir']) +# Write non-default PORTDIR/DISTDIR/PKGDIR settings to make.conf +for x in ['target_portdir', 'target_distdir', 'target_pkgdir']: +if self.settings[x] != confdefaults[x]: +varname = x.split('_')[1].upper() +myf.write(f'{varname}="{self.settings[x]}"\n') + if setup: # Setup the portage overlay if "portage_overlay" in self.settings: diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index b7d493eb..11b27d90 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -114,20 +114,19 @@ The name of the main repository (e.g. gentoo). The git repository at snapshot. *target_distdir*:: -This is the target distfiles directory location for the stage being created. -This is important because this value will be stored in the stage's make.conf -and will become the default location used if it is not edited by users. -The default location is `/var/cache/distfiles`. +Defines the location of the local source file repository in the +target. This will be written to the target's make.conf if it is not +the default value of `/var/cache/distfiles`. *target_pkgdir*:: -This is the target packages directory for storing binpkgs in the stage being -built. This location is stored in the make.conf of the stage being built. -The default location is `/var/cache/binpkgs` +Defines the location of binary packages in the target. This will be +written to the target's make.conf if it is not the default value of +`/var/cache/binpkgs`. *target_portdir*:: -Defines the location of main repository in the target stages. This -location is stored in make.conf of the stage being built. -The default location is `/var/db/repos/gentoo` +Defines the location of the main ebuild repository in the target. +This will be written to the target's make.conf if it is not the +default value of `/var/db/repos/gentoo`. Other settings ~~
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/
commit: 73d70b4f433147d94d6c1e458fe2341dba1dcff7 Author: Matt Turner gentoo org> AuthorDate: Wed Apr 22 00:57:34 2020 + Commit: Matt Turner gentoo org> CommitDate: Thu Apr 30 22:56:12 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=73d70b4f catalyst: Only write out non-default paths to make.conf Signed-off-by: Matt Turner gentoo.org> catalyst/base/stagebase.py | 11 +++ doc/catalyst-config.5.txt | 19 +-- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index 1f091829..affdabbe 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -14,7 +14,7 @@ from snakeoil.osutils import pjoin from DeComp.compress import CompressMap from catalyst import log -from catalyst.defaults import (MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN) +from catalyst.defaults import (confdefaults, MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN) from catalyst.support import (CatalystError, file_locate, normpath, cmd, read_makeconf, ismount, file_check) from catalyst.base.targetbase import TargetBase @@ -1055,9 +1055,12 @@ class StageBase(TargetBase, ClearBase, GenBase): myf.write(hostuseexpand + '="' + ' '.join(myuseexpandvars[hostuseexpand]) + '"\n') -myf.write('PORTDIR="%s"\n' % self.settings['target_portdir']) -myf.write('DISTDIR="%s"\n' % self.settings['target_distdir']) -myf.write('PKGDIR="%s"\n' % self.settings['target_pkgdir']) +# Write non-default PORTDIR/DISTDIR/PKGDIR settings to make.conf +for x in ['target_portdir', 'target_distdir', 'target_pkgdir']: +if self.settings[x] != confdefaults[x]: +varname = x.split('_')[1].upper() +myf.write(f'{varname}="{self.settings[x]}"\n') + if setup: # Setup the portage overlay if "portage_overlay" in self.settings: diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index b7d493eb..11b27d90 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -114,20 +114,19 @@ The name of the main repository (e.g. gentoo). The git repository at snapshot. *target_distdir*:: -This is the target distfiles directory location for the stage being created. -This is important because this value will be stored in the stage's make.conf -and will become the default location used if it is not edited by users. -The default location is `/var/cache/distfiles`. +Defines the location of the local source file repository in the +target. This will be written to the target's make.conf if it is not +the default value of `/var/cache/distfiles`. *target_pkgdir*:: -This is the target packages directory for storing binpkgs in the stage being -built. This location is stored in the make.conf of the stage being built. -The default location is `/var/cache/binpkgs` +Defines the location of binary packages in the target. This will be +written to the target's make.conf if it is not the default value of +`/var/cache/binpkgs`. *target_portdir*:: -Defines the location of main repository in the target stages. This -location is stored in make.conf of the stage being built. -The default location is `/var/db/repos/gentoo` +Defines the location of the main ebuild repository in the target. +This will be written to the target's make.conf if it is not the +default value of `/var/db/repos/gentoo`. Other settings ~~
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/, etc/
commit: 7904c59e78590a1fd0a07a9a439fb5c4e9865425 Author: Matt Turner gentoo org> AuthorDate: Wed Apr 22 01:15:59 2020 + Commit: Matt Turner gentoo org> CommitDate: Wed Apr 22 02:09:59 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=7904c59e catalyst: Make distdir default to host's DISTDIR Signed-off-by: Matt Turner gentoo.org> catalyst/defaults.py | 4 +++- doc/catalyst-config.5.txt | 5 ++--- etc/catalyst.conf | 4 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/catalyst/defaults.py b/catalyst/defaults.py index a0b87f96..0c18f9f5 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -1,4 +1,6 @@ +import portage + from collections import OrderedDict from DeComp.definitions import DECOMPRESSOR_SEARCH_ORDER @@ -54,7 +56,7 @@ confdefaults = { "compressor_options": XATTRS_OPTIONS['linux'], "decomp_opt": DECOMPRESSOR_PROGRAM_OPTIONS['linux'], "decompressor_search_order": DECOMPRESSOR_SEARCH_ORDER, -"distdir": "/var/cache/distfiles", +"distdir": portage.settings['DISTDIR'], "icecream": "/var/cache/icecream", 'list_xattrs_opt': LIST_XATTRS_OPTIONS['linux'], "local_overlay": "/var/db/repos/local", diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index 14b7dd5a..df3f25f9 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -122,9 +122,8 @@ Repository settings ~~~ *distdir*:: -Source distfiles location used in generation of the stages. This is usually the -hosts distfiles location. `/var/cache/distfiles` should work for most -default installations, but it should be set to match your hosts configuration. +Defines the location of your local source file repository. +Defaults to the host's DISTDIR. *portdir*:: Source Gentoo tree location (primary repo). `/var/db/repos/gentoo/` should work for most diff --git a/etc/catalyst.conf b/etc/catalyst.conf index 2e511cce..60a4dd93 100644 --- a/etc/catalyst.conf +++ b/etc/catalyst.conf @@ -12,10 +12,6 @@ # digests="blake2b sha512" -# distdir specifies where your distfiles are located. This setting should -# work fine for most default installations. -distdir="/var/cache/distfiles" - # envscript allows users to set options such as http proxies, MAKEOPTS, # GENTOO_MIRRORS, or any other environment variables needed for building. # The envscript file sets environment variables like so:
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/, etc/, /
commit: 6565ad2eb90fe0840047d097fa5637d176a7d580 Author: Matt Turner gentoo org> AuthorDate: Fri Apr 17 03:21:34 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri Apr 17 17:20:18 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6565ad2e catalyst: Use hashlib instead of external tools Signed-off-by: Matt Turner gentoo.org> README | 1 - catalyst/base/genbase.py | 18 ++- catalyst/base/stagebase.py | 8 +-- catalyst/hash_utils.py | 126 - catalyst/main.py | 42 ++- doc/catalyst-config.5.txt | 8 +-- etc/catalyst.conf | 18 +++ 7 files changed, 35 insertions(+), 186 deletions(-) diff --git a/README b/README index eb75ba67..1a039fca 100644 --- a/README +++ b/README @@ -20,7 +20,6 @@ Requirements - Python 3.6 or greater - An ebuild repository snapshot (or an ebuild tree to create one) - A generic stage3 tarball for your architecture -- shash for digest support What is catalyst? diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py index 08076460..632ee0d9 100644 --- a/catalyst/base/genbase.py +++ b/catalyst/base/genbase.py @@ -1,4 +1,5 @@ +import hashlib import io import os @@ -11,6 +12,20 @@ class GenBase(): def __init__(self, myspec): self.settings = myspec +@staticmethod +def generate_hash(filepath, name): +h = hashlib.new(name) + +with open(filepath, 'rb') as f: +while True: +data = f.read(8192) +if not data: +break +h.update(data) + +filename = os.path.split(filepath)[1] +return f'# {name.upper()} HASH\n{h.hexdigest()} {filename}\n' + def gen_contents_file(self, path): contents = path + ".CONTENTS" if os.path.exists(contents): @@ -29,11 +44,10 @@ class GenBase(): if os.path.exists(digests): os.remove(digests) if "digests" in self.settings: -hash_map = self.settings["hash_map"] if os.path.exists(path): with io.open(digests, "w", encoding='utf-8') as myf: for f in [path, path + '.CONTENTS']: if os.path.exists(f): for i in self.settings["digests"].split(): -digest = hash_map.generate_hash(f, hash_=i) +digest = self.generate_hash(f, name=i) myf.write(digest) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index 5f3fa1d0..71bf1ef9 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -442,9 +442,7 @@ class StageBase(TargetBase, ClearBase, GenBase): # XXX: Is this even necessary if the previous check passes? if os.path.exists(self.settings["source_path"]): self.settings["source_path_hash"] = \ -self.settings["hash_map"].generate_hash( -self.settings["source_path"], -hash_="sha1") +self.generate_hash(self.settings["source_path"], "sha1") log.notice('Source path set to %s', self.settings['source_path']) def set_dest_path(self): @@ -469,9 +467,7 @@ class StageBase(TargetBase, ClearBase, GenBase): ) log.info('SNAPSHOT_PATH set to: %s', self.settings['snapshot_path']) self.settings["snapshot_path_hash"] = \ -self.settings["hash_map"].generate_hash( -self.settings["snapshot_path"], -hash_="sha1") +self.generate_hash(self.settings["snapshot_path"], "sha1") def set_snapcache_path(self): self.settings["snapshot_cache_path"] = \ diff --git a/catalyst/hash_utils.py b/catalyst/hash_utils.py deleted file mode 100644 index 3aae890e.. --- a/catalyst/hash_utils.py +++ /dev/null @@ -1,126 +0,0 @@ - -import os -from collections import namedtuple -from subprocess import Popen, PIPE - -from catalyst import log -from catalyst.support import CatalystError - - -# Use HashMap.fields for the value legend -# fields = ["func", "cmd", "args", "id"] -HASH_DEFINITIONS = { -"adler32" :["calc_hash2", "shash", ["-a", "ADLER32"], "ADLER32"], -"blake2" :["calc_hash", "b2sum", [ ], "BLAKE2"], -"crc32":["calc_hash2", "shash", ["-a", "CRC32"], "CRC32"], -"crc32b" :["calc_hash2", "shash", ["-a", "CRC32B"], "CRC32B"], -"gost" :["calc_hash2", "shash", ["-a", "GOST"], "GOST"], -"haval128" :["calc_hash2", "shash", ["-a", "HAVAL128"], "HAVAL128"], -"haval160" :["calc_hash2", "shash", ["-a", "HAVAL160"], "HAVAL160"], -"haval192" :["calc_hash2", "shash", ["-a", "HAVAL192"], "HAVAL192"], -"haval224" :["calc_hash2", "shash", ["-a", "HAVAL224"], "HAVAL224"], -
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/, etc/
commit: 45edc6510ce9cbf9cbe9ab5d2b7b8cd9f65f84cb Author: Matt Turner gentoo org> AuthorDate: Fri Apr 17 06:38:14 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri Apr 17 17:03:47 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=45edc651 catalyst: Remove the 'hash_function' config option Switch to SHA1, which is plenty fast. The next commit switches from the external shash/b2sum tools to Python's hashlib, and the removal of this config option will simplify that since the crc32 hash is in a different python module (zlib). Signed-off-by: Matt Turner gentoo.org> catalyst/base/stagebase.py | 4 ++-- catalyst/defaults.py | 2 -- catalyst/main.py | 16 doc/catalyst-config.5.txt | 6 -- etc/catalyst.conf | 10 -- 5 files changed, 2 insertions(+), 36 deletions(-) diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py index f673382f..5f3fa1d0 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -444,7 +444,7 @@ class StageBase(TargetBase, ClearBase, GenBase): self.settings["source_path_hash"] = \ self.settings["hash_map"].generate_hash( self.settings["source_path"], -hash_=self.settings["hash_function"]) +hash_="sha1") log.notice('Source path set to %s', self.settings['source_path']) def set_dest_path(self): @@ -471,7 +471,7 @@ class StageBase(TargetBase, ClearBase, GenBase): self.settings["snapshot_path_hash"] = \ self.settings["hash_map"].generate_hash( self.settings["snapshot_path"], -hash_=self.settings["hash_function"]) +hash_="sha1") def set_snapcache_path(self): self.settings["snapshot_cache_path"] = \ diff --git a/catalyst/defaults.py b/catalyst/defaults.py index e60980d3..f292c211 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -40,7 +40,6 @@ valid_config_file_values.extend([ "digests", "distcc", "envscript", -"hash_function", "options", "snapshot_cache", "VERBOSE", @@ -63,7 +62,6 @@ confdefaults = { "decomp_opt": DECOMPRESSOR_PROGRAM_OPTIONS['linux'], "decompressor_search_order": DECOMPRESSOR_SEARCH_ORDER, "distdir": DISTDIR[:], -"hash_function": "crc32", "icecream": "/var/cache/icecream", 'list_xattrs_opt': LIST_XATTRS_OPTIONS['linux'], "local_overlay": REPODIR[:] + "/local", diff --git a/catalyst/main.py b/catalyst/main.py index ba6e3fcc..cb3cd3f7 100644 --- a/catalyst/main.py +++ b/catalyst/main.py @@ -384,22 +384,6 @@ def _main(parser, opts): # Now reload the config with our updated value. conf_values['digests'] = ' '.join(digests) -if "hash_function" in conf_values: -if conf_values["hash_function"] not in HASH_DEFINITIONS: -log.critical( -'%s is not a valid hash_function entry\n' -'Valid hash_function entries:\n' -'%s', conf_values["hash_function"], HASH_DEFINITIONS.keys()) -try: -process.find_binary( -hash_map.hash_map[conf_values["hash_function"]].cmd) -except process.CommandNotFound: -log.critical( -'The "%s" binary needed by hash_function "%s" was not found. ' -'It needs to be in your system path.', -hash_map.hash_map[conf_values['hash_function']].cmd, -conf_values['hash_function']) - addlargs = {} if myspecfile: diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index 220bb528..d5e8c128 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -66,12 +66,6 @@ variables using POSIX shell notation: export FOO="bar" - -*hash_function*:: -Internal hash function catalyst should use for things like autoresume, -seedcache, etc. The default and fastest is `crc32`. You should not -ever need to change this unless your OS does not support it. See the -*SUPPORTED HASHES* section for a list of supported hashes. - **options*:: Set different build-time options (example: `autoresume bindist kerncache pkgcache seedcache snapcache`). Supported values: diff --git a/etc/catalyst.conf b/etc/catalyst.conf index a3c22049..4c4d491e 100644 --- a/etc/catalyst.conf +++ b/etc/catalyst.conf @@ -24,16 +24,6 @@ distdir="/var/cache/distfiles" # export FOO="bar" envscript="/etc/catalyst/catalystrc" -# Internal hash function catalyst should use for things like autoresume, -# seedcache, etc. The default and fastest is crc32. You should not ever need -# to change this unless your OS does not support it. -# Supported hashes: -# adler32, blake2, crc32, crc32b, gost, haval128, haval160, haval192, haval224, -# haval256, md2, md4, md5, ripemd128, ripemd160, ripemd256,
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/
commit: bb21b8615e64cb31fa9aa9d533ef328dc1374e45 Author: Matt Turner gentoo org> AuthorDate: Fri Apr 17 18:03:02 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri Apr 17 19:51:32 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=bb21b861 catalyst: gzip the .CONTENTS file Other algorithms give better compression ratios, but the difference is not meaningful for a 2MiB text file. In my testing bzip2 gave a better compression ratio of 15:1 vs gzip's 11:1, but that ends up being only a size difference of 50KiB (136 vs 187) which is only an additional 2.5% savings from the uncompressed input. Choose gzip because transparent decompression is widely supported by web servers and clients. Closes: https://bugs.gentoo.org/630284 Signed-off-by: Matt Turner gentoo.org> catalyst/base/genbase.py | 5 +++-- catalyst/support.py | 2 +- doc/HOWTO.txt| 6 +++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py index eb09a4e0..3db20f84 100644 --- a/catalyst/base/genbase.py +++ b/catalyst/base/genbase.py @@ -2,6 +2,7 @@ import hashlib import io import os +import gzip class GenBase(): """ @@ -28,7 +29,7 @@ class GenBase(): def gen_contents_file(self, path): c = self.settings['contents_map'] -with io.open(path + '.CONTENTS', 'w', encoding='utf-8') as file: +with gzip.open(path + '.CONTENTS.gz', 'w', encoding='utf-8') as file: file.write(c.contents(path, '', verbose=self.settings['VERBOSE'])) def gen_digest_file(self, path): @@ -36,6 +37,6 @@ class GenBase(): return with io.open(path + '.DIGESTS', 'w', encoding='utf-8') as file: -for f in [path, path + '.CONTENTS']: +for f in [path, path + '.CONTENTS.gz']: for i in self.settings['digests']: file.write(self.generate_hash(f, name=i)) diff --git a/catalyst/support.py b/catalyst/support.py index 654b23aa..988a81f5 100644 --- a/catalyst/support.py +++ b/catalyst/support.py @@ -65,7 +65,7 @@ def file_check(filepath, extensions=None, strict=True): files = glob.glob("%s.*" % filepath) # remove any false positive files files = [x for x in files if not x.endswith( -".CONTENTS") and not x.endswith(".DIGESTS")] +".CONTENTS") and not x.endswith(".CONTENTS.gz") and not x.endswith(".DIGESTS")] if len(files) == 1: return files[0] if len(files) > 1 and strict: diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt index 960b5761..7b759121 100644 --- a/doc/HOWTO.txt +++ b/doc/HOWTO.txt @@ -22,7 +22,7 @@ Create a snapshot of your current Portage tree (you may want to # catalyst --snapshot 20130131 # ls /var/tmp/catalyst/snapshots/ portage-20130131.tar.bz2 -portage-20130131.tar.bz2.CONTENTS +portage-20130131.tar.bz2.CONTENTS.gz portage-20130131.tar.bz2.DIGESTS where the storage location is relative to the default @@ -44,7 +44,7 @@ For example, Grab the tarball and put it where catalyst will find it: # wget http://…/stage3-amd64-20121213.tar.bz2 -# wget http://…/stage3-amd64-20121213.tar.bz2.CONTENTS +# wget http://…/stage3-amd64-20121213.tar.bz2.CONTENTS.gz # wget http://…/stage3-amd64-20121213.tar.bz2.DIGESTS.asc # sha512sum -c stage3-amd64-20121213.tar.bz2.DIGESTS.asc # gpg --verify stage3-amd64-20121213.tar.bz2.DIGESTS.asc @@ -89,7 +89,7 @@ which will build the target and install something like: # ls /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.* /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2 -/var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.CONTENTS +/var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.CONTENTS.gz /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.DIGESTS The name is an expansion of
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: aa656fef421a285809a06e184389052fbe063f48 Author: Matt Turner gentoo org> AuthorDate: Fri Apr 10 04:58:39 2020 + Commit: Matt Turner gentoo org> CommitDate: Fri Apr 10 06:12:16 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=aa656fef doc: Update boot/kernel//console documentation Signed-off-by: Matt Turner gentoo.org> doc/catalyst-spec.5.txt | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index 2b8413ec..58f0a9f0 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -253,10 +253,10 @@ use this on the official media. `` should match one of the labels given to *boot/kernel*. *boot/kernel//console*:: -This is only supported on ppc64 currently. This entry sets up the -console boot parameters required for sending the output to the -appropriate console (example: `tty0 ttyS0`). `` should match -one of the labels given to *boot/kernel*. +This is only supported on with grub currently (x86, amd64, ia64, ppc, +sparc). This entry sets up the console boot parameters required for +sending the output to the appropriate console (example: `tty0 ttyS0`). +`` should match one of the labels given to *boot/kernel*. */modblacklist*:: This is for blacklisting modules from being hotplugged that are known
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: f07e9148213a499251e0c81db083eccee4eb8546 Author: Matt Turner gentoo org> AuthorDate: Thu Apr 9 00:28:43 2020 + Commit: Matt Turner gentoo org> CommitDate: Thu Apr 9 18:47:23 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f07e9148 doc: Update catalyst-spec.5 for netboot2 rename Signed-off-by: Matt Turner gentoo.org> doc/catalyst-spec.5.txt | 31 +++ 1 file changed, 7 insertions(+), 24 deletions(-) diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index 03fdeecc..2b8413ec 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -115,10 +115,6 @@ explained below. This setting is supported by the livecd targets. -*/builddate*:: -Set the build date of the `` (example: `20060107`). This -setting is supported by the `netboot2` target. - */readme*:: This is for the README.txt on the root of the CD. For Gentoo releases, we use a default README.txt, and this will be used on your @@ -223,14 +219,12 @@ the labels that will be used by the CD bootloader to refer to each kernel image (example: `gentoo`). *boot/kernel//sources*:: -*netboot/kernel/sources*:: This option tells catalyst which kernel sources to merge for this kernel label (example: `gentoo-sources`). This can use normal portage atoms to specify a specific version. `` should match one of the labels given to *boot/kernel*. *boot/kernel//config*:: -*netboot/kernel/config*:: This option is the full path and filename to a kernel `.config` file that is used by genkernel to compile the kernel this label applies to. `` should match one of the labels given to *boot/kernel*. @@ -271,7 +265,6 @@ will keep it from being auto-loaded, even if it is detected by hotplug. This setting is supported by the `stage4` and `livecd` targets. -*netboot/kernel/use*:: *boot/kernel//use*:: This option sets the `USE` flags used to build the kernel and also any packages which are defined under this kernel label (example: `pcmcia @@ -293,16 +286,13 @@ to enable keymap selection. Netboot ~~~ -*/busybox_config*:: +*netboot/busybox_config*:: The netboot target builds busybox for its root filesystem. This option is where you specify the full path and filename to your busybox -configuration (example: `/tmp/busybox.config`). This setting is -supported by the `netboot` and `netboot2` targets. +configuration (example: `/tmp/busybox.config`). -*netboot/base_tarball*:: -This is the full path and filename to the tarball to use as the base -for the netboot image (example: -`/usr/share/catalyst/netboot/netboot-base.tar.bz2`). +*netboot/builddate*:: +Set the build date of the `` (example: `20060107`). Runlevels ~ @@ -343,7 +333,7 @@ quite a few problems with these, so be careful with whatever `USE` flags you add here. This is generally used for adding some functionality that we do not want on by default for all Gentoo users, but that we want on by default in our binaries. This setting is -supported by the `stage4` and `netboot2` targets. +supported by the `stage4` and `netboot` targets. */packages*:: This is the set of packages that we will merge into the stage4 tarball @@ -352,22 +342,15 @@ fxload irssi wpa_supplicant`). They will be built with the `USE` flags configured above. These packages must not depend on a configured kernel. If the package requires a configured kernel, then it will be defined elsewhere. This setting is supported by the -`stage4`, and `netboot2` targets. - -*netboot/packages*:: -These package names are also labels used later when determining what -files to copy into your netboot image (example: `raidtools -e2fsprogs`). +`stage4`, and `netboot` targets. -*/packages//files*:: +*netboot/packages//files*:: This is where you tell catalyst which files from each package to copy into the netboot image. `` should match one of the labels given to *netboot/packages*. For example: netboot/packages/raidtools/files: /sbin/raidstart /sbin/mkraid -This option is supported by the `netboot` and `netboot2` targets. - *netboot/extra_files*:: This is a list of any other files, not belonging to the above packages, that you would wish to have copied into your netboot image
[gentoo-commits] proj/catalyst:master commit in: doc/, examples/, targets/support/
commit: f82226cf8a9f09099557ef344dce60b7d4d6f663 Author: Matt Turner gentoo org> AuthorDate: Tue Mar 31 03:25:48 2020 + Commit: Matt Turner gentoo org> CommitDate: Tue Mar 31 03:54:29 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f82226cf catalyst: Remove references to spind spind was deleted from livecd-tools in 2011. See: https://gitweb.gentoo.org/proj/livecd-tools.git/commit/?id=7d4d363bae7313e46de960a45888907059b0f2d6 Signed-off-by: Matt Turner gentoo.org> doc/catalyst-spec.5.txt | 2 +- examples/livecd-stage2_template.spec | 2 +- examples/stage4_template.spec| 2 +- targets/support/rc-update.sh | 7 --- 4 files changed, 3 insertions(+), 10 deletions(-) diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index 4c0be357..ecea9ba7 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -319,7 +319,7 @@ the `stage4` and `livecd` targets. This is for adding init scripts to runlevels. The syntax for the init script is the script name, followed by a pipe, followed by the runlevel in which you want the script to run. It looks like -`spind|default` and is space delimited. We do not use this on the +`acpid|default` and is space delimited. We do not use this on the official media, as catalyst sets up the runlevels correctly for us. This setting is supported by the `stage4` and `livecd` targets. diff --git a/examples/livecd-stage2_template.spec b/examples/livecd-stage2_template.spec index fe9b36e7..0c45bd67 100644 --- a/examples/livecd-stage2_template.spec +++ b/examples/livecd-stage2_template.spec @@ -188,7 +188,7 @@ livecd/modblacklist: # This is for adding init scripts to runlevels. The syntax for the init script # is the script name, followed by a pipe, followed by the runlevel in which you -# want the script to run. It looks like spind|default and is space delimited. +# want the script to run. It looks like acpid|default and is space delimited. # We do not use this on the official media, as catalyst sets up the runlevels # correctly for us. Since we do not use this, it is left blank below. # This option will automatically create missing runlevels diff --git a/examples/stage4_template.spec b/examples/stage4_template.spec index fe394f3e..f3531f90 100644 --- a/examples/stage4_template.spec +++ b/examples/stage4_template.spec @@ -148,7 +148,7 @@ stage4/modblacklist: # This is for adding init scripts to runlevels. The syntax for the init script # is the script name, followed by a pipe, followed by the runlevel in which you -# want the script to run. It looks like spind|default and is space delimited. +# want the script to run. It looks like acpid|default and is space delimited. # We do not use this on the official media, as catalyst sets up the runlevels # correctly for us. Since we do not use this, it is left blank below. # example: diff --git a/targets/support/rc-update.sh b/targets/support/rc-update.sh index 3941f2e0..6e739966 100755 --- a/targets/support/rc-update.sh +++ b/targets/support/rc-update.sh @@ -18,16 +18,9 @@ then # Do some livecd_type specific rc-update changes case ${clst_livecd_type} in - gentoo-gamecd) - rc-update add spind default - ;; gentoo-release-live*) - rc-update add spind default rc-update add xdm default ;; - generic-livecd) - rc-update add spind default - ;; esac fi
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: 724b49a0407303f5f9459238d1cb2f0ab290844f Author: Matt Turner gentoo org> AuthorDate: Mon Mar 30 16:53:49 2020 + Commit: Matt Turner gentoo org> CommitDate: Mon Mar 30 23:47:16 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=724b49a0 doc: Fix some typos Signed-off-by: Matt Turner gentoo.org> doc/HOWTO.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt index b1d315e6..960b5761 100644 --- a/doc/HOWTO.txt +++ b/doc/HOWTO.txt @@ -1,4 +1,4 @@ -Catalyst is a release-buildcing tool for Gentoo. If you use Gentoo +Catalyst is a release-building tool for Gentoo. If you use Gentoo and want to roll your own live CD or bootable USB stick, this is the way to go. First, get a Gentoo development box and install the necessary tools: @@ -54,7 +54,7 @@ where the storage dir is `$storedir/builds/$source_subpath` (`$storedir` from `catalyst.conf`, `$source_subpath` from your `*.spec` file). -`.*spec` files +`*.spec` files ~~ `.*spec` files tell catalyst about the system you're trying to build.
[gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/, /, examples/, targets/grp/, catalyst/targets/
commit: 9eac042d5e39ea30a6d26468218fa2151faf2ad8 Author: Matt Turner gentoo org> AuthorDate: Sun Mar 29 00:44:10 2020 + Commit: Matt Turner gentoo org> CommitDate: Sun Mar 29 01:35:14 2020 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9eac042d catalyst: Delete Gentoo Reference Platform target Unused for 12 years. Signed-off-by: Matt Turner gentoo.org> README | 1 - catalyst/defaults.py | 2 +- catalyst/targets/grp.py| 97 -- doc/catalyst-spec.5.txt| 24 + doc/catalyst.1.txt | 6 +-- examples/grp_template.spec | 117 - targets/grp/chroot.sh | 21 targets/grp/controller.sh | 34 targets/grp/preclean-chroot.sh | 7 --- 9 files changed, 5 insertions(+), 304 deletions(-) diff --git a/README b/README index 6dc6a149..b398fe30 100644 --- a/README +++ b/README @@ -34,7 +34,6 @@ Catalyst is capable of: - Building installation stages - Building bootable LiveCDs -- Building GRP (Gentoo Reference Platform) sets - Setting up a Tinderbox target for test building - Building netboot images diff --git a/catalyst/defaults.py b/catalyst/defaults.py index 705ae931..568d8566 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -17,7 +17,7 @@ required_build_targets = ["targetbase", "generic_stage_target"] # new build types should be added here valid_build_targets = ["stage1_target", "stage2_target", "stage3_target", - "stage4_target", "grp_target", "livecd_stage1_target", "livecd_stage2_target", + "stage4_target", "livecd_stage1_target", "livecd_stage2_target", "embedded_target", "tinderbox_target", "snapshot_target", "netboot_target", "netboot2_target" ] diff --git a/catalyst/targets/grp.py b/catalyst/targets/grp.py deleted file mode 100644 index d47654d0.. --- a/catalyst/targets/grp.py +++ /dev/null @@ -1,97 +0,0 @@ -""" -Gentoo Reference Platform (GRP) target -""" -# NOTE: That^^ docstring has influence catalyst-spec(5) man page generation. - -import os -import glob - -from catalyst import log -from catalyst.support import (CatalystError, normpath, cmd) -from catalyst.base.stagebase import StageBase - - -class grp(StageBase): - """ - The builder class for GRP (Gentoo Reference Platform) builds. - """ - def __init__(self,spec,addlargs): - self.required_values=["version_stamp","target","subarch",\ - "rel_type","profile","snapshot","source_subpath"] - - self.valid_values=self.required_values[:] - self.valid_values.extend(["grp/use"]) - if "grp" not in addlargs: - raise CatalystError("Required value \"grp\" not specified in spec.") - - self.required_values.extend(["grp"]) - if isinstance(addlargs['grp'], str): - addlargs["grp"]=[addlargs["grp"]] - - if "grp/use" in addlargs: - if isinstance(addlargs['grp/use'], str): - addlargs["grp/use"]=[addlargs["grp/use"]] - - for x in addlargs["grp"]: - self.required_values.append("grp/"+x+"/packages") - self.required_values.append("grp/"+x+"/type") - - StageBase.__init__(self,spec,addlargs) - - def run_local(self): - for pkgset in self.settings["grp"]: - # example call: "grp.sh run pkgset cd1 xmms vim sys-apps/gleep" - try: - cmd([self.settings['controller_file'], 'run', - self.settings['grp/' + pkgset + '/type'], - pkgset] + self.settings['grp/' + pkgset + '/packages'], - env=self.env) - - except CatalystError: - self.unbind() - raise CatalystError("GRP build aborting due to error.", - print_traceback=True) - - def set_mounts(self): - self.mounts.append("/tmp/grp") - self.mountmap["/tmp/grp"]=self.settings["target_path"] - - def generate_digests(self): - for pkgset in self.settings["grp"]: - if self.settings["grp/"+pkgset+"/type"] == "pkgset": - destdir=normpath(self.settings["target_path"]+"/"+pkgset+"/All") - log.notice('Digesting files in the pkgset...') - digests=glob.glob(destdir+'/*.DIGESTS') - for i in digests: - if os.path.exists(i): - os.remove(i) - -
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: 0f3b5d033719787f8ef8764b640c2b148e5b7db4 Author: Rick Farina (Zero_Chaos) gentoo org> AuthorDate: Thu Apr 26 15:56:10 2018 + Commit: Richard Farina gentoo org> CommitDate: Thu Apr 26 15:56:10 2018 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=0f3b5d03 fix shebang in doc creation per report by hanetzer on irc, this shebang work for prefix installs and per floppym it is valid and correct doc/make_subarch_table_guidexml.py | 2 +- doc/make_target_table.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/make_subarch_table_guidexml.py b/doc/make_subarch_table_guidexml.py index f571e001..84624dc1 100755 --- a/doc/make_subarch_table_guidexml.py +++ b/doc/make_subarch_table_guidexml.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # Copyright (C) 2011 Sebastian Pipping# Copyright (C) 2013 Brian dolbec # Licensed under GPL v2 or later diff --git a/doc/make_target_table.py b/doc/make_target_table.py index 9eb072b1..c4b49ad9 100755 --- a/doc/make_target_table.py +++ b/doc/make_target_table.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # Copyright (C) 2012 W. Trevor King # Copyright (C) 2012 Sebastian Pipping # Copyright (C) 2013 Brian dolbec
[gentoo-commits] proj/catalyst:master commit in: doc/, examples/, targets/support/, targets/livecd-stage2/
commit: 189a1ce2cfb24efad8bc33c43a955c82547b4364 Author: Rick Farina (Zero_Chaos) gentoo org> AuthorDate: Thu Apr 5 15:57:52 2018 + Commit: Richard Farina gentoo org> CommitDate: Thu Apr 5 15:57:52 2018 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=189a1ce2 make livecd depclean settings optional allow livecd to keep build deps or just not run depclean at all doc/catalyst-spec.5.txt | 9 + examples/livecd-stage2_template.spec | 7 +++ targets/livecd-stage2/livecd-stage2-controller.sh | 2 +- targets/support/depclean.sh | 6 +- 4 files changed, 22 insertions(+), 2 deletions(-) diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt index a1559faf..5e9cf87a 100644 --- a/doc/catalyst-spec.5.txt +++ b/doc/catalyst-spec.5.txt @@ -383,6 +383,15 @@ This is a list of any other files, not belonging to the above packages, that you would wish to have copied into your netboot image (example: `/lib/libresolv.so.2 /lib/libnss_compat.so.2`). +*livecd/depclean*:: +This feature controls the depclean run after fsscript and before unmerge. +The default is unset, and will run emerge --depclean --with-bdeps=n which results +in the smallest possible livecd. For some use cases it may be nice to not run depclean at all, +or to keep build deps. For those cases, the following two special cases are available: +livecd/depclean: no +livecd/depclean: keepbdeps +This setting is only supported by the livecd target. + */unmerge*:: This is a list of packages that will be unmerged after all the kernels have been built (example: `autoconf automake libtool m4 bison`). diff --git a/examples/livecd-stage2_template.spec b/examples/livecd-stage2_template.spec index a90b7aac..a19812ae 100644 --- a/examples/livecd-stage2_template.spec +++ b/examples/livecd-stage2_template.spec @@ -336,6 +336,13 @@ boot/kernel/gentoo/console: # Currently this feature will be enabled if livecd/verify is defined to *any* value, leave commented to disable. #livecd/verify: true +# This feature controls the depclean run after fsscript and before unmerge. +# The default is unset, and will run emerge --depclean --with-bdeps=n which results +# in the smallest possible livecd. For some use cases it may be nice to not run depclean at all, +# or to keep build deps. For those cases, the following two special cases are available: +# livecd/depclean: no +# livecd/depclean: keepbdeps + # This is a list of packages that will be unmerged after all the kernels have # been built. There are no checks on these packages, so be careful what you # add here. They can potentially break your CD. diff --git a/targets/livecd-stage2/livecd-stage2-controller.sh b/targets/livecd-stage2/livecd-stage2-controller.sh index 845d6d55..2f0af1f9 100755 --- a/targets/livecd-stage2/livecd-stage2-controller.sh +++ b/targets/livecd-stage2/livecd-stage2-controller.sh @@ -187,7 +187,7 @@ case $1 in ;; unmerge) - exec_in_chroot ${clst_shdir}/support/depclean.sh + [ "${clst_depclean}" != "no" ] && exec_in_chroot ${clst_shdir}/support/depclean.sh shift export clst_packages="$*" exec_in_chroot ${clst_shdir}/support/unmerge.sh diff --git a/targets/support/depclean.sh b/targets/support/depclean.sh index aa66c9fd..b3640958 100755 --- a/targets/support/depclean.sh +++ b/targets/support/depclean.sh @@ -4,6 +4,10 @@ source /tmp/chroot-functions.sh # If the user enabled PRESERVE_LIBS in options, tell portage to preserve them. [ -n "${clst_PRESERVE_LIBS}" ] && FEATURES="${clst_myfeatures} preserve-libs" -run_merge --depclean --with-bdeps=n +if [ "${clst_depclean}" = "keepbdeps" ]; then + run_merge --depclean --with-bdeps=y +else + run_merge --depclean --with-bdeps=n +fi exit 0
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: d7dbe6e1b2fd3148a652843ef43ab80415959842 Author: Rick Farina (Zero_Chaos) gentoo org> AuthorDate: Sat Mar 11 02:35:39 2017 + Commit: Richard Farina gentoo org> CommitDate: Sat Mar 11 02:35:39 2017 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d7dbe6e1 would be nice if the ebuild worked again doc/make_subarch_table_guidexml.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/make_subarch_table_guidexml.py b/doc/make_subarch_table_guidexml.py index 0699d2a..f571e00 100755 --- a/doc/make_subarch_table_guidexml.py +++ b/doc/make_subarch_table_guidexml.py @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#!/usr/bin/python # Copyright (C) 2011 Sebastian Pipping# Copyright (C) 2013 Brian dolbec # Licensed under GPL v2 or later
[gentoo-commits] proj/catalyst:master commit in: doc/
commit: 6a38e62e143df9bda84da1ce8a54948c57e833fa Author: Mike Frysinger gentoo org> AuthorDate: Mon Oct 12 00:55:05 2015 + Commit: Mike Frysinger gentoo org> CommitDate: Wed Oct 28 16:49:56 2015 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6a38e62e make_target_table: rewrite for py3/stability doc/make_target_table.py | 32 ++-- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/doc/make_target_table.py b/doc/make_target_table.py index f127c37..9eb072b 100755 --- a/doc/make_target_table.py +++ b/doc/make_target_table.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/python # Copyright (C) 2012 W. Trevor King# Copyright (C) 2012 Sebastian Pipping # Copyright (C) 2013 Brian dolbec @@ -10,34 +10,38 @@ from __future__ import print_function -import sys as _sys - import glob -import re +import locale +import os +import sys -def key_netboot_before_netboot2((target_name, _module)): - return target_name + '1' +def main(_argv): + source_root = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + # Force consistent sorting order. + locale.setlocale(locale.LC_COLLATE, 'C') -if __name__ == '__main__': - extractor = re.compile('^catalyst/targets/(([^ ]+)).py$') targets = list() - for filename in sorted(glob.glob('catalyst/targets/*.py')): + for filename in glob.glob(os.path.join(source_root, 'catalyst/targets/*.py')): if '__init__' in filename: continue - match = extractor.match(filename) - target_name = match.group(2).replace('_', '-') - module_name = 'catalyst.targets.' + match.group(1) + name = os.path.basename(filename)[0:-3] + target_name = name.replace('_', '-') + module_name = 'catalyst.targets.' + name __import__(module_name) - module = _sys.modules[module_name] + module = sys.modules[module_name] targets.append((target_name, module)) - for target_name, module in sorted(targets, key=key_netboot_before_netboot2): + for target_name, module in sorted(targets, key=lambda x: x[0]): print('`%s`;;' % target_name) # Replace blank lines with `+` (asciidoc list item continuation) print(module.__doc__.strip().replace('\n\n', '\n+\n')) print('') + + +if __name__ == '__main__': + main(sys.argv[1:])
[gentoo-commits] proj/catalyst:master commit in: /, doc/, catalyst/
commit: b3cfe6e0c7754a9e17b681581b2a317b7861b486 Author: Mike Frysinger gentoo org> AuthorDate: Mon Oct 12 03:59:28 2015 + Commit: Mike Frysinger gentoo org> CommitDate: Wed Oct 28 16:49:56 2015 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b3cfe6e0 lint: fix redefined-outer-name warnings The doc module just needs a main func to hold all the variables instead of coding it all in global scope. .pylintrc | 3 +-- catalyst/log.py| 4 doc/make_subarch_table_guidexml.py | 9 +++-- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/.pylintrc b/.pylintrc index 2a03f23..e657daf 100644 --- a/.pylintrc +++ b/.pylintrc @@ -32,10 +32,9 @@ load-plugins= # bad-continuation -- might be hard with tab indentation policy # invalid-name -- need to manage constants better # line-too-long -- figure out a length and stick to it -# redefined-outer-name -- clean up code to not do this # super-init-not-called -- fix the classes __init__ structure # no-init -- update classes w/missing __init__ functions -disable=missing-docstring, too-many-lines, too-many-branches, too-many-statements, too-few-public-methods, too-many-instance-attributes, too-many-public-methods, too-many-locals, too-many-arguments, locally-enabled, locally-disabled, fixme, broad-except, bad-whitespace, bad-continuation, invalid-name, line-too-long, redefined-outer-name, super-init-not-called, no-init +disable=missing-docstring, too-many-lines, too-many-branches, too-many-statements, too-few-public-methods, too-many-instance-attributes, too-many-public-methods, too-many-locals, too-many-arguments, locally-enabled, locally-disabled, fixme, broad-except, bad-whitespace, bad-continuation, invalid-name, line-too-long, super-init-not-called, no-init [REPORTS] diff --git a/catalyst/log.py b/catalyst/log.py index 5938199..d640dec 100644 --- a/catalyst/log.py +++ b/catalyst/log.py @@ -98,6 +98,10 @@ class CatalystFormatter(logging.Formatter): return msg +# We define |debug| in global scope so people can call log.debug(), but it +# makes the linter complain when we have a |debug| keyword. Since we don't +# use that func in here, it's not a problem, so silence the warning. +# pylint: disable=redefined-outer-name def setup_logging(level, output=None, debug=False, color=None): """Initialize the logging module using the |level| level""" # The incoming level will be things like "info", but setLevel wants diff --git a/doc/make_subarch_table_guidexml.py b/doc/make_subarch_table_guidexml.py index a6a9022..0699d2a 100755 --- a/doc/make_subarch_table_guidexml.py +++ b/doc/make_subarch_table_guidexml.py @@ -6,6 +6,7 @@ import os import re +import sys import textwrap @@ -99,11 +100,11 @@ def dump(subarch_title_to_subarch_id, subarch_id_to_pattern_arch_genericrch_id): f.close() -if __name__ == '__main__': +def main(_argv): subarch_title_to_subarch_id = dict() subarch_id_to_pattern_arch_genericrch_id = dict() - for (dirpath, dirnames, filenames) in os.walk('catalyst/arch'): + for dirpath, _dirnames, filenames in os.walk('catalyst/arch'): for _fn in filenames: if not _fn.endswith('.py'): continue @@ -114,3 +115,7 @@ if __name__ == '__main__': handle_file(fn, subarch_title_to_subarch_id, subarch_id_to_pattern_arch_genericrch_id) dump(subarch_title_to_subarch_id, subarch_id_to_pattern_arch_genericrch_id) + + +if __name__ == '__main__': + main(sys.argv[1:])
[gentoo-commits] proj/catalyst:master commit in: doc/, etc/, catalyst/
commit: 6a4a77f2b22783b9b0fbad557ea5a8b63422bbda Author: Mike Frysinger gentoo org> AuthorDate: Tue Oct 6 20:18:31 2015 + Commit: Mike Frysinger gentoo org> CommitDate: Tue Oct 6 20:18:31 2015 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6a4a77f2 catalyst: add support for digests=auto Rather than require people to list all possible hashes, add support for an "auto" keyword. This serves two purposes: - any digest that is missing system support (e.g. missing prog), then that digest will be silently skipped - if it's the only value, then all available digests will be added catalyst/main.py | 47 +++ doc/catalyst-config.5.txt | 2 +- etc/catalyst.conf | 6 -- 3 files changed, 40 insertions(+), 15 deletions(-) diff --git a/catalyst/main.py b/catalyst/main.py index ebec4d3..65e34ef 100644 --- a/catalyst/main.py +++ b/catalyst/main.py @@ -259,23 +259,46 @@ def main(): # Start checking that digests are valid now that hash_map is initialized if "digests" in conf_values: - for i in conf_values["digests"].split(): - if i not in HASH_DEFINITIONS: - print - print i+" is not a valid digest entry" - print "Valid digest entries:" - print HASH_DEFINITIONS.keys() - print - print "Catalyst aborting" - sys.exit(2) - if find_binary(hash_map.hash_map[i].cmd) == None: + digests = set(conf_values['digests'].split()) + valid_digests = set(HASH_DEFINITIONS.keys()) + + # Use the magic keyword "auto" to use all algos that are available. + skip_missing = False + if 'auto' in digests: + skip_missing = True + digests.remove('auto') + if not digests: + digests = set(valid_digests) + + # First validate all the requested digests are valid keys. + if digests - valid_digests: + print + print "These are not a valid digest entries:" + print ', '.join(digests - valid_digests) + print "Valid digest entries:" + print ', '.join(sorted(valid_digests)) + print + print "Catalyst aborting" + sys.exit(2) + + # Then check for any programs that the hash func requires. + for digest in digests: + if find_binary(hash_map.hash_map[digest].cmd) == None: + # In auto mode, just ignore missing support. + if skip_missing: + digests.remove(digest) + continue print - print "digest=" + i - print "\tThe " + hash_map.hash_map[i].cmd + \ + print "digest=" + digest + print "\tThe " + hash_map.hash_map[digest].cmd + \ " binary was not found. It needs to be in your system path" print print "Catalyst aborting" sys.exit(2) + + # Now reload the config with our updated value. + conf_values['digests'] = ' '.join(digests) + if "hash_function" in conf_values: if conf_values["hash_function"] not in HASH_DEFINITIONS: print diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt index 31c4fe9..f778a80 100644 --- a/doc/catalyst-config.5.txt +++ b/doc/catalyst-config.5.txt @@ -24,7 +24,7 @@ and empty lines are interpreted as comments. For example: - # /etc/catalyst/catalyst.conf -digests="md5 sha1 sha512 whirlpool" +digests="auto" contents="auto" distdir="/usr/portage/distfiles" envscript="/etc/catalyst/catalystrc" diff --git a/etc/catalyst.conf b/etc/catalyst.conf index a2abb4a..939e941 100644 --- a/etc/catalyst.conf +++ b/etc/catalyst.conf @@ -4,13 +4,15 @@ # documentation for more information. # Creates a .DIGESTS file containing the hash output from any of the supported -# options below. Adding them all may take a long time. +# options below. Adding them all may take a long time on slower systems. The +# special "auto" keyword will skip digests that the system does not support, +# and if it's the only keyword given, will default to enabling all digests. # Supported hashes: # adler32, crc32, crc32b,
[gentoo-commits] proj/catalyst:master commit in: /, doc/, catalyst/
commit: b48733ee81b9d68ef1629f772071924142979e31 Author: Mike Frysinger gentoo org> AuthorDate: Tue Oct 6 15:02:47 2015 + Commit: Mike Frysinger gentoo org> CommitDate: Tue Oct 6 15:02:47 2015 + URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b48733ee lint: clean up superfluous parens These don't need the parens, so omit them. In the case of setup.py, we were expecting a print function, not a keyword, so make sure to import that module. catalyst/lock.py | 6 +++--- doc/make_subarch_table_guidexml.py | 6 +++--- setup.py | 1 + 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/catalyst/lock.py b/catalyst/lock.py index d079b2d..3d50c06 100644 --- a/catalyst/lock.py +++ b/catalyst/lock.py @@ -179,7 +179,7 @@ class LockDir(object): if not os.path.exists(self.lockfile): print "lockfile does not exist '%s'" % self.lockfile #print "fcntl_unlock() , self.myfd:", self.myfd, type(self.myfd) - if (self.myfd != None): + if self.myfd != None: #print "fcntl_unlock() trying to close it " try: os.close(self.myfd) @@ -236,7 +236,7 @@ class LockDir(object): #if type(lockfilename) == types.StringType: #os.close(myfd) #print "fcntl_unlock() trying a last ditch close", self.myfd - if (self.myfd != None): + if self.myfd != None: os.close(self.myfd) self.myfd=None self.locked=False @@ -256,7 +256,7 @@ class LockDir(object): start_time = time.time() reported_waiting = False - while(time.time() < (start_time + max_wait)): + while time.time() < (start_time + max_wait): # We only need it to exist. self.myfd = os.open(self.myhardlock, os.O_CREAT|os.O_RDWR,0660) os.close(self.myfd) diff --git a/doc/make_subarch_table_guidexml.py b/doc/make_subarch_table_guidexml.py index 54e0a4a..a6a9022 100755 --- a/doc/make_subarch_table_guidexml.py +++ b/doc/make_subarch_table_guidexml.py @@ -30,7 +30,7 @@ def handle_line(line, subarch_title_to_subarch_id, subarch_id_to_pattern_arch_ge # Apply alias grouping arch = _pattern_arch_genericliases.get(arch, arch) - assert(subarch not in subarch_id_to_pattern_arch_genericrch_id) + assert subarch not in subarch_id_to_pattern_arch_genericrch_id subarch_id_to_pattern_arch_genericrch_id[subarch] = arch return @@ -40,7 +40,7 @@ def handle_line(line, subarch_title_to_subarch_id, subarch_id_to_pattern_arch_ge child_subarch = x.group(1) parent_subarch = x.group(2) - assert(child_subarch not in subarch_id_to_pattern_arch_genericrch_id) + assert child_subarch not in subarch_id_to_pattern_arch_genericrch_id subarch_id_to_pattern_arch_genericrch_id[child_subarch] = subarch_id_to_pattern_arch_genericrch_id[parent_subarch] return @@ -49,7 +49,7 @@ def handle_line(line, subarch_title_to_subarch_id, subarch_id_to_pattern_arch_ge subarch_title = x.group(1) subarch_id = x.group(2) - assert(subarch_title not in subarch_title_to_subarch_id) + assert subarch_title not in subarch_title_to_subarch_id subarch_title_to_subarch_id[subarch_title] = subarch_id diff --git a/setup.py b/setup.py index e4569ee..a875db1 100755 --- a/setup.py +++ b/setup.py @@ -1,5 +1,6 @@ """Catalyst is a release building tool used by Gentoo Linux""" +from __future__ import print_function import codecs as _codecs from distutils.core import setup as _setup, Command as _Command
[gentoo-commits] proj/catalyst:master commit in: /, doc/
commit: 773ae7f9ee04cedba218a9abdc9f9b7e8360f055 Author: Brian Dolbec dolsen AT gentoo DOT org AuthorDate: Fri Jan 25 04:26:38 2013 + Commit: Brian Dolbec brian.dolbec AT gmail DOT com CommitDate: Wed Apr 2 20:04:15 2014 + URL: http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=773ae7f9 Commit a testpath file to source with instructions on it's use. Make testpath automagic via dwfeed's tweak Update HOWTO on how to run the git checkout code directly without being installed. --- doc/HOWTO.txt | 10 -- testpath | 18 ++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt index 8fe63c1..cec22c3 100644 --- a/doc/HOWTO.txt +++ b/doc/HOWTO.txt @@ -233,9 +233,15 @@ Running catalyst from a Git checkout If you're developing catalyst, you'll want to test your altered version. An easy way to run it without reinstalling is to setup a -local configuration file and run: +local configuration file. The checkout also includes a testpath file +which when sourced adds it's bin/ to PATH and it's checkout dir to +PYTHONPATH. That terminal will then use the checkout bin/catalyst and +the checkout's python modules. -# ./catalyst -c catalyst.conf -f path/to/your.spec +cd into the checkout directory and run: + +# source ./testpath +# catalyst -c catalyst.conf -f path/to/your.spec The local configuration file can use all the defaults except for `sharedir`, which you should change to point to your development diff --git a/testpath b/testpath new file mode 100644 index 000..37af9c9 --- /dev/null +++ b/testpath @@ -0,0 +1,18 @@ + +# edit the following 2 paths to point to your catalyst git checkout if nessecary +# then in a terminal window you wish to run the git code in... +# cd ~/catalyst +# source ./testpath +# edit a config catalyst.conf file. Make similar path changes for it to +# find and use/run the git checkout code. +# The catalyst code can then be run on a system without being installed or +# replacing an existing catalyst install which may be used for production. +# +# then run catalyst from there +# eg: +# +# catalyst -c catalyst.conf -p -f stage1.spec + +export PATH=$(dirname ${BASH_SOURCE[0]})/bin:${PATH} + +export PYTHONPATH=$(dirname ${BASH_SOURCE[0]})${PYTHONPATH:+:}${PYTHONPATH}