[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/base/, doc/, catalyst/, targets/support/
commit: d89f8aae2ed865e2c568473ab59dd72bb62c7b68
Author: Matt Turner gentoo org>
AuthorDate: Thu May 14 02:37:11 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Thu May 21 21:40:43 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d89f8aae
catalyst: Set jobs/load-average via catalyst.conf
We currently have two mechanisms of setting MAKEOPTS: in spec files and
in catalystrc.
Setting makeopts in spec files doesn't make sense. The spec should
describe the thing that's being built and not contain options that are
specific to the build system.
Setting makeopts via catalystrc is better, but it only applies to the
actual build system invocations, leaving emerge to run jobs serially or
again requiring configuration specific to the build machine to be put
into the spec file. For example:
update_seed_command: ... --jobs 5 --load-average 5
With jobs and load-average specified in catalyst.conf, catalyst has the
information required to configure both emerge and the build systems
emerge executes.
This removes the undocumented makeopts spec file option and replaces it
with jobs and load-average settings in catalyst.conf.
Signed-off-by: Matt Turner gentoo.org>
catalyst/base/stagebase.py | 12 +---
catalyst/defaults.py| 2 ++
doc/catalyst-config.5.txt | 15 ---
etc/catalyst.conf | 8
etc/catalystrc | 3 ---
targets/support/chroot-functions.sh | 8
6 files changed, 35 insertions(+), 13 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5a8cd1df..bc721ad4 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -56,7 +56,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"hostuse",
"kerncache_path",
"ldflags",
-"makeopts",
"pkgcache_path",
"portage_confdir",
"portage_overlay",
@@ -1290,12 +1289,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
continue
log.warning("Not making envar for '%s', is a dict", x)
-if "makeopts" in self.settings:
-if isinstance(self.settings["makeopts"], str):
-self.env["MAKEOPTS"] = self.settings["makeopts"]
-else:
-# ensure makeopts is a string
-self.env["MAKEOPTS"] = ' '.join(self.settings["makeopts"])
+makeopts = []
+for flag, setting in {'j': 'jobs', 'l': 'load-average'}.items():
+if setting in self.settings:
+makeopts.append(f'-{flag}{self.settings[setting]}')
+self.env['MAKEOPTS'] = ' '.join(makeopts)
log.debug('setup_environment(); env = %r', self.env)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 14f671fe..b31d5b50 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -12,6 +12,8 @@ valid_config_file_values = frozenset([
"digests",
"distdir",
"envscript",
+"jobs",
+"load-average",
"options",
"port_logdir",
"repo_basedir",
diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 7ac9a2a3..cbef6092 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -48,9 +48,9 @@ $ python3 -c 'import hashlib;
print(hashlib.algorithms_available)'
*envscript*::
Environment script location, which allows users to set options such as
-HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
-variables needed for building. The envscript file sets environment
-variables using POSIX shell notation:
+HTTP proxies, `GENTOO_MIRRORS`, or any other environment variables
+needed for building. The envscript file sets environment variables
+using POSIX shell notation:
+
-
export FOO="bar"
@@ -136,6 +136,15 @@ written to the target's make.conf if it is not the default
value of
Other settings
~~
+*jobs*::
+Integral value passed to *emerge(1)* as the parameter to --jobs and is
+used to define *MAKEOPTS* during the target build.
+
+*load-average*::
+Floating-point value passed to *emerge(1)* as the parameter to
+--load-average and is used to define *MAKEOPTS* during the target
+build.
+
*sharedir*::
Catalyst runtime script location. `/usr/share/catalyst` should work for
most default installations. If you are running catalyst from a Git
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index 2272cb86..81693c25 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -76,3 +76,11 @@ options = [
# WARNING: If you use too much RAM everything will fail horribly and it is not
our fault.
# set size of /var/tmp/portage tmpfs in gigabytes
# var_tmpfs_portage = 16
+
+# Integral value passed to emerge as the parameter to --jobs and is used to
+# define MAKEOPTS during the target build.
+# jobs = 4
+
+# Floating-point value passed to
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/base/, catalyst/, doc/, targets/support/
commit: cb431104aaf4de58d88f383aedb7e65f3d6fc3e7
Author: Matt Turner gentoo org>
AuthorDate: Thu May 14 02:37:11 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Wed May 20 01:52:18 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=cb431104
catalyst: Set jobs/load-average via catalyst.conf
We currently have two mechanisms of setting MAKEOPTS: in spec files and
in catalystrc.
Setting makeopts in spec files doesn't make sense. The spec should
describe the thing that's being built and not contain options that are
specific to the build system.
Setting makeopts via catalystrc is better, but it only applies to the
actual build system invocations, leaving emerge to run jobs serially or
again requiring configuration specific to the build machine to be put
into the spec file. For example:
update_seed_command: ... --jobs 5 --load-average 5
With jobs and load-average specified in catalyst.conf, catalyst has the
information required to configure both emerge and the build systems
emerge executes.
This removes the undocumented makeopts spec file option and replaces it
with jobs and load-average settings in catalyst.conf.
Signed-off-by: Matt Turner gentoo.org>
catalyst/base/stagebase.py | 12 +---
catalyst/defaults.py| 2 ++
doc/catalyst-config.5.txt | 15 ---
etc/catalystrc | 3 ---
targets/support/chroot-functions.sh | 8
5 files changed, 27 insertions(+), 13 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 9410f151..a53a58dd 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -56,7 +56,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"hostuse",
"kerncache_path",
"ldflags",
-"makeopts",
"pkgcache_path",
"portage_confdir",
"portage_overlay",
@@ -1308,12 +1307,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:
self.env[varname] = "false"
-if "makeopts" in self.settings:
-if isinstance(self.settings["makeopts"], str):
-self.env["MAKEOPTS"] = self.settings["makeopts"]
-else:
-# ensure makeopts is a string
-self.env["MAKEOPTS"] = ' '.join(self.settings["makeopts"])
+makeopts = []
+for flag, setting in {'j': 'jobs', 'l': 'load-average'}.items():
+if setting in self.settings:
+makeopts.append(f'-{flag}{self.settings[setting]}')
+self.env['MAKEOPTS'] = ' '.join(makeopts)
log.debug('setup_environment(); env = %r', self.env)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 14f671fe..b31d5b50 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -12,6 +12,8 @@ valid_config_file_values = frozenset([
"digests",
"distdir",
"envscript",
+"jobs",
+"load-average",
"options",
"port_logdir",
"repo_basedir",
diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 7ac9a2a3..cbef6092 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -48,9 +48,9 @@ $ python3 -c 'import hashlib;
print(hashlib.algorithms_available)'
*envscript*::
Environment script location, which allows users to set options such as
-HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
-variables needed for building. The envscript file sets environment
-variables using POSIX shell notation:
+HTTP proxies, `GENTOO_MIRRORS`, or any other environment variables
+needed for building. The envscript file sets environment variables
+using POSIX shell notation:
+
-
export FOO="bar"
@@ -136,6 +136,15 @@ written to the target's make.conf if it is not the default
value of
Other settings
~~
+*jobs*::
+Integral value passed to *emerge(1)* as the parameter to --jobs and is
+used to define *MAKEOPTS* during the target build.
+
+*load-average*::
+Floating-point value passed to *emerge(1)* as the parameter to
+--load-average and is used to define *MAKEOPTS* during the target
+build.
+
*sharedir*::
Catalyst runtime script location. `/usr/share/catalyst` should work for
most default installations. If you are running catalyst from a Git
diff --git a/etc/catalystrc b/etc/catalystrc
index bcd729af..e7904128 100755
--- a/etc/catalystrc
+++ b/etc/catalystrc
@@ -1,5 +1,2 @@
#!/bin/bash
# This is an example catalystrc. As such, it doesn't actually *do* anything.
-
-# Uncomment the following to increase the number of threads used to compile.
-# export MAKEOPTS="-j16"
diff --git a/targets/support/chroot-functions.sh
b/targets/support/chroot-functions.sh
index b6e221af..02fd8fef 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -130,6 +130,14 @@
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/base/, catalyst/
commit: adfb01ca3a36e74f500e384177404563023aa12b
Author: Matt Turner gentoo org>
AuthorDate: Sat May 16 21:53:11 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Wed May 20 01:49:37 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=adfb01ca
catalyst: Remove support for source_matching="loose"
This does not seem like a useful feature to me.
Signed-off-by: Matt Turner gentoo.org>
catalyst/base/stagebase.py | 3 ---
catalyst/defaults.py | 1 -
catalyst/support.py| 6 +++---
etc/catalyst.conf | 12
4 files changed, 3 insertions(+), 19 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index f39895fe..febaf969 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -130,8 +130,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
decomp_opt=self.settings["decomp_opt"])
self.accepted_extensions = self.decompressor.search_order_extensions(
self.settings["decompressor_search_order"])
-log.notice("Source file specification matching setting is: %s",
- self.settings["source_matching"])
log.notice("Accepted source file extensions search order: %s",
self.accepted_extensions)
# save resources, it is not always needed
@@ -409,7 +407,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
normpath(self.settings["storedir"] + "/builds/" +
self.settings["source_subpath"]),
self.accepted_extensions,
-self.settings["source_matching"] in ["strict"]
)
log.debug('Source path returned from file_check is: %s',
self.settings["source_path"])
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 412cb956..14f671fe 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -45,7 +45,6 @@ confdefaults = {
"repos": "%(storedir)s/repos",
"sharedir": "/usr/share/catalyst",
"shdir": "%(sharedir)s/targets",
-"source_matching": "strict",
"storedir": "/var/tmp/catalyst",
"target_distdir": "/var/cache/distfiles",
"target_pkgdir": "/var/cache/binpkgs",
diff --git a/catalyst/support.py b/catalyst/support.py
index c4a5c797..a6a6854a 100644
--- a/catalyst/support.py
+++ b/catalyst/support.py
@@ -59,7 +59,7 @@ def cmd(mycmd, env=None, debug=False, fail_func=None):
print_traceback=False)
-def file_check(filepath, extensions=None, strict=True):
+def file_check(filepath, extensions=None):
'''Check for the files existence and that only one exists
if others are found with various extensions
'''
@@ -73,8 +73,8 @@ def file_check(filepath, extensions=None, strict=True):
".CONTENTS") and not x.endswith(".CONTENTS.gz") and not
x.endswith(".DIGESTS")]
if len(files) == 1:
return files[0]
-if len(files) > 1 and strict:
-msg = "Ambiguos Filename: %s\nPlease specify the correct extension as
well" % filepath
+if len(files) > 1:
+msg = "Ambiguous Filename: %s\nPlease specify the correct extension as
well" % filepath
raise CatalystError(msg, print_traceback=False)
target_file = None
for ext in extensions:
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index f64fe971..d33be15f 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -50,18 +50,6 @@ envscript="/etc/catalyst/catalystrc"
# (These options can be used together)
options="autoresume bindist kerncache pkgcache seedcache"
-# source_matching specifies how catalyst will match non-specific file names
-# if the filename is not found as an exact match.
-# ie: a filename without the extension specified. "/path/to/foo"
-#
-# possible values are:
-# "strict" meaning if more than one file of that name is present with any
-#file extension, then it will raise an exception.
-# "loose" meaning it will search for an existing filename with an added
-#extension from an ordered list of extensions determined from the
-#decompressor_search_order specification in the spec file or
(default)
-source_matching="strict"
-
# port_logdir is where all build logs will be kept. This dir will be
automatically cleaned
# of all logs over 30 days old. If left undefined the logs will remain in the
build directory
# as usual and get cleaned every time a stage build is restarted.
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/
commit: 69ed1d18c01ee1a6ed0d5261fd6b9551449dae56
Author: Matt Turner gentoo org>
AuthorDate: Sat May 16 03:18:21 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Tue May 19 23:30:23 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=69ed1d18
catalyst: Convert catalyst.conf to TOML
Signed-off-by: Matt Turner gentoo.org>
catalyst/main.py | 49 +--
etc/catalyst.conf | 88 ---
2 files changed, 66 insertions(+), 71 deletions(-)
diff --git a/catalyst/main.py b/catalyst/main.py
index be06ccd7..a20fb0d7 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -5,6 +5,8 @@ import os
import sys
import textwrap
+import toml
+
from snakeoil.process import namespaces
from DeComp.definitions import (COMPRESS_DEFINITIONS, DECOMPRESS_DEFINITIONS,
@@ -17,8 +19,7 @@ from catalyst.defaults import confdefaults, option_messages,
DEFAULT_CONFIG_FILE
from catalyst.support import CatalystError
from catalyst.version import get_version
-
-conf_values = {}
+conf_values = confdefaults
def version():
@@ -30,42 +31,20 @@ def version():
def parse_config(config_files):
-# search a couple of different areas for the main config file
-myconf = {}
-
-# try and parse the config file "config_file"
for config_file in config_files:
log.notice('Loading configuration file: %s', config_file)
try:
-config = catalyst.config.ConfigParser(config_file)
-myconf.update(config.get_values())
+conf_values.update(toml.load(config_file))
except Exception as e:
log.critical('Could not find parse configuration file: %s: %s',
config_file, e)
-# now, load up the values into conf_values so that we can use them
-for x in list(confdefaults):
-if x in myconf:
-if x == 'options':
-conf_values[x] = set(myconf[x].split())
-elif x in ["decompressor_search_order"]:
-conf_values[x] = myconf[x].split()
-else:
-conf_values[x] = myconf[x]
-else:
-conf_values[x] = confdefaults[x]
-
# print out any options messages
for opt in conf_values['options']:
if opt in option_messages:
log.info(option_messages[opt])
-for key in ["digests", "envscript", "var_tmpfs_portage", "port_logdir",
-"local_overlay", "repos"]:
-if key in myconf:
-conf_values[key] = myconf[key]
-
-if "envscript" in myconf:
+if "envscript" in conf_values:
log.info('Envscript support enabled.')
# take care of any variable substitutions that may be left
@@ -297,17 +276,17 @@ def _main(parser, opts):
conf_values['DEBUG'] = opts.debug
conf_values['VERBOSE'] = opts.debug or opts.verbose
-options = set()
+options = []
if opts.fetchonly:
-options.add('fetch')
+options.append('fetch')
if opts.purge:
-options.add('purge')
+options.append('purge')
if opts.purgeonly:
-options.add('purgeonly')
+options.append('purgeonly')
if opts.purgetmponly:
-options.add('purgetmponly')
+options.append('purgetmponly')
if opts.clear_autoresume:
-options.add('clear-autoresume')
+options.append('clear-autoresume')
# Make sure we have some work before moving further.
if not myspecfile and not mycmdline:
@@ -316,9 +295,9 @@ def _main(parser, opts):
# made it this far so start by outputting our version info
version()
# import configuration file and import our main module using those settings
-parse_config(myconfigs)
+conf_values = parse_config(myconfigs)
-conf_values["options"].update(options)
+conf_values["options"].extend(options)
log.notice('conf_values[options] = %s', conf_values['options'])
# initialize our contents generator
@@ -335,7 +314,7 @@ def _main(parser, opts):
if "digests" in conf_values:
valid_digests = hashlib.algorithms_available
-digests = set(conf_values['digests'].split())
+digests = set(conf_values['digests'])
conf_values['digests'] = digests
# First validate all the requested digests are valid keys.
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index d33be15f..2272cb86 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -10,53 +10,69 @@
#
# $ python3 -c 'import hashlib; print(hashlib.algorithms_available)'
#
-digests="blake2b sha512"
+digests = ["blake2b", "sha512"]
# envscript allows users to set options such as http proxies, MAKEOPTS,
# GENTOO_MIRRORS, or any other environment variables needed for building.
# The envscript file sets environment variables like so:
# export FOO="bar"
-envscript="/etc/catalyst/catalystrc"
-
-# options set different build-time options for catalyst. Some
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/, catalyst/base/, doc/
commit: 4109de01e511e420d612281b0555b458ca14c098
Author: Matt Turner gentoo org>
AuthorDate: Thu May 14 02:37:11 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Tue May 19 23:04:19 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=4109de01
catalyst: Set jobs/load-average via catalyst.conf
We currently have two mechanisms of setting MAKEOPTS: in spec files and
in catalystrc.
Setting makeopts in spec files doesn't make sense. The spec should
describe the thing that's being built and not contain options that are
specific to the build system.
Setting makeopts via catalystrc is better, but it only applies to the
actual build system invocations, leaving emerge to run jobs serially or
again requiring configuration specific to the build machine to be put
into the spec file. For example:
update_seed_command: ... --jobs 5 --load-average 5
With jobs and load-average specified in catalyst.conf, catalyst has the
information required to configure both emerge and the build systems
emerge executes.
This removes the undocumented makeopts spec file option and replaces it
with jobs and load-average settings in catalyst.conf.
Signed-off-by: Matt Turner gentoo.org>
catalyst/base/stagebase.py | 12 +---
catalyst/defaults.py | 2 ++
doc/catalyst-config.5.txt | 24 +++-
etc/catalystrc | 3 ---
4 files changed, 22 insertions(+), 19 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 537ab752..2e6b349a 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -56,7 +56,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
"hostuse",
"kerncache_path",
"ldflags",
-"makeopts",
"pkgcache_path",
"portage_confdir",
"portage_overlay",
@@ -1308,12 +1307,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:
self.env[varname] = "false"
-if "makeopts" in self.settings:
-if isinstance(self.settings["makeopts"], str):
-self.env["MAKEOPTS"] = self.settings["makeopts"]
-else:
-# ensure makeopts is a string
-self.env["MAKEOPTS"] = ' '.join(self.settings["makeopts"])
+makeopts = []
+for flag, setting in {'j': 'jobs', 'l': 'load-average'}.items():
+if setting in self.settings:
+makeopts.append(f'-{flag}{self.settings[setting]}')
+self.env['MAKEOPTS'] = ' '.join(makeopts)
log.debug('setup_environment(); env = %r', self.env)
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 14f671fe..b31d5b50 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -12,6 +12,8 @@ valid_config_file_values = frozenset([
"digests",
"distdir",
"envscript",
+"jobs",
+"load-average",
"options",
"port_logdir",
"repo_basedir",
diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 7ac9a2a3..4ea4690f 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -48,9 +48,9 @@ $ python3 -c 'import hashlib;
print(hashlib.algorithms_available)'
*envscript*::
Environment script location, which allows users to set options such as
-HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
-variables needed for building. The envscript file sets environment
-variables using POSIX shell notation:
+HTTP proxies, `GENTOO_MIRRORS`, or any other environment variables
+needed for building. The envscript file sets environment variables
+using POSIX shell notation:
+
-
export FOO="bar"
@@ -136,6 +136,18 @@ written to the target's make.conf if it is not the default
value of
Other settings
~~
+*jobs*::
+Number of jobs to execute simultaneously.
+
+*load-average*::
+Load-average
+
+*port_logdir*::
+Location for build logs (example: `/var/tmp/catalyst/tmp`). This dir
+will be automatically cleaned of all logs over 30 days old. If left
+undefined the logs will remain in the build directory as usual and get
+cleaned every time a stage build is restarted.
+
*sharedir*::
Catalyst runtime script location. `/usr/share/catalyst` should work for
most default installations. If you are running catalyst from a Git
@@ -145,12 +157,6 @@ checkout, you should change this to point to your checkout
directory.
Location for built seeds, temporary files, and caches (example:
`/var/tmp/catalyst`).
-*port_logdir*::
-Location for build logs (example: `/var/tmp/catalyst/tmp`). This dir
-will be automatically cleaned of all logs over 30 days old. If left
-undefined the logs will remain in the build directory as usual and get
-cleaned every time a stage build is restarted.
-
*var_tmpfs_portage*::
Set the size of a `/var/tmp/portage` tmpfs in gigabytes (example:
`16`). If set, this
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/base/, catalyst/
commit: b387321f6eaf3075914f2d4953f757bc04f159b6
Author: Matt Turner gentoo org>
AuthorDate: Sat May 16 21:53:11 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Tue May 19 22:54:21 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b387321f
catalyst: Remove support for source_matching="loose"
This does not seem like a useful feature to me.
Signed-off-by: Matt Turner gentoo.org>
catalyst/base/stagebase.py | 3 ---
catalyst/defaults.py | 1 -
catalyst/support.py| 6 +++---
etc/catalyst.conf | 12
4 files changed, 3 insertions(+), 19 deletions(-)
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 651bf4e4..537ab752 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -130,8 +130,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
decomp_opt=self.settings["decomp_opt"])
self.accepted_extensions = self.decompressor.search_order_extensions(
self.settings["decompressor_search_order"])
-log.notice("Source file specification matching setting is: %s",
- self.settings["source_matching"])
log.notice("Accepted source file extensions search order: %s",
self.accepted_extensions)
# save resources, it is not always needed
@@ -409,7 +407,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
normpath(self.settings["storedir"] + "/builds/" +
self.settings["source_subpath"]),
self.accepted_extensions,
-self.settings["source_matching"] in ["strict"]
)
log.debug('Source path returned from file_check is: %s',
self.settings["source_path"])
diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 412cb956..14f671fe 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -45,7 +45,6 @@ confdefaults = {
"repos": "%(storedir)s/repos",
"sharedir": "/usr/share/catalyst",
"shdir": "%(sharedir)s/targets",
-"source_matching": "strict",
"storedir": "/var/tmp/catalyst",
"target_distdir": "/var/cache/distfiles",
"target_pkgdir": "/var/cache/binpkgs",
diff --git a/catalyst/support.py b/catalyst/support.py
index c4a5c797..a6a6854a 100644
--- a/catalyst/support.py
+++ b/catalyst/support.py
@@ -59,7 +59,7 @@ def cmd(mycmd, env=None, debug=False, fail_func=None):
print_traceback=False)
-def file_check(filepath, extensions=None, strict=True):
+def file_check(filepath, extensions=None):
'''Check for the files existence and that only one exists
if others are found with various extensions
'''
@@ -73,8 +73,8 @@ def file_check(filepath, extensions=None, strict=True):
".CONTENTS") and not x.endswith(".CONTENTS.gz") and not
x.endswith(".DIGESTS")]
if len(files) == 1:
return files[0]
-if len(files) > 1 and strict:
-msg = "Ambiguos Filename: %s\nPlease specify the correct extension as
well" % filepath
+if len(files) > 1:
+msg = "Ambiguous Filename: %s\nPlease specify the correct extension as
well" % filepath
raise CatalystError(msg, print_traceback=False)
target_file = None
for ext in extensions:
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index f64fe971..d33be15f 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -50,18 +50,6 @@ envscript="/etc/catalyst/catalystrc"
# (These options can be used together)
options="autoresume bindist kerncache pkgcache seedcache"
-# source_matching specifies how catalyst will match non-specific file names
-# if the filename is not found as an exact match.
-# ie: a filename without the extension specified. "/path/to/foo"
-#
-# possible values are:
-# "strict" meaning if more than one file of that name is present with any
-#file extension, then it will raise an exception.
-# "loose" meaning it will search for an existing filename with an added
-#extension from an ordered list of extensions determined from the
-#decompressor_search_order specification in the spec file or
(default)
-source_matching="strict"
-
# port_logdir is where all build logs will be kept. This dir will be
automatically cleaned
# of all logs over 30 days old. If left undefined the logs will remain in the
build directory
# as usual and get cleaned every time a stage build is restarted.
[gentoo-commits] proj/catalyst:wip/mattst88 commit in: etc/, catalyst/
commit: 1b2dd33da134cb8f6fbaa1b000e15b0b2836ef93
Author: Matt Turner gentoo org>
AuthorDate: Sat May 16 03:18:21 2020 +
Commit: Matt Turner gentoo org>
CommitDate: Tue May 19 22:54:21 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=1b2dd33d
catalyst: Convert catalyst.conf to TOML
Signed-off-by: Matt Turner gentoo.org>
catalyst/main.py | 27 +
etc/catalyst.conf | 88 ---
2 files changed, 59 insertions(+), 56 deletions(-)
diff --git a/catalyst/main.py b/catalyst/main.py
index be06ccd7..a2444eeb 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -5,6 +5,8 @@ import os
import sys
import textwrap
+import toml
+
from snakeoil.process import namespaces
from DeComp.definitions import (COMPRESS_DEFINITIONS, DECOMPRESS_DEFINITIONS,
@@ -30,31 +32,16 @@ def version():
def parse_config(config_files):
-# search a couple of different areas for the main config file
-myconf = {}
+config = confdefaults
-# try and parse the config file "config_file"
for config_file in config_files:
log.notice('Loading configuration file: %s', config_file)
try:
-config = catalyst.config.ConfigParser(config_file)
-myconf.update(config.get_values())
+config.update(toml.load(config_file))
except Exception as e:
log.critical('Could not find parse configuration file: %s: %s',
config_file, e)
-# now, load up the values into conf_values so that we can use them
-for x in list(confdefaults):
-if x in myconf:
-if x == 'options':
-conf_values[x] = set(myconf[x].split())
-elif x in ["decompressor_search_order"]:
-conf_values[x] = myconf[x].split()
-else:
-conf_values[x] = myconf[x]
-else:
-conf_values[x] = confdefaults[x]
-
# print out any options messages
for opt in conf_values['options']:
if opt in option_messages:
@@ -62,10 +49,10 @@ def parse_config(config_files):
for key in ["digests", "envscript", "var_tmpfs_portage", "port_logdir",
"local_overlay", "repos"]:
-if key in myconf:
-conf_values[key] = myconf[key]
+if key in config:
+conf_values[key] = config[key]
-if "envscript" in myconf:
+if "envscript" in config:
log.info('Envscript support enabled.')
# take care of any variable substitutions that may be left
diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index d33be15f..2272cb86 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -10,53 +10,69 @@
#
# $ python3 -c 'import hashlib; print(hashlib.algorithms_available)'
#
-digests="blake2b sha512"
+digests = ["blake2b", "sha512"]
# envscript allows users to set options such as http proxies, MAKEOPTS,
# GENTOO_MIRRORS, or any other environment variables needed for building.
# The envscript file sets environment variables like so:
# export FOO="bar"
-envscript="/etc/catalyst/catalystrc"
-
-# options set different build-time options for catalyst. Some examples are:
-# autoresume = Attempt to resume a failed build, clear the autoresume flags
with
-# the -a option to the catalyst cmdline. -p will clear the autoresume
flags
-# as well as your pkgcache and kerncache
-# ( This option is not fully tested, bug reports welcome )
-# bindist = enables the bindist USE flag, please see package specific
definition,
-# however, it is suggested to enable this if redistributing builds.
-# This optional USE flag is normally cleaned from the make.conf file on
-# completion of the stage. For a non-cleaned version,
-# use sticky-config also (see below)
-# ccache = enables build time ccache support
-# distcc = enable distcc support for building. You have to set distcc_hosts in
-# your spec file.
-# icecream = enables icecream compiler cluster support for building
-# keepwork = Prevents the removal of the working chroot path and any autoresume
-# files or points.
-# kerncache = keeps a tbz2 of your built kernel and modules (useful if your
-# build stops in livecd-stage2)
-# pkgcache = keeps a tbz2 of every built package (useful if your build stops
-# prematurely)
-# seedcache = use the build output of a previous target if it exists to speed
up
-# the copy
-# sticky-config = enables the code that will keep any internal 'catalyst_use'
flags
-# added to the USE= for building the stage. These are usually added for
legal
-# or specific needs in building the the early stage. Mostly it is the
-# 'bindist' USE flag option that is used for legal reasons, please see its
-# specific definition. It will also keep any /etc/portage/package.*
-# files or directories.
-#
-# (These options can be used together)
-options="autoresume
