[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: 1314c345cacfce8a5cd6036d66da0b5ab65c6adb
Author: Eli Schwartz  gmail  com>
AuthorDate: Thu Feb 22 20:40:26 2024 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Sun Mar  3 18:50:45 2024 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=1314c345

avoid chrooting and running the $PATH to bash from outside the chroot

The API filename for bash is /bin/bash and that is where it is
guaranteed to be. On merged-usr systems, `shutil.which('bash')` will
search PATH and maybe find /usr/bin/bash first, since it is resolved as
a symlink, but that path will then fail in the chroot if the chroot is
not merged-usr.

Even on merged-usr systems, app-shells/bash always, always, always
installs to /bin/bash and lets portage handle the symlink.

Signed-off-by: Eli Schwartz  gmail.com>
Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index dd00af45..4bcab30c 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1458,11 +1458,12 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def enter_chroot(self):
 chroot = command('chroot')
-bash = command('bash')
+# verify existence only
+command(os.path.join(self.settings['chroot_path'], '/bin/bash'))
 
 log.notice("Entering chroot")
 try:
-cmd([chroot, self.settings['chroot_path'], bash, '-l'],
+cmd([chroot, self.settings['chroot_path'], '/bin/bash', '-l'],
 env=self.env)
 except CatalystError:
 pass

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: e9d2b62ea8300aae67a483c7e260d3ff7bc80fc5
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Sun Dec 24 21:17:59 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Sun Dec 24 21:41:33 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=e9d2b62e

base/stagebase.py: Do not touch subarch if chost is set

Currently, if you set chost in a stage1 spec file, catalyst resets the
subarch to the first part of the triple (i.e. aarch64 for arm64).

With this change, it leaves the subarch alone at the value defined
in the spec file. This makes it much easier to override CHOST for,
say, musl, without having to provide a separate section in arch/* ...

Impact of the commit on the existing releng stage1 specs:
amd64, arm64, x86, ppc, ppc64, ppc64le are all fine (the definitions
are idencal, or subarch = first part of triplet)
power9le is funny because right now the chost (powerpc64le) resets the
subarch and drops the power9le optimizations, so there the commit is
actually an improvement!

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 3cbdf489..dd00af45 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -105,10 +105,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 self.makeconf = {}
 
-if "chost" in self.settings:
-host = self.settings["chost"].split("-")[0]
-else:
-host = self.settings["subarch"]
+host = self.settings["subarch"]
 self.settings["hostarch"] = host
 
 if "cbuild" in self.settings:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: 1abd94188e9e9eecd5fe16de3aeaceebac1ec4e0
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Tue Sep  5 01:43:19 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Tue Sep  5 01:43:47 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=1abd9418

Turn binrepos.conf into a folder with gentoobinhost.conf inside

Closes: https://bugs.gentoo.org/913660
Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index fe307eab..3cbdf489 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1200,9 +1200,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # Write out binrepos.conf (for the chroot)
 binrpath = normpath(self.settings["chroot_path"] +
 self.settings["binrepos_conf"])
+Path(binrpath).mkdir(mode=0o755, parents=True, exist_ok=True)
 
-with open(binrpath, "w") as myb:
-log.notice("Writing the stage binrepos.conf to: %s" % binrpath)
+binrfile = binrpath + "/gentoobinhost.conf"
+with open(binrfile, "w") as myb:
+log.notice("Writing the stage binrepo config to: %s" % 
binrfile)
 myb.write("# These settings were set by the catalyst build 
script "
 "that automatically\n# built this stage.\n")
 myb.write("# Please consider using a local mirror.\n\n")
@@ -1211,8 +1213,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 myb.write("sync-uri = " + self.settings["binhost"] + \
 self.settings["binrepo_path"] + "\n")
 
-return
-
 def fsscript(self):
 if "autoresume" in self.settings["options"] \
 and self.resume.is_enabled("fsscript"):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, targets/support/, targets/livecd-stage2/

[Andreas K. Hüttel]

commit: 3d4fb24383716958dadea95d100911037bb2b557
Author: Azamat H. Hackimov  gmail  com>
AuthorDate: Tue Aug  1 08:18:31 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Tue Aug 22 20:36:37 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=3d4fb243

Implementing distkernel installation

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py  | 22 +++---
 targets/livecd-stage2/controller.sh |  5 +++
 targets/support/bootloader-setup.sh | 19 -
 targets/support/kmerge.sh   | 85 +++--
 targets/support/pre-distkmerge.sh   |  7 +++
 5 files changed, 116 insertions(+), 22 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index a9147241..fe307eab 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -694,6 +694,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "boot/kernel/" + x + "/aliases",
 "boot/kernel/" + x + "/config",
 "boot/kernel/" + x + "/console",
+"boot/kernel/" + x + "/distkernel",
+"boot/kernel/" + x + "/dracut_args",
 "boot/kernel/" + x + "/extraversion",
 "boot/kernel/" + x + "/gk_action",
 "boot/kernel/" + x + "/gk_kernargs",
@@ -716,6 +718,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.settings["gk_mainargs"] = self.settings[gk_mainargs]
 del self.settings[gk_mainargs]
 
+dracut_mainargs = prefix + "/dracut_args"
+if dracut_mainargs in self.settings:
+self.settings["dracut_args"] = self.settings[dracut_mainargs]
+del self.settings[dracut_mainargs]
+
 # Ask genkernel to include b2sum if /verify is set
 verify = prefix + "/verify"
 if verify in self.settings:
@@ -1584,7 +1591,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.resume.enable("build_packages")
 
 def build_kernel(self):
-'''Build all configured kernels'''
+"""Build all configured kernels"""
 if "autoresume" in self.settings["options"] \
 and self.resume.is_enabled("build_kernel"):
 log.notice(
@@ -1595,19 +1602,23 @@ class StageBase(TargetBase, ClearBase, GenBase):
 mynames = self.settings["boot/kernel"]
 if isinstance(mynames, str):
 mynames = [mynames]
-# Execute the script that sets up the kernel build environment
-cmd([self.settings['controller_file'], 'pre-kmerge'], env=self.env)
 for kname in [sanitize_name(name) for name in mynames]:
+if "boot/kernel/" + kname + "/distkernel" in self.settings:
+cmd([self.settings['controller_file'], 'pre-distkmerge'], 
env=self.env)
+else:
+# Execute the script that sets up the kernel build 
environment
+cmd([self.settings['controller_file'], 'pre-kmerge'], 
env=self.env)
 self._build_kernel(kname=kname)
 self.resume.enable("build_kernel")
 
 def _build_kernel(self, kname):
-"Build a single configured kernel by name"
+"""Build a single configured kernel by name"""
 if "autoresume" in self.settings["options"] \
 and self.resume.is_enabled("build_kernel_" + kname):
 log.notice('Resume point detected, skipping build_kernel '
'for %s operation...', kname)
 return
+
 self._copy_kernel_config(kname=kname)
 
 key = 'boot/kernel/' + kname + '/extraversion'
@@ -1617,8 +1628,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self._copy_initramfs_overlay(kname=kname)
 
 # Execute the script that builds the kernel
-cmd([self.settings['controller_file'], 'kernel', kname],
-env=self.env)
+cmd([self.settings['controller_file'], 'kernel', kname], env=self.env)
 
 if "boot/kernel/" + kname + "/initramfs_overlay" in self.settings:
 log.notice('Cleaning up temporary overlay dir')

diff --git a/targets/livecd-stage2/controller.sh 
b/targets/livecd-stage2/controller.sh
index f6522e63..57d947a1 100755
--- a/targets/livecd-stage2/controller.sh
+++ b/targets/livecd-stage2/controller.sh
@@ -20,6 +20,11 @@ case $1 in
extract_modules ${clst_chroot_path} ${kname}
;;
 
+   pre-distkmerge)
+   # Install dracut
+   exec_in_chroot ${clst_shdir}/support/pre-distkmerge.sh
+   ;;
+
preclean)
# Move over the motd (if applicable)
case ${clst_livecd_type} in

diff --git a/targets/support/bootloader-setup.sh 
b/targets/support/bootloader-setup.sh
index d6d5f96e..455a4f07 100755
--- a/targets/support/bootloader-setup.sh
+++ 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, etc/, catalyst/

[Andreas K. Hüttel]

commit: 750947eed34fe51a9502b16585a59f34bd47b1a1
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Wed Aug  9 21:22:55 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Fri Aug 11 19:55:12 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=750947ee

Add configuration options and code to write out binrepos.conf

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 32 
 catalyst/defaults.py   |  3 +++
 etc/catalyst.conf  |  5 +
 3 files changed, 40 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 74d197b5..a9147241 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -62,6 +62,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 ])
 self.valid_values |= self.required_values | frozenset([
 "asflags",
+"binrepo_path",
 "catalyst_use",
 "cbuild",
 "cflags",
@@ -1074,6 +1075,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if os.path.exists(hosts_file):
 os.rename(hosts_file, hosts_file + '.catalyst')
 shutil.copy('/etc/hosts', hosts_file)
+
 # write out the make.conf
 try:
 self.write_make_conf(setup=True)
@@ -1081,6 +1083,16 @@ class StageBase(TargetBase, ClearBase, GenBase):
 raise CatalystError('Could not write %s: %s' % (
 normpath(self.settings["chroot_path"] +
  self.settings["make_conf"]), e)) from e
+
+# write out the binrepos.conf
+# we do this here for later user convenience, but normally
+# it should not affect stage builds (which only get --usepkg,
+# but never --getbinpkg as emerge parameters).
+try:
+self.write_binrepos_conf()
+except OSError as e:
+raise CatalystError('Could not write binrepos.conf: %s' % ( e )) 
from e
+
 self.resume.enable("chroot_setup")
 
 def write_make_conf(self, setup=True):
@@ -1174,6 +1186,26 @@ class StageBase(TargetBase, ClearBase, GenBase):
 '# Please keep this setting intact when reporting bugs.\n'
 'LC_MESSAGES=C.utf8\n')
 
+def write_binrepos_conf(self):
+# only if catalyst.conf defines the host and the spec defines the 
path...
+if self.settings["binhost"] != '' and "binrepo_path" in self.settings:
+
+# Write out binrepos.conf (for the chroot)
+binrpath = normpath(self.settings["chroot_path"] +
+self.settings["binrepos_conf"])
+
+with open(binrpath, "w") as myb:
+log.notice("Writing the stage binrepos.conf to: %s" % binrpath)
+myb.write("# These settings were set by the catalyst build 
script "
+"that automatically\n# built this stage.\n")
+myb.write("# Please consider using a local mirror.\n\n")
+myb.write("[gentoobinhost]\n")
+myb.write("priority = 1\n")
+myb.write("sync-uri = " + self.settings["binhost"] + \
+self.settings["binrepo_path"] + "\n")
+
+return
+
 def fsscript(self):
 if "autoresume" in self.settings["options"] \
 and self.resume.is_enabled("fsscript"):

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 39b71c34..f4d48fef 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -9,6 +9,7 @@ from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, 
LIST_XATTRS_OPTIONS
 
 
 valid_config_file_values = frozenset([
+"binhost",
 "compression_mode",
 "digests",
 "digest_format",
@@ -31,6 +32,7 @@ valid_config_file_values = frozenset([
 ])
 
 confdefaults = {
+"binhost": '',
 "comp_prog": COMPRESSOR_PROGRAM_OPTIONS['linux'],
 "compression_mode": 'lbzip2',
 "compressor_arch": None,
@@ -42,6 +44,7 @@ confdefaults = {
 "icecream": "/var/cache/icecream",
 'list_xattrs_opt': LIST_XATTRS_OPTIONS['linux'],
 "port_conf": "/etc/portage",
+"binrepos_conf": "%(port_conf)s/binrepos.conf",
 "make_conf": "%(port_conf)s/make.conf",
 "repos_conf": "%(port_conf)s/repos.conf",
 "options": set(),

diff --git a/etc/catalyst.conf b/etc/catalyst.conf
index b0b284fa..8745d0b5 100644
--- a/etc/catalyst.conf
+++ b/etc/catalyst.conf
@@ -83,3 +83,8 @@ options = [
 # Floating-point value passed to emerge as the parameter to --load-average and
 # is used to define MAKEOPTS during the target build.
 # load-average = 4.0
+
+# If you want catalyst to drop a binrepos.conf into /etc/portage, then
+# define the binhost here. This value is concatenated with the configuration
+# option binrepo_path in the spec file to obtain the src-uri.
+# binhost = "https://gentoo.osuosl.org/releases/;

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: 6272f501c0101a49fcb35f60ba75778ded696b67
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Sat Jul  1 17:39:01 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Sat Jul  1 17:45:11 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6272f501

stagebase: Add blank linkes in make.conf (e.g. before CHOST block)

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 812ac4c8..65f4306e 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1114,13 +1114,13 @@ class StageBase(TargetBase, ClearBase, GenBase):
   % (flags, self.settings[flags]))
 
 if "CBUILD" in self.settings:
-myf.write("# This should not be changed unless you know 
exactly"
+myf.write("\n# This should not be changed unless you know 
exactly"
   " what you are doing.  You\n# should probably be "
   "using a different stage, instead.\n")
 myf.write('CBUILD="' + self.settings["CBUILD"] + '"\n')
 
 if "CHOST" in self.settings:
-myf.write("# WARNING: Changing your CHOST is not something "
+myf.write("\n# WARNING: Changing your CHOST is not something "
   "that should be done lightly.\n# Please consult "
   
"https://wiki.gentoo.org/wiki/Changing_the_CHOST_variable "
   "before changing.\n")

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Andreas K. Hüttel]

commit: 9d6e495ca2ee55c21befc183a9a56fc5062a63c7
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Sat Jul  1 17:44:41 2023 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Sat Jul  1 19:26:14 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9d6e495c

stagebase: Fix and improve port_logdir config option

NOTE: The cleaning of log files does not work yet (and probably
never did).

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 7 +++
 catalyst/defaults.py   | 2 ++
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 65f4306e..74d197b5 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -271,9 +271,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 if "port_logdir" in self.settings:
 self.mount['port_logdir']['enable'] = True
-self.mount['port_logdir']['source'] = self.settings['port_logdir']
-self.env["PORT_LOGDIR"] = self.settings["port_logdir"]
-self.env["PORT_LOGDIR_CLEAN"] = PORT_LOGDIR_CLEAN
+self.mount['port_logdir']['source'] = 
normpath(self.settings['port_logdir'] + "/" + self.settings["target_subpath"] + 
"/")
+self.env["PORTAGE_LOGDIR"] = self.settings["target_logdir"]
 
 def override_cbuild(self):
 if "CBUILD" in self.makeconf:
@@ -1002,7 +1001,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # We may need to create the source of the bind mount. 
E.g., in the
 # case of an empty package cache we must create the 
directory that
 # the binary packages will be stored into.
-source_path.mkdir(mode=0o755, exist_ok=True)
+source_path.mkdir(mode=0o755, parents=True, exist_ok=True)
 
 Path(target).mkdir(mode=0o755, parents=True, exist_ok=True)
 

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 2cede562..39b71c34 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -25,6 +25,7 @@ valid_config_file_values = frozenset([
 "sharedir",
 "storedir",
 "target_distdir",
+"target_logdir",
 "target_pkgdir",
 "var_tmpfs_portage",
 ])
@@ -53,6 +54,7 @@ confdefaults = {
 "shdir": "%(sharedir)s/targets",
 "storedir": "/var/tmp/catalyst",
 "target_distdir": "/var/cache/distfiles",
+"target_logdir": "/var/log/portage",
 "target_pkgdir": "/var/cache/binpkgs",
 }
 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Ben Kohler]

commit: d7e25003b3657b8fb9dd69787efbdb6dec6cb491
Author: Ben Kohler  gentoo  org>
AuthorDate: Wed May 10 18:10:11 2023 +
Commit: Ben Kohler  gentoo  org>
CommitDate: Wed May 10 18:10:11 2023 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d7e25003

stagebase.py: switch to LC_MESSAGES=C.utf8 in make.conf

Signed-off-by: Ben Kohler  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 732c5939..812ac4c8 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1173,7 +1173,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 '\n'
 '# This sets the language of build output to English.\n'
 '# Please keep this setting intact when reporting bugs.\n'
-'LC_MESSAGES=C\n')
+'LC_MESSAGES=C.utf8\n')
 
 def fsscript(self):
 if "autoresume" in self.settings["options"] \

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: aa52470e86ebd0578c379813a9144611c97431a8
Author: Matt Turner  gentoo  org>
AuthorDate: Tue Nov 22 03:26:59 2022 +
Commit: Matt Turner  gentoo  org>
CommitDate: Tue Nov 22 21:19:03 2022 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=aa52470e

catalyst: Use correct arch setting for setarch testing

Previously we were testing against the 'host' or subarchitecture (e.g.
i486) instead of the architecture (e.g. x86). This manifests on x86
builds on x86_64; on ppc on ppc64 and sparc on sparc64 the arch and
subarch values happen to line up.

Fixes: cd58b37b ("catalyst: Access setarch data correctly")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5a3bd375..732c5939 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -129,7 +129,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # Search for a subarchitecture in each arch in the arch_config
 for arch in [x for x in arch_config if x.startswith(name) and 
host in arch_config[x]]:
 self.settings.update(arch_config[arch][host])
-setarch = arch_config.get('setarch', {}).get(host, {})
+setarch = arch_config.get('setarch', {}).get(arch, {})
 break
 else:
 # Didn't find a matching subarchitecture, keep searching

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: f2263466c535d0019be0bcbf1f08128b2399d6ea
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Mon Nov 14 00:05:58 2022 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Mon Nov 14 00:09:33 2022 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f2263466

stagebase: Also clean out profile/package.use.force profile/package.use.mask

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index bcf596ae..5a3bd375 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1230,7 +1230,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "sticky-config" not in self.settings["options"]):
 log.debug("clean(), portage_preix = %s, no sticky-config",
   self.settings["portage_prefix"])
-for _dir in "package.accept_keywords", "package.keywords", 
"package.mask", "package.unmask", "package.use", "package.env", "env":
+for _dir in "package.accept_keywords", "package.keywords", 
"package.mask", "package.unmask", "package.use", "package.env", "env", 
"profile/package.use.force", "profile/package.use.mask":
 target = pjoin(self.settings["stage_path"],
"etc/portage/%s" % _dir,
self.settings["portage_prefix"])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: 64d2993d8ca38cefd76df7a69a44accd60284cd4
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Sat Nov 12 20:42:03 2022 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Sun Nov 13 02:17:29 2022 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=64d2993d

Allow "interpreter" parameter as space-separated list of files

I added this feature to the 3.0 branch long ago and forgot about it.
Needed for multilib emulation, e.g., qemu-mips qemu-mipsn32 qemu-mips64
Tested on demeter "in production"

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 48 +++---
 1 file changed, 24 insertions(+), 24 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index eb869c70..bcf596ae 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1036,26 +1036,24 @@ class StageBase(TargetBase, ClearBase, GenBase):
 shutil.copy('/etc/resolv.conf',
 self.settings['chroot_path'] + '/etc/')
 
-# Copy over the binary interpreter (qemu), if applicable; note that 
it's given
-# as full path and goes to the same place in the chroot
+# Copy over the binary interpreter(s) (qemu), if applicable; note that 
they are given
+# as space-separated list of full paths and go to the same place in 
the chroot
 if "interpreter" in self.settings:
-if not os.path.exists(self.settings["interpreter"]):
-raise CatalystError(
-"Can't find interpreter " +
-self.settings["interpreter"],
-print_traceback=True)
+if isinstance(self.settings["interpreter"], str):
+myints = [self.settings["interpreter"]]
+else:
+myints = self.settings["interpreter"]
 
-log.notice('Copying binary interpreter %s into chroot',
-   self.settings['interpreter'])
+for myi in myints:
+if not os.path.exists(myi):
+raise CatalystError("Can't find interpreter " + myi, 
print_traceback=True)
 
-if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter']):
-os.rename(
-self.settings['chroot_path'] +
-'/' + self.settings['interpreter'],
-self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst')
+log.notice('Copying binary interpreter %s into chroot', myi)
 
-shutil.copy(self.settings['interpreter'],
-self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+if os.path.exists(self.settings['chroot_path'] + '/' + myi):
+os.rename(self.settings['chroot_path'] + '/' + myi, 
self.settings['chroot_path'] + '/' + myi + '.catalyst')
+
+shutil.copy(myi, self.settings['chroot_path'] + '/' + myi)
 
 # Copy over the envscript, if applicable
 if "envscript" in self.settings:
@@ -1214,16 +1212,18 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if os.path.exists(hosts_file + '.catalyst'):
 os.rename(hosts_file + '.catalyst', hosts_file)
 
-# optionally clean up binary interpreter
+# optionally clean up binary interpreter(s)
 if "interpreter" in self.settings:
-if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst'):
-os.rename(
-self.settings['chroot_path'] + '/' +
-self.settings['interpreter'] + '.catalyst',
-self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+if isinstance(self.settings["interpreter"], str):
+myints = [self.settings["interpreter"]]
 else:
-os.remove(
-self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+myints = self.settings["interpreter"]
+
+for myi in myints:
+if os.path.exists(self.settings['chroot_path'] + '/' + myi + 
'.catalyst'):
+os.rename(self.settings['chroot_path'] + '/' + myi + 
'.catalyst', self.settings['chroot_path'] + '/' + myi)
+else:
+os.remove(self.settings['chroot_path'] + '/' + myi)
 
 # optionally clean up portage configs
 if ("portage_prefix" in self.settings and

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 1c46e66ef2c9ab24f383eebd3c6cb1d02492b8fc
Author: Matt Turner  gentoo  org>
AuthorDate: Wed Apr 20 23:23:07 2022 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Apr 20 23:23:07 2022 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=1c46e66e

catalyst: Replace a tab

Fixes: 5be6069b ("catalyst: support 3 new options")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 76feb5f0..5c7e9adb 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -597,7 +597,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.settings["groups"] = []
 log.info('groups to create: %s' % self.settings["groups"])
 
-   def set_users(self):
+def set_users(self):
 users = self.settings["spec_prefix"] + "/users"
 if users in self.settings:
 if isinstance(self.settings[users], str):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 5be6069bcbd5a7fa3f114f28366597bc5ddbb891
Author: Patrice Clement  gentoo  org>
AuthorDate: Tue Mar  8 21:02:55 2022 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Apr  2 23:50:16 2022 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=5be6069b

catalyst: support 3 new options

* stage4/groups: create a a list of groups.
* stage4/users: create a list of users. users can also be added to
  groups using the "foo.bar=wheel,audio,baz" format.
* stage4/ssh_public_keys: copy an SSH public key into the stage4 user's home
  (.ssh/authorized_keys) and set the file permission to 0644.

Bug: https://bugs.gentoo.org/236905
Signed-off-by: Patrice Clement  gentoo.org>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 70 ++
 1 file changed, 70 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index de1e30ef..76feb5f0 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -201,6 +201,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.set_packages()
 self.set_rm()
 self.set_linuxrc()
+self.set_groups()
+self.set_users()
+self.set_ssh_public_keys()
 self.set_busybox_config()
 self.set_overlay()
 self.set_repos()
@@ -583,6 +586,39 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.settings[self.settings["spec_prefix"] + "/linuxrc"]
 del self.settings[self.settings["spec_prefix"] + "/linuxrc"]
 
+def set_groups(self):
+groups = self.settings["spec_prefix"] + "/groups"
+if groups in self.settings:
+if isinstance(self.settings[groups], str):
+self.settings["groups"] = self.settings[groups].split(",")
+self.settings["groups"] = self.settings[groups]
+del self.settings[groups]
+else:
+self.settings["groups"] = []
+log.info('groups to create: %s' % self.settings["groups"])
+
+   def set_users(self):
+users = self.settings["spec_prefix"] + "/users"
+if users in self.settings:
+if isinstance(self.settings[users], str):
+self.settings["users"] = self.settings[users].split(",")
+self.settings["users"] = self.settings[users]
+del self.settings[users]
+else:
+self.settings["users"] = []
+log.info('users to create: %s' % self.settings["users"])
+
+def set_ssh_public_keys(self):
+ssh_public_keys = self.settings["spec_prefix"] + "/ssh_public_keys"
+if ssh_public_keys in self.settings:
+if isinstance(self.settings[ssh_public_keys], str):
+self.settings["ssh_public_keys"] = 
self.settings[ssh_public_keys].split(",")
+self.settings["ssh_public_keys"] = self.settings[ssh_public_keys]
+del self.settings[ssh_public_keys]
+else:
+self.settings["ssh_public_keys"] = []
+log.info('ssh public keys to copy: %s' % 
self.settings["ssh_public_keys"])
+
 def set_busybox_config(self):
 if self.settings["spec_prefix"] + "/busybox_config" in self.settings:
 if isinstance(self.settings[self.settings['spec_prefix'] + 
'/busybox_config'], str):
@@ -894,6 +930,40 @@ class StageBase(TargetBase, ClearBase, GenBase):
 cmd(['rsync', '-a', x + '/', self.settings['stage_path']],
 env=self.env)
 
+def groups(self):
+for x in self.settings["groups"].split():
+log.notice("Creating group: '%s'", x)
+cmd(["groupadd", "-R", self.settings['chroot_path'], x], 
env=self.env)
+
+def users(self):
+for x in self.settings["users"]:
+usr, grp = '', ''
+try:
+usr, grp = x.split("=")
+except ValueError:
+usr = x
+log.debug("users: '=' separator not found on line " + x)
+log.debug("users: missing separator means no groups found")
+uacmd = ["useradd", "-R", self.settings['chroot_path'], "-m", x]
+if grp != '':
+uacmd = ["useradd", "-R", self.settings['chroot_path'], "-m", 
"-G", grp, usr]
+log.notice("Creating user: '%s'", f"{usr}={grp}")
+cmd(uacmd, env=self.env)
+
+def ssh_public_keys(self):
+for x in self.settings["ssh_public_keys"]:
+usr, pub_key_src = '', ''
+try:
+usr, pub_key_src = x.split("=")
+except ValueError:
+raise CatalystError(f"ssh_public_keys: '=' separator not found 
on line {x}")
+log.notice("Copying SSH public key for user: '%s'", usr)
+pub_key_dest = self.settings['chroot_path'] + 
f"/home/{usr}/.ssh/authorized_keys"
+cpcmd = ["cp", "-av", pub_key_src, pub_key_dest]
+cmd(cpcmd, 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, doc/

[Georgy Yakovlev]

commit: 7457cd3b1a5f3ed4a566bbf23e36c939af06967c
Author: Georgy Yakovlev  gentoo  org>
AuthorDate: Fri Nov  5 02:14:00 2021 +
Commit: Georgy Yakovlev  gentoo  org>
CommitDate: Thu Nov 25 02:43:21 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=7457cd3b

catalyst: generate .sha256 file if any digest is enabled

checksum format is simple one, identical to one
sha256sum from coreutils produces, lines starting with # are ignored.

example:[1]

# SHA256 HASH
.  stage3-tar.xz

systemd upstream calls it suse-style .sha256 files.[0]
infra already supports inline signing of files.

Bug: https://bugs.gentoo.org/821568
[0] 
https://github.com/systemd/systemd/blob/aedec452b9e5dd197881f2164fb205dfe8bfdcec/src/import/pull-common.c#L236
[1] 
https://mirrors.edge.kernel.org/opensuse/distribution/leap/15.0/iso/openSUSE-Leap-15.0-DVD-x86_64.iso.sha256
Signed-off-by: Georgy Yakovlev  gentoo.org>

 catalyst/base/genbase.py  | 3 +++
 doc/catalyst-config.5.txt | 7 ---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 6d039a8f..1643b1dc 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -43,3 +43,6 @@ class GenBase():
 for f in [path, path + '.CONTENTS.gz']:
 for i in self.settings['digests']:
 file.write(self.generate_hash(f, name=i))
+
+with io.open(path + '.sha256', 'w', encoding='utf-8') as sha256file:
+sha256file.write(self.generate_hash(path, name='sha256'))

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 003d40b6..ca9335d6 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -38,9 +38,10 @@ Basic configuration
 *digests*::
 Create a `.DIGESTS` file containing the hash output from any of the
 supported options below.  Adding them all may take a long time.
-(example: `md5 sha1 sha512 whirlpool`).  The set of supported hashes
-is dependent on the version of Python. To see the set of hashes
-supported by the version of Python in use, run
+(example: `md5 sha1 sha512 whirlpool`).  Enabling ANY digest will
+generate `.sha256` file in addition `.DIGESTS` file.  The set of
+supported hashes is dependent on the version of Python. To see the
+set of hashes supported by the version of Python in use, run
 +
 -
 $ python3 -c 'import hashlib; print(hashlib.algorithms_available)'

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, targets/support/

[Matt Turner]

commit: ad04eab48c96350f0866d6c1fe76f289499df21b
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Jul 29 01:10:58 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Jul 29 01:13:52 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=ad04eab4

catalyst: Remove boot/kernel/*/kernelopts support

This undocumented option was only wired up for hppa, and was used by the
minimal livecd to set panic=30.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py  | 11 ---
 targets/support/bootloader-setup.sh |  8 +---
 2 files changed, 1 insertion(+), 18 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 10cffd4c..4a1b4eb6 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -662,7 +662,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "boot/kernel/" + x + "/gk_action",
 "boot/kernel/" + x + "/gk_kernargs",
 "boot/kernel/" + x + "/initramfs_overlay",
-"boot/kernel/" + x + "/kernelopts",
 "boot/kernel/" + x + "/packages",
 "boot/kernel/" + x + "/softlevel",
 "boot/kernel/" + x + "/sources",
@@ -1508,16 +1507,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 return
 self._copy_kernel_config(kname=kname)
 
-# If we need to pass special options to the bootloader
-# for this kernel put them into the environment
-key = 'boot/kernel/' + kname + '/kernelopts'
-if key in self.settings:
-myopts = self.settings[key]
-
-if not isinstance(myopts, str):
-myopts = ' '.join(myopts)
-self.env[kname + "_kernelopts"] = myopts
-
 key = 'boot/kernel/' + kname + '/extraversion'
 self.settings.setdefault(key, '')
 self.env["clst_kextraversion"] = self.settings[key]

diff --git a/targets/support/bootloader-setup.sh 
b/targets/support/bootloader-setup.sh
index a998a420..08d25b59 100755
--- a/targets/support/bootloader-setup.sh
+++ b/targets/support/bootloader-setup.sh
@@ -70,16 +70,10 @@ case ${clst_hostarch} in
boot_kernel_common_name=${first/%32/}
boot_kernel_common_name=${boot_kernel_common_name/%64/}
 
-   for x in ${clst_boot_kernel}
-   do
-   eval kopt=\$clst_boot_kernel_${x}_kernelopts
-   kopts+=(${kopt})
-   done
-
# copy the bootloader for the final image
cp /usr/share/palo/iplboot $1/boot/
 
-   echo "--commandline=0/${boot_kernel_common_name} 
initrd=${first}.igz ${default_append_line[@]} ${kopts[@]}" >> ${icfg}
+   echo "--commandline=0/${boot_kernel_common_name} 
initrd=${first}.igz ${default_append_line[@]}" >> ${icfg}
echo "--bootloader=boot/iplboot" >> ${icfg}
echo "--ramdisk=boot/${first}.igz" >> ${icfg}
for x in ${clst_boot_kernel}

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: dc22f3540c6879e88e5f3bd2ff8999066387d3ea
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Mar 25 03:02:39 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Mar 25 03:02:39 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=dc22f354

catalyst: Fix indentation

Fixes: 48338534 ("Add option to generate BSD-style tagged hashes ...")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 4a4c3efa..b89e7c93 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -26,9 +26,9 @@ class GenBase():
 filename = os.path.split(filepath)[1]
 
 if self.settings['digest_format'] == 'bsd':
-return f'# {name.upper()} HASH\n{name.upper()} ({filename}) = 
{h.hexdigest()}\n'
+return f'# {name.upper()} HASH\n{name.upper()} ({filename}) = 
{h.hexdigest()}\n'
 else:
-return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
+return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
 
 def gen_contents_file(self, path):
 c = self.settings['contents_map']

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 7c4d1c5f3aab4a32795797332f33d5a759f72a9d
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Mar 25 03:02:53 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Mar 25 03:02:53 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=7c4d1c5f

catalyst: Remove @staticmethod from generate_hash()

Fixes: 48338534 ("Add option to generate BSD-style tagged hashes ...")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index b89e7c93..6d039a8f 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -12,8 +12,7 @@ class GenBase():
 def __init__(self, myspec):
 self.settings = myspec
 
-@staticmethod
-def generate_hash(filepath, name):
+def generate_hash(self, filepath, name):
 h = hashlib.new(name)
 
 with open(filepath, 'rb') as f:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 91dba2e6199b24c7d0b52b8bd6204b4a5f6e5137
Author: Matt Turner  gentoo  org>
AuthorDate: Tue Mar  9 16:17:33 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Tue Mar  9 16:17:33 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=91dba2e6

catalyst: Remove tabs

Fixes: 48338534 ("Add option to generate BSD-style tagged hashes ...")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 52418877..4a4c3efa 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -25,10 +25,10 @@ class GenBase():
 
 filename = os.path.split(filepath)[1]
 
-   if self.settings['digest_format'] == 'bsd':
-   return f'# {name.upper()} HASH\n{name.upper()} ({filename}) = 
{h.hexdigest()}\n'
-   else:
-   return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
+if self.settings['digest_format'] == 'bsd':
+return f'# {name.upper()} HASH\n{name.upper()} ({filename}) = 
{h.hexdigest()}\n'
+else:
+return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
 
 def gen_contents_file(self, path):
 c = self.settings['contents_map']

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 71037a931ff23141875c40d37605e8614a31077e
Author: Daniel Cordero  0xdc  io>
AuthorDate: Thu Feb 25 13:09:12 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Mar  6 16:57:09 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=71037a93

catalyst: Handle specifying target_profiles in repo_name:path format

Fixes: a5004005 ("catalyst: Call config_profile_link for all targets")
Signed-off-by: Daniel Cordero  0xdc.io>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index a4da7eb3..448d6265 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -814,10 +814,15 @@ class StageBase(TargetBase, ClearBase, GenBase):
 make_profile = Path(self.settings['chroot_path'] + 
self.settings['port_conf'],
 'make.profile')
 make_profile.unlink(missing_ok=True)
+try:
+repo_name, target_profile = 
self.settings['target_profile'].split(":", 1)
+except ValueError:
+repo_name = self.settings['repo_name']
+target_profile = self.settings['target_profile']
 make_profile.symlink_to(Path('../..' + self.settings['repo_basedir'],
- self.settings['repo_name'],
+ repo_name,
  'profiles',
- self.settings['target_profile']),
+ target_profile),
 target_is_directory=True)
 
 def setup_confdir(self):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Andreas K. Hüttel]

commit: 483385348b32fab2230f596816b6e1d0b6489962
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Wed Feb 24 21:26:58 2021 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Wed Feb 24 21:26:58 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=48338534

Add option to generate BSD-style tagged hashes (can be verified by modern 
coreutils)

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/genbase.py | 6 +-
 catalyst/defaults.py | 2 ++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index c7dd87bc..52418877 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -24,7 +24,11 @@ class GenBase():
 h.update(data)
 
 filename = os.path.split(filepath)[1]
-return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
+
+   if self.settings['digest_format'] == 'bsd':
+   return f'# {name.upper()} HASH\n{name.upper()} ({filename}) = 
{h.hexdigest()}\n'
+   else:
+   return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
 
 def gen_contents_file(self, path):
 c = self.settings['contents_map']

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index ccb0a584..2cede562 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -11,6 +11,7 @@ from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, 
LIST_XATTRS_OPTIONS
 valid_config_file_values = frozenset([
 "compression_mode",
 "digests",
+"digest_format",
 "distcc_hosts",
 "distdir",
 "envscript",
@@ -35,6 +36,7 @@ confdefaults = {
 "compressor_options": XATTRS_OPTIONS['linux'],
 "decomp_opt": DECOMPRESSOR_PROGRAM_OPTIONS['linux'],
 "decompressor_search_order": DECOMPRESSOR_SEARCH_ORDER,
+"digest_format": 'linux',
 "distdir": portage.settings['DISTDIR'],
 "icecream": "/var/cache/icecream",
 'list_xattrs_opt': LIST_XATTRS_OPTIONS['linux'],

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: c1f5707a9c9f27f730aad908c8f3c86075201fc9
Author: Daniel Cordero  0xdc  io>
AuthorDate: Thu Feb  4 11:07:19 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sun Feb 21 02:05:39 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=c1f5707a

catalyst: Remove /etc/machine-id in all stages

Machine IDs are unique per installation, but if they are retained in a
stage tarball, all installations from that stage share the same machine
id.

Signed-off-by: Daniel Cordero  0xdc.io>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index bc0eccd0..a4da7eb3 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -465,6 +465,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def set_cleanables(self):
 self.settings['cleanables'] = [
+"/etc/machine-id",
 "/etc/resolv.conf",
 "/var/tmp/*",
 "/tmp/*",

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 5f34226550e221921189ca2184a0afe2de1e8821
Author: Felix Bier  rohde-schwarz  com>
AuthorDate: Thu Feb  4 00:45:27 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Feb 20 21:27:29 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=5f342265

Drop PORTDIR from make.conf

This commit drops PORTDIR from being included in the generated
make.conf. The variable is no longer needed, since the existing code
will now generate a repos.conf entry for the main repo when needed
(i.e. when the location is non-default, which is the same condition
that was applied for PORTDIR in the code removed by this commit).

The PORTDIR variable is considered deprecated in favor of
repos.conf entries (reference: https://wiki.gentoo.org/wiki/PORTDIR).

Signed-off-by: Felix Bier  rohde-schwarz.com>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 6 --
 1 file changed, 6 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 97e2318c..bc0eccd0 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1092,12 +1092,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 myf.write(hostuseexpand + '="' +
   ' '.join(myuseexpandvars[hostuseexpand]) + '"\n')
 
-# Write non-default PORTDIR/DISTDIR/PKGDIR settings to make.conf
-if (self.settings['repo_basedir'], self.settings['repo_name']) != \
-   (confdefaults['repo_basedir'], confdefaults['repo_name']):
-myf.write('PORTDIR="%s/%s"\n' % (self.settings['repo_basedir'],
- self.settings['repo_name']))
-
 for x in ['target_distdir', 'target_pkgdir']:
 if self.settings[x] != confdefaults[x]:
 varname = x.split('_')[1].upper()

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Matt Turner]

commit: 65d49f1028b49fed6e011526429e553ebb6c903e
Author: Felix Bier  rohde-schwarz  com>
AuthorDate: Thu Feb  4 00:38:36 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Feb 20 21:27:29 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=65d49f10

Unify handling of main repo and other repos

This commit unifies the handling of the main repo and the other repos.
All are stored in one common list. A mount entry is created for
each entry in the list (previously a mount entry was only created
for the main repo and the other repos were copied into the chroot).
This means each non-main repo can now be either a directory or a
squash files (previously the non-main repos had to be directories).
The existing mount logic will bind-mount the repos that are stored as
directories, removing the need for copying.

A repos.conf entry will be created for each entry in the list.
This means a repos.conf entry for the main repo can now be created
(previously repos.conf entries were only created for the non-main
repos). The repos.conf entry will only be created if the target
location for the main repo is non-default, i.e. unequal to
/var/db/repos/gentoo. This mirrors the behavior of write_make_conf,
which only writes the PORTDIR variable to make.conf if the location
of the main repo is non-default.

As a side effect, the PORTDIR variable is now no longer needed,
and will be removed from write_make_conf in a separate commit.

Signed-off-by: Felix Bier  rohde-schwarz.com>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 88 +++---
 catalyst/defaults.py   |  5 ---
 2 files changed, 60 insertions(+), 33 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index fedc8f87..97e2318c 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -219,8 +219,17 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # Setup our mount points.
 self.mount = copy.deepcopy(MOUNT_DEFAULTS)
 
-self.mount['portdir']['source'] = self.snapshot
-self.mount['portdir']['target'] = self.settings['repo_basedir'] + '/' 
+ self.settings['repo_name']
+# Create mount entry for each repository
+for path, name, _ in self.repos:
+name = get_repo_name(path)
+mount_id = f'repo_{name}'
+
+self.mount[mount_id] = {
+'enable': True,
+'source': path,
+'target': self.get_repo_location(name)
+}
+
 self.mount['distdir']['source'] = self.settings['distdir']
 self.mount["distdir"]['target'] = self.settings['target_distdir']
 
@@ -587,13 +596,41 @@ class StageBase(TargetBase, ClearBase, GenBase):
   "/busybox_config"]
 
 def set_repos(self):
+
+# Each entry in this list will be a tuple of the form
+# (source, name, default)
+#
+# source: the location of the repo on the host system,
+# either a directory or a squashfs file.
+#
+# name: the repository name parsed from the repo.
+# This is just a caching mechanism to avoid parsing the name
+# every time the source is processed.
+#
+# default: Default location where the repo is expected in the
+# target system. If this matches the path where we mount the repo to
+# (as per get_repo_location), then we can skip generating a repos.conf
+# entry for that repo. Currently this mechanism is only used for
+# the main repo, which has a default location hard-coded in
+# /usr/share/portage/config/repos.conf. For the other repos,
+# the default is set to None.
+self.repos = []
+
+# Create entry for snapshot
+default_location = Path(confdefaults['repo_basedir'], 
confdefaults['repo_name'])
+self.repos.append((self.snapshot, get_repo_name(self.snapshot), 
default_location))
+
+# Create entry for every other repo
 if 'repos' in self.settings:
 if isinstance(self.settings['repos'], str):
 self.settings['repos'] = \
 self.settings['repos'].split()
-log.info('repos directories are set to: %s',
+log.info('repos are set to: %s',
  ' '.join(self.settings['repos']))
 
+get_info = lambda repo: (repo, get_repo_name(repo), None)
+self.repos.extend(map(get_info, self.settings['repos']))
+
 def set_overlay(self):
 if self.settings["spec_prefix"] + "/overlay" in self.settings:
 if isinstance(self.settings[self.settings['spec_prefix'] + 
'/overlay'], str):
@@ -832,24 +869,19 @@ class StageBase(TargetBase, ClearBase, GenBase):
 raise CatalystError(f'Could not write {repo_conf_chroot}: {e}') 
from e
 
 def process_repos(self):
-""" We copy the contents of our repos to 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: 27d7c7898bb0cccf6962b84058b0e2789bc314c1
Author: Matt Turner  gentoo  org>
AuthorDate: Mon Jan 18 03:20:41 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Jan 23 16:19:07 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=27d7c789

catalyst: Deduplicate the common build_sequence steps

This also provides a single location to add a function to all targets'
build_sequence.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 11 ++-
 catalyst/targets/embedded.py  |  3 ---
 catalyst/targets/livecd_stage1.py |  3 ---
 catalyst/targets/livecd_stage2.py |  3 ---
 catalyst/targets/netboot.py   |  3 ---
 catalyst/targets/stage4.py|  3 ---
 6 files changed, 6 insertions(+), 20 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 3b8d3a6e..40b60af3 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -90,7 +90,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.setup_confdir,
 self.portage_overlay,
 ]
-self.build_sequence = []
+self.build_sequence = [
+self.bind,
+self.chroot_setup,
+self.setup_environment,
+]
 self.finish_sequence = []
 
 self.set_valid_build_kernel_vars(addlargs)
@@ -497,16 +501,13 @@ class StageBase(TargetBase, ClearBase, GenBase):
 Or it calls the normal set_action_sequence() for the target stage.
 """
 if "purgeonly" in self.settings["options"]:
-self.build_sequence.append(self.remove_chroot)
+self.build_sequence = [self.remove_chroot]
 return
 self.set_action_sequence()
 
 def set_action_sequence(self):
 """Set basic stage1, 2, 3 action sequences"""
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.run_local,
 self.preclean,
 ])

diff --git a/catalyst/targets/embedded.py b/catalyst/targets/embedded.py
index 83b17ad2..b677b226 100644
--- a/catalyst/targets/embedded.py
+++ b/catalyst/targets/embedded.py
@@ -42,9 +42,6 @@ class embedded(StageBase):
 
 def set_action_sequence(self):
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.build_kernel,
 self.build_packages,
 self.root_overlay,

diff --git a/catalyst/targets/livecd_stage1.py 
b/catalyst/targets/livecd_stage1.py
index 9fe95a00..dbfa54ed 100644
--- a/catalyst/targets/livecd_stage1.py
+++ b/catalyst/targets/livecd_stage1.py
@@ -24,9 +24,6 @@ class livecd_stage1(StageBase):
 
 def set_action_sequence(self):
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.build_packages,
 ])
 self.finish_sequence.extend([

diff --git a/catalyst/targets/livecd_stage2.py 
b/catalyst/targets/livecd_stage2.py
index ab8f4a0e..e90e9f53 100644
--- a/catalyst/targets/livecd_stage2.py
+++ b/catalyst/targets/livecd_stage2.py
@@ -87,9 +87,6 @@ class livecd_stage2(StageBase):
 
 def set_action_sequence(self):
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.run_local,
 self.build_kernel
 ])

diff --git a/catalyst/targets/netboot.py b/catalyst/targets/netboot.py
index cb5e7ae4..a2a9fcb3 100644
--- a/catalyst/targets/netboot.py
+++ b/catalyst/targets/netboot.py
@@ -159,9 +159,6 @@ class netboot(StageBase):
 
 def set_action_sequence(self):
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.build_packages,
 self.root_overlay,
 self.copy_files_to_image,

diff --git a/catalyst/targets/stage4.py b/catalyst/targets/stage4.py
index 9e53f070..f8383f75 100644
--- a/catalyst/targets/stage4.py
+++ b/catalyst/targets/stage4.py
@@ -40,9 +40,6 @@ class stage4(StageBase):
 
 def set_action_sequence(self):
 self.build_sequence.extend([
-self.bind,
-self.chroot_setup,
-self.setup_environment,
 self.build_packages,
 self.build_kernel,
 self.bootloader,

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: 4f0fa29d5a2499b7178dc4d19f7693e52eb8def9
Author: Matt Turner  gentoo  org>
AuthorDate: Mon Jan 18 03:15:02 2021 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Jan 23 16:19:07 2021 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=4f0fa29d

catalyst: Deduplicate prepare_sequence assignments

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 13 ++---
 catalyst/targets/embedded.py  |  6 --
 catalyst/targets/livecd_stage1.py |  6 --
 catalyst/targets/livecd_stage2.py |  6 --
 catalyst/targets/netboot.py   |  6 --
 catalyst/targets/stage4.py|  6 --
 6 files changed, 6 insertions(+), 37 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 53b0a224..3b8d3a6e 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -84,7 +84,12 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "portage_overlay",
 "portage_prefix",
 ])
-self.prepare_sequence = []
+self.prepare_sequence = [
+self.unpack,
+self.config_profile_link,
+self.setup_confdir,
+self.portage_overlay,
+]
 self.build_sequence = []
 self.finish_sequence = []
 
@@ -498,12 +503,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def set_action_sequence(self):
 """Set basic stage1, 2, 3 action sequences"""
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

diff --git a/catalyst/targets/embedded.py b/catalyst/targets/embedded.py
index 7d328808..83b17ad2 100644
--- a/catalyst/targets/embedded.py
+++ b/catalyst/targets/embedded.py
@@ -41,12 +41,6 @@ class embedded(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

diff --git a/catalyst/targets/livecd_stage1.py 
b/catalyst/targets/livecd_stage1.py
index eb6bb0f0..9fe95a00 100644
--- a/catalyst/targets/livecd_stage1.py
+++ b/catalyst/targets/livecd_stage1.py
@@ -23,12 +23,6 @@ class livecd_stage1(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

diff --git a/catalyst/targets/livecd_stage2.py 
b/catalyst/targets/livecd_stage2.py
index 5fecff85..ab8f4a0e 100644
--- a/catalyst/targets/livecd_stage2.py
+++ b/catalyst/targets/livecd_stage2.py
@@ -86,12 +86,6 @@ class livecd_stage2(StageBase):
 print_traceback=True)
 
 def set_action_sequence(self):
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

diff --git a/catalyst/targets/netboot.py b/catalyst/targets/netboot.py
index 9a7e59e5..cb5e7ae4 100644
--- a/catalyst/targets/netboot.py
+++ b/catalyst/targets/netboot.py
@@ -158,12 +158,6 @@ class netboot(StageBase):
 self.resume.enable("empty")
 
 def set_action_sequence(self):
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

diff --git a/catalyst/targets/stage4.py b/catalyst/targets/stage4.py
index 997139a5..9e53f070 100644
--- a/catalyst/targets/stage4.py
+++ b/catalyst/targets/stage4.py
@@ -39,12 +39,6 @@ class stage4(StageBase):
 self.settings["cleanables"].remove('/etc/resolv.conf')
 
 def set_action_sequence(self):
-self.prepare_sequence.extend([
-self.unpack,
-self.config_profile_link,
-self.setup_confdir,
-self.portage_overlay,
-])
 self.build_sequence.extend([
 self.bind,
 self.chroot_setup,

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: 2c28062446246e246c992ea7084a3950b1337827
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Oct 29 00:32:34 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=2c280624

catalyst: Use .extend() and .append() for action_sequence

Ensures that we don't overwrite and lose some settings.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 6 +++---
 catalyst/targets/embedded.py  | 4 ++--
 catalyst/targets/livecd_stage1.py | 4 ++--
 catalyst/targets/livecd_stage2.py | 8 
 catalyst/targets/netboot.py   | 4 ++--
 catalyst/targets/stage4.py| 4 ++--
 6 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 801df2fb..46b7c59c 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -478,13 +478,13 @@ class StageBase(TargetBase, ClearBase, GenBase):
 Or it calls the normal set_action_sequence() for the target stage.
 """
 if "purgeonly" in self.settings["options"]:
-self.action_sequence = ["remove_chroot"]
+self.action_sequence.append("remove_chroot")
 return
 self.set_action_sequence()
 
 def set_action_sequence(self):
 """Set basic stage1, 2, 3 action sequences"""
-self.action_sequence = [
+self.action_sequence.extend([
 "unpack",
 "setup_confdir",
 "portage_overlay",
@@ -495,7 +495,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "preclean",
 "unbind",
 "clean",
-]
+])
 self.set_completion_action_sequences()
 
 def set_completion_action_sequences(self):

diff --git a/catalyst/targets/embedded.py b/catalyst/targets/embedded.py
index 3899cf1b..75eb68e4 100644
--- a/catalyst/targets/embedded.py
+++ b/catalyst/targets/embedded.py
@@ -41,7 +41,7 @@ class embedded(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.action_sequence = [
+self.action_sequence.extend([
 "unpack",
 "config_profile_link",
 "setup_confdir",
@@ -60,7 +60,7 @@ class embedded(StageBase):
 "clean",
 "capture",
 "clear_autoresume",
-]
+])
 
 def set_stage_path(self):
 self.settings["stage_path"] = normpath(

diff --git a/catalyst/targets/livecd_stage1.py 
b/catalyst/targets/livecd_stage1.py
index b8c26cb1..9dbfa506 100644
--- a/catalyst/targets/livecd_stage1.py
+++ b/catalyst/targets/livecd_stage1.py
@@ -23,7 +23,7 @@ class livecd_stage1(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.action_sequence = [
+self.action_sequence.extend([
 "unpack",
 "config_profile_link",
 "setup_confdir",
@@ -34,7 +34,7 @@ class livecd_stage1(StageBase):
 "build_packages",
 "unbind",
 "clean",
-]
+])
 self.set_completion_action_sequences()
 
 def set_spec_prefix(self):

diff --git a/catalyst/targets/livecd_stage2.py 
b/catalyst/targets/livecd_stage2.py
index cac16b6e..c9b5ce08 100644
--- a/catalyst/targets/livecd_stage2.py
+++ b/catalyst/targets/livecd_stage2.py
@@ -87,7 +87,7 @@ class livecd_stage2(StageBase):
 print_traceback=True)
 
 def set_action_sequence(self):
-self.action_sequence = [
+self.action_sequence.extend([
 "unpack",
 "config_profile_link",
 "setup_confdir",
@@ -97,9 +97,9 @@ class livecd_stage2(StageBase):
 "setup_environment",
 "run_local",
 "build_kernel"
-]
+])
 if "fetch" not in self.settings["options"]:
-self.action_sequence += [
+self.action_sequence.extend([
 "bootloader",
 "preclean",
 "livecd_update",
@@ -114,5 +114,5 @@ class livecd_stage2(StageBase):
 "target_setup",
 "setup_overlay",
 "create_iso",
-]
+])
 self.action_sequence.append("clear_autoresume")

diff --git a/catalyst/targets/netboot.py b/catalyst/targets/netboot.py
index 61583f0d..e5c6d43c 100644
--- a/catalyst/targets/netboot.py
+++ b/catalyst/targets/netboot.py
@@ -160,7 +160,7 @@ class netboot(StageBase):
 self.resume.enable("empty")
 
 def set_action_sequence(self):
-self.action_sequence = [
+self.action_sequence.extend([
 "unpack",
 "config_profile_link",
 "setup_confdir",
@@ -179,4 +179,4 @@ class netboot(StageBase):
 "unbind",
 "clean",
 "clear_autoresume",
-]
+])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: 3d85d537b1a00d05234cea064f863602b83ffaab
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Oct 29 03:08:04 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=3d85d537

catalyst: Split action_sequence into prepare/build/finish

We want to run the "build" sequence in a different mount namespace from
the "prepare" and "finish" sequences, so this splits action_sequence
into those groupings.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 24 +++-
 catalyst/targets/embedded.py  |  6 +-
 catalyst/targets/livecd_stage1.py |  6 +-
 catalyst/targets/livecd_stage2.py | 10 +++---
 catalyst/targets/netboot.py   |  6 +-
 catalyst/targets/stage1.py| 12 ++--
 catalyst/targets/stage4.py|  6 +-
 7 files changed, 48 insertions(+), 22 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 46b7c59c..75c84baa 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -64,7 +64,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "portage_overlay",
 "portage_prefix",
 ])
-self.action_sequence = []
+self.prepare_sequence = []
+self.build_sequence = []
+self.finish_sequence = []
 
 self.set_valid_build_kernel_vars(addlargs)
 TargetBase.__init__(self, myspec, addlargs)
@@ -478,36 +480,40 @@ class StageBase(TargetBase, ClearBase, GenBase):
 Or it calls the normal set_action_sequence() for the target stage.
 """
 if "purgeonly" in self.settings["options"]:
-self.action_sequence.append("remove_chroot")
+self.build_sequence.append("remove_chroot")
 return
 self.set_action_sequence()
 
 def set_action_sequence(self):
 """Set basic stage1, 2, 3 action sequences"""
-self.action_sequence.extend([
+self.prepare_sequence.extend([
 "unpack",
 "setup_confdir",
 "portage_overlay",
+])
+self.build_sequence.extend([
 "bind",
 "chroot_setup",
 "setup_environment",
 "run_local",
 "preclean",
 "unbind",
+])
+self.finish_sequence.extend([
 "clean",
 ])
 self.set_completion_action_sequences()
 
 def set_completion_action_sequences(self):
 if "fetch" not in self.settings["options"]:
-self.action_sequence.append("capture")
+self.finish_sequence.append("capture")
 if "keepwork" in self.settings["options"]:
-self.action_sequence.append("clear_autoresume")
+self.finish_sequence.append("clear_autoresume")
 elif "seedcache" in self.settings["options"]:
-self.action_sequence.append("remove_autoresume")
+self.finish_sequence.append("remove_autoresume")
 else:
-self.action_sequence.append("remove_autoresume")
-self.action_sequence.append("remove_chroot")
+self.finish_sequence.append("remove_autoresume")
+self.finish_sequence.append("remove_chroot")
 
 def set_use(self):
 use = self.settings["spec_prefix"] + "/use"
@@ -1381,7 +1387,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.purge()
 
 failure = False
-for x in self.action_sequence:
+for x in self.prepare_sequence + self.build_sequence + 
self.finish_sequence:
 log.notice('--- Running action sequence: %s', x)
 sys.stdout.flush()
 try:

diff --git a/catalyst/targets/embedded.py b/catalyst/targets/embedded.py
index 75eb68e4..1314ce7c 100644
--- a/catalyst/targets/embedded.py
+++ b/catalyst/targets/embedded.py
@@ -41,11 +41,13 @@ class embedded(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.action_sequence.extend([
+self.prepare_sequence.extend([
 "unpack",
 "config_profile_link",
 "setup_confdir",
 "portage_overlay",
+])
+self.build_sequence.extend([
 "bind",
 "chroot_setup",
 "setup_environment",
@@ -55,6 +57,8 @@ class embedded(StageBase):
 "fsscript",
 "unmerge",
 "unbind",
+])
+self.finish_sequence.extend([
 "remove",
 "empty",
 "clean",

diff --git a/catalyst/targets/livecd_stage1.py 
b/catalyst/targets/livecd_stage1.py
index 9dbfa506..81367053 100644
--- a/catalyst/targets/livecd_stage1.py
+++ b/catalyst/targets/livecd_stage1.py
@@ -23,16 +23,20 @@ class livecd_stage1(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 595ab9922f4ed8f88148ca7fbc9fe533db63fb4c
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Oct 29 14:39:35 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=595ab992

catalyst: Remove mount_safety_check()

mount_safety_check() exists to prevent bad things from happening if a
previous catalyst invocation left bind mounts active in the chroot.
E.g., a previous catalyst invocation is interrupted without unmounting
the bind mounts. A new catalyst invocation runs and cleans the old
chroot, which inadvertently deletes files outside of the chroot via the
bind mounts.

With all the mounts now inside a namespace, it is no longer possible to
have mounts accessible outside the build sequence. In fact, I think this
code has been unnecessary since commit e5a53e42 ("catalyst: create
namespaces for building").

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 37 -
 1 file changed, 37 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index bd5ba8d0..b9c220d0 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -638,39 +638,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 assert self.settings[verify] == "blake2"
 self.settings.setdefault("gk_mainargs", []).append("--b2sum")
 
-def mount_safety_check(self):
-"""
-Check and verify that none of our paths in mypath are mounted. We don't
-want to clean up with things still mounted, and this allows us to 
check.
-Returns 1 on ok, 0 on "something is still mounted" case.
-"""
-
-if not os.path.exists(self.settings["chroot_path"]):
-return
-
-log.debug('self.mount = %s', self.mount)
-for x in [x for x in self.mount if self.mount[x]['enable']]:
-target = normpath(self.settings['chroot_path'] +
-  self.mount[x]['target'])
-log.debug('mount_safety_check() x = %s %s', x, target)
-if not os.path.exists(target):
-continue
-
-if ismount(target):
-# Something is still mounted
-try:
-log.warning(
-'%s is still mounted; performing auto-bind-umount...', 
target)
-# Try to umount stuff ourselves
-self.unbind()
-if ismount(target):
-raise CatalystError("Auto-unbind failed for " + target)
-log.notice('Auto-unbind successful...')
-except CatalystError:
-raise CatalystError("Unable to auto-unbind " + target)
-
 def unpack(self):
-
 clst_unpack_hash = self.resume.get("unpack")
 
 # Set up all unpack info settings
@@ -755,8 +723,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 % self.settings["source_path"])
 
 if _unpack:
-self.mount_safety_check()
-
 if invalid_chroot:
 if "autoresume" in self.settings["options"]:
 log.notice(
@@ -1363,9 +1329,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 def run(self):
 self.chroot_lock.write_lock()
 
-# Check for mounts right away and abort if we cannot unmount them
-self.mount_safety_check()
-
 if "clear-autoresume" in self.settings["options"]:
 self.clear_autoresume()
 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Matt Turner]

commit: 488b06bf5dbe1eba68ac11de95f56feeb6cead83
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Oct 29 15:00:42 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=488b06bf

catalyst: Run the build sequence in new mount namespace

Catalyst has a lot of code to unmount the bind mounts it's made, and
then more to try harder when something fails. This is important because
if bind mounts still exist within the chroot when clean up happens,
files outside of the chroot on the host system can inadvertently be
deleted. E.g., distfiles, binpkgs, kerncache.

Running the build sequence (the steps that need bind mounts) within a
mount namespace and exiting the mount namespace when finished ensures
that clean up can never accidentally delete files outside the chroot.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 7 ---
 catalyst/main.py   | 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 06ec8727..caec5935 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -15,6 +15,7 @@ from snakeoil.osutils import pjoin
 from DeComp.compress import CompressMap
 
 from catalyst import log
+from catalyst.context import namespace
 from catalyst.defaults import (confdefaults, MOUNT_DEFAULTS, PORT_LOGDIR_CLEAN)
 from catalyst.support import (CatalystError, file_locate, normpath,
   cmd, read_makeconf, ismount, file_check,
@@ -1405,9 +1406,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if not self.run_sequence(self.prepare_sequence):
 return False
 
-if not self.run_sequence(self.build_sequence):
-self.unbind()
-return False
+with namespace(mount=True):
+if not self.run_sequence(self.build_sequence):
+return False
 
 if not self.run_sequence(self.finish_sequence):
 return False

diff --git a/catalyst/main.py b/catalyst/main.py
index 93a4a0d3..5536471a 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -355,7 +355,7 @@ def _main(parser, opts):
 # use pid & user namespaces, but snakeoil's namespace module has signal
 # transfer issues (CTRL+C doesn't propagate), and user namespaces need
 # more work due to Gentoo build process (uses sudo/root/portage).
-with namespace(mount=True, uts=True, ipc=True, hostname='catalyst'):
+with namespace(uts=True, ipc=True, hostname='catalyst'):
 # everything is setup, so the build is a go
 try:
 success = build_target(addlargs)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: d5747d138a0ce706d4b18be60d99ea55bdbc4120
Author: Matt Turner  gentoo  org>
AuthorDate: Sat May 16 21:44:37 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d5747d13

catalyst: Use libmount for handling mounts

Handle the mounts/unmounts in all in process rather than shelling out
(pun intended!) to an external program.

While we're here, change some log.notice to log.debug since those cases
are normal and expected.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 57 --
 1 file changed, 35 insertions(+), 22 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index e71ce344..73eacfbe 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -6,6 +6,7 @@ import sys
 
 from pathlib import Path
 
+import libmount
 import toml
 
 from snakeoil import fileutils
@@ -853,7 +854,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 source = str(self.mount[x]['source'])
 target = self.settings['chroot_path'] + 
str(self.mount[x]['target'])
-mount = ['mount']
+fstype = ''
+options = ''
 
 log.debug('bind %s: "%s" -> "%s"', x, source, target)
 
@@ -861,18 +863,20 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if 'var_tmpfs_portage' not in self.settings:
 continue
 
-mount += ['-t', 'tmpfs', '-o',
-  f"size={self.settings['var_tmpfs_portage']}G"]
+fstype = 'tmpfs'
+options = f"size={self.settings['var_tmpfs_portage']}G"
 elif source == 'tmpfs':
-mount += ['-t', 'tmpfs']
+fstype = 'tmpfs'
 elif source == 'shm':
-mount += ['-t', 'tmpfs', '-o', 'noexec,nosuid,nodev']
+fstype = 'tmpfs'
+options = 'noexec,nosuid,nodev'
 else:
 source_path = Path(self.mount[x]['source'])
 if source_path.suffix == '.sqfs':
-mount += ['-o', 'ro']
+fstype = 'squashfs'
+options = 'ro,loop'
 else:
-mount.append('--bind')
+options = 'bind'
 
 # We may need to create the source of the bind mount. 
E.g., in the
 # case of an empty package cache we must create the 
directory that
@@ -881,38 +885,47 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 Path(target).mkdir(mode=0o755, parents=True, exist_ok=True)
 
-cmd(mount + [source, target], env=self.env, fail_func=self.unbind)
+try:
+cxt = libmount.Context(source=source, target=target,
+   fstype=fstype, options=options)
+cxt.mount()
+except OSError as e:
+self.unbind()
+raise CatalystError(f"Couldn't mount: {source}, {e.strerror}")
 
 def unbind(self):
-ouch = 0
-mypath = self.settings["chroot_path"]
+chroot_path = self.settings["chroot_path"]
+umount_failed = False
 
 # Unmount in reverse order
-for x in [x for x in reversed(self.mount) if self.mount[x]['enable']]:
-target = normpath(mypath + self.mount[x]['target'])
-if not os.path.exists(target):
-log.notice('%s does not exist. Skipping', target)
+for target in [Path(chroot_path + self.mount[x]['target'])
+   for x in reversed(self.mount)
+   if self.mount[x]['enable']]:
+if not target.exists():
+log.debug('%s does not exist. Skipping', target)
 continue
 
 if not ismount(target):
-log.notice('%s is not a mount point. Skipping', target)
+log.debug('%s is not a mount point. Skipping', target)
 continue
 
 try:
-cmd(['umount', target], env=self.env)
-except CatalystError:
+cxt = libmount.Context(target=str(target))
+cxt.umount()
+except OSError:
 log.warning('First attempt to unmount failed: %s', target)
 log.warning('Killing any pids still running in the chroot')
 
 self.kill_chroot_pids()
 
 try:
-cmd(['umount', target], env=self.env)
-except CatalystError:
-ouch = 1
-log.warning("Couldn't umount bind mount: %s", target)
+cxt.umount()
+except OSError as e:
+umount_failed = True
+log.warning("Couldn't umount: %s, %s", target,
+

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: d9010dead74470b2be148412e5b9ed9cca7a8b45
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Oct 29 13:03:33 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Oct 30 22:40:52 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d9010dea

catalyst: Factor out run_sequence()

This is preparation for the next patch, which will run the build
sequence in a separate mount namespace.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 40 +++-
 1 file changed, 23 insertions(+), 17 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 75c84baa..06ec8727 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1362,6 +1362,22 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 log.debug('setup_environment(); env = %r', self.env)
 
+def run_sequence(self, sequence):
+for func in sequence:
+log.notice('--- Running action sequence: %s', func)
+sys.stdout.flush()
+try:
+getattr(self, func)()
+except LockInUse:
+log.error('Unable to aquire the lock...')
+return False
+except Exception:
+log.error('Exception running action sequence %s',
+  func, exc_info=True)
+return False
+
+return True
+
 def run(self):
 self.chroot_lock.write_lock()
 
@@ -1386,26 +1402,16 @@ class StageBase(TargetBase, ClearBase, GenBase):
 log.info('StageBase: run() purge')
 self.purge()
 
-failure = False
-for x in self.prepare_sequence + self.build_sequence + 
self.finish_sequence:
-log.notice('--- Running action sequence: %s', x)
-sys.stdout.flush()
-try:
-getattr(self, x)()
-except LockInUse:
-log.error('Unable to aquire the lock...')
-failure = True
-break
-except Exception:
-log.error('Exception running action sequence %s',
-  x, exc_info=True)
-failure = True
-break
+if not self.run_sequence(self.prepare_sequence):
+return False
 
-if failure:
-log.notice('Cleaning up... Running unbind()')
+if not self.run_sequence(self.build_sequence):
 self.unbind()
 return False
+
+if not self.run_sequence(self.finish_sequence):
+return False
+
 return True
 
 def unmerge(self):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 5d01bda685236a8543da3a8136a4fe809f6fd9ae
Author: Felix Bier  rohde-schwarz  com>
AuthorDate: Wed Oct 14 13:38:21 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Oct 14 17:47:48 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=5d01bda6

catalyst: Fix spec file USE flag parsing

In stagebase, the set_use function applies .split() to the use flags
passed from the spec file, if the value is a string. However, the
result is immediately overwritten after the if-statement. Therefore the
.split() is ineffectual.

This results in self.settings["use"] holding a string,
which is then regarded as a list of characters in write_make_conf.
This fix ensures that the result of the split is not overwritten
(matching the similar code in set_catalyst_use).

For example, setting "stage4/use: abc" in a spec file results
in USE="a b c ..." in the generated make.conf.
With this fix, the generated make.conf contains the expected
USE="abc ...".

Fixes: b30dd97d ("Unify all make.conf settings and writing")
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index df1cb844..2e313bd8 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -512,7 +512,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if use in self.settings:
 if isinstance(self.settings[use], str):
 self.settings["use"] = self.settings[use].split()
-self.settings["use"] = self.settings[use]
+else:
+self.settings["use"] = self.settings[use]
 del self.settings[use]
 else:
 self.settings["use"] = []

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: a1efe958a133a49275b359e78cde0f222166cbcd
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Aug 15 21:11:10 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Aug 15 21:11:10 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=a1efe958

stagebase: Run autopep8 to remove tabs

Fixes: 22f59e6d758b (Add "interpreter" parameter for qemu-user build support)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 49 +-
 1 file changed, 27 insertions(+), 22 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 345fe1db..df1cb844 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -939,23 +939,26 @@ class StageBase(TargetBase, ClearBase, GenBase):
 shutil.copy('/etc/resolv.conf',
 self.settings['chroot_path'] + '/etc/')
 
-   # Copy over the binary interpreter (qemu), if 
applicable; note that it's given
-   # as full path and goes to the same place in the chroot
-   if "interpreter" in self.settings:
-   if not 
os.path.exists(self.settings["interpreter"]):
-   raise CatalystError(
-   "Can't find interpreter " + 
self.settings["interpreter"],
-   print_traceback=True)
+# Copy over the binary interpreter (qemu), if applicable; note 
that it's given
+# as full path and goes to the same place in the chroot
+if "interpreter" in self.settings:
+if not os.path.exists(self.settings["interpreter"]):
+raise CatalystError(
+"Can't find interpreter " +
+self.settings["interpreter"],
+print_traceback=True)
 
-   log.notice('Copying binary interpreter %s into 
chroot', self.settings['interpreter'])
+log.notice('Copying binary interpreter %s into chroot',
+   self.settings['interpreter'])
 
-   if os.path.exists(self.settings['chroot_path'] 
+ '/' + self.settings['interpreter']):
-   os.rename(
-   self.settings['chroot_path'] + 
'/' + self.settings['interpreter'], 
-   self.settings['chroot_path'] + 
'/' + self.settings['interpreter'] + '.catalyst')
+if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter']):
+os.rename(
+self.settings['chroot_path'] +
+'/' + self.settings['interpreter'],
+self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst')
 
-   shutil.copy(self.settings['interpreter'],
-   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+shutil.copy(self.settings['interpreter'],
+self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
 
 # Copy over the envscript, if applicable
 if "envscript" in self.settings:
@@ -1124,14 +1127,16 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if os.path.exists(hosts_file + '.catalyst'):
 os.rename(hosts_file + '.catalyst', hosts_file)
 
-   # optionally clean up binary interpreter
-   if "interpreter" in self.settings:
-   if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst'):
-   os.rename(
-   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst', 
-   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
-   else:
-   os.remove(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+# optionally clean up binary interpreter
+if "interpreter" in self.settings:
+if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst'):
+os.rename(
+self.settings['chroot_path'] + '/' +
+self.settings['interpreter'] + '.catalyst',
+self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+else:
+os.remove(
+self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: b267ba9b28953df3ba6528e8cf189527b037fc1d
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Thu Jul 23 22:25:17 2020 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Fri Jul 24 13:34:54 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b267ba9b

stagebase: Extend cleanup logic to more directories

Needed for FEATURES management (switching off pid namespaces in qemu)
and for testing build system hacks (like building python single-threaded
to avoid hangs).

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 0d3a418b..345fe1db 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1138,13 +1138,19 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "sticky-config" not in self.settings["options"]):
 log.debug("clean(), portage_preix = %s, no sticky-config",
   self.settings["portage_prefix"])
-for _dir in "accept_keywords", "keywords", "mask", "unmask", "use":
+for _dir in "package.accept_keywords", "package.keywords", 
"package.mask", "package.unmask", "package.use", "package.env", "env":
 target = pjoin(self.settings["destpath"],
-   "etc/portage/package.%s" % _dir,
+   "etc/portage/%s" % _dir,
self.settings["portage_prefix"])
 log.notice("Clearing portage_prefix target: %s", target)
 clear_path(target)
 
+# Remove hacks that should *never* go into stages
+target = pjoin(self.settings["destpath"], "etc/portage/patches")
+if os.path.exists(target):
+log.warning("You've been hacking. Clearing target patches: %s", 
target)
+clear_path(target)
+
 # Remove our overlay
 overlay = normpath(
 self.settings["chroot_path"] + self.settings["local_overlay"])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Andreas K. Hüttel]

commit: 22f59e6d758b3c4801a984ea6a3b4574407143c6
Author: Andreas K. Hüttel  gentoo  org>
AuthorDate: Mon Jul 20 20:39:53 2020 +
Commit: Andreas K. Hüttel  gentoo  org>
CommitDate: Mon Jul 20 20:39:53 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=22f59e6d

Add "interpreter" parameter for qemu-user build support

Signed-off-by: Andreas K. Hüttel  gentoo.org>

 catalyst/base/stagebase.py | 28 
 1 file changed, 28 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index c2cc2c3e..0d3a418b 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -55,6 +55,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "fflags",
 "hostuse",
 "install_mask",
+"interpreter",
 "kerncache_path",
 "ldflags",
 "makeopts",
@@ -938,6 +939,24 @@ class StageBase(TargetBase, ClearBase, GenBase):
 shutil.copy('/etc/resolv.conf',
 self.settings['chroot_path'] + '/etc/')
 
+   # Copy over the binary interpreter (qemu), if 
applicable; note that it's given
+   # as full path and goes to the same place in the chroot
+   if "interpreter" in self.settings:
+   if not 
os.path.exists(self.settings["interpreter"]):
+   raise CatalystError(
+   "Can't find interpreter " + 
self.settings["interpreter"],
+   print_traceback=True)
+
+   log.notice('Copying binary interpreter %s into 
chroot', self.settings['interpreter'])
+
+   if os.path.exists(self.settings['chroot_path'] 
+ '/' + self.settings['interpreter']):
+   os.rename(
+   self.settings['chroot_path'] + 
'/' + self.settings['interpreter'], 
+   self.settings['chroot_path'] + 
'/' + self.settings['interpreter'] + '.catalyst')
+
+   shutil.copy(self.settings['interpreter'],
+   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+
 # Copy over the envscript, if applicable
 if "envscript" in self.settings:
 if not os.path.exists(self.settings["envscript"]):
@@ -1105,6 +1124,15 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if os.path.exists(hosts_file + '.catalyst'):
 os.rename(hosts_file + '.catalyst', hosts_file)
 
+   # optionally clean up binary interpreter
+   if "interpreter" in self.settings:
+   if os.path.exists(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst'):
+   os.rename(
+   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'] + '.catalyst', 
+   self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+   else:
+   os.remove(self.settings['chroot_path'] + '/' + 
self.settings['interpreter'])
+
 # optionally clean up portage configs
 if ("portage_prefix" in self.settings and
 "sticky-config" not in self.settings["options"]):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 834e1caa4d78ffc7607c6508f54fd192a5a3830c
Author: Daniel Cordero  0xdc  io>
AuthorDate: Thu May 21 17:32:45 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Jun  5 21:03:26 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=834e1caa

stagebase: allow specfiles to define install_mask

There is some code that uses this option - set_install_mask() - but it was not
added to the list of valid specfile options.

The only mention of use is in the embedded target, but it may also be useful
for other stages.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5a8cd1df..5c59c4d2 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -54,6 +54,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 "fcflags",
 "fflags",
 "hostuse",
+"install_mask",
 "kerncache_path",
 "ldflags",
 "makeopts",

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: fd1903e5cd80a13d397ae07c6472d08ac6675cba
Author: Daniel Cordero  0xdc  io>
AuthorDate: Thu May 21 17:32:46 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Jun  5 21:08:36 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=fd1903e5

embedded: use destpath to clean mergeroot

When using remove() with a capture subpath (e.g. the embedded target), 
specifying
files in rm removes the files from the seed stage, not the captured stage.

Use destpath as a base for removals. When not using a subpath, destpath
equals chroot_path.

Also unhardcode log prefix to match the target.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5c59c4d2..ab1d409b 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1168,8 +1168,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 for x in self.settings[self.settings["spec_prefix"] + "/rm"]:
 # We're going to shell out for all these cleaning
 # operations, so we get easy glob handling.
-log.notice('livecd: removing %s', x)
-clear_path(self.settings["chroot_path"] + x)
+log.notice('%s: removing %s', 
self.settings["spec_prefix"], x)
+clear_path(self.settings["destpath"] + x)
 try:
 if os.path.exists(self.settings["controller_file"]):
 cmd([self.settings['controller_file'], 'clean'],

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: d2dd86b8f21337133d9d8c744c5f3fdd96867cd5
Author: Daniel Cordero  0xdc  io>
AuthorDate: Thu May 21 17:32:47 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Jun  5 21:08:42 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d2dd86b8

embedded: copy root overlay into destpath

When using a merge subpath (e.g. the embedded target's /tmp/mergeroot),
root_overlay() copies the overlay into the seed stage, rather than the subpath.

Copy the files into destpath. When not using a subpath, destpath equals
chroot_path (so other stages are not affected).

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index ab1d409b..c2cc2c3e 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -838,7 +838,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
"/root_overlay"]:
 if os.path.exists(x):
 log.info('Copying root_overlay: %s', x)
-cmd(['rsync', '-a', x + '/', self.settings['chroot_path']],
+cmd(['rsync', '-a', x + '/', self.settings['destpath']],
 env=self.env)
 
 def bind(self):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 4d2ed2e6ea5bf5e2d79a7a9e9726748752986a35
Author: Matt Turner  gentoo  org>
AuthorDate: Fri May 15 01:22:28 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri May 15 03:39:56 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=4d2ed2e6

catalyst: Make parent directories of snapshots/

Otherwise, if /var/tmp/catalyst does not exist when the snapshot target
is executed it will fail to make the snapshots/ directory.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/targetbase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/targetbase.py b/catalyst/base/targetbase.py
index 5bcea920..3e338bee 100644
--- a/catalyst/base/targetbase.py
+++ b/catalyst/base/targetbase.py
@@ -24,7 +24,7 @@ class TargetBase(ABC):
 def set_snapshot(self, treeish=None):
 # Make snapshots directory
 snapshot_dir = Path(self.settings['storedir'], 'snapshots')
-snapshot_dir.mkdir(mode=0o755, exist_ok=True)
+snapshot_dir.mkdir(mode=0o755, parents=True, exist_ok=True)
 
 repo_name = self.settings['repo_name']
 if treeish is None:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 202556d8a04852f2fa1c39eaca11da3c1421fbfe
Author: Matt Turner  gentoo  org>
AuthorDate: Thu May 14 03:27:47 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri May 15 03:39:56 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=202556d8

catalyst: Drop unnecessary clear_path()

open(..., 'w') truncates the file.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 964a5129..68945da8 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -974,7 +974,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # Modify and write out make.conf (for the chroot)
 makepath = normpath(self.settings["chroot_path"] +
 self.settings["make_conf"])
-clear_path(makepath)
 with open(makepath, "w") as myf:
 log.notice("Writing the stage make.conf to: %s" % makepath)
 myf.write("# These settings were set by the catalyst build script "

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 9f60d7d97a9fb72d10b756801a79ad5878a9bfa2
Author: Etienne Buira  free  fr>
AuthorDate: Wed May  6 16:46:10 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed May  6 17:41:15 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9f60d7d9

catalyst: Skip only var_tmpfs_portage mount

If var_tmpfs_portage is not in settings, we want to skip the tmpfs mount
only. Prior to this commit subsequent mounts would have been incorrectly
skipped.

Fixes: 4c4a82badb5e (catalyst: Clean up bind())
Signed-off-by: Etienne Buira  free.fr>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 951500d8..964a5129 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -857,7 +857,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 if source == 'maybe_tmpfs':
 if 'var_tmpfs_portage' not in self.settings:
-return
+continue
 
 _cmd = ['mount', '-t', 'tmpfs', '-o', 'size=' +
 self.settings['var_tmpfs_portage'] + 'G', source,

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 7e392bb6998dfb726f72b3c7bddcedbce685261c
Author: Matt Turner  gentoo  org>
AuthorDate: Tue May  5 05:20:15 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Tue May  5 05:20:15 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=7e392bb6

catalyst: Remove never-used "crosscompile"

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 80cc16f2..951500d8 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -111,15 +111,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
 chroot = 'chroot'
 self.settings["CHROOT"] = chroot
 
-self.settings["crosscompile"] = \
-build != host and not chroot.startswith('setarch')
-
 log.notice('Using target: %s', self.settings['target'])
 # Print a nice informational message
-if self.settings["crosscompile"]:
-log.info('Cross-compiling on %s for different machine type %s',
- build, host)
-elif chroot.startswith('setarch'):
+if chroot.startswith('setarch'):
 log.info('Building on %s for alternate personality type %s',
  build, host)
 else:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, examples/, targets/support/, doc/, targets/embedded/

[Matt Turner]

commit: 9ad9eed7833829cc1150d715d48ea32539a5848a
Author: Matt Turner  gentoo  org>
AuthorDate: Fri May  1 06:28:58 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon May  4 02:41:16 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9ad9eed7

targets: Drop most fstypes

This removes support for various file systems from the embedded target
(a target for producing images for embedded systems, as far as I
understand) and for ISOs. For ISOs, squashfs is great and everyone uses
it -- it provides better performance from a CD than the alternatives
like zisofs, normal, or noloop. See [1] for performance data of squashfs
vs other methods.

For embedded, it's unclear whether the target is used at all. There are
some very old specs in releng.git that use 'rel_type: embedded', but I'm
not sure if the target is used at all these days. To that end, I've
asked in #gentoo-embedded if anyone uses it. I've removed what I believe
to be the file system options that don't provide any value, leaving
jffs2 for now.

[1] https://elinux.org/Squash_Fs_Comparisons

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py  |   4 +-
 doc/catalyst-spec.5.txt |   9 +--
 examples/livecd-stage2_template.spec|   8 +--
 targets/embedded/fs-runscript.sh|  15 -
 targets/support/create-iso.sh   | 114 +++-
 targets/support/filesystem-functions.sh |  65 --
 targets/support/functions.sh|  14 
 targets/support/target_image_setup.sh   |  20 --
 8 files changed, 58 insertions(+), 191 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 7c82029a..bb008744 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -386,10 +386,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
 del self.settings[self.settings["spec_prefix"] + "/fstype"]
 
 if "fstype" not in self.settings:
-self.settings["fstype"] = "normal"
+self.settings["fstype"] = "squashfs"
 for x in self.valid_values:
 if x == self.settings["spec_prefix"] + "/fstype":
-log.info('%s/fstype is being set to the default of 
"normal"',
+log.info('%s/fstype is being set to the default of 
"squashfs"',
  self.settings['spec_prefix'])
 
 def set_fsops(self):

diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index e269e16d..40453e0f 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -170,18 +170,11 @@ Filesystem
 *livecd/fstype*::
 The fstype is used to determine what sort of CD we should build.  This
 is used to set the type of loopback filesystem that we will use on our
-CD.  Possible values are as follows:
-  `squashfs`;; This gives the best compression, but requires a kernel patch.
-  `zisofs`;; This uses in-kernel compression and is supported on all platforms.
-  `normal`;; This creates a loop without compression.
-  `noloop`;; This copies the files to the CD directly, without using a
- loopback.
+CD.  The only possible value is `squashfs`.
 
 *livecd/fsops*::
 The fsops are a list of optional parameters that can be passed to the
 tool which will create the filesystem specified in *livecd/fstype*
-It is valid for the following fstypes: `squashfs`, `jffs`, `jffs2`,
-and `cramfs`.
 
 *livecd/iso*::
 This is the full path and filename to the ISO image that the

diff --git a/examples/livecd-stage2_template.spec 
b/examples/livecd-stage2_template.spec
index 3b9ca1da..e9758909 100644
--- a/examples/livecd-stage2_template.spec
+++ b/examples/livecd-stage2_template.spec
@@ -83,18 +83,14 @@ kerncache_path:
 
 # The fstype is used to determine what sort of CD we should build.  This is
 # used to set the type of loopback filesystem that we will use on our CD.
-# Possible options are as follows:
-# squashfs - This gives the best compression, but requires a kernel patch.
-# zisofs - This uses in-kernel compression and is supported on all platforms.
-# normal - This creates a loop without compression.
-# noloop - This copies the files to the CD directly, without using a loopback.
+# Possible options are as follows: squashfs
 # example:
 # livecd/fstype: squashfs
 livecd/fstype:
 
 # The fsops are a list of optional parameters that can be passed to the tool
 # which will create the filesystem specified in livecd/fstype.  It is valid for
-# the following fstypes: squashfs, jffs, jffs2, cramfs
+# the following fstypes: squashfs
 livecd/fsops:
 
 # The cdtar is essentially the bootloader for the CD.  It also holds the main

diff --git a/targets/embedded/fs-runscript.sh b/targets/embedded/fs-runscript.sh
index 4746a909..fb17d3ba 100755
--- a/targets/embedded/fs-runscript.sh
+++ b/targets/embedded/fs-runscript.sh
@@ -13,32 +13,17 @@ fs_check() {
 }
 
 case ${1} in
-   jffs)
-   fs_check 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: cd58b37bcd07433e93b80b4a93c9e79ec9395e26
Author: Matt Turner  gentoo  org>
AuthorDate: Fri May  1 05:55:43 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri May  1 05:55:43 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=cd58b37b

catalyst: Access setarch data correctly

This prevented the setarch data in the TOML files from being applied
correctly.

Fixes: b368f6aa5e29 (catalyst: Use arch data from TOML, not python
 modules)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index affdabbe..7c82029a 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -95,7 +95,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 # Search for a subarchitecture in each arch in the arch_config
 for arch in [x for x in arch_config if x.startswith(name) and 
host in arch_config[x]]:
 self.settings.update(arch_config[arch][host])
-setarch = arch_config.get('setarch', {})
+setarch = arch_config.get('setarch', {}).get(host, {})
 break
 else:
 # Didn't find a matching subarchitecture, keep searching

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 4c4a82badb5ef8bd472dae7b530ef87a4c1f9127
Author: Matt Turner  gentoo  org>
AuthorDate: Sun Apr 19 01:10:51 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Apr 23 18:20:35 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=4c4a82ba

catalyst: Clean up bind()

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 62 +-
 1 file changed, 34 insertions(+), 28 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 275c07eb..9aecf013 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -906,35 +906,41 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def bind(self):
 for x in [x for x in self.mount if self.mount[x]['enable']]:
-log.debug('bind(); x = %s', x)
-target = normpath(self.settings['chroot_path'] +
-  self.mount[x]['target'])
-ensure_dirs(target, mode=0o755)
-
-if not os.path.exists(self.mount[x]['source']):
-if self.mount[x]['source'] not in ("maybe_tmpfs", "tmpfs", 
"shmfs"):
-ensure_dirs(self.mount[x]['source'], mode=0o755)
-
-src = self.mount[x]['source']
-log.debug('bind(); src = %s', src)
-if src == "maybe_tmpfs":
-if "var_tmpfs_portage" in self.settings:
-_cmd = ['mount', '-t', 'tmpfs',
-'-o', 'size=' +
-self.settings['var_tmpfs_portage'] + 'G',
-src, target]
-elif src == "tmpfs":
-_cmd = ['mount', '-t', 'tmpfs', src, target]
+if str(self.mount[x]['source']) == 'config':
+raise CatalystError(f'"{x}" bind mount source is not 
configured')
+if str(self.mount[x]['target']) == 'config':
+raise CatalystError(f'"{x}" bind mount target is not 
configured')
+
+source = str(self.mount[x]['source'])
+target = self.settings['chroot_path'] + 
str(self.mount[x]['target'])
+
+log.debug('bind %s: "%s" -> "%s"', x, source, target)
+
+if source == 'maybe_tmpfs':
+if 'var_tmpfs_portage' not in self.settings:
+return
+
+_cmd = ['mount', '-t', 'tmpfs', '-o', 'size=' +
+self.settings['var_tmpfs_portage'] + 'G', source,
+target]
+elif source == 'tmpfs':
+_cmd = ['mount', '-t', 'tmpfs', source, target]
+elif source == 'shmfs':
+_cmd = ['mount', '-t', 'tmpfs', '-o', 'noexec,nosuid,nodev',
+'shm', target]
 else:
-if src == "shmfs":
-_cmd = ['mount', '-t', 'tmpfs', '-o',
-'noexec,nosuid,nodev', 'shm', target]
-else:
-_cmd = ['mount', '--bind', src, target]
-if _cmd:
-log.debug('bind(); _cmd = %s', _cmd)
-cmd(_cmd, env=self.env, fail_func=self.unbind)
-log.debug('bind(); finished :D')
+_cmd = ['mount', '--bind', source, target]
+
+source = Path(self.mount[x]['source'])
+
+# We may need to create the source of the bind mount. E.g., in 
the
+# case of an empty package cache we must create the directory 
that
+# the binary packages will be stored into.
+source.mkdir(mode=0o755, exist_ok=True)
+
+Path(target).mkdir(mode=0o755, parents=True, exist_ok=True)
+
+cmd(_cmd, env=self.env, fail_func=self.unbind)
 
 def unbind(self):
 ouch = 0

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, targets/support/, targets/stage2/, catalyst/, doc/

[Matt Turner]

commit: bec6650faa65d734fbbde63cb5d652e45bff8cac
Author: Matt Turner  gentoo  org>
AuthorDate: Wed Apr 22 02:09:19 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Apr 30 22:56:12 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=bec6650f

catalyst: Remove repo_basedir setting

Its purpose was very confused.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 7 ++-
 catalyst/defaults.py   | 1 -
 doc/catalyst-config.5.txt  | 4 
 targets/stage2/chroot.sh   | 2 +-
 targets/support/livecdfs-update.sh | 2 +-
 5 files changed, 4 insertions(+), 12 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index e1477bb4..1f091829 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -192,7 +192,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.mount = MOUNT_DEFAULTS.copy()
 
 self.mount['portdir']['source'] = self.snapshot
-self.mount['portdir']['target'] = self.settings['repo_basedir'] + '/' 
+ self.settings['repo_name']
+self.mount['portdir']['target'] = self.settings['target_portdir']
 self.mount['distdir']['source'] = self.settings['distdir']
 self.mount["distdir"]['target'] = self.settings['target_distdir']
 
@@ -1054,11 +1054,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 for hostuseexpand in myuseexpandvars:
 myf.write(hostuseexpand + '="' +
   ' '.join(myuseexpandvars[hostuseexpand]) + '"\n')
-# write out a shipable version
-target_portdir = normpath(self.settings["repo_basedir"] + "/" +
-  self.settings["repo_name"])
 
-myf.write('PORTDIR="%s"\n' % target_portdir)
+myf.write('PORTDIR="%s"\n' % self.settings['target_portdir'])
 myf.write('DISTDIR="%s"\n' % self.settings['target_distdir'])
 myf.write('PKGDIR="%s"\n' % self.settings['target_pkgdir'])
 if setup:

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 9878b426..bbefa3a8 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -65,7 +65,6 @@ confdefaults = {
 "pkgdir": "/var/cache/binpkgs",
 "port_tmpdir": "/var/tmp/portage",
 "PythonDir": "./catalyst",
-"repo_basedir": "/var/db/repos",
 "repo_name": "gentoo",
 "repos": "%(storedir)s/repos",
 "sharedir": "/usr/share/catalyst",

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index c1ebe9dd..b7d493eb 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -108,10 +108,6 @@ Defaults to the host's DISTDIR.
 The directory in which git repositories exist for use by the snapshot target.
 Defaults to `${storedir}/repos`.
 
-*repo_basedir*::
-The target repository directory to contain the primary repo (gentoo repo) and
-any overlays.  The default location is `/var/db/repos`.
-
 *repo_name*::
 The name of the main repository (e.g. gentoo). The git repository at
 `${repos}/${repo_name}.git` will be used to produce the portdir sqfs

diff --git a/targets/stage2/chroot.sh b/targets/stage2/chroot.sh
index bf98d328..aac9a92d 100755
--- a/targets/stage2/chroot.sh
+++ b/targets/stage2/chroot.sh
@@ -9,7 +9,7 @@ export CONFIG_PROTECT="-* /etc/locale.gen"
 echo "$locales" > /etc/locale.gen
 
 ## START BUILD
-${clst_repo_basedir}/${clst_repo_name}/scripts/bootstrap.sh ${bootstrap_opts} 
|| exit 1
+${clst_target_portdir}/scripts/bootstrap.sh ${bootstrap_opts} || exit 1
 
 # Replace modified /etc/locale.gen with default
 etc-update --automode -5

diff --git a/targets/support/livecdfs-update.sh 
b/targets/support/livecdfs-update.sh
index b1049671..8297e60d 100755
--- a/targets/support/livecdfs-update.sh
+++ b/targets/support/livecdfs-update.sh
@@ -257,7 +257,7 @@ case ${clst_livecd_type} in
USE="-* $(cat /var/db/pkg/sys-libs/glibc*/USE)" emerge -eqp 
@system | grep -e '^\[ebuild' | sed -e 's:^\[ebuild .\+\] ::' -e 's: .\+$::' > 
/usr/livecd/systempkgs.txt
 
# This is my hack to reduce tmpfs usage
-   cp -r ${clst_repo_basedir}/${clst_repo_name}/{profiles,eclass} 
/usr/livecd
+   cp -r ${clst_target_portdir}/{profiles,eclass} /usr/livecd
rm -rf 
/usr/livecd/profiles/{co*,default-{1*,a*,b*,d*,h*,i*,m*,p*,s*,x*},g*,hardened-*,n*,x*}
mv -f /etc/gconf /usr/livecd
ln -sf /usr/livecd/gconf /etc/gconf

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 992cf27d631b064a612c203169f1bc2446e6c6fc
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Apr 23 02:05:11 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Apr 23 02:05:11 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=992cf27d

catalyst: Fix Path() usage

When an Path argument has a leading /, all previous arguments are
ignored. This is surprising behavior to me...

Fixes: ddc1b50d47ff (catalyst: Simplify config_profile_link())
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 0531cf61..1c0d98c6 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -906,12 +906,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def config_profile_link(self):
 log.info('Configuring profile link...')
-make_profile = Path(self.settings['chroot_path'],
-self.settings['port_conf'],
+make_profile = Path(self.settings['chroot_path'] + 
self.settings['port_conf'],
 'make.profile')
 make_profile.unlink()
-make_profile.symlink_to(Path('../..',
- self.settings['portdir'],
+make_profile.symlink_to(Path('../..' + self.settings['portdir'],
  'profiles',
  self.settings['target_profile']),
 target_is_directory=True)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: ddc1b50d47ffde9b9bbceb83ce0b5a6cb7045e1b
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 18 23:32:08 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Apr 22 05:51:22 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=ddc1b50d

catalyst: Simplify config_profile_link()

I see no use in making this a resume point since making a symlink is a
cheap operation.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 31 +--
 1 file changed, 13 insertions(+), 18 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index cc997688..0531cf61 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -3,6 +3,9 @@ import os
 import platform
 import shutil
 import sys
+
+from pathlib import Path
+
 import toml
 
 from snakeoil import fileutils
@@ -902,24 +905,16 @@ class StageBase(TargetBase, ClearBase, GenBase):
 self.snapcache_lock.unlock()
 
 def config_profile_link(self):
-if "autoresume" in self.settings["options"] \
-and self.resume.is_enabled("config_profile_link"):
-log.notice(
-'Resume point detected, skipping config_profile_link 
operation...')
-else:
-# TODO: zmedico and I discussed making this a directory and pushing
-# in a parent file, as well as other user-specified configuration.
-log.info('Configuring profile link...')
-clear_path(self.settings['chroot_path'] +
-   self.settings['port_conf'] + '/make.profile')
-ensure_dirs(self.settings['chroot_path'] +
-self.settings['port_conf'])
-cmd(['ln', '-sf',
- '../..' + self.settings['portdir'] +
- '/profiles/' + self.settings['target_profile'],
- self.settings['chroot_path'] + self.settings['port_conf'] + 
'/make.profile'],
-env=self.env)
-self.resume.enable("config_profile_link")
+log.info('Configuring profile link...')
+make_profile = Path(self.settings['chroot_path'],
+self.settings['port_conf'],
+'make.profile')
+make_profile.unlink()
+make_profile.symlink_to(Path('../..',
+ self.settings['portdir'],
+ 'profiles',
+ self.settings['target_profile']),
+target_is_directory=True)
 
 def setup_confdir(self):
 if "autoresume" in self.settings["options"] \

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: d3a8f202bf8eb82cc1d61e2b2b9da4eadc550c90
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 18 00:36:46 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Apr 18 20:29:57 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d3a8f202

catalyst: Reformat some array definitions

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 17 +
 catalyst/targets/embedded.py  | 29 +++--
 catalyst/targets/livecd_stage1.py | 17 +
 catalyst/targets/livecd_stage2.py |  6 +++---
 catalyst/targets/netboot.py   | 26 +-
 catalyst/targets/stage4.py| 28 ++--
 6 files changed, 95 insertions(+), 28 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 71bf1ef9..60cb891a 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -531,10 +531,19 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
 def set_action_sequence(self):
 """Set basic stage1, 2, 3 action sequences"""
-self.settings["action_sequence"] = ["unpack", "unpack_snapshot",
-"setup_confdir", "portage_overlay",
-"bind", "chroot_setup", 
"setup_environment",
-"run_local", "preclean", "unbind", 
"clean"]
+self.settings['action_sequence'] = [
+"unpack",
+"unpack_snapshot",
+"setup_confdir",
+"portage_overlay",
+"bind",
+"chroot_setup",
+"setup_environment",
+"run_local",
+"preclean",
+"unbind",
+"clean",
+]
 self.set_completion_action_sequences()
 
 def set_completion_action_sequences(self):

diff --git a/catalyst/targets/embedded.py b/catalyst/targets/embedded.py
index e288228e..189eb722 100644
--- a/catalyst/targets/embedded.py
+++ b/catalyst/targets/embedded.py
@@ -40,12 +40,29 @@ class embedded(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.settings["action_sequence"] = ["dir_setup", "unpack", 
"unpack_snapshot",
-"config_profile_link", 
"setup_confdir",
-"portage_overlay", "bind", 
"chroot_setup",
-"setup_environment", 
"build_kernel", "build_packages",
-"bootloader", "root_overlay", 
"fsscript", "unmerge",
-"unbind", "remove", "empty", 
"clean", "capture", "clear_autoresume"]
+self.settings['action_sequence'] = [
+"dir_setup",
+"unpack",
+"unpack_snapshot",
+"config_profile_link",
+"setup_confdir",
+"portage_overlay",
+"bind",
+"chroot_setup",
+"setup_environment",
+"build_kernel",
+"build_packages",
+"bootloader",
+"root_overlay",
+"fsscript",
+"unmerge",
+"unbind",
+"remove",
+"empty",
+"clean",
+"capture",
+"clear_autoresume",
+]
 
 def set_stage_path(self):
 self.settings["stage_path"] = normpath(

diff --git a/catalyst/targets/livecd_stage1.py 
b/catalyst/targets/livecd_stage1.py
index f4f6b512..727d95f2 100644
--- a/catalyst/targets/livecd_stage1.py
+++ b/catalyst/targets/livecd_stage1.py
@@ -23,10 +23,19 @@ class livecd_stage1(StageBase):
 StageBase.__init__(self, spec, addlargs)
 
 def set_action_sequence(self):
-self.settings["action_sequence"] = ["unpack", "unpack_snapshot",
-"config_profile_link", 
"setup_confdir", "portage_overlay",
-"bind", "chroot_setup", 
"setup_environment", "build_packages",
-"unbind", "clean"]
+self.settings['action_sequence'] = [
+"unpack",
+"unpack_snapshot",
+"config_profile_link",
+"setup_confdir",
+"portage_overlay",
+"bind",
+"chroot_setup",
+"setup_environment",
+"build_packages",
+"unbind",
+"clean",
+]
 self.set_completion_action_sequences()
 
 def set_spec_prefix(self):

diff --git a/catalyst/targets/livecd_stage2.py 
b/catalyst/targets/livecd_stage2.py
index d41321e6..09de22fb 100644
--- a/catalyst/targets/livecd_stage2.py
+++ b/catalyst/targets/livecd_stage2.py
@@ -88,7 +88,7 @@ class livecd_stage2(StageBase):
 print_traceback=True)
 
 def 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 060df92357aed97af955aa36caf47e8518b01e05
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 18 20:25:38 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Apr 18 20:26:38 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=060df923

catalyst: Open CONTENTS file in text mode

Fixes: bb21b8615e64 (catalyst: gzip the .CONTENTS file)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 3db20f84..c7dd87bc 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -29,7 +29,7 @@ class GenBase():
 def gen_contents_file(self, path):
 c = self.settings['contents_map']
 
-with gzip.open(path + '.CONTENTS.gz', 'w', encoding='utf-8') as file:
+with gzip.open(path + '.CONTENTS.gz', 'wt', encoding='utf-8') as file:
 file.write(c.contents(path, '', verbose=self.settings['VERBOSE']))
 
 def gen_digest_file(self, path):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 907f5a20e28ef9fed45595bdc68daaebd36d5281
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 17 07:03:14 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 17 07:08:24 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=907f5a20

catalyst: Remove guard rails on duplicate digests

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index e75e6e3c..f55b113f 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -37,17 +37,13 @@ class GenBase():
 hash_map = self.settings["hash_map"]
 if os.path.exists(path):
 with io.open(digests, "w", encoding='utf-8') as myf:
-keys = {}
-for i in self.settings["digests"].split():
-keys[i] = 1
-array = sorted(keys.keys())
 for f in [path, path + '.CONTENTS']:
 if os.path.exists(f):
-if "all" in array:
+if "all" in self.settings["digests"].split():
 for k in list(hash_map.hash_map):
 digest = hash_map.generate_hash(f, hash_=k)
 myf.write(digest)
 else:
-for j in array:
+for j in self.settings["digests"].split():
 digest = hash_map.generate_hash(f, hash_=j)
 myf.write(digest)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 46e6f497dcd0c21cd5197e5d9cf1ef6e196af10a
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 17 05:37:42 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 17 05:51:15 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=46e6f497

catalyst: Generate array outside of loop

I'm going to change all of this code anyway, but I had to make this a
separate commit since I was so flabbergasted by the absurdity of it.
After digging through the history across multiple renames, I found that
this mistake was introduced in 2006.

Fixes: d128d2a554b0 (digests function uses raw output from hash function now)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 1164fd55..e75e6e3c 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -22,7 +22,7 @@ class GenBase():
 keys = {}
 for i in self.settings["contents"].split():
 keys[i] = 1
-array = sorted(keys.keys())
+array = sorted(keys.keys())
 for j in array:
 contents = contents_map.contents(path, j,
  
verbose=self.settings["VERBOSE"])
@@ -40,7 +40,7 @@ class GenBase():
 keys = {}
 for i in self.settings["digests"].split():
 keys[i] = 1
-array = sorted(keys.keys())
+array = sorted(keys.keys())
 for f in [path, path + '.CONTENTS']:
 if os.path.exists(f):
 if "all" in array:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Matt Turner]

commit: bcffd2ef3fc347d6ce346a1737ddbd0ed0b7dff6
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 17 17:27:08 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 17 18:55:23 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=bcffd2ef

catalyst: Make conf_values['digests'] a set

This makes it automatically deduplicate values (which I honestly don't
really care about) but also makes it so users don't need to .split()
themselves.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 2 +-
 catalyst/main.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index bc7f9375..eb09a4e0 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -37,5 +37,5 @@ class GenBase():
 
 with io.open(path + '.DIGESTS', 'w', encoding='utf-8') as file:
 for f in [path, path + '.CONTENTS']:
-for i in self.settings['digests'].split():
+for i in self.settings['digests']:
 file.write(self.generate_hash(f, name=i))

diff --git a/catalyst/main.py b/catalyst/main.py
index d8e791c4..c91fc8b6 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -341,8 +341,9 @@ def _main(parser, opts):
 # TODO add capability to config/spec new definitions
 
 if "digests" in conf_values:
-digests = set(conf_values['digests'].split())
 valid_digests = hashlib.algorithms_available
+digests = set(conf_values['digests'].split())
+conf_values['digests'] = digests
 
 # First validate all the requested digests are valid keys.
 if digests - valid_digests:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: f0e78af28a5e1c6275cd5b6b3f8c602e7fd16f7f
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 17 17:17:49 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 17 18:55:18 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f0e78af2

catalyst: Simplify GenBase

So much was wrong with this code. Deleting a file if it exists only to
open it with 'w' (which would have truncated it anyway!), checking for
presence of files that must exist and doing it in the wrong place, etc.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 36 
 1 file changed, 12 insertions(+), 24 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index 632ee0d9..bc7f9375 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -3,10 +3,9 @@ import hashlib
 import io
 import os
 
-
 class GenBase():
 """
-This class does generation of the contents and digests files.
+Generates CONTENTS and DIGESTS files.
 """
 
 def __init__(self, myspec):
@@ -27,27 +26,16 @@ class GenBase():
 return f'# {name.upper()} HASH\n{h.hexdigest()}  {filename}\n'
 
 def gen_contents_file(self, path):
-contents = path + ".CONTENTS"
-if os.path.exists(contents):
-os.remove(contents)
-
-contents_map = self.settings["contents_map"]
-if os.path.exists(path):
-with io.open(contents, "w", encoding='utf-8') as myf:
-contents = contents_map.contents(path, '',
- 
verbose=self.settings["VERBOSE"])
-if contents:
-myf.write(contents)
+c = self.settings['contents_map']
+
+with io.open(path + '.CONTENTS', 'w', encoding='utf-8') as file:
+file.write(c.contents(path, '', verbose=self.settings['VERBOSE']))
 
 def gen_digest_file(self, path):
-digests = path + ".DIGESTS"
-if os.path.exists(digests):
-os.remove(digests)
-if "digests" in self.settings:
-if os.path.exists(path):
-with io.open(digests, "w", encoding='utf-8') as myf:
-for f in [path, path + '.CONTENTS']:
-if os.path.exists(f):
-for i in self.settings["digests"].split():
-digest = self.generate_hash(f, name=i)
-myf.write(digest)
+if 'digests' not in self.settings:
+return
+
+with io.open(path + '.DIGESTS', 'w', encoding='utf-8') as file:
+for f in [path, path + '.CONTENTS']:
+for i in self.settings['digests'].split():
+file.write(self.generate_hash(f, name=i))

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: fe133e8058ecfcad654ae80bdfbf7359237d4ec5
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 17 07:11:33 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 17 07:14:44 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=fe133e80

catalyst: Remove dead 'all' digests case

Generating close to 30 digests is not a sensible thing to do in general,
but worse this code was not possible: 'all' was not a valid digest
option since it is not a key in HASH_DEFINITIONS.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/genbase.py | 11 +++
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py
index f55b113f..2dc8bad0 100644
--- a/catalyst/base/genbase.py
+++ b/catalyst/base/genbase.py
@@ -39,11 +39,6 @@ class GenBase():
 with io.open(digests, "w", encoding='utf-8') as myf:
 for f in [path, path + '.CONTENTS']:
 if os.path.exists(f):
-if "all" in self.settings["digests"].split():
-for k in list(hash_map.hash_map):
-digest = hash_map.generate_hash(f, hash_=k)
-myf.write(digest)
-else:
-for j in self.settings["digests"].split():
-digest = hash_map.generate_hash(f, hash_=j)
-myf.write(digest)
+for i in self.settings["digests"].split():
+digest = hash_map.generate_hash(f, hash_=i)
+myf.write(digest)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, /

[Matt Turner]

commit: 2731f5f4571c91acb0e459ddcec6582bba25f5ce
Author: Matt Turner  gentoo  org>
AuthorDate: Wed Apr 15 20:03:16 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Apr 15 20:03:16 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=2731f5f4

catalyst: Fix and enable bad-continuation checks

Signed-off-by: Matt Turner  gentoo.org>

 .pylintrc  | 2 --
 catalyst/base/stagebase.py | 8 
 setup.py   | 2 +-
 3 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/.pylintrc b/.pylintrc
index 8caafef1..5428e349 100644
--- a/.pylintrc
+++ b/.pylintrc
@@ -26,7 +26,6 @@ load-plugins=
 # We should clean up things so we can enable:
 # missing-docstring -- add lots of docstrings everywhere!
 # bad-whitespace -- fix spacing everywhere
-# bad-continuation -- might be hard with tab indentation policy
 # invalid-name -- need to manage constants better
 # line-too-long -- figure out a length and stick to it
 # super-init-not-called -- fix the classes __init__ structure
@@ -46,7 +45,6 @@ disable=
fixme,
broad-except,
bad-whitespace,
-   bad-continuation,
invalid-name,
line-too-long,
super-init-not-called,

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d2f16f9b..f673382f 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -443,8 +443,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 if os.path.exists(self.settings["source_path"]):
 self.settings["source_path_hash"] = \
 self.settings["hash_map"].generate_hash(
-self.settings["source_path"],
-hash_=self.settings["hash_function"])
+self.settings["source_path"],
+hash_=self.settings["hash_function"])
 log.notice('Source path set to %s', self.settings['source_path'])
 
 def set_dest_path(self):
@@ -470,8 +470,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
 log.info('SNAPSHOT_PATH set to: %s', self.settings['snapshot_path'])
 self.settings["snapshot_path_hash"] = \
 self.settings["hash_map"].generate_hash(
-self.settings["snapshot_path"],
-hash_=self.settings["hash_function"])
+self.settings["snapshot_path"],
+hash_=self.settings["hash_function"])
 
 def set_snapcache_path(self):
 self.settings["snapshot_cache_path"] = \

diff --git a/setup.py b/setup.py
index fef09622..7a97b30c 100755
--- a/setup.py
+++ b/setup.py
@@ -93,7 +93,7 @@ _setup(
 long_description=_codecs.open(
 _os.path.join(_this_dir, 'README'), 'r', 'utf-8').read(),
 classifiers=[
-'Development Status :: 5 - Production/Stable',
+'Development Status :: 5 - Production/Stable',
 'License :: OSI Approved :: GNU General Public License v2 or later 
(GPLv2+)',
 'Intended Audience :: System Administrators',
 'Operating System :: POSIX',

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 051596bc5da38d0d1b6ccc1f349cec783d9599b3
Author: Matt Turner  gentoo  org>
AuthorDate: Wed Apr 15 19:57:41 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Apr 15 19:57:41 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=051596bc

catalyst: Remove unused import and reorder

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d1e638be..d2f16f9b 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1,10 +1,9 @@
 
 import os
-import imp
-import toml
 import platform
 import shutil
 import sys
+import toml
 
 from snakeoil import fileutils
 from snakeoil.osutils import pjoin

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: 92af7453a386fb5bd1b03ba5afae6d0165fe4364
Author: Matt Turner  gentoo  org>
AuthorDate: Wed Apr 15 18:00:15 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Wed Apr 15 18:04:03 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=92af7453

catalyst: Remove dead base_dirs() function

Prior to commit 8ef6e85a57ec (catalyst: Drop BSD remnants) this was used
for some FreeBSD-specific logic.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 5 +
 catalyst/targets/stage1.py | 3 ---
 2 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index b01e4825..d993641f 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -528,7 +528,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
"""Set basic stage1, 2, 3 action sequences"""
self.settings["action_sequence"] = ["unpack", "unpack_snapshot",
"setup_confdir", "portage_overlay",
-   "base_dirs", "bind", "chroot_setup", 
"setup_environment",
+   "bind", "chroot_setup", "setup_environment",
"run_local", "preclean", "unbind", "clean"]
self.set_completion_action_sequences()
 
@@ -940,9 +940,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
cmd(['rsync', '-a', x + '/', 
self.settings['chroot_path']],
env=self.env)
 
-   def base_dirs(self):
-   pass
-
def bind(self):
for x in self.mounts:
log.debug('bind(); x = %s', x)

diff --git a/catalyst/targets/stage1.py b/catalyst/targets/stage1.py
index 4ce0d0b4..a1a29cff 100644
--- a/catalyst/targets/stage1.py
+++ b/catalyst/targets/stage1.py
@@ -77,9 +77,6 @@ class stage1(StageBase):
"If you break it, you buy it.  Don't complain 
to us about it.\n"
"Don't say we did not warn you.")
 
-   def base_dirs(self):
-   pass
-
def set_mounts(self):
# stage_path/proc probably doesn't exist yet, so create it
ensure_dirs(self.settings["stage_path"]+"/proc")

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: f0c42aff30385e2e37b96bb0b568ed049d40443b
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 11 21:59:07 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:43:10 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f0c42aff

catalyst: Require that subclasses implement needed properties

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/targetbase.py | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/targetbase.py b/catalyst/base/targetbase.py
index 0977bad3..a09abc73 100644
--- a/catalyst/base/targetbase.py
+++ b/catalyst/base/targetbase.py
@@ -1,8 +1,10 @@
 import os
 
+from abc import ABC, abstractmethod
+
 from catalyst.support import addl_arg_parse
 
-class TargetBase():
+class TargetBase(ABC):
"""
The toplevel class for all targets. This is about as generic as we get.
"""
@@ -13,3 +15,15 @@ class TargetBase():
'PATH': '/bin:/sbin:/usr/bin:/usr/sbin',
'TERM': os.getenv('TERM', 'dumb'),
}
+
+   @property
+   @classmethod
+   @abstractmethod
+   def required_values(cls):
+   return NotImplementedError
+
+   @property
+   @classmethod
+   @abstractmethod
+   def valid_values(cls):
+   return NotImplementedError

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: e28d8c5cc9e80003ae9eb8ba401601845f81a413
Author: Matt Turner  gentoo  org>
AuthorDate: Sun Apr 12 01:01:31 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:43:10 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=e28d8c5c

catalyst: Use platform.machine() instead of uname

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d1dd14b4..d18f28af 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1,6 +1,7 @@
 
 import os
 import imp
+import platform
 import shutil
 import sys
 
@@ -131,7 +132,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
if "cbuild" in self.settings:
buildmachine = self.settings["cbuild"].split("-")[0]
else:
-   buildmachine = os.uname()[4]
+   buildmachine = platform.machine()
if buildmachine not in machinemap:
raise CatalystError("Unknown build machine type " + 
buildmachine)
self.settings["buildarch"] = machinemap[buildmachine]

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Matt Turner]

commit: dd9e1cda4772f33fa0c3d07578c5ca6808c6dd22
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 11 22:23:54 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:43:10 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=dd9e1cda

catalyst: Clean assignments of {valid/required}_values

Making them class variables will allow us to verify that all subclasses
of TargetBase have defined them (in the next commit).

While we're moving them, change them to frozensets, since testing
membership is the thing they're useed for.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py| 68 ++-
 catalyst/targets/embedded.py  | 24 ++
 catalyst/targets/livecd_stage1.py | 11 ---
 catalyst/targets/livecd_stage2.py | 48 +++
 catalyst/targets/netboot.py   | 29 -
 catalyst/targets/snapshot.py  | 11 +--
 catalyst/targets/stage1.py| 10 --
 catalyst/targets/stage2.py|  7 ++--
 catalyst/targets/stage3.py|  5 +--
 catalyst/targets/stage4.py| 26 +++
 10 files changed, 159 insertions(+), 80 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index b4275355..27516af6 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -27,16 +27,36 @@ class StageBase(TargetBase, ClearBase, GenBase):
the driver class for pretty much everything that Catalyst does.
"""
def __init__(self,myspec,addlargs):
-   self.required_values.extend(["version_stamp", "target", 
"subarch",
-   "rel_type", "profile", "snapshot", "source_subpath"])
-
-   self.valid_values.extend(["version_stamp", "target", "subarch",
-   "rel_type", "profile", "snapshot", "source_subpath",
-   "portage_confdir", "portage_prefix", "portage_overlay",
-   "cflags", "cxxflags", "fcflags", "fflags", "ldflags", 
"asflags",
-   "common_flags", "cbuild", "hostuse", "catalyst_use",
-   "distcc_hosts", "makeopts", "pkgcache_path", 
"kerncache_path",
-   "compression_mode", "decompression_mode"])
+   self.required_values |= frozenset([
+   "profile",
+   "rel_type",
+   "snapshot",
+   "source_subpath",
+   "subarch",
+   "target",
+   "version_stamp",
+   ])
+   self.valid_values |= self.required_values | frozenset([
+   "asflags",
+   "catalyst_use",
+   "cbuild",
+   "cflags",
+   "common_flags",
+   "compression_mode",
+   "cxxflags",
+   "decompression_mode",
+   "distcc_hosts",
+   "fcflags",
+   "fflags",
+   "hostuse",
+   "kerncache_path",
+   "ldflags",
+   "makeopts",
+   "pkgcache_path",
+   "portage_confdir",
+   "portage_overlay",
+   "portage_prefix",
+   ])
 
self.set_valid_build_kernel_vars(addlargs)
TargetBase.__init__(self, myspec, addlargs)
@@ -422,7 +442,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
def set_fsops(self):
if "fstype" in self.settings:
-   self.valid_values.append("fsops")
+   self.valid_values |= {"fsops"}
if self.settings["spec_prefix"] + "/fsops" in 
self.settings:
self.settings["fsops"] = \

self.settings[self.settings["spec_prefix"] + "/fsops"]
@@ -647,18 +667,20 @@ class StageBase(TargetBase, ClearBase, GenBase):
loopy = addlargs["boot/kernel"]
 
for x in loopy:
-   self.valid_values.append("boot/kernel/" + x + 
"/aliases")
-   self.valid_values.append("boot/kernel/" + x + 
"/config")
-   self.valid_values.append("boot/kernel/" + x + 
"/console")
-   self.valid_values.append("boot/kernel/" + x + 
"/extraversion")
-   self.valid_values.append("boot/kernel/" + x + 
"/gk_action")
-   self.valid_values.append("boot/kernel/" + x + 
"/gk_kernargs")
-   self.valid_values.append("boot/kernel/" + x + 
"/initramfs_overlay")
-   

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 403ab9c13eda83e841316a15dff4877be4eda698
Author: Matt Turner  gentoo  org>
AuthorDate: Mon Apr 13 19:44:27 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:42:48 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=403ab9c1

catalyst: Fix appending --b2sum

... again. If gk_mainargs is set in the spec, then
self.settings[gk_mainargs] is an array, not a string, so adding a string
adds the individual characters as array elements. However, if
gk_mainargs is not set, then adding a string makes it a string and
everything works.

Fixes: 4b430dc36fdb (catalyst: Fix appending --b2sum)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 6f0a1bc3..b4275355 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -676,8 +676,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
verify = prefix + "/verify"
if verify in self.settings:
assert self.settings[verify] == "blake2"
-   self.settings.setdefault("gk_mainargs", "")
-   self.settings["gk_mainargs"] += " --b2sum"
+   self.settings.setdefault("gk_mainargs", 
[]).append("--b2sum")
 
def kill_chroot_pids(self):
log.info('Checking for processes running in chroot and killing 
them.')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: a711659173fa2d1b3763370bb8b76b990a2d2df4
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 11 21:59:07 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:35:35 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=a7116591

catalyst: Require that subclasses implement needed properties

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/targetbase.py | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/targetbase.py b/catalyst/base/targetbase.py
index 0977bad3..a09abc73 100644
--- a/catalyst/base/targetbase.py
+++ b/catalyst/base/targetbase.py
@@ -1,8 +1,10 @@
 import os
 
+from abc import ABC, abstractmethod
+
 from catalyst.support import addl_arg_parse
 
-class TargetBase():
+class TargetBase(ABC):
"""
The toplevel class for all targets. This is about as generic as we get.
"""
@@ -13,3 +15,15 @@ class TargetBase():
'PATH': '/bin:/sbin:/usr/bin:/usr/sbin',
'TERM': os.getenv('TERM', 'dumb'),
}
+
+   @property
+   @classmethod
+   @abstractmethod
+   def required_values(cls):
+   return NotImplementedError
+
+   @property
+   @classmethod
+   @abstractmethod
+   def valid_values(cls):
+   return NotImplementedError

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 36980e93e59ede79aa3ecbf72dfedfd2014d692b
Author: Matt Turner  gentoo  org>
AuthorDate: Mon Apr 13 19:44:27 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:35:35 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=36980e93

catalyst: Fix appending --b2sum

... again. If gk_mainargs is set in the spec, then
self.settings[gk_mainargs] is an array, not a string, so adding a string
adds the individual characters as array elements. However, if
gk_mainargs is not set, then adding a string makes it a string and
everything works.

Fixes: 4b430dc36fdb (catalyst: Fix appending --b2sum)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 6f0a1bc3..829f9fb7 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -676,8 +676,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
verify = prefix + "/verify"
if verify in self.settings:
assert self.settings[verify] == "blake2"
-   self.settings.setdefault("gk_mainargs", "")
-   self.settings["gk_mainargs"] += " --b2sum"
+   self.settings.setdefault("gk_mainargs", 
[]).append("--b2um")
 
def kill_chroot_pids(self):
log.info('Checking for processes running in chroot and killing 
them.')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Matt Turner]

commit: 2fb8d95e7c83a6e2123e9575f9fd253aae962b43
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Apr 11 22:30:03 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:35:35 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=2fb8d95e

catalyst: Just import things where they're needed

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/resume.py| 3 ++-
 catalyst/base/stagebase.py | 3 ++-
 catalyst/fileops.py| 7 +--
 3 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/catalyst/base/resume.py b/catalyst/base/resume.py
index a0770b9d..82f97cd3 100644
--- a/catalyst/base/resume.py
+++ b/catalyst/base/resume.py
@@ -9,9 +9,10 @@ set, unset, is_set, is_unset, enabled, clear_all
 import os
 
 from snakeoil import fileutils
+from snakeoil.osutils import pjoin, listdir_files
 
 from catalyst import log
-from catalyst.fileops import ensure_dirs, pjoin, listdir_files, clear_dir
+from catalyst.fileops import ensure_dirs, clear_dir
 
 
 class AutoResume():

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index a90fc843..683e5989 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -5,6 +5,7 @@ import shutil
 import sys
 
 from snakeoil import fileutils
+from snakeoil.osutils import pjoin
 
 from DeComp.compress import CompressMap
 
@@ -17,7 +18,7 @@ from catalyst.base.targetbase import TargetBase
 from catalyst.base.clearbase import ClearBase
 from catalyst.base.genbase import GenBase
 from catalyst.lock import LockDir, LockInUse
-from catalyst.fileops import ensure_dirs, pjoin, clear_dir, clear_path
+from catalyst.fileops import ensure_dirs, clear_dir, clear_path
 from catalyst.base.resume import AutoResume
 
 

diff --git a/catalyst/fileops.py b/catalyst/fileops.py
index fffcfc6a..671d8d9f 100644
--- a/catalyst/fileops.py
+++ b/catalyst/fileops.py
@@ -10,12 +10,7 @@ import os
 import shutil
 from stat import ST_UID, ST_GID, ST_MODE
 
-# NOTE: pjoin and listdir_files are imported here for export
-# to other catalyst modules
-# pylint: disable=unused-import
-from snakeoil.osutils import (ensure_dirs as snakeoil_ensure_dirs,
-   pjoin, listdir_files)
-# pylint: enable=unused-import
+from snakeoil.osutils import ensure_dirs as snakeoil_ensure_dirs
 
 from catalyst import log
 from catalyst.support import CatalystError

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: af046b2ae52d510496c8896c3750c3dae3640b71
Author: Matt Turner  gentoo  org>
AuthorDate: Sun Apr 12 01:01:31 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Apr 13 20:35:35 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=af046b2a

catalyst: Use platform.machine() instead of uname

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 683e5989..210527a9 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1,6 +1,7 @@
 
 import os
 import imp
+import platform
 import shutil
 import sys
 
@@ -131,7 +132,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
if "cbuild" in self.settings:
buildmachine = self.settings["cbuild"].split("-")[0]
else:
-   buildmachine = os.uname()[4]
+   buildmachine = platform.machine()
if buildmachine not in machinemap:
raise CatalystError("Unknown build machine type " + 
buildmachine)
self.settings["buildarch"] = machinemap[buildmachine]

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/, catalyst/, targets/stage1/

[Matt Turner]

commit: ecdf556b4c00325bb299e012024f97e8af33c788
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 10 16:51:36 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 10 21:03:34 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=ecdf556b

catalyst: Fix most pylint warnings

- consider-iterating-dictionary
- consider-using-enumerate
- inconsistent-return-statements
- len-as-condition
- literal-comparison
- no-else-raise
- no-else-return
- useless-return

Unfortunately there are more and they're a bit trickier.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 10 --
 catalyst/builder.py|  2 --
 catalyst/config.py |  2 +-
 catalyst/defaults.py   |  2 --
 catalyst/fileops.py|  2 +-
 catalyst/log.py|  3 +--
 catalyst/main.py   |  5 ++---
 catalyst/support.py| 23 +++
 catalyst/targets/stage1.py |  5 -
 targets/stage1/build.py|  6 +++---
 10 files changed, 23 insertions(+), 37 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 021f3beb..6f0a1bc3 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -474,7 +474,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
"/snapshots/" + self.settings["snapshot_name"] +
self.settings["snapshot"]),
self.accepted_extensions,
-   self.settings["source_matching"] is "strict"
+   self.settings["source_matching"] == "strict"
)
log.info('SNAPSHOT_PATH set to: %s', 
self.settings['snapshot_path'])
self.settings["snapshot_path_hash"] = \
@@ -557,7 +557,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:

self.settings["action_sequence"].append("remove_autoresume")
self.settings["action_sequence"].append("remove_chroot")
-   return
 
def set_use(self):
use = self.settings["spec_prefix"] + "/use"
@@ -716,8 +715,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.unbind()
if ismount(target):
raise 
CatalystError("Auto-unbind failed for " + target)
-   else:
-   log.notice('Auto-unbind 
successful...')
+   log.notice('Auto-unbind successful...')
except CatalystError:
raise CatalystError("Unable to 
auto-unbind " + target)
 
@@ -1105,7 +1103,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
continue
myf.write("# %s is unsupported.  USE AT 
YOUR OWN RISK!\n"
% flags)
-   if flags not in self.settings or (flags is not 
"COMMON_FLAGS" and
+   if flags not in self.settings or (flags != 
"COMMON_FLAGS" and
self.settings[flags] == 
self.settings["COMMON_FLAGS"]):
myf.write('%s="${COMMON_FLAGS}"\n' % 
flags)
elif isinstance(self.settings[flags], list):
@@ -1442,7 +1440,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
if "purgetmponly" in self.settings["options"]:
self.purge()
-   return
+   return True
 
if "purgeonly" in self.settings["options"]:
log.info('StageBase: run() purgeonly')

diff --git a/catalyst/builder.py b/catalyst/builder.py
index e9f468d7..64e14c8d 100644
--- a/catalyst/builder.py
+++ b/catalyst/builder.py
@@ -1,5 +1,3 @@
-import os
-
 class generic():
def __init__(self,myspec):
self.settings=myspec

diff --git a/catalyst/config.py b/catalyst/config.py
index 5e70de41..1571db9d 100644
--- a/catalyst/config.py
+++ b/catalyst/config.py
@@ -22,7 +22,7 @@ class ParserBase():
 
def dump(self):
dump = ""
-   for x in self.values.keys():
+   for x in self.values:
dump += x + " = " + repr(self.values[x]) + "\n"
return dump
 

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index bcb59796..349f16f6 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -1,6 +1,4 @@
 
-import os
-
 from DeComp.definitions import DECOMPRESSOR_SEARCH_ORDER
 from DeComp.definitions import COMPRESSOR_PROGRAM_OPTIONS, XATTRS_OPTIONS
 from DeComp.definitions import DECOMPRESSOR_PROGRAM_OPTIONS, 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 481d0321fc1278e9a1a0af328c0b4cf14013ff3c
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 10 07:34:19 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 10 07:34:19 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=481d0321

catalyst: Remove some python-2 left-overs

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 53d39536..320e9d53 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -20,11 +20,6 @@ from catalyst.lock import LockDir, LockInUse
 from catalyst.fileops import ensure_dirs, pjoin, clear_dir, clear_path
 from catalyst.base.resume import AutoResume
 
-if sys.version_info[0] >= 3:
-   py_input = input
-else:
-   py_input = raw_input  # pylint: disable=undefined-variable
-
 
 class StageBase(TargetBase, ClearBase, GenBase):
"""
@@ -1697,6 +1692,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
@staticmethod
def _debug_pause_():
-   py_input("press any key to continue: ")
+   input("press any key to continue: ")
 
 # vim: ts=4 sw=4 sta et sts=4 ai

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 4b430dc36fdb7fdecb768f617b67fab1baacb9cd
Author: Matt Turner  gentoo  org>
AuthorDate: Fri Apr 10 05:35:35 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Fri Apr 10 05:37:06 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=4b430dc3

catalyst: Fix appending --b2sum

I fixed this locally on my testing machine and failed to sync the
changes back.

Fixes: c3b572d0ed5e (catalyst: Automatically specify --b2sum ...)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 00596264..72ef57d3 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -683,7 +683,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
verify = prefix + "/verify"
if verify in self.settings:
assert self.settings[verify] == "blake2"
-   self.settings.setdefault("gk_mainargs", "").append(" 
--b2sum")
+   self.settings.setdefault("gk_mainargs", "")
+   self.settings["gk_mainargs"] += " --b2sum"
 
def kill_chroot_pids(self):
log.info('Checking for processes running in chroot and killing 
them.')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: c3b572d0ed5ebf29017da5e894e98fe9ab150126
Author: Matt Turner  gentoo  org>
AuthorDate: Thu Apr  9 19:04:28 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Thu Apr  9 19:23:43 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=c3b572d0

catalyst: Automatically specify --b2sum if /verify is set

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 16 
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 4855c9f0..00596264 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -672,10 +672,18 @@ class StageBase(TargetBase, ClearBase, GenBase):

[addlargs["boot/kernel/" + x + "/packages"]]
 
def set_build_kernel_vars(self):
-   if self.settings["spec_prefix"] + "/gk_mainargs" in 
self.settings:
-   self.settings["gk_mainargs"] = \
-   self.settings[self.settings["spec_prefix"] + 
"/gk_mainargs"]
-   del self.settings[self.settings["spec_prefix"] + 
"/gk_mainargs"]
+   prefix = self.settings["spec_prefix"]
+
+   gk_mainargs = prefix + "/gk_mainargs"
+   if gk_mainargs in self.settings:
+   self.settings["gk_mainargs"] = 
self.settings[gk_mainargs]
+   del self.settings[gk_mainargs]
+
+   # Ask genkernel to include b2sum if /verify is set
+   verify = prefix + "/verify"
+   if verify in self.settings:
+   assert self.settings[verify] == "blake2"
+   self.settings.setdefault("gk_mainargs", "").append(" 
--b2sum")
 
def kill_chroot_pids(self):
log.info('Checking for processes running in chroot and killing 
them.')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: a4f0612d4162d937652c34200e3de0925d113e05
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Mar 28 22:00:59 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sun Mar 29 01:35:14 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=a4f0612d

catalyst: Skip "maybe_tmpfs" directory creation

Fixes: 8441372c2857 (fix /var/tmp/portage not being forced into a tmpfs)
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 939e03e2..4855c9f0 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -963,7 +963,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
ensure_dirs(target, mode=0o755)
 
if not os.path.exists(self.mountmap[x]):
-   if self.mountmap[x] not in ["tmpfs", "shmfs"]:
+   if self.mountmap[x] not in ("maybe_tmpfs", 
"tmpfs", "shmfs"):
ensure_dirs(self.mountmap[x], 
mode=0o755)
 
src = self.mountmap[x]

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, examples/, targets/support/, livecd/cdtar/, doc/

[Matt Turner]

commit: 34a82f07bd3e0f968c99369e794db4dc8c747b38
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Mar 28 07:25:05 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Mar 28 07:33:21 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=34a82f07

Remove yaboot leftovers

GRUB replaced yaboot on PPC in commit e9c0a0394d15 (ppc: Generate Grub
entries)

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py   |   1 -
 doc/catalyst-spec.5.txt  |   6 --
 examples/livecd-stage2_template.spec |   6 --
 livecd/cdtar/yaboot-1.3.13-cdtar.tar.bz2 | Bin 51088 -> 0 bytes
 targets/support/bootloader-setup.sh  |   1 -
 5 files changed, 14 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d29b48fc..d2b4cd94 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -662,7 +662,6 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.valid_values.append("boot/kernel/" + x + 
"/gk_action")
self.valid_values.append("boot/kernel/" + x + 
"/gk_kernargs")
self.valid_values.append("boot/kernel/" + x + 
"/initramfs_overlay")
-   self.valid_values.append("boot/kernel/" + x + 
"/machine_type")
self.valid_values.append("boot/kernel/" + x + 
"/sources")
self.valid_values.append("boot/kernel/" + x + 
"/softlevel")
self.valid_values.append("boot/kernel/" + x + 
"/use")

diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 250acded..6c71b57b 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -264,12 +264,6 @@ sources to keep the modules from overwriting each other.  
We do not
 use this on the official media.  `` should match one of the
 labels given to *boot/kernel*.
 
-*boot/kernel//machine_type*::
-This option is only for ppc64 machines (example: `ibm`).  If used it
-will create the `/etc/yaboot.conf` entry used for booting an ibm
-powerpc machine.  `` should match one of the labels given to
-*boot/kernel*.
-
 *boot/kernel//console*::
 This is only supported on ppc64 currently.  This entry sets up the
 console boot parameters required for sending the output to the

diff --git a/examples/livecd-stage2_template.spec 
b/examples/livecd-stage2_template.spec
index 7dd64c3f..fe9b36e7 100644
--- a/examples/livecd-stage2_template.spec
+++ b/examples/livecd-stage2_template.spec
@@ -317,12 +317,6 @@ boot/kernel/gentoo/extraversion:
 # boot/kernel/gentoo/packages: pcmcia-cs speedtouch slmodem globespan-adsl 
hostap-driver hostap-utils ipw2100 ipw2200 fritzcapi fcdsl cryptsetup
 boot/kernel/gentoo/packages:
 
-# This option is only for ppc64 machines.  If used it will create the 
/etc/yaboot.conf
-# entry used for booting a ibm powerpc machine.
-# example:
-# boot/kernel/gentoo/machine_type: ibm
-boot/kernel/gentoo/machine_type:
-
 # This is only supported on ppc64 currently.  This entry sets up the console=
 # boot parameters required for sending the output to the appropriate console.
 # example:

diff --git a/livecd/cdtar/yaboot-1.3.13-cdtar.tar.bz2 
b/livecd/cdtar/yaboot-1.3.13-cdtar.tar.bz2
deleted file mode 100644
index a4bec73b..
Binary files a/livecd/cdtar/yaboot-1.3.13-cdtar.tar.bz2 and /dev/null differ

diff --git a/targets/support/bootloader-setup.sh 
b/targets/support/bootloader-setup.sh
index e4735782..68e1355e 100755
--- a/targets/support/bootloader-setup.sh
+++ b/targets/support/bootloader-setup.sh
@@ -93,7 +93,6 @@ case ${clst_hostarch} in
for x in ${clst_boot_kernel}
do
eval 
"clst_kernel_console=\$clst_boot_kernel_${x}_console"
-   eval 
"clst_kernel_machine_type=\$clst_boot_kernel_${x}_machine_type"
eval custom_kopts=\$${x}_kernelopts
 
echo "menuentry 'Boot LiveCD (kernel: ${x})' --class 
gnu-linux --class os {"  >> ${iacfg}

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: 671ba2f22c9ac335c5f24d69d75b75a1050f543d
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Mar 28 00:52:57 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Mar 28 07:34:51 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=671ba2f2

catalyst: Trim annoying warning

> WARNING : You have been warned.

is awfully silly.

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d2b4cd94..0a825700 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1068,11 +1068,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
print_traceback=True)
 
log.warning(
-   'Overriding certain env variables may 
cause catastrophic failure.\n'
-   'If your build fails look here first as 
the possible problem.\n'
-   'Catalyst assumes you know what you are 
doing when setting these variables.\n'
-   'Catalyst Maintainers use VERY minimal 
envscripts, if used at all.\n'
-   'You have been warned.')
+   'env variables in catalystrc may cause 
catastrophic failure.\n'
+   'If your build fails look here first as 
the possible problem.')
 
shutil.copy(self.settings['envscript'],
self.settings['chroot_path'] + 
'/tmp/envscript')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, targets/stage1/, bin/

[Matt Turner]

commit: 911c083c92587de5a2998c453b33e4890b4a07e3
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Mar 28 07:00:58 2020 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Mar 28 07:35:39 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=911c083c

catalyst: Use python3 shebangs

Signed-off-by: Matt Turner  gentoo.org>

 bin/catalyst| 2 +-
 bin/catalyst.git| 2 +-
 bin/pylint  | 2 +-
 catalyst/base/resume.py | 2 +-
 targets/stage1/build.py | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/bin/catalyst b/bin/catalyst
index 228a913e..97cbb0f9 100755
--- a/bin/catalyst
+++ b/bin/catalyst
@@ -1,4 +1,4 @@
-#!/usr/bin/python -OO
+#!/usr/bin/python3 -OO
 
 import sys
 

diff --git a/bin/catalyst.git b/bin/catalyst.git
index 733ade15..9b3deaa1 100755
--- a/bin/catalyst.git
+++ b/bin/catalyst.git
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 # -*- coding: utf-8 -*-
 # Copyright 1999-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2

diff --git a/bin/pylint b/bin/pylint
index 6c718134..98108f5f 100755
--- a/bin/pylint
+++ b/bin/pylint
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 # Copyright 1999-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 

diff --git a/catalyst/base/resume.py b/catalyst/base/resume.py
index 4234f8b9..6a4ebdfa 100644
--- a/catalyst/base/resume.py
+++ b/catalyst/base/resume.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 
 '''resume.py
 

diff --git a/targets/stage1/build.py b/targets/stage1/build.py
index fa4fd136..4373126b 100755
--- a/targets/stage1/build.py
+++ b/targets/stage1/build.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 
 import os
 import sys

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Rick Farina]

commit: 8441372c285726de85c71a83035a9e11cc47f037
Author: Rick Farina (Zero_Chaos)  gentoo  org>
AuthorDate: Mon Jan 27 18:50:42 2020 +
Commit: Rick Farina  gentoo  org>
CommitDate: Mon Jan 27 18:52:55 2020 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=8441372c

fix /var/tmp/portage not being forced into a tmpfs

2e91db10 accidently revealed that src tmpfs was being being used when we
meant "this might be a tmpfs, or it might not".  Fix preexisting code to
be more readable and adjust src tmpfs to be what a sane person would
expect

Signed-off-by: Rick Farina (Zero_Chaos)  gentoo.org>

 catalyst/base/stagebase.py | 6 +++---
 catalyst/defaults.py   | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 85e30450..d29b48fc 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -973,13 +973,13 @@ class StageBase(TargetBase, ClearBase, GenBase):
if "snapcache" in self.settings["options"] and x == 
"portdir":
self.snapcache_lock.read_lock()
_cmd = None
-   if src == "tmpfs":
+   if src == "maybe_tmpfs":
if "var_tmpfs_portage" in self.settings:
_cmd = ['mount', '-t', 'tmpfs',
'-o', 'size=' + 
self.settings['var_tmpfs_portage'] + 'G',
src, target]
-   else:
-   _cmd = ['mount', '-t', 'tmpfs', src, 
target]
+   elif src == "tmpfs":
+   _cmd = ['mount', '-t', 'tmpfs', src, target]
else:
if os.uname()[0] == "FreeBSD":
if src == "/dev":

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index cb5f84e3..705ae931 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -99,7 +99,7 @@ SOURCE_MOUNT_DEFAULTS = {
"devpts": "/dev/pts",
"distdir": DISTDIR[:],
"portdir": PORTDIR[:],
-   "port_tmpdir": "tmpfs",
+   "port_tmpdir": "maybe_tmpfs",
"proc": "/proc",
"shm": "shmfs",
"run": "tmpfs",

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: f06836c23696fb6eb4bcae3b343129e2d4b4d1ac
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Oct 19 23:11:07 2019 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Oct 19 23:11:07 2019 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f06836c2

Improve debugging messages

- Move start message out of loop
- Print loaded module name in the loop

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index c68c278d..2ad53381 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -75,10 +75,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.subarchmap = {}
machinemap = {}
arch_dir = self.settings["archdir"] + "/"
+   log.debug("Begin loading arch modules...")
for x in [
x[:-3] for x in os.listdir(arch_dir) if 
x.endswith(".py")
and x != "__init__.py"]:
-   log.debug("Begin loading arch modules...")
+   log.debug("\tLoading %s", x)
try:
fh = open(arch_dir + x + ".py")
# This next line loads the plugin as a module 
and

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: f08bb56fb240da5860418b47b726a2954587a85a
Author: Michael Everitt  2e0cer  net>
AuthorDate: Mon Oct 14 20:14:18 2019 +
Commit: Matt Turner  gentoo  org>
CommitDate: Mon Oct 14 20:14:50 2019 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f08bb56f

Demote warning about missing/failed unmounts to 'notice'

Signed-off-by: Michael Everitt  2e0cer.net>
Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 3c274587..c68c278d 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1001,11 +1001,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
for x in myrevmounts:
target = normpath(mypath + self.target_mounts[x])
if not os.path.exists(target):
-   log.warning('%s does not exist. Skipping', 
target)
+   log.notice('%s does not exist. Skipping', 
target)
continue
 
if not ismount(target):
-   log.warning('%s is not a mount point. 
Skipping', target)
+   log.notice('%s is not a mount point. Skipping', 
target)
continue
 
try:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Matt Turner]

commit: f97a050114eeef0550eec0052c526706a6bf3f0a
Author: Matt Turner  gentoo  org>
AuthorDate: Sat Oct  5 22:30:38 2019 +
Commit: Matt Turner  gentoo  org>
CommitDate: Sat Oct  5 22:30:38 2019 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=f97a0501

Warn if skipping unbinding a path

Signed-off-by: Matt Turner  gentoo.org>

 catalyst/base/stagebase.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 4e81a9b1..3c274587 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1001,9 +1001,11 @@ class StageBase(TargetBase, ClearBase, GenBase):
for x in myrevmounts:
target = normpath(mypath + self.target_mounts[x])
if not os.path.exists(target):
+   log.warning('%s does not exist. Skipping', 
target)
continue
 
if not ismount(target):
+   log.warning('%s is not a mount point. 
Skipping', target)
continue
 
try:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Rick Farina]

commit: 44fcaf16a9eb5b17b6f0b6a61647c97ccc92cd25
Author: Rick Farina (Zero_Chaos)  gentoo  org>
AuthorDate: Wed Nov  7 19:47:16 2018 +
Commit: Rick Farina  gentoo  org>
CommitDate: Wed Nov  7 19:49:47 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=44fcaf16

allow custom COMMON_FLAGS

instead of forcing the user to override *FLAGS, just let them set
COMMON_FLAGS directly

Signed-off-by: Rick Farina (Zero_Chaos)  gentoo.org>

 catalyst/base/stagebase.py | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 7665834c..54a1f389 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -39,7 +39,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
"rel_type", "profile", "snapshot", "source_subpath",
"portage_confdir", "portage_prefix", "portage_overlay",
"cflags", "cxxflags", "fcflags", "fflags", "ldflags", 
"asflags",
-   "cbuild", "hostuse", "catalyst_use",
+   "common_flags", "cbuild", "hostuse", "catalyst_use",
"distcc_hosts", "makeopts", "pkgcache_path", 
"kerncache_path",
"compression_mode", "decompression_mode"])
 
@@ -305,6 +305,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
if "ASFLAGS" in self.makeconf:
self.settings["ASFLAGS"] = self.makeconf["ASFLAGS"]
 
+   def override_common_flags(self):
+   if "COMMON_FLAGS" in self.makeconf:
+   self.settings["COMMON_FLAGS"] = 
self.makeconf["COMMON_FLAGS"]
+
def set_install_mask(self):
if "install_mask" in self.settings:
if not isinstance(self.settings['install_mask'], str):
@@ -1042,6 +1046,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.override_fflags()
self.override_ldflags()
self.override_asflags()
+   self.override_common_flags()
if "autoresume" in self.settings["options"] \
and self.resume.is_enabled("chroot_setup"):
log.notice('Resume point detected, skipping 
chroot_setup operation...')

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 9cfdfd6076471edbc422ba2e73dc103a93c9e917
Author: Michael Everitt  iee  org>
AuthorDate: Fri Sep  7 17:11:34 2018 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Mon Oct  1 16:21:36 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9cfdfd60

Update parsing of 'makeopts' in specs to permit multiple arguments

edit by Brian Dolbec  gentoo.org>:
Make elif an else so it will accept any iterable.

 catalyst/base/stagebase.py | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 6d78f937..7665834c 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1420,7 +1420,12 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.env[varname] = 
"false"
 
if "makeopts" in self.settings:
-   self.env["MAKEOPTS"] = self.settings["makeopts"]
+   if isinstance(self.settings["makeopts"], str):
+   self.env["MAKEOPTS"] = self.settings["makeopts"]
+   else:
+   # ensure makeopts is a string
+   self.env["MAKEOPTS"] = ' 
'.join(self.settings["makeopts"])
+
log.debug('setup_environment(); env = %r', self.env)
 
def run(self):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/arch/

[Ben Kohler]

commit: 14a5425fdf918f7299706196a32ee3e722315376
Author: Ben Kohler  gentoo  org>
AuthorDate: Wed Sep 19 15:42:11 2018 +
Commit: Ben Kohler  gentoo  org>
CommitDate: Wed Sep 19 15:53:12 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=14a5425f

First implementation of COMMON_FLAGS for make.conf

This introduces a new variable COMMON_FLAGS for make.conf.  In catalyst,
individual arches' arch/${arch}.py files will set COMMON_FLAGS and this
will be expanded to set default values for otherwise unset {C,CXX,F,FC}FLAGS.

Eg:

COMMON_FLAGS="-O2 -pipe"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"

Individual (sub)arches can choose to set specific values for those other
*FLAGS and then they will not be set to ${COMMON_FLAGS}.

Implementation thanks to dwfreed.  Should resolve the following bugs:

https://bugs.gentoo.org/575446
https://bugs.gentoo.org/665758
https://bugs.gentoo.org/654080

 catalyst/arch/alpha.py |  18 +++
 catalyst/arch/amd64.py |  14 ++---
 catalyst/arch/arm.py   |  28 +-
 catalyst/arch/arm64.py |   3 +-
 catalyst/arch/hppa.py  |   9 ++--
 catalyst/arch/ia64.py  |   3 +-
 catalyst/arch/m68k.py  |   4 +-
 catalyst/arch/mips.py  | 132 ++---
 catalyst/arch/powerpc.py   |  36 ++---
 catalyst/arch/s390.py  |   4 +-
 catalyst/arch/sh.py|  24 -
 catalyst/arch/sparc.py |   4 +-
 catalyst/arch/x86.py   |  30 +--
 catalyst/base/stagebase.py |  14 ++---
 14 files changed, 160 insertions(+), 163 deletions(-)

diff --git a/catalyst/arch/alpha.py b/catalyst/arch/alpha.py
index a4e7cafa..813b0020 100644
--- a/catalyst/arch/alpha.py
+++ b/catalyst/arch/alpha.py
@@ -7,55 +7,55 @@ class generic_alpha(builder.generic):
"abstract base class for all alpha builders"
def __init__(self,myspec):
builder.generic.__init__(self,myspec)
-   self.settings["CFLAGS"]="-mieee -pipe"
+   self.settings["COMMON_FLAGS"]="-mieee -pipe"
 
 class arch_alpha(generic_alpha):
"builder class for generic alpha (ev4+)"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev4"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev4"
self.settings["CHOST"]="alpha-unknown-linux-gnu"
 
 class arch_ev4(generic_alpha):
"builder class for alpha ev4"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev4"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev4"
self.settings["CHOST"]="alphaev4-unknown-linux-gnu"
 
 class arch_ev45(generic_alpha):
"builder class for alpha ev45"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev45"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev45"
self.settings["CHOST"]="alphaev45-unknown-linux-gnu"
 
 class arch_ev5(generic_alpha):
"builder class for alpha ev5"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev5"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev5"
self.settings["CHOST"]="alphaev5-unknown-linux-gnu"
 
 class arch_ev56(generic_alpha):
"builder class for alpha ev56 (ev5 plus BWX)"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev56"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev56"
self.settings["CHOST"]="alphaev56-unknown-linux-gnu"
 
 class arch_pca56(generic_alpha):
"builder class for alpha pca56 (ev5 plus BWX & MAX)"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=pca56"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=pca56"
self.settings["CHOST"]="alphaev56-unknown-linux-gnu"
 
 class arch_ev6(generic_alpha):
"builder class for alpha ev6"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev6"
+   self.settings["COMMON_FLAGS"]+=" -O2 -mcpu=ev6"
self.settings["CHOST"]="alphaev6-unknown-linux-gnu"
self.settings["HOSTUSE"]=["ev6"]
 
@@ -63,7 +63,7 @@ class arch_ev67(generic_alpha):
"builder class for alpha ev67 (ev6 plus CIX)"
def __init__(self,myspec):
generic_alpha.__init__(self,myspec)
-   self.settings["CFLAGS"]+=" -O2 -mcpu=ev67"
+   self.settings["COMMON_FLAGS"]+=" -O2 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: c142d2f5c70d254d197343c2d86043a0e8616a46
Author: Brian Dolbec dolsen  gentoo  org>
AuthorDate: Fri Jul 20 16:29:26 2018 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Fri Jul 20 16:36:15 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=c142d2f5

stagebase.py: Relocate ccache location away from /root/

Signed-off-by: Brian Dolbec dolsen  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index e98a289f..d30ed5ce 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -253,7 +253,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
ccdir = os.environ["CCACHE_DIR"]
del os.environ["CCACHE_DIR"]
else:
-   ccdir = "/root/.ccache"
+   ccdir = "/var/tmp/ccache"
if not os.path.isdir(ccdir):
raise CatalystError(
"Compiler cache support can't be 
enabled (can't find " + ccdir+")")

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: bb1aad33b000e39689a66243bbb446c6c25c1e88
Author: Brian Dolbec dolsen  gentoo  org>
AuthorDate: Fri Jul 20 16:30:19 2018 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Fri Jul 20 16:36:15 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=bb1aad33

stagebase.py: Remove /root/* from default cleanables

This was not something that should need to be cleaned.
With ccache removed, it is safe to leave alone.
Livecd images may have a .bashrc to start to start the desktop, etc..

Signed-off-by: Brian Dolbec dolsen  gentoo.org>

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index d30ed5ce..a66229c2 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -465,7 +465,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
def set_cleanables(self):
self.settings["cleanables"] = ["/etc/resolv.conf", 
"/var/tmp/*", "/tmp/*",
-   "/root/*", self.settings["repo_basedir"] + "/" +
+   self.settings["repo_basedir"] + "/" +
self.settings["repo_name"]]
 
def set_snapshot_path(self):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Richard Farina]

commit: 85646148865ae50e4e5702f2c707f816060ac5cf
Author: Rick Farina (Zero_Chaos)  gentoo  org>
AuthorDate: Thu Feb 22 03:53:39 2018 +
Commit: Richard Farina  gentoo  org>
CommitDate: Thu Feb 22 03:53:39 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=85646148

fix copypasta, sorry

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index c249295a..e98a289f 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1231,7 +1231,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
if os.path.exists(overlay):
clear_path(overlay)
 
-   if "sticky-config" not in self.settings["options"]):
+   if "sticky-config" not in self.settings["options"]:
# re-write the make.conf to be sure it is clean
self.write_make_conf(setup=False)
 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Richard Farina]

commit: 6dce47a175ec2508eac1e04e5df42efc58baf398
Author: Rick Farina (Zero_Chaos)  gentoo  org>
AuthorDate: Thu Feb 22 03:44:54 2018 +
Commit: Richard Farina  gentoo  org>
CommitDate: Thu Feb 22 03:44:54 2018 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6dce47a1

do not rewrite make.conf if sticky-config is set

we don't want to re-write the user's make.conf if sticky-config is set,
it may have been modified during fsscript and the user asked for their
config to be preserved

 catalyst/base/stagebase.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index eaed73fc..c249295a 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1231,8 +1231,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
if os.path.exists(overlay):
clear_path(overlay)
 
-   # re-write the make.conf to be sure it is clean
-   self.write_make_conf(setup=False)
+   if "sticky-config" not in self.settings["options"]):
+   # re-write the make.conf to be sure it is clean
+   self.write_make_conf(setup=False)
 
# Clean up old and obsoleted files in /etc
if os.path.exists(self.settings["stage_path"]+"/etc"):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 73219eb1c1878d2dd160ba067c6096ff5aefff95
Author: Yuta SATOH  gmail  com>
AuthorDate: Fri Dec 29 01:55:05 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Fri Dec 29 02:05:15 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=73219eb1

stagebase.py: Update bind() for Gentoo/FreeBSD tmpfs/shmfs # 363577

Bug url: https://bugs.gentoo.org/363577

 catalyst/base/stagebase.py | 27 ++-
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 32cc3df1..eaed73fc 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -967,21 +967,22 @@ class StageBase(TargetBase, ClearBase, GenBase):
if "snapcache" in self.settings["options"] and x == 
"portdir":
self.snapcache_lock.read_lock()
_cmd = None
-   if os.uname()[0] == "FreeBSD":
-   if src == "/dev":
-   _cmd = ['mount', '-t', 'devfs', 'none', 
target]
-   else:
-   _cmd = ['mount_nullfs', src, target]
+   if src == "tmpfs":
+   if "var_tmpfs_portage" in self.settings:
+   _cmd = ['mount', '-t', 'tmpfs',
+   '-o', 'size=' + 
self.settings['var_tmpfs_portage'] + 'G',
+   src, target]
else:
-   if src == "tmpfs":
-   if "var_tmpfs_portage" in self.settings:
-   _cmd = ['mount', '-t', 'tmpfs',
-   '-o', 'size=' + 
self.settings['var_tmpfs_portage'] + 'G',
-   src, target]
-   elif src == "shmfs":
-   _cmd = ['mount', '-t', 'tmpfs', '-o', 
'noexec,nosuid,nodev', 'shm', target]
+   if os.uname()[0] == "FreeBSD":
+   if src == "/dev":
+   _cmd = ['mount', '-t', 'devfs', 
'none', target]
+   else:
+   _cmd = ['mount_nullfs', src, 
target]
else:
-   _cmd = ['mount', '--bind', src, target]
+   if src == "shmfs":
+   _cmd = ['mount', '-t', 'tmpfs', 
'-o', 'noexec,nosuid,nodev', 'shm', target]
+   else:
+   _cmd = ['mount', '--bind', src, 
target]
if _cmd:
log.debug('bind(); _cmd = %s', _cmd)
cmd(_cmd, env=self.env, fail_func=self.unbind)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 16cb2fcfa1fac91f877a5f41ddb97eee68208466
Author: Yuta SATOH  gmail  com>
AuthorDate: Fri Dec 29 01:42:12 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Fri Dec 29 02:05:15 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=16cb2fcf

stagebase.py: Update umount cmd, add env=self.env  # 363577

Bug url: https://bugs.gentoo.org/363577

 catalyst/base/stagebase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 8f9a5dbd..32cc3df1 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1002,7 +1002,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
continue
 
try:
-   cmd(['umount', target])
+   cmd(['umount', target], env=self.env)
except CatalystError:
log.warning('First attempt to unmount failed: 
%s', target)
log.warning('Killing any pids still running in 
the chroot')
@@ -1010,7 +1010,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.kill_chroot_pids()
 
try:
-   cmd(['umount', target])
+   cmd(['umount', target], env=self.env)
except CatalystError:
ouch = 1
log.warning("Couldn't umount bind 
mount: %s", target)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: d1dbb555a08951b65c21f900ace740b1a76efa51
Author: Brian Dolbec  gentoo  org>
AuthorDate: Thu Dec 28 20:58:29 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Fri Dec 29 00:34:30 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d1dbb555

stagebase.py: Fix cleaning portage_prefix errors

Remove outer [ ] pair in the pjoin.
Remove leading / in the "/etc/portage/package.%s"  Ben Kohler  
gmail.com>
Add "accept_keywords" package.* entry to clean. Ben Kohler  
gmail.com>
Add extra logging info.
Change from using chroot_path to destpath in case it is different (stage1).
Use normpath() on some other cleanables to remove "//"'s

 catalyst/base/stagebase.py | 17 ++---
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index a6a32f5a..8f9a5dbd 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1207,7 +1207,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:
for x in self.settings["cleanables"]:
log.notice('Cleaning chroot: %s', x)
-   clear_path(self.settings["destpath"] + x)
+   clear_path(normpath(self.settings["destpath"] + 
x))
 
# Put /etc/hosts back into place
hosts_file = self.settings['chroot_path'] + '/etc/hosts'
@@ -1217,15 +1217,18 @@ class StageBase(TargetBase, ClearBase, GenBase):
# optionally clean up portage configs
if ("portage_prefix" in self.settings and
"sticky-config" not in self.settings["options"]):
-   for _dir in "keywords", "mask", "unmask", "use":
-   target = pjoin([self.settings['chroot_path'],
-   "/etc/portage/package.%s" % _dir,
-   self.settings["portage_prefix"]])
+   log.debug("clean(), portage_preix = %s, no 
sticky-config", self.settings["portage_prefix"])
+   for _dir in "accept_keywords", "keywords", "mask", 
"unmask", "use":
+   target = pjoin(self.settings["destpath"],
+   "etc/portage/package.%s" % _dir,
+   self.settings["portage_prefix"])
+   log.notice("Clearing portage_prefix target: 
%s", target)
clear_path(target)
 
# Remove our overlay
-   if os.path.exists(self.settings["chroot_path"] + 
self.settings["local_overlay"]):
-   clear_path(self.settings["chroot_path"] + 
self.settings["local_overlay"])
+   overlay = normpath(self.settings["chroot_path"] + 
self.settings["local_overlay"])
+   if os.path.exists(overlay):
+   clear_path(overlay)
 
# re-write the make.conf to be sure it is clean
self.write_make_conf(setup=False)

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, doc/, catalyst/

[Robin H. Johnson]

commit: d0fc244d633375faeaf4da81350c9818a1b291f0
Author: Robin H. Johnson  gentoo  org>
AuthorDate: Wed Dec  6 08:45:30 2017 +
Commit: Robin H. Johnson  gentoo  org>
CommitDate: Wed Dec  6 18:05:00 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d0fc244d

Allow unversioned cache directories

The generation of cachedirs up to this point includes the version_stamp
from the specfile.

If the version is being automatically incremented (eg daily builds),
then it generates unique cache directories for each build, and doesn't
reuse old cache.

Example versioned cache directories:
/release/buildroot/amd64-dev/packages/default/stage1-amd64-20171203
/release/buildroot/amd64-dev/packages/default/stage1-amd64-20171204
/release/buildroot/amd64-dev/packages/default/stage1-amd64-20171206

This should be using a common cache directory:
/release/buildroot/amd64-dev/packages/default/stage1-amd64

Signed-off-by: Robin H. Johnson  gentoo.org>
Signed-off-by: Brian Dolbec  gentoo.org>

 catalyst/base/stagebase.py | 23 ++-
 catalyst/main.py   |  8 
 doc/catalyst-config.5.txt  |  9 +
 3 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 7a41973c..a6a32f5a 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -318,9 +318,15 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.settings["target_profile"] = self.settings["profile"]
 
def set_target_subpath(self):
-   self.settings["target_subpath"] = self.settings["rel_type"] + 
"/" + \
-   self.settings["target"] + "-" + 
self.settings["subarch"] + "-" + \
-   self.settings["version_stamp"] + '/'
+   common = self.settings["rel_type"] + "/" + \
+   self.settings["target"] + "-" + 
self.settings["subarch"]
+   self.settings["target_subpath"] = \
+   common + \
+   "-" + self.settings["version_stamp"] + \
+   "/"
+   self.settings["target_subpath_unversioned"] = \
+   common + \
+   "/"
 
def set_source_subpath(self):
if not isinstance(self.settings['source_subpath'], str):
@@ -333,19 +339,26 @@ class StageBase(TargetBase, ClearBase, GenBase):
if not isinstance(self.settings['pkgcache_path'], str):
self.settings["pkgcache_path"] = \
normpath(self.settings["pkgcache_path"])
-   else:
+   elif "versioned_cache" in self.settings["options"]:
self.settings["pkgcache_path"] = \
normpath(self.settings["storedir"] + 
"/packages/" + \
self.settings["target_subpath"] + "/")
+   else:
+   self.settings["pkgcache_path"] = \
+   normpath(self.settings["storedir"] + 
"/packages/" + \
+   self.settings["target_subpath_unversioned"] + 
"/")
 
def set_kerncache_path(self):
if "kerncache_path" in self.settings:
if not isinstance(self.settings['kerncache_path'], str):
self.settings["kerncache_path"] = \

normpath(self.settings["kerncache_path"])
-   else:
+   elif "versioned_cache" in self.settings["options"]:
self.settings["kerncache_path"] = 
normpath(self.settings["storedir"] + \
"/kerncache/" + self.settings["target_subpath"])
+   else:
+   self.settings["kerncache_path"] = 
normpath(self.settings["storedir"] + \
+   "/kerncache/" + 
self.settings["target_subpath_unversioned"])
 
def set_target_path(self):
self.settings["target_path"] = 
normpath(self.settings["storedir"] + \

diff --git a/catalyst/main.py b/catalyst/main.py
index b8ae662d..3a428d3a 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -190,6 +190,14 @@ $ catalyst -f stage1-specfile.spec"""
group.add_argument('-T', '--purgetmponly',
default=False, action='store_true',
help='clear tmp dirs and autoresume flags and exit')
+group.add_argument('--versioned-cachedir',
+dest='versioned_cachedir', action='store_true',
+help='use stage version on cache directory name')
+group.add_argument('--unversioned-cachedir',
+dest='versioned_cachedir', action='store_false',
+help='do not use stage version on cache directory name')
+

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 753d18b58afd38cb282b405e75406e20dd9c14be
Author: Brian Dolbec  gentoo  org>
AuthorDate: Thu Mar  9 05:38:28 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Thu Mar  9 05:38:28 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=753d18b5

base/stagebase.py: Fix 48856f7e10e due to a change to AutoResume.get()

Changing the formatting to match in the log messages broke the hash matching 
preventing
them from ever matching.

 catalyst/base/stagebase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 2557fe8..b857a64 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -744,7 +744,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
# check seed source
if os.path.isfile(self.settings["source_path"]) and not 
invalid_chroot:
-   if self.settings["source_path_hash"] == 
clst_unpack_hash:
+   if 
self.settings["source_path_hash"].replace("\n", " ") == clst_unpack_hash:
# Seed tarball has not changed, chroot 
is valid
_unpack = False
invalid_chroot = False

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 48856f7e10e04d1720b35d32a6151ff2e0d50b31
Author: Brian Dolbec  gentoo  org>
AuthorDate: Thu Mar  9 05:04:47 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Thu Mar  9 05:04:47 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=48856f7e

base/stagebase.py: Cleanup and fix the autoresume validation in the unpack()

The previous logic was convoluted and had many errors, most notably, not 
checking
the chroot is a directory, but instead was checking the seed source.
I broke up and simplified the logic into two sections, one checks the chroot 
for a valid resume
point, the other checks for seed tarball changes which would invalidate the 
existing chroot.

Also the non-autoresume code block code was mostly not needed and was also 
simplified.

 catalyst/base/resume.py|  2 +-
 catalyst/base/stagebase.py | 92 ++
 2 files changed, 45 insertions(+), 49 deletions(-)

diff --git a/catalyst/base/resume.py b/catalyst/base/resume.py
index 70d9a4f..b210073 100644
--- a/catalyst/base/resume.py
+++ b/catalyst/base/resume.py
@@ -75,7 +75,7 @@ class AutoResume(object):
with open(self._points[point], 'r') as myf:
data = myf.read()
if data and no_lf:
-   data = data.replace('\n', '')
+   data = data.replace('\n', ' ')
except OSError as e:
log.error('AutoResumeError: %s', e)
return None

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index a6233b2..2557fe8 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -416,6 +416,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.settings["source_subpath"] + "/")):
self.settings["source_path"] = 
normpath(self.settings["storedir"] +
"/tmp/" + self.settings["source_subpath"] + "/")
+   log.debug("source_subpath is: %s", 
self.settings["source_path"])
else:
log.debug('Checking source path existence and '
'get the final filepath. subpath: %s',
@@ -693,10 +694,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
raise CatalystError("Unable to 
auto-unbind " + target)
 
def unpack(self):
-   _unpack = True
 
clst_unpack_hash = self.resume.get("unpack")
 
+   # Set up all unpack info settings
unpack_info = self.decompressor.create_infodict(
source = self.settings["source_path"],
destination = self.settings["chroot_path"],
@@ -721,56 +722,51 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:
# No SEEDCACHE, use tar
unpack_info['source'] = 
file_check(unpack_info['source'])
-   # endif "seedcache"
+   # end of unpack_info settings
 
+   # set defaults,
+   # only change them if the resume point is proven to be good
+   _unpack = True
+   invalid_chroot = True
+   # Begin autoresume validation
if "autoresume" in self.settings["options"]:
-   if os.path.isdir(self.settings["source_path"]) \
-   and self.resume.is_enabled("unpack"):
-   # Autoresume is valid, SEEDCACHE is valid
-   _unpack = False
-   invalid_snapshot = False
-   log.notice('Resume point "unpack" valid...')
-
-   elif os.path.isfile(self.settings["source_path"]) \
-   and self.settings["source_path_hash"] == 
clst_unpack_hash:
-   # Autoresume is valid, tarball is valid
-   _unpack = False
-   invalid_snapshot = False
-   log.notice('Resume point "source_path_hash" 
valid...')
-
-   elif os.path.isdir(self.settings["source_path"]) \
-   and self.resume.is_disabled("unpack"):
-   # Autoresume is invalid, SEEDCACHE
-   _unpack = True
-   invalid_snapshot = True
-   log.notice('Resume point "unpack is disabled" 
is True, invalidating snapshot... :(')
-
-   elif os.path.isfile(self.settings["source_path"]) \
-   and self.settings["source_path_hash"] != 
clst_unpack_hash:
-   # 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: faa1349963f41c39c4fbf5da5445c526fab3430f
Author: Brian Dolbec  gentoo  org>
AuthorDate: Wed Mar  8 23:13:59 2017 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Wed Mar  8 23:14:50 2017 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=faa13499

base/stagebase.py: Add additional log messages for autoresume point invalidation

 catalyst/base/stagebase.py | 4 
 1 file changed, 4 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 0d36e33..a6233b2 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -729,24 +729,28 @@ class StageBase(TargetBase, ClearBase, GenBase):
# Autoresume is valid, SEEDCACHE is valid
_unpack = False
invalid_snapshot = False
+   log.notice('Resume point "unpack" valid...')
 
elif os.path.isfile(self.settings["source_path"]) \
and self.settings["source_path_hash"] == 
clst_unpack_hash:
# Autoresume is valid, tarball is valid
_unpack = False
invalid_snapshot = False
+   log.notice('Resume point "source_path_hash" 
valid...')
 
elif os.path.isdir(self.settings["source_path"]) \
and self.resume.is_disabled("unpack"):
# Autoresume is invalid, SEEDCACHE
_unpack = True
invalid_snapshot = True
+   log.notice('Resume point "unpack is disabled" 
is True, invalidating snapshot... :(')
 
elif os.path.isfile(self.settings["source_path"]) \
and self.settings["source_path_hash"] != 
clst_unpack_hash:
# Autoresume is invalid, tarball
_unpack = True
invalid_snapshot = True
+   log.notice('Resume point "source_path_hash" is 
invalid... :(')
unpack_info['source'] = 
file_check(unpack_info['source'])
 
else:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 6b4f6861ad9847fe5c1be5c60167578f2404e11d
Author: Brian Dolbec  gentoo  org>
AuthorDate: Mon Sep 19 03:27:14 2016 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Mon Sep 19 03:27:14 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=6b4f6861

base/stagebase.py: Add 2 more debug logs for arch loading modules.

 catalyst/base/stagebase.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 904103f..68ccb79 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -74,6 +74,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
machinemap = {}
arch_dir = self.settings["archdir"] + "/"
for x in [x[:-3] for x in os.listdir(arch_dir) if 
x.endswith(".py") and x != "__init__.py"]:
+   log.debug("Begin loading arch modules...")
try:
fh=open(arch_dir + x + ".py")
# This next line loads the plugin as a module 
and assigns it to
@@ -94,6 +95,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
# the dir should load just fine. If it doesn't, 
it's probably a
# syntax error in the module
log.warning("Can't find/load %s.py plugin in 
%s", x, arch_dir)
+   log.debug("Loaded arch module: %s", self.archmap[x])
 
if "chost" in self.settings:
hostmachine = self.settings["chost"].split("-")[0]

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 8cf9e10774576c75777633e79bd2173c291e95cc
Author: Brian Dolbec  gentoo  org>
AuthorDate: Sat Jun 25 14:59:20 2016 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Sat Jun 25 14:59:20 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=8cf9e107

stagebase.py: Add support for a single pkg to be listed in the specs 
stage?/packages

The code previously only supported packages as a list, but teh spec parser 
always makes a 
single item into a string, only multiple entries were a list.
This allows the packages list to be either a single string or a list of strings.

 catalyst/base/stagebase.py | 10 +++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 6695ac4..904103f 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1473,14 +1473,18 @@ class StageBase(TargetBase, ClearBase, GenBase):
log.notice('Resume point detected, skipping 
build_packages operation...')
else:
if self.settings["spec_prefix"]+"/packages" in 
self.settings:
+   target_pkgs = self.settings["spec_prefix"] + 
'/packages'
if "autoresume" in self.settings["options"] \
and 
self.resume.is_enabled("build_packages"):
log.notice('Resume point detected, 
skipping build_packages operation...')
else:
+   command = 
[self.settings['controller_file'], 'build_packages']
+   if 
isinstance(self.settings[target_pkgs], str):
+   
command.append(self.settings[target_pkgs])
+   else:
+   
command.extend(self.settings[target_pkgs])
try:
-   
cmd([self.settings['controller_file'], 'build_packages'] +
-   
self.settings[self.settings["spec_prefix"] + '/packages'],
-   env=self.env)
+   cmd(command, env=self.env)

fileutils.touch(build_packages_resume)

self.resume.enable("build_packages")
except CatalystError:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/targets/

[Mike Frysinger]

commit: 415c65624ff62f5ab139666cd5447bc35e1e9e8b
Author: Mike Frysinger  gentoo  org>
AuthorDate: Fri May 20 05:42:43 2016 +
Commit: Mike Frysinger  gentoo  org>
CommitDate: Fri May 20 05:42:43 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=415c6562

replace ad-hoc implementations of clear_dir

A few places copy & paste the logic we have in the clear_dir func.
Punt them with a simple call to clear_dir.

 catalyst/base/stagebase.py   |  9 +
 catalyst/targets/netboot2.py | 10 ++
 catalyst/targets/snapshot.py | 20 ++--
 3 files changed, 5 insertions(+), 34 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 78e5b94..f0f3ba9 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -3,7 +3,6 @@ import os
 import imp
 import shutil
 import sys
-from stat import ST_UID, ST_GID, ST_MODE
 
 from snakeoil import fileutils
 
@@ -1202,13 +1201,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
log.warning('not a directory or 
does not exist, skipping "empty" operation: %s', x)
continue
log.info('Emptying directory %s', x)
-   # stat the dir, delete the dir, 
recreate the dir and set
-   # the proper perms and ownership
-   mystat=os.stat(myemp)
-   shutil.rmtree(myemp)
-   ensure_dirs(myemp, mode=0o755)
-   
os.chown(myemp,mystat[ST_UID],mystat[ST_GID])
-   os.chmod(myemp,mystat[ST_MODE])
+   clear_dir(myemp)
self.resume.enable("empty")
 
def remove(self):

diff --git a/catalyst/targets/netboot2.py b/catalyst/targets/netboot2.py
index da856ba..87dada3 100644
--- a/catalyst/targets/netboot2.py
+++ b/catalyst/targets/netboot2.py
@@ -4,12 +4,10 @@ netboot target, version 2
 # NOTE: That^^ docstring has influence catalyst-spec(5) man page generation.
 
 import os
-import shutil
-from stat import ST_UID, ST_GID, ST_MODE
 
 from catalyst import log
 from catalyst.support import (CatalystError, normpath, cmd)
-from catalyst.fileops import ensure_dirs, clear_path
+from catalyst.fileops import (ensure_dirs, clear_dir, clear_path)
 
 from catalyst.base.stagebase import StageBase
 
@@ -152,11 +150,7 @@ class netboot2(StageBase):
log.info('Emptying directory %s', x)
# stat the dir, delete the dir, 
recreate the dir and set
# the proper perms and ownership
-   mystat=os.stat(myemp)
-   shutil.rmtree(myemp)
-   ensure_dirs(myemp, mode=0o755)
-   
os.chown(myemp,mystat[ST_UID],mystat[ST_GID])
-   os.chmod(myemp,mystat[ST_MODE])
+   clear_dir(myemp)
self.resume.enable("empty")
 
def set_action_sequence(self):

diff --git a/catalyst/targets/snapshot.py b/catalyst/targets/snapshot.py
index 7ba94b2..8a9acdd 100644
--- a/catalyst/targets/snapshot.py
+++ b/catalyst/targets/snapshot.py
@@ -2,17 +2,13 @@
 Snapshot target
 """
 
-import os
-import shutil
-from stat import ST_UID, ST_GID, ST_MODE
-
 from DeComp.compress import CompressMap
 
 from catalyst import log
 from catalyst.support import normpath, cmd
 from catalyst.base.targetbase import TargetBase
 from catalyst.base.genbase import GenBase
-from catalyst.fileops import ensure_dirs
+from catalyst.fileops import (clear_dir, ensure_dirs)
 
 
 class snapshot(TargetBase, GenBase):
@@ -102,16 +98,4 @@ class snapshot(TargetBase, GenBase):
log.info('Cleaning up ...')
 
def purge(self):
-   myemp=self.settings["tmp_path"]
-   if os.path.isdir(myemp):
-   log.notice('Emptying directory %s', myemp)
-   # stat the dir, delete the dir, recreate the dir and set
-   # the proper perms and ownership
-   mystat=os.stat(myemp)
-   # There's no easy way to change flags recursively in 
python
-   if os.uname()[0] == "FreeBSD":
-   os.system("chflags -R noschg "+myemp)
-   shutil.rmtree(myemp)
-   ensure_dirs(myemp, mode=0o755)
-   os.chown(myemp,mystat[ST_UID],mystat[ST_GID])
-   os.chmod(myemp,mystat[ST_MODE])
+   clear_dir(self.settings['tmp_path'])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Mike Frysinger]

commit: 64255565bf45bd07a66405d876bbbe58b7a0c441
Author: Mike Frysinger  gentoo  org>
AuthorDate: Fri May 20 05:03:34 2016 +
Commit: Mike Frysinger  gentoo  org>
CommitDate: Fri May 20 05:31:57 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=64255565

use native code in more places for file ops

Rather than shell out to cp/rm/mv, use python native code to do things.

 catalyst/base/stagebase.py | 39 +++
 1 file changed, 15 insertions(+), 24 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 2009ab6..0b25516 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -1013,8 +1013,7 @@ class StageBase(TargetBase, ClearBase, GenBase):
else:
log.notice('Setting up chroot...')
 
-   cmd("cp /etc/resolv.conf " + 
self.settings["chroot_path"] + "/etc/",
-   env=self.env)
+   shutil.copy('/etc/resolv.conf', 
self.settings['chroot_path'] + '/etc/')
 
# Copy over the envscript, if applicable
if "envscript" in self.settings:
@@ -1030,18 +1029,15 @@ class StageBase(TargetBase, ClearBase, GenBase):
'Catalyst Maintainers use VERY minimal 
envscripts, if used at all.\n'
'You have been warned.')
 
-   cmd("cp "+self.settings["envscript"]+" "+\
-   
self.settings["chroot_path"]+"/tmp/envscript",\
-   env=self.env)
+   shutil.copy(self.settings['envscript'],
+   self.settings['chroot_path'] + 
'/tmp/envscript')
 
# Copy over /etc/hosts from the host in case there are 
any
# specialties in there
-   if 
os.path.exists(self.settings["chroot_path"]+"/etc/hosts"):
-   cmd("mv 
"+self.settings["chroot_path"]+"/etc/hosts "+\
-   
self.settings["chroot_path"]+"/etc/hosts.catalyst",\
-   env=self.env)
-   cmd("cp /etc/hosts 
"+self.settings["chroot_path"]+"/etc/hosts",\
-   env=self.env)
+   hosts_file = self.settings['chroot_path'] + '/etc/hosts'
+   if os.path.exists(hosts_file):
+   os.rename(hosts_file, hosts_file + '.catalyst')
+   shutil.copy('/etc/hosts', hosts_file)
 
# Modify and write out make.conf (for the chroot)
makepath = normpath(self.settings["chroot_path"] +
@@ -1162,10 +1158,9 @@ class StageBase(TargetBase, ClearBase, GenBase):
clear_path(self.settings["destpath"] + x)
 
# Put /etc/hosts back into place
-   if 
os.path.exists(self.settings["chroot_path"]+"/etc/hosts.catalyst"):
-   cmd("mv -f 
"+self.settings["chroot_path"]+"/etc/hosts.catalyst "+\
-   self.settings["chroot_path"]+"/etc/hosts",\
-   env=self.env)
+   hosts_file = self.settings['chroot_path'] + '/etc/hosts'
+   if os.path.exists(hosts_file + '.catalyst'):
+   os.rename(hosts_file + '.catalyst', hosts_file)
 
# Remove our overlay
if os.path.exists(self.settings["chroot_path"] + 
self.settings["local_overlay"]):
@@ -1565,11 +1560,8 @@ class StageBase(TargetBase, ClearBase, GenBase):
env=self.env)
 
if "boot/kernel/"+kname+"/initramfs_overlay" in self.settings:
-   if os.path.exists(self.settings["chroot_path"]+\
-   "/tmp/initramfs_overlay/"):
-   log.notice('Cleaning up temporary overlay dir')
-   cmd("rm -R "+self.settings["chroot_path"]+\
-   "/tmp/initramfs_overlay/",env=self.env)
+   log.notice('Cleaning up temporary overlay dir')
+   clear_dir(self.settings['chroot_path'] + 
'/tmp/initramfs_overlay/')
 
self.resume.is_enabled("build_kernel_"+kname)
 
@@ -1586,11 +1578,10 @@ class StageBase(TargetBase, ClearBase, GenBase):
self.settings[key])
 
try:
-   cmd('cp ' + self.settings[key] + ' ' +
-   self.settings['chroot_path'] + 
'/var/tmp/' + kname + '.config',
-   env=self.env)
+   shutil.copy(self.settings[key],
+ 

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/, catalyst/

[Mike Frysinger]

commit: 125ca190d30215de85fc3343c95784e658f75c37
Author: Mike Frysinger  gentoo  org>
AuthorDate: Fri May 20 05:43:14 2016 +
Commit: Mike Frysinger  gentoo  org>
CommitDate: Fri May 20 05:43:14 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=125ca190

replace os.system with cmd

Use the existing cmd() helper for running external programs.

 catalyst/base/stagebase.py | 11 ++-
 catalyst/fileops.py|  4 ++--
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index f0f3ba9..6695ac4 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -967,16 +967,17 @@ class StageBase(TargetBase, ClearBase, GenBase):
if not ismount(target):
continue
 
-   retval=os.system("umount " + target)
-
-   if retval!=0:
+   try:
+   cmd(['umount', target])
+   except CatalystError:
log.warning('First attempt to unmount failed: 
%s', target)
log.warning('Killing any pids still running in 
the chroot')
 
self.kill_chroot_pids()
 
-   retval2 = os.system("umount " + target)
-   if retval2!=0:
+   try:
+   cmd(['umount', target])
+   except CatalystError:
ouch=1
log.warning("Couldn't umount bind 
mount: %s", target)
 

diff --git a/catalyst/fileops.py b/catalyst/fileops.py
index 4b9e200..6971911 100644
--- a/catalyst/fileops.py
+++ b/catalyst/fileops.py
@@ -22,7 +22,7 @@ from snakeoil.osutils import (ensure_dirs as 
snakeoil_ensure_dirs,
 # pylint: enable=unused-import
 
 from catalyst import log
-from catalyst.support import CatalystError
+from catalyst.support import (cmd, CatalystError)
 
 
 def ensure_dirs(path, gid=-1, uid=-1, mode=0o755, minimal=True,
@@ -79,7 +79,7 @@ def clear_dir(target, mode=0o755, chg_flags=False, 
remove=False,
mystat = os.stat(target)
# There's no easy way to change flags recursively in 
python
if chg_flags and os.uname()[0] == "FreeBSD":
-   os.system("chflags -R noschg " + target)
+   cmd(['chflags', '-R', 'noschg', target])
log.debug('shutil.rmtree()')
shutil.rmtree(target)
except Exception:

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Mike Frysinger]

commit: d8c463e167f5be77bedf767646d9485a9a8f4f8f
Author: Mike Frysinger  gentoo  org>
AuthorDate: Thu May 19 19:22:13 2016 +
Commit: Mike Frysinger  gentoo  org>
CommitDate: Thu May 19 19:39:22 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d8c463e1

clear_dir: use kwargs everywhere

Don't pass args by position when clear_args uses optional args.
This makes it easy to screw up the calls when changing the API.

 catalyst/base/clearbase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/catalyst/base/clearbase.py b/catalyst/base/clearbase.py
index 9a4c625..644a385 100644
--- a/catalyst/base/clearbase.py
+++ b/catalyst/base/clearbase.py
@@ -30,13 +30,13 @@ class ClearBase(object):
def clear_chroot(self):
self.chroot_lock.unlock()
log.notice('Clearing the chroot path ...')
-   clear_dir(self.settings["chroot_path"], 0o755, True)
+   clear_dir(self.settings["chroot_path"], mode=0o755, 
chg_flags=True)
 
 
def remove_chroot(self):
self.chroot_lock.unlock()
log.notice('Removing the chroot path ...')
-   clear_dir(self.settings["chroot_path"], 0o755, True, 
remove=True)
+   clear_dir(self.settings["chroot_path"], mode=0o755, 
chg_flags=True, remove=True)
 
 
def clear_packages(self, remove=False):

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 558064f6752e8446414f2e9aaad83b15660cb84a
Author: Brian Dolbec  gentoo  org>
AuthorDate: Thu Mar 24 14:35:57 2016 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Thu Mar 24 14:35:57 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=558064f6

stagebase.py: Change a second unpack_info to use the 
decompressor.create_infodict()

I missed one of the unpack_info definitions in the previous commit.
This eliminates the need to manually add in CompressMap global variables which 
are
automatically added to the info_dict.
Fixes commit: 966bff4dae7ed88aa807282ae299c3c0da1f966d

 catalyst/base/stagebase.py | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index b7fefb5..b6dd08d 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -800,13 +800,12 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
unpack_errmsg="Error unpacking snapshot using mode %(mode)s"
 
-   unpack_info = {
-   'source': self.settings["snapshot_path"],
-   'destination': self.settings["snapshot_cache_path"],
-   'mode': None,
-   'auto-ext': False,
-   'other_options': self.settings["compressor_options"],
-   }
+   unpack_info = self.decompressor.create_infodict(
+   source=self.settings["snapshot_path"],
+   destination=self.settings["snapshot_cache_path"],
+   arch=self.settings["compressor_arch"],
+   other_options=self.settings["compressor_options"],
+   )
 
target_portdir = normpath(self.settings["chroot_path"] +
self.settings["repo_basedir"] + "/" + 
self.settings["repo_name"])

[gentoo-commits] proj/catalyst:master commit in: catalyst/base/

[Brian Dolbec]

commit: 70457ffd8cd98593510d4baeb3010aad2f209065
Author: Brian Dolbec  gentoo  org>
AuthorDate: Wed Mar 23 21:14:33 2016 +
Commit: Brian Dolbec  gentoo  org>
CommitDate: Wed Mar 23 21:14:33 2016 +
URL:https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=70457ffd

stagebase.py: Change unpack_info to use the decompressor.create_infodict()

This eliminates the need to manually add in CompressMap global variables which 
are
automatically added to the info_dict.
Fixes commit: 966bff4dae7ed88aa807282ae299c3c0da1f966d

 catalyst/base/stagebase.py | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index 5e87f44..b7fefb5 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -685,13 +685,12 @@ class StageBase(TargetBase, ClearBase, GenBase):
 
clst_unpack_hash = self.resume.get("unpack")
 
-   unpack_info = {
-   'source': self.settings["source_path"],
-   "destination": self.settings["chroot_path"],
-   'mode': None,
-   'auto-ext': False,
-   'other_options': self.settings["compressor_options"],
-   }
+   unpack_info = self.decompressor.create_infodict(
+   source=self.settings["source_path"],
+   destination=self.settings["chroot_path"],
+   arch=self.settings["compressor_arch"],
+   other_options=self.settings["compressor_options"],
+   )
 
display_msg = (
'Starting %(mode)s from %(source)s\nto '