On Tue, Nov 3, 2020 at 5:56 AM Daniel Cordero <gentoo.catal...@xxoo.ws> wrote: > > On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote: > > The catalyst-auto automation scripts live in a repo separate from > > catalyst. That increases the difficulty of changing catalyst's > > interface, and it doesn't seem to offer any advantages otherwise. > > (Keeping build specs in a separate repo allows them to be updated > > independent of catalyst and that is valuable). Additionally, since the > > primary way catalyst is used is via this automation, it makes sense to > > support this workflow in catalyst directly. > > > > What would be more heavily impacted are those users who may not already > have infra set up to do builds or just starting out using catalyst for > the first time and haven't written their own automation. > > I suggest prioritising the collection of up-to-date documentation, > especially regarding running catalyst manually, since it'll be > completely different to the literature that's currently out there.
I'm a bit unsure what you mean. Do you suggest prioritizing documenting the current method of running catalyst before changing it? > > But to get there, there are some changes to catalyst that I think are > > improvements on their own and simplify the path to integrating > > automation capabilities directly into catalyst. That's what I'd like > > to discuss here. > > > > I'd like to: > > > > 1) Replace the custom .spec file format with TOML > > > > Fine. Aside from the extra quotes and commas, I'd be happy with any well > defined format that can handle strings and lists properly. > > > 2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 -> > > livecd-stage1 -> livecd-stage2) into a single file. I suggest naming > > this a ".build" file. This will also allow us to remove the redundant > > information that currently has to be specified in stage1.spec, > > stage2.spec, stage3.spec, like rel_type, version, profile, etc. It > > also means that we remove the nonsensical ability to change settings > > from one stage to the next that should not change (e.g., rel_type, > > version). > > > > How would a target that depends on a different rel_type work? Forks in > the dependency tree. > > > 3) Add ability to denote which stage builds produce artifacts we care > > about (and want to save and/or upload) and which are just temporary. > > If they're temporary (e.g., a stage1 build) we can delete the artifact > > after the build sequence has no further use of it, and we can skip > > compressing the result, etc. > > > > This feature should (haven't tested) already exist - it's just not > documented. > > compression_mode: rsync > options=['seedcache'] Hah! I was completely unaware of this. Thanks. > or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence. > > > > > To that end, I'm starting by figuring out what I would like the new > > spec file format to look like. Below are some open questions and then > > a strawman new-style spec file. > > > > • The .spec files in releng.git are really templates that are not > > directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would > > be nice if they were directly usable as that would reduce confusion > > from users. > > • Can we make them directly usable? > > • Perhaps we can make catalyst handle the replacements directly? > > • Calculating @TIMESTAMP@ is trivially doable—we do it today (see below) > > Maybe a strftime() template, or even fstring-like tokens? > (e.g. "{year}-{month}-{day}") One goal I have is to make it more transparent what is actually in a particular stage tarball or ISO and along with that to make it easier to reproduce the result. Obviously we'll want to keep the ability to specify a particular version, as you describe, but I think for Gentoo releases we will want to continue using a timestamp that's unambiguously tied to the git SHA1 of gentoo.git as is possible. > > • We could configure @REPO_DIR@ in catalyst.conf and let catalyst > > do the replacement, or we could just make the field relative to some > > path specified in catalyst.conf? > > > > While nice to have, I don't agree with locking users into a particular > repository layout. Can you explain what you mean? I don't know how what I said would require a particular repository layout. Perhaps you're confused by the @REPO_DIR@ name? It is the path to the releng.git repository (containing the .specs and the /etc/portage/ files) on the build machine and is not in any way connected with the ebuild repositories. The name predates my involvement, so don't blame me :) > > • In the current automation scripts, we generate a value for > > @TIMESTAMP@ from the git HEAD used in creating the snapshot. > > • Would be nice to remove the dependence on the squashfs snapshot > > generation—not difficult to do > > > > I have no comment on this. > > > • Can we generate and upload a .build file with replacements done to > > make stage builds more easily reproducible? Seems easy. > > > > These can just be artifacts from the build. Yes, that's what I'm thinking too.