Hi I again did some work on building the cloud images. I did work on how to schedule builds, how to perform builds and how to get data where we need it. The whole thing uses infrastructure Debian provides, the exception is the image release step.
The components are: - salsa.debian.org with GitLab, responsible for storing code, scheduling builds, storing logs and integrity details for eternity and storing image data for a short amount of time. - casulana.debian.org with kvm, responsible for providing the environment to perform builds, perform tests. - Somewhere for storing data with user access. - Some environments may need additional systems (EC2 can only use snapshots of existing disks, so the image release step must run there). All builds are orchestrated by GitLab CI, running on salsa.debian.org. All the build definition resides in the same Git repo as the FAI config and scripts, so as much information as possible is distributed in one place. Builds are split up into jobs, each run independently and depending on another. For now I'd like to define the following jobs: - Build images for all supported environments and dists. On each push, regardless of the branch, a subset of these builds is performed. All the other builds are only performed on the scheduled runs. On casulana we should be able to do a full set of builds in about 10 minutes. Right now the compression of the build result dominates the required cpu time, as it uses xz -9 to get the output down to a usable size for storage. Each build runs in a scratch Docker environment via a special configured GitLab Runner. The builds need access to loop devices, which is by default not allowed. (Yes, I'm aware that neither Docker, nor GitLab Runner, have suitable versions in Debian Stretch.) Some of these builds will currently fail (Azure on Buster, GCE on Buster and Sid), so there status will be ignored. I have a test repo that performs this operation already: https://salsa.debian.org/waldi/fai-cloud-images/ Full build: https://salsa.debian.org/waldi/fai-cloud-images/pipelines/3155 - Tests for images. I'm not sure if scheduled builds should perform detail tests on all platforms, or if this should be restricted to releases and explicit triggers. At least we should do a minimal test and look if the system boots to userspace (call qemu and read serial output until systemd shows itself; this weeds out broken bootloader, kernel and filesytem). Even without kvm this takes less then 30 seconds, so is an easy to perform test. - For all scheduled runs, upload images and metadata to a user accessible storage. - A manual job to release the images. This triggers a pipeline from a different project. This new pipeline contains the following jobs: - Upload image to platforms. - Test new instance using the images. - Publish images. - Notify debian-cloud@. While this proposal introduces some complexity, it uses common components and does all the special stuff itself. It allows a user to setup his own environment with only a small amount of effort: - Setup GitLab or use an existing instance. - Setup a GitLab Runner with Docker to run the builds, using some documented config options. - Configure some build options. Regards, Bastian -- Our missions are peaceful -- not for conquest. When we do battle, it is only because we have no choice. -- Kirk, "The Squire of Gothos", stardate 2124.5
