@athos-ribeiro is preparing a PPA of the previous working version with a version higher than the current archive version
steps for installing the PPA into launchpad- buildd/lpbuildd/target/build_oci.py https://git.launchpad.net/launchpad- buildd/tree/lpbuildd/target/build_oci.py#n72 1. add-apt-repository $PPA --yes 2. apt install docker.io we are adding an `epoch` to the start of the version string. this will mean it will always sort _higher_ than any version pushed to the archive. to revert, remove the line adding the PPA # FUTURE STEPS the OCI image format is described here: https://github.com/opencontainers/image-spec/blob/main/image-layout.md . an Image must contain a blobs folder. a blob is _not_ necessarily a layer. Each blob is tracked in the top level manifest.json, which describes all blobs. The layers will be in the Layers key. Example manifest[10] the launchpad-buildd code walks a tarfile created by a `docker save` command.[1] it checks for directories and continues its walk, and looks for `layer.tar`. This is then read from the tar, and written to a new file $current_dir.tar.gz. in the new OCI format, there is no `layer.tar`, each blob name is named the HASHSUM of the file. There is a `digest.json` created, which I found documented internally as: Build digests.json * If a diff_id exists in the retrieved lookup files, use that digest and add the source value. * If it does not exist, create a new digest by taking a sha256 of the <layer_id>.tar.gz. This is not consistent with the method the docker engine uses to create a digest, but is sufficient for now. The digest is optionally validated by the client, currently the docker client will accept these digests. That may change in future. * Add the layer_id from the manifest for the matching diff_id However, the primary downstream consumer of these builds, the ROCKS team, does not appear to be utilizing the digest in any fashion. As it stands, the `manifest.json` contains the same information, so I am unsure as to the reason to continue providing digest.json (retrieved by looking at CPC Jenkins and cpc-build-tools, non-public resources) To create the analogous, launchpad-buildd could 1. read the manifest.json 2. identify which blob/$HASH_ALGO/$HASH are layers 3. turn those into tar.gz files 4. construct a digest in a matching format, taking the sha256 of the tar.gz Skipping any of that will cause `launchpad` to have failures, specifically in launchpad/lib/buildmaster/model/buildfamjobbehaviour.py[2]. That specifically references the `digest.json` related to extra files self._saveBuildSpecificFiles which leads to launchpad/lib/lp/oci/models/ocirecipebuild.py[3] The ROCKS team creating these files no longer utilizes the Registry Upload portions of the recipe, instead downloading the files from each build. This means that the layer functionality, used to push individual changes to registries, is not being utilized[4]. Instead the ROCKS team downloads the docker save, runs an operation to convert it to an OCI recipe that matches what is output in docker.io > 25 [5]. ### Big However The ROCKS team producing the base docker images is not the only team utilizing the OCI build recipes. Canonical maintains a number of OCI build recipes[6]. This means in fixing the issue, if we are adopting the new format, the entire codebase in launchpad-buildd and launchpad-itself will need to be redone and examined 1. we currently only push _new_ layers to the librarian and to registries 2. This is done on our side by figuring out the diff_ids, creating a registry manifest, and asking the registry to _mount_ the diffs we upload this saves transmission space in a couple place: between buildd and the librarian (which do not co-reside), and up to the registries. The registries themselves don't care all that much, as they de-dupe on their side (NOTE: assumption, as this is what I've understood to this day, but may be missing details). we can either 1. accept the transmission and size costs or; 2. do a refactoring to convert the OCI format to the old layer and diff digest.json method or; 3. do a much more thorough investigation on how the OCI format is handled by registries to do something in _spirit_ the same as 2, but fitting into the new world. [1] https://git.launchpad.net/launchpad-buildd/tree/lpbuildd/oci.py#n158 [2] https://git.launchpad.net/launchpad/tree/lib/lp/buildmaster/model/buildfarmjobbehaviour.py#n436 [3] https://git.launchpad.net/launchpad/tree/lib/lp/oci/model/ocirecipebuild.py#n104 [4] I'm not 100% clear here yet on 1) that being why layers are split from the image this way; 2) that _all_ users of the recipes are _not_ utilizing this. [5] PROPRIETARY CODE: sorry, most will be unable to see the reference, but it basically does what I described in reverse above in turning an OCI compliant image into an old format : https://git.launchpad.net/~cloudware/cloudware/+git/cpc_build_tools/tree/oci_lp_convert.py [6] https://launchpad.net/~ubuntu-docker-images/+oci-recipes [10] [ { "Config": "blobs/sha256/eebd1f6f54a921620cb11bba2dab2a8e83d875a1cf160eff1709e24eb4fa7e23", "RepoTags": null, "Layers": [ "blobs/sha256/9c8b3e5cca6750d7c05a8adc6bc4706048b72f003b0e60f74f2d8c0fc07bad7b", "blobs/sha256/6202c40a2ac7973af00544724a43dc550ecfdc7f060dbdcd410572a67da02802", "blobs/sha256/f8dc411581c29e8e0e0b56b38150de7c0783dbe97869dfbcf301fc7300c70a05" ], "LayerSources": { "sha256:6202c40a2ac7973af00544724a43dc550ecfdc7f060dbdcd410572a67da02802": { "mediaType": "application/vnd.oci.image.layer.v1.tar", "size": 87362560, "digest": "sha256:6202c40a2ac7973af00544724a43dc550ecfdc7f060dbdcd410572a67da02802" }, "sha256:9c8b3e5cca6750d7c05a8adc6bc4706048b72f003b0e60f74f2d8c0fc07bad7b": { "mediaType": "application/vnd.oci.image.layer.v1.tar", "size": 82568192, "digest": "sha256:9c8b3e5cca6750d7c05a8adc6bc4706048b72f003b0e60f74f2d8c0fc07bad7b" }, "sha256:f8dc411581c29e8e0e0b56b38150de7c0783dbe97869dfbcf301fc7300c70a05": { "mediaType": "application/vnd.oci.image.layer.v1.tar", "size": 4096, "digest": "sha256:f8dc411581c29e8e0e0b56b38150de7c0783dbe97869dfbcf301fc7300c70a05" } } } ] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2098106 Title: Launchpad OCI builds are failing after the docker.io-app SRU update to 26.x on Focal To manage notifications about this bug go to: https://bugs.launchpad.net/launchpad/+bug/2098106/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
