Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, Jun 5, 2023 at 8:35 AM Josh Boyer wrote: > On Mon, Jun 5, 2023 at 6:28 AM Debarshi Ray via devel > wrote: > > I wanted to wrap up this sub-thread on-list, after Owen and I chatted > > about it off-list. > > > > I am fine with having the fedora-toolbox OCI images being defined as > > kickstart files in the Fedora infrastructure and built by ImageFactory > > and published as another base image, just like the fedora base image > > currently is. > > Just an FYI, but we're trying to move away from ImageFactory in the > general sense. I'm not sure if/how that will manifest itself in > Fedora, but ImageFactory is pretty terrible for debugging things when > it goes wrong. The options we're looking at elsewhere are Image > Builder, OSBS2, and RHTAP. > If we want to move Fedora images to ImageBuilder, then the right thing to do here is still to merge the toolbox image into fedora-kickstarts and then work on migrating that, rather than have it be a one-off outlier. (*) - Owen (*) Yes, I know Fedora-IOT uses ImageBuilder. but in the context of the main Fedora compose. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, Jun 5, 2023 at 6:28 AM Debarshi Ray via devel wrote: > > Hey, > > I wanted to wrap up this sub-thread on-list, after Owen and I chatted > about it off-list. > > I am fine with having the fedora-toolbox OCI images being defined as > kickstart files in the Fedora infrastructure and built by ImageFactory > and published as another base image, just like the fedora base image > currently is. Just an FYI, but we're trying to move away from ImageFactory in the general sense. I'm not sure if/how that will manifest itself in Fedora, but ImageFactory is pretty terrible for debugging things when it goes wrong. The options we're looking at elsewhere are Image Builder, OSBS2, and RHTAP. Honestly, for base container images the simplest solution is just buildah but nobody has put a lot of time into making that workable in the ways that many projects would view as "official". josh > This means that the downstream sources of the images in Fedora that are > actually used to make them available to users will differ from the > upstream sources that are used for testing and CI and are currently > represented as Containerfiles. If we can't use ImageFactory for > upstream testing and CI, we will have to carefully keep them > synchronized whenever there are changes. If this is the price to pay > for making the fedora-toolbox images release-blocking deliverables, > then so be it. :) > > That said, my understanding is that Owen is still not settled on all > the tooling changes for building Fedora Flatpaks. So, we may or may > not end up using ImageFactory for the fedora-toolbox images in the end. > > Cheers, > Rishi > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey, I wanted to wrap up this sub-thread on-list, after Owen and I chatted about it off-list. I am fine with having the fedora-toolbox OCI images being defined as kickstart files in the Fedora infrastructure and built by ImageFactory and published as another base image, just like the fedora base image currently is. This means that the downstream sources of the images in Fedora that are actually used to make them available to users will differ from the upstream sources that are used for testing and CI and are currently represented as Containerfiles. If we can't use ImageFactory for upstream testing and CI, we will have to carefully keep them synchronized whenever there are changes. If this is the price to pay for making the fedora-toolbox images release-blocking deliverables, then so be it. :) That said, my understanding is that Owen is still not settled on all the tooling changes for building Fedora Flatpaks. So, we may or may not end up using ImageFactory for the fedora-toolbox images in the end. Cheers, Rishi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
I think it's worth to mention that many layered container images are published directly on the Fedora's Quay.io organisation ( https://quay.io/organization/fedora). The builds are not happening in OSBS or the Fedora infrastructure but IIRC it's a GitHub Action that does the build and pushes the container image. There could be an option to leverage Quay.io build capabilities too. On Tue, 30 May 2023, 17:02 Owen Taylor, wrote: > > > On Tue, May 30, 2023 at 9:47 AM Debarshi Ray wrote: > >> Hey Owen, >> >> On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote: >> > On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel >> > wrote: >> > > >> > > My main concern, which I had brought up in the Release Engineering >> > > tickets before [1,2] is whether the fedora-toolbox images would >> > > continue to be defined as a Docker/Containerfile. I am asking >> > > because the fedora base images are defined as kickstart files [3]. >> > > >> > > There's some value in continuing to use a Docker/Containerfile >> > > because it's widely known and leads to a common workflow. It is >> > > used for the official Toolbx images for other distributions like >> > > Arch Linux, RHEL and Ubuntu, and a growing list of third-party >> > > Toolbx images [4]. A Docker/Containerfile is easy to use with >> > > 'podman build' as part of the upstream CI, which makes it easier to >> > > test all the different parts. >> > >> > Unfortunately, if we switched to using ImageFactory or pretty much >> > anything other than OSBS, we'd no longer be able to define the Fedora >> > toolbox image as a Dockerfile. It certainly is a disadvantage that >> > it's no longer connected to the way that the other toolbox images are >> > defined, >> >> It's an advantage to have the Fedora images be defined in the same >> source language as the Arch Linux and Ubuntu images, and certainly the >> RHEL images. However, ultimately, I see it only as a convenience. >> Folks can take the Fedora images as a reference and tweak them to fit >> their own distribution, or copy paste the files across Git >> repositories, etc.. If we lose that, then I would learn to live with >> it. :) >> > > I guess in this scenario, the Fedora images are no longer the reference > for creating things for other distributions and something else (RHEL, > Ubuntu, ...) would have to take over that role. For "tweaking", I'd think > we'd definitely want to promote "FROM fedora-toolbox:latest". > > >> If we do move away from Container/Dockerfiles, then my main question >> would be whether it'll still be easy to locally build the images. Will >> we need something a lot more elaborate than the simplicity of a 'podman >> build'? >> > > I haven't actually tried building the Fedora container images locally, > but it it looks relatively documented and straightforward, and someone > could write a shell script to automate it pretty easily. But it is still > going to require tools (ImageFactory) that people don't have installed and > probably take a while to run. It seems like more of a "want to contribute > to the the Fedora toolbox image" thing than a "would do it to run tests > when contributing to Toolbx" thing. > > Toolbx has two parts. A somewhat distro-agnostic toolbox(1) binary, >> and an OCI image for a distribution. Both work together to provide an >> interactive CLI environment, and hence both should be tested together. >> >> If a contributor attempts to change one or both of those two, then it >> should be easy for them to verify whether both would continue to work >> together. Currently, they can do that by doing a local 'podman build' >> on the image definitions in toolbox.git and try to use them with their >> modified toolbox(1) binary. I want to formalize this by making the >> upstream CI run against both: >> (a) images that are currently published on the OCI registries and >> (b) images that are built on-the-fly from the sources in toolbox.git >> > > In this scenario the Fedora Toolbx image definition would live in > fedora-kickstarts, so it doesn't really make sense for me that Toolbox CI > would *recreate it*, rather than just using the version from the repository. > > Ideally, we'd run the Toolbox tests against the newly built Toolbx image > as part of QE for the Fedora compose. > > >> ... so that it will validate any changes to the image sources in >> toolbox.git, and alert us to any changes in the underlying distribution >> (eg., packages, dependencies, base images, etc.) that could break >> future image rebuilds. >> >> Currently, the upstream CI runs only against (a). >> >> If we lose the simplicity of a 'podman build' then we lose the testing. >> That's my main worry. >> >> If the new set-up offers a similarly simple way of building the images >> from source, then I am happy. >> >> > but maybe that's bearable if we can substantially improve the >> > workflow? >> >> The phrase "we can substantially improve the workflow" makes me >> curious. Any details or pointers? >> >
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, May 30, 2023 at 9:47 AM Debarshi Ray wrote: > Hey Owen, > > On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote: > > On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel > > wrote: > > > > > > My main concern, which I had brought up in the Release Engineering > > > tickets before [1,2] is whether the fedora-toolbox images would > > > continue to be defined as a Docker/Containerfile. I am asking > > > because the fedora base images are defined as kickstart files [3]. > > > > > > There's some value in continuing to use a Docker/Containerfile > > > because it's widely known and leads to a common workflow. It is > > > used for the official Toolbx images for other distributions like > > > Arch Linux, RHEL and Ubuntu, and a growing list of third-party > > > Toolbx images [4]. A Docker/Containerfile is easy to use with > > > 'podman build' as part of the upstream CI, which makes it easier to > > > test all the different parts. > > > > Unfortunately, if we switched to using ImageFactory or pretty much > > anything other than OSBS, we'd no longer be able to define the Fedora > > toolbox image as a Dockerfile. It certainly is a disadvantage that > > it's no longer connected to the way that the other toolbox images are > > defined, > > It's an advantage to have the Fedora images be defined in the same > source language as the Arch Linux and Ubuntu images, and certainly the > RHEL images. However, ultimately, I see it only as a convenience. > Folks can take the Fedora images as a reference and tweak them to fit > their own distribution, or copy paste the files across Git > repositories, etc.. If we lose that, then I would learn to live with > it. :) > I guess in this scenario, the Fedora images are no longer the reference for creating things for other distributions and something else (RHEL, Ubuntu, ...) would have to take over that role. For "tweaking", I'd think we'd definitely want to promote "FROM fedora-toolbox:latest". > If we do move away from Container/Dockerfiles, then my main question > would be whether it'll still be easy to locally build the images. Will > we need something a lot more elaborate than the simplicity of a 'podman > build'? > I haven't actually tried building the Fedora container images locally, but it it looks relatively documented and straightforward, and someone could write a shell script to automate it pretty easily. But it is still going to require tools (ImageFactory) that people don't have installed and probably take a while to run. It seems like more of a "want to contribute to the the Fedora toolbox image" thing than a "would do it to run tests when contributing to Toolbx" thing. Toolbx has two parts. A somewhat distro-agnostic toolbox(1) binary, > and an OCI image for a distribution. Both work together to provide an > interactive CLI environment, and hence both should be tested together. > > If a contributor attempts to change one or both of those two, then it > should be easy for them to verify whether both would continue to work > together. Currently, they can do that by doing a local 'podman build' > on the image definitions in toolbox.git and try to use them with their > modified toolbox(1) binary. I want to formalize this by making the > upstream CI run against both: > (a) images that are currently published on the OCI registries and > (b) images that are built on-the-fly from the sources in toolbox.git > In this scenario the Fedora Toolbx image definition would live in fedora-kickstarts, so it doesn't really make sense for me that Toolbox CI would *recreate it*, rather than just using the version from the repository. Ideally, we'd run the Toolbox tests against the newly built Toolbx image as part of QE for the Fedora compose. > ... so that it will validate any changes to the image sources in > toolbox.git, and alert us to any changes in the underlying distribution > (eg., packages, dependencies, base images, etc.) that could break > future image rebuilds. > > Currently, the upstream CI runs only against (a). > > If we lose the simplicity of a 'podman build' then we lose the testing. > That's my main worry. > > If the new set-up offers a similarly simple way of building the images > from source, then I am happy. > > > but maybe that's bearable if we can substantially improve the > > workflow? > > The phrase "we can substantially improve the workflow" makes me > curious. Any details or pointers? > What I mean by this was simply the obvious - that the Toolbx image is built as part of the compose, and can be tested as part of the compose validation. That nobody is sitting there manually typing 'fedpkg container-build' and creating Bodhi updates. (*) - Owen (*) I'm not saying that this level of integration is impossible with OSBS - in fact there is some level of Pungi support for OSBS container builds already. Just that it's basically *already* there in Fedora for the base images. ___ devel mailing
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey Owen, On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote: > On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel > wrote: > > > > My main concern, which I had brought up in the Release Engineering > > tickets before [1,2] is whether the fedora-toolbox images would > > continue to be defined as a Docker/Containerfile. I am asking > > because the fedora base images are defined as kickstart files [3]. > > > > There's some value in continuing to use a Docker/Containerfile > > because it's widely known and leads to a common workflow. It is > > used for the official Toolbx images for other distributions like > > Arch Linux, RHEL and Ubuntu, and a growing list of third-party > > Toolbx images [4]. A Docker/Containerfile is easy to use with > > 'podman build' as part of the upstream CI, which makes it easier to > > test all the different parts. > > Unfortunately, if we switched to using ImageFactory or pretty much > anything other than OSBS, we'd no longer be able to define the Fedora > toolbox image as a Dockerfile. It certainly is a disadvantage that > it's no longer connected to the way that the other toolbox images are > defined, It's an advantage to have the Fedora images be defined in the same source language as the Arch Linux and Ubuntu images, and certainly the RHEL images. However, ultimately, I see it only as a convenience. Folks can take the Fedora images as a reference and tweak them to fit their own distribution, or copy paste the files across Git repositories, etc.. If we lose that, then I would learn to live with it. :) If we do move away from Container/Dockerfiles, then my main question would be whether it'll still be easy to locally build the images. Will we need something a lot more elaborate than the simplicity of a 'podman build'? Toolbx has two parts. A somewhat distro-agnostic toolbox(1) binary, and an OCI image for a distribution. Both work together to provide an interactive CLI environment, and hence both should be tested together. If a contributor attempts to change one or both of those two, then it should be easy for them to verify whether both would continue to work together. Currently, they can do that by doing a local 'podman build' on the image definitions in toolbox.git and try to use them with their modified toolbox(1) binary. I want to formalize this by making the upstream CI run against both: (a) images that are currently published on the OCI registries and (b) images that are built on-the-fly from the sources in toolbox.git ... so that it will validate any changes to the image sources in toolbox.git, and alert us to any changes in the underlying distribution (eg., packages, dependencies, base images, etc.) that could break future image rebuilds. Currently, the upstream CI runs only against (a). If we lose the simplicity of a 'podman build' then we lose the testing. That's my main worry. If the new set-up offers a similarly simple way of building the images from source, then I am happy. > but maybe that's bearable if we can substantially improve the > workflow? The phrase "we can substantially improve the workflow" makes me curious. Any details or pointers? > With some refactoring of the kickstarts for Fedora containers, we > could also remove some of the hackiness in the Dockerfile where it is > undoing choices in the base image. *nod* > When I ask this question, I do have a hidden agenda - I'm thinking > that it might be possible to move Flatpak generation to a Koji plugin > that runs directly on the builders, but that would leave the toolbox > as the sole user of the OSBS cluster, which seems dubiously > sustainable. So maybe we can take Toolbox off of OSBS as well and > retire OSBS? Yes, I saw you mention this elsewhere. I totally understand and agree with your motivation. Trying to consolidate the infrastructure does sound like a good idea. > Now, if we could instead move to OSBS 2 and have that actively > maintained and across architecture, that could be better, but I don't > really see where the resources are going to come from to do that. *nod* Cheers, Rishi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel < devel@lists.fedoraproject.org> wrote: > Hey Owen, > > On Wed, 2023-05-24 at 13:50 -0400, Owen Taylor wrote: > > > > What if we made the Toolbox container image just one more base image > > and built it with ImageFactory? > > > > - Integrated into the compose process > > - Across all architectures > > - No OSBS dependency > > > > The main disadvantage is that it is no longer layered, so *if* you > > happen to have the exact same Fedora image version around for some > > other reason (a big if), you save a fraction of space: > > > > Fedora 38 container - 71M compressed, 201M uncompressed > > Toolbox add-on layer - 232M compressed, 753M uncompressed > > Toolbox squashed - 291M compressed, 884M uncompressed > > I am not too attached to the bandwidth or space savings due to the > fedora-toolbox image being a layered image. > > [ In future, I would like to explore if we can ship the fedora-toolbox > image as part of the Silverblue and Workstation ISOs, to avoid the need > for a sizable download just to set up the default Toolbx. That could > unlock things like defaulting to a Toolbx shell or generally help > promoting Toolbx as a primary interactive CLI environment. Anyway, > that's a big 'if'. ] > > My main concern, which I had brought up in the Release Engineering > tickets before [1,2] is whether the fedora-toolbox images would > continue to be defined as a Docker/Containerfile. I am asking because > the fedora base images are defined as kickstart files [3]. > > There's some value in continuing to use a Docker/Containerfile because > it's widely known and leads to a common workflow. It is used for the > official Toolbx images for other distributions like Arch Linux, RHEL > and Ubuntu, and a growing list of third-party Toolbx images [4]. A > Docker/Containerfile is easy to use with 'podman build' as part of the > upstream CI, which makes it easier to test all the different parts. Unfortunately, if we switched to using ImageFactory or pretty much anything other than OSBS, we'd no longer be able to define the Fedora toolbox image as a Dockerfile. It certainly is a disadvantage that it's no longer connected to the way that the other toolbox images are defined, but maybe that's bearable if we can substantially improve the workflow? With some refactoring of the kickstarts for Fedora containers, we could also remove some of the hackiness in the Dockerfile where it is undoing choices in the base image. When I ask this question, I do have a hidden agenda - I'm thinking that it might be possible to move Flatpak generation to a Koji plugin that runs directly on the builders, but that would leave the toolbox as the sole user of the OSBS cluster, which seems dubiously sustainable. So maybe we can take Toolbox off of OSBS as well and retire OSBS? Now, if we could instead move to OSBS 2 and have that actively maintained and across architecture, that could be better, but I don't really see where the resources are going to come from to do that. - Owen ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey Owen, On Wed, 2023-05-24 at 13:50 -0400, Owen Taylor wrote: > > What if we made the Toolbox container image just one more base image > and built it with ImageFactory? > > - Integrated into the compose process > - Across all architectures > - No OSBS dependency > > The main disadvantage is that it is no longer layered, so *if* you > happen to have the exact same Fedora image version around for some > other reason (a big if), you save a fraction of space: > > Fedora 38 container - 71M compressed, 201M uncompressed > Toolbox add-on layer - 232M compressed, 753M uncompressed > Toolbox squashed - 291M compressed, 884M uncompressed I am not too attached to the bandwidth or space savings due to the fedora-toolbox image being a layered image. [ In future, I would like to explore if we can ship the fedora-toolbox image as part of the Silverblue and Workstation ISOs, to avoid the need for a sizable download just to set up the default Toolbx. That could unlock things like defaulting to a Toolbx shell or generally help promoting Toolbx as a primary interactive CLI environment. Anyway, that's a big 'if'. ] My main concern, which I had brought up in the Release Engineering tickets before [1,2] is whether the fedora-toolbox images would continue to be defined as a Docker/Containerfile. I am asking because the fedora base images are defined as kickstart files [3]. There's some value in continuing to use a Docker/Containerfile because it's widely known and leads to a common workflow. It is used for the official Toolbx images for other distributions like Arch Linux, RHEL and Ubuntu, and a growing list of third-party Toolbx images [4]. A Docker/Containerfile is easy to use with 'podman build' as part of the upstream CI, which makes it easier to test all the different parts. Currently the Fedora and RHEL images that are shipped to users aren't built straight from the upstream Git repository, but from the separate copies on Fedora and Red Hat dist-git. Even then, it's a lot easier to manage two copies of the sources in the same format, as opposed to one using Docker/Containerfiles and the other something else like kickstart files. Cheers, Rishi [1] https://pagure.io/releng/issue/11399#comment-853047 Attached to this Change proposal [2] https://pagure.io/releng/issue/11189#comment-840689 Past discussion that led to this Change [3] https://pagure.io/fedora-kickstarts [4] https://github.com/toolbx-images/images ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Wed, May 24, 2023 at 01:50:21PM -0400, Owen Taylor wrote: > > What if we made the Toolbox container image just one more base image and > built it with ImageFactory? > > - Integrated into the compose process > - Across all architectures > - No OSBS dependency > > The main disadvantage is that it is no longer layered, so *if* you happen > to have the exact same Fedora image version around for some other reason (a > big if), you save a fraction of space: > > Fedora 38 container - 71M compressed, 201M uncompressed > Toolbox add-on layer - 232M compressed, 753M uncompressed > Toolbox squashed - 291M compressed, 884M uncompressed > > But generally seems like it would be a win. osbuild/kiwi/whatever can be > left as a separate project. I would support this fully. ;) It would make it a lot easier and then it wouldn't depend on the module build pipeline. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, May 9, 2023 at 11:52 AM Kevin Fenzi wrote: > Just a general answer/info here at the bottom of the thread... > > I realize our container build pipeline is not great, but it's currently > working and I will keep it working until we replace it. > > I agree we should replace it, and there's lots of options, but I don't > think this thread is the place to go back and forth about them. > > I know of at least kiwi, osbuild, some other build systems that don't > fully exist yet, switching to use quay.io, osbs2 (based on openshift4), > and probibly others. > What if we made the Toolbox container image just one more base image and built it with ImageFactory? - Integrated into the compose process - Across all architectures - No OSBS dependency The main disadvantage is that it is no longer layered, so *if* you happen to have the exact same Fedora image version around for some other reason (a big if), you save a fraction of space: Fedora 38 container - 71M compressed, 201M uncompressed Toolbox add-on layer - 232M compressed, 753M uncompressed Toolbox squashed - 291M compressed, 884M uncompressed But generally seems like it would be a win. osbuild/kiwi/whatever can be left as a separate project. - Owen ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Wed, 2023-05-10 at 21:35 +0200, Clement Verna wrote: > > On Tue, 9 May 2023, 15:42 Debarshi Ray via devel, > wrote: > > > > On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote: > > > > > > If we do this, we should also make the container base images > > > release blocking since AFAIK they are currently not release > > > blocking. > > > > Yes, that's a good point. > > > > The OCI base images aren't currently listed as release-blocking > > deliverables: > > https://docs.fedoraproject.org/en-US/releases/f39/blocking/ > > > > ... but if this Change goes through, then they will implicitly > > become release-blocking deliverables because the fedora-toolbox > > images are based on them. > > > > Should we explicitly mention this in the Change? Or, something > > else? Or, is it fine as it is? > > I think mentioning explicitly in this change is good enough :) > Okay! I add a note to the Detailed Description section of the Change. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, 9 May 2023, 17:51 Kevin Fenzi, wrote: > Just a general answer/info here at the bottom of the thread... > > I realize our container build pipeline is not great, but it's currently > working and I will keep it working until we replace it. > > I agree we should replace it, and there's lots of options, but I don't > think this thread is the place to go back and forth about them. > > I know of at least kiwi, osbuild, some other build systems that don't > fully exist yet, switching to use quay.io, osbs2 (based on openshift4), > and probibly others. > I also agree that we should replace it but that's not an easy task unfortunately. My understanding of osbuild is that it focuses on base images and not layered images so that probably would not be a good candidate for replacing OSBS. I might also be wrong :-) > > kevin > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, 9 May 2023, 15:42 Debarshi Ray via devel, < devel@lists.fedoraproject.org> wrote: > Hey Clement, > > On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote: > > > > > > On Mon, 8 May 2023 at 22:11, Kevin Fenzi wrote: > > > On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski- > > > Szmek wrote: > > > > > > > > I think we need some clarity wrt. to the dependency order here. > > > > Let's say we: > > > > > In order to do this at branch point, we will need to move > > > > > building this > > > > > image into the compose process and mark it blocking. > > > > OK, so we build things, but then we need to publish to > > > > registry.fp.o, > > > > which is asynchrounous (?). When we test the newly built ISOs, do > > > > > > No, it happens at the end of the compose (if no blocking > > > deliverables failed causing the compose to fail) > > > > If we do this, we should also make the container base images release > > blocking since AFAIK they are currently not release blocking. > > Yes, that's a good point. > > The OCI base images aren't currently listed as release-blocking > deliverables: > https://docs.fedoraproject.org/en-US/releases/f39/blocking/ > > ... but if this Change goes through, then they will implicitly become > release-blocking deliverables because the fedora-toolbox images are > based on them. > > Should we explicitly mention this in the Change? Or, something else? > Or, is it fine as it is? > I think mentioning explicitly in this change is good enough :) > > Thanks, > Rishi > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, May 09, 2023 at 03:46:21PM +0200, Debarshi Ray via devel wrote: > Hey Zbigniew, > > On Mon, 2023-05-08 at 19:57 +, Zbigniew Jędrzejewski-Szmek wrote: > > On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote: > > > I'm broadly in favor here, some comments in line... > > > > > > ...snip... > > > > First, we want to ensure that there are up to date > > > > [https://src.fedoraproject.org/container/fedora-toolbox > > > > fedora-toolbox] OCI images published on > > > > [https://registry.fedoraproject.org/ registry.fedoraproject.org] > > > > as > > > > release-blocking deliverables at critical points in the > > > > development > > > > schedule, just like the installation ISOs for the Editions from > > > > [https://download.fedoraproject.org/pub/fedora/linux/releases/ > > > > download.fedoraproject.org]. This must at least happen when an > > > > upcoming Fedora release is branched from Rawhide, and for the > > > > Beta and > > > > Final release candidates. If possible, they should be updated > > > > more > > > > frequently as part of the nightly composes. We do not expect > > > > this to > > > > happen after a Fedora release has gone GA. > > > > I think we need some clarity wrt. to the dependency order here. > > Let's say we: > > > In order to do this at branch point, we will need to move building > > > this > > > image into the compose process and mark it blocking. > > OK, so we build things, but then we need to publish to registry.fp.o, > > which is asynchrounous (?). When we test the newly built ISOs, do > > we test them also with the latest image that we get from > > registry.fp.o? > > And if we find a bug anywhere in this pipeline, we respin everything? > > That's our expectation, yes. > > When we apply the test criteria to a compose, if the freshly composed > fedora-toolbox OCI images don't meet the test criteria on hosts running > the freshly composed Fedora ISOs, then we track down the bugs, get them > fixed, re-compose, and try again. Currently, this will happen for the > Beta and Final release candidates. Thanks. Makes sense. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Just a general answer/info here at the bottom of the thread... I realize our container build pipeline is not great, but it's currently working and I will keep it working until we replace it. I agree we should replace it, and there's lots of options, but I don't think this thread is the place to go back and forth about them. I know of at least kiwi, osbuild, some other build systems that don't fully exist yet, switching to use quay.io, osbs2 (based on openshift4), and probibly others. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On 5/9/23 07:53, Stephen Smoogen wrote: > On Mon, 8 May 2023 at 19:35, Demi Marie Obenour > wrote: > >> On 5/8/23 19:09, Neal Gompa wrote: >>> On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour >> wrote: On 5/8/23 18:49, Kevin Fenzi wrote: > On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: >> Dear Kevin, >> Hmm, quoting from https://pagure.io/releng/issue/11092: >> Also the aarch64 cluster is running on Fedora 33 boxes, so we >> should probably try to do a full redeploy :-( > We can't upgrade it from f33 because docker is no longer in f34+ >> and > openshift origin / 3.11 doesn't support any newer either. Is this still true? I don't think we want to make the Fedora release process contingent on something that requires F33. >>> >>> yes, it's still true. Note thats the aarch64 osbs cluster. >>> The x86_64 one is rhel7. >> >> Might it be possible to replace Docker with CRI-O on the OpenShift >> cluster? > > Nope. openshift 3 / osbs1 uses docker. :( > > kevin alias docker=podman >>> >>> Using Podman with it through Podman's implementation of the Docker >>> Host protocol *may* work, but a version of Podman that has it is not >>> currently available for RHEL 7 (which is what OpenShift 3.11 runs on). >> >> Time for an OpenShift upgrade? >> >> > I realize people are trying to help with suggestions, but the problem is > that this is not a simple solution or it would have been done years ago. > > The Fedora Build System is a very complicated set of applications all tied > together with custom scripting, schema, and various updates over time. > There are somewhere around 40 core services and 30 more subsidiary ones > which need to be continually working to allow for the daily builds for > multiple releases. Many of those services have been added in at different > times with different operating systems and rules. In order to work with > each other a lot of 'glue' scripting, libraries and such are needed. > > You upgrade podman/docker in one section and you find that the message bus > isn't sending in another because an API call isn't there anymore. You try > putting in something which says it has a koji plugin and you find it > doesn't work with our koji and then you need to extend the plugin 40 ways > to work with bodhi, pdc, fedmsg and fedora-messaging, and a dozen other > things which all are the actual build system. Once those get fixed, you > find that it is now causing other glue parts to fail in odd ways no one > remembers why without some archaeological coding. Having ancient versions of OpenShift is almost certainly a security problem. If there is stuff that cannot be kept on a supported version it should be removed. > When dealing with the Build system side of things there is currently 1 > senior system administrator and 1 senior release engineer. There are some > volunteers who can work on subparts but mainly have day jobs doing other > things. Looking at IRC, most of the 10-12 hour work day is dealing with > compose problems, build issues, hardware failures, and similar things. Most > of the 40 component subsystems like the Open Shift Build System (OSBS) are > brought in by a dedicated team who have a budget time to get it working and > into place. Once that time is done, any fixes are on the Fedora staff or > working with available time from whatever team. This is the actual problem IMO. What will it take to fix it? > tl;dr: The Fedora Build System is in total a very complicated set of tools > where changing or upgrading any one part tends to cascade to fixing lots of > glue throughout the system. Instead of looking to upgrade things to add > more deliverables, it may be better to look at other ways to get those > deliverables built. Do you mean outside of the Fedora Build System? -- Sincerely, Demi Marie Obenour (she/her/hers) OpenPGP_0xB288B55FFF9C22C1.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey Kevin, On Mon, 2023-05-08 at 09:24 -0700, Kevin Fenzi wrote: > I'm broadly in favor here, some comments in line... > > ...snip... > > It will be beneficial to consider the > > [https://src.fedoraproject.org/container/fedora-toolbox > > fedora-toolbox] images as release-blocking deliverables because > > Fedora's [https://opencontainers.org/ OCI] infrastructure is often > > broken. Here are [https://pagure.io/releng/issue/11092 two] > > [https://pagure.io/releng/issue/11367 recent] examples of > > fedpkg > > container-build not working. In the second case, it was > > preventing the images from being rebuilt to pull in an > > [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important] > > bug-fix. The broken infrastructure prevents regular Fedora > > contributors from jumping in to rebuild and publish the images at > > critical points in the development schedule. Making them > > release-blocking deliverables would attract greater attention and > > scrutiny from release engineering and ensure that a Fedora > > development > > cycle does not proceed with broken or outdated or missing > > fedora-toolbox images. > > I'd like to note that making this blocking doesn't waive any kind of > magic wand that makes our infrastructure more reliable. ;) > The container build pipeline is a long collection of fragile things. > It may well result in us slipping more based on things not working. > ;( Yes, I understand. Currently, what happens is that someone (very often Jens) jumps in to update the fedora-toolbox images from time to time, sometimes because users started to complain about missing images after branching or some other problem, then it turns out that Fedora's OCI infrastructure is broken, then it leads to a pagure.io/releng issue, and then someone (very often you, Kevin) jumps in to frantically debug and get things going again. I was hoping that by making the fedora-toolbox images release-blocking deliverables, we will avoid some of the ad-hoc frantic last-minute fire-fighting and get more eye balls on this part of the Fedora infrastructure. Most importantly, Fedora users would get a more robust and smoother user experience because they won't have to scream and yell on bug trackers to get new images pushed out the door. Also, it's not really possible to test the toolbox(1) binary unless the corresponding fedora-toolbox images are updated and published reliably. In practical terms, if this means that the Toolbx developers need to get involved to help out with Fedora's OCI infrastructure, then so be it. :) Cheers, Rishi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey Zbigniew, On Mon, 2023-05-08 at 19:57 +, Zbigniew Jędrzejewski-Szmek wrote: > On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote: > > I'm broadly in favor here, some comments in line... > > > > ...snip... > > > First, we want to ensure that there are up to date > > > [https://src.fedoraproject.org/container/fedora-toolbox > > > fedora-toolbox] OCI images published on > > > [https://registry.fedoraproject.org/ registry.fedoraproject.org] > > > as > > > release-blocking deliverables at critical points in the > > > development > > > schedule, just like the installation ISOs for the Editions from > > > [https://download.fedoraproject.org/pub/fedora/linux/releases/ > > > download.fedoraproject.org]. This must at least happen when an > > > upcoming Fedora release is branched from Rawhide, and for the > > > Beta and > > > Final release candidates. If possible, they should be updated > > > more > > > frequently as part of the nightly composes. We do not expect > > > this to > > > happen after a Fedora release has gone GA. > > I think we need some clarity wrt. to the dependency order here. > Let's say we: > > In order to do this at branch point, we will need to move building > > this > > image into the compose process and mark it blocking. > OK, so we build things, but then we need to publish to registry.fp.o, > which is asynchrounous (?). When we test the newly built ISOs, do > we test them also with the latest image that we get from > registry.fp.o? > And if we find a bug anywhere in this pipeline, we respin everything? That's our expectation, yes. When we apply the test criteria to a compose, if the freshly composed fedora-toolbox OCI images don't meet the test criteria on hosts running the freshly composed Fedora ISOs, then we track down the bugs, get them fixed, re-compose, and try again. Currently, this will happen for the Beta and Final release candidates. Cheers, Rishi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Hey Clement, On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote: > > > On Mon, 8 May 2023 at 22:11, Kevin Fenzi wrote: > > On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski- > > Szmek wrote: > > > > > > I think we need some clarity wrt. to the dependency order here. > > > Let's say we: > > > > In order to do this at branch point, we will need to move > > > > building this > > > > image into the compose process and mark it blocking. > > > OK, so we build things, but then we need to publish to > > > registry.fp.o, > > > which is asynchrounous (?). When we test the newly built ISOs, do > > > > No, it happens at the end of the compose (if no blocking > > deliverables failed causing the compose to fail) > > If we do this, we should also make the container base images release > blocking since AFAIK they are currently not release blocking. Yes, that's a good point. The OCI base images aren't currently listed as release-blocking deliverables: https://docs.fedoraproject.org/en-US/releases/f39/blocking/ ... but if this Change goes through, then they will implicitly become release-blocking deliverables because the fedora-toolbox images are based on them. Should we explicitly mention this in the Change? Or, something else? Or, is it fine as it is? Thanks, Rishi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, 8 May 2023 at 19:35, Demi Marie Obenour wrote: > On 5/8/23 19:09, Neal Gompa wrote: > > On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour > wrote: > >> > >> On 5/8/23 18:49, Kevin Fenzi wrote: > >>> On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: > Dear Kevin, > > >> Hmm, quoting from https://pagure.io/releng/issue/11092: > Also the aarch64 cluster is running on Fedora 33 boxes, so we > should probably try to do a full redeploy :-( > >>> We can't upgrade it from f33 because docker is no longer in f34+ > and > >>> openshift origin / 3.11 doesn't support any newer either. > >> > >> Is this still true? I don't think we want to make the Fedora release > >> process contingent on something that requires F33. > > > > yes, it's still true. Note thats the aarch64 osbs cluster. > > The x86_64 one is rhel7. > > Might it be possible to replace Docker with CRI-O on the OpenShift > cluster? > >>> > >>> Nope. openshift 3 / osbs1 uses docker. :( > >>> > >>> kevin > >> > >> alias docker=podman > > > > Using Podman with it through Podman's implementation of the Docker > > Host protocol *may* work, but a version of Podman that has it is not > > currently available for RHEL 7 (which is what OpenShift 3.11 runs on). > > Time for an OpenShift upgrade? > > I realize people are trying to help with suggestions, but the problem is that this is not a simple solution or it would have been done years ago. The Fedora Build System is a very complicated set of applications all tied together with custom scripting, schema, and various updates over time. There are somewhere around 40 core services and 30 more subsidiary ones which need to be continually working to allow for the daily builds for multiple releases. Many of those services have been added in at different times with different operating systems and rules. In order to work with each other a lot of 'glue' scripting, libraries and such are needed. You upgrade podman/docker in one section and you find that the message bus isn't sending in another because an API call isn't there anymore. You try putting in something which says it has a koji plugin and you find it doesn't work with our koji and then you need to extend the plugin 40 ways to work with bodhi, pdc, fedmsg and fedora-messaging, and a dozen other things which all are the actual build system. Once those get fixed, you find that it is now causing other glue parts to fail in odd ways no one remembers why without some archaeological coding. When dealing with the Build system side of things there is currently 1 senior system administrator and 1 senior release engineer. There are some volunteers who can work on subparts but mainly have day jobs doing other things. Looking at IRC, most of the 10-12 hour work day is dealing with compose problems, build issues, hardware failures, and similar things. Most of the 40 component subsystems like the Open Shift Build System (OSBS) are brought in by a dedicated team who have a budget time to get it working and into place. Once that time is done, any fixes are on the Fedora staff or working with available time from whatever team. tl;dr: The Fedora Build System is in total a very complicated set of tools where changing or upgrading any one part tends to cascade to fixing lots of glue throughout the system. Instead of looking to upgrade things to add more deliverables, it may be better to look at other ways to get those deliverables built. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Tue, May 9, 2023 at 6:19 AM Peter Robinson wrote: > > On Mon, May 8, 2023 at 9:11 PM Kevin Fenzi wrote: > > > > On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > > > I think we need some clarity wrt. to the dependency order here. > > > Let's say we: > > > > In order to do this at branch point, we will need to move building this > > > > image into the compose process and mark it blocking. > > > OK, so we build things, but then we need to publish to registry.fp.o, > > > which is asynchrounous (?). When we test the newly built ISOs, do > > > > No, it happens at the end of the compose (if no blocking deliverables > > failed causing the compose to fail) > > > > > we test them also with the latest image that we get from registry.fp.o? > > > And if we find a bug anywhere in this pipeline, we respin everything? > > > > Good question. I'll note that currently we do not do any specific > > testing after branch point. We freeze things to get a compose to > > complete, but then we move on. It's not like Beta or Final. > > > > > > I'd like to note that making this blocking doesn't waive any kind of > > > > magic wand that makes our infrastructure more reliable. ;) > > > > The container build pipeline is a long collection of fragile things. > > > > It may well result in us slipping more based on things not working. ;( > > > > > > Hmm, quoting from https://pagure.io/releng/issue/11092: > > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > > > >> should probably try to do a full redeploy :-( > > > > We can't upgrade it from f33 because docker is no longer in f34+ and > > > > openshift origin / 3.11 doesn't support any newer either. > > > > > > Is this still true? I don't think we want to make the Fedora release > > > process contingent on something that requires F33. > > > > yes, it's still true. Note thats the aarch64 osbs cluster. > > The x86_64 one is rhel7. > > > > So, perhaps it would make sense to only make the x86_64 one blocking? > > Or possibly build it with osbuild, the infra there is pretty stable > now for IoT, it's not perfect, but it does have x86_64/aarch64. We have multiple options that aren't the OSBS stuff: * kiwi (there's a koji plugin for it) * osbuild * lorax I'd be happy to help with kiwi (as one of the upstream developers of it). The Fedora Asahi, CentOS Hyperscale, and CentOS Alternative Image teams have been using kiwi to build images for a while now. :) -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 8, 2023 at 9:11 PM Kevin Fenzi wrote: > > On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > I think we need some clarity wrt. to the dependency order here. > > Let's say we: > > > In order to do this at branch point, we will need to move building this > > > image into the compose process and mark it blocking. > > OK, so we build things, but then we need to publish to registry.fp.o, > > which is asynchrounous (?). When we test the newly built ISOs, do > > No, it happens at the end of the compose (if no blocking deliverables > failed causing the compose to fail) > > > we test them also with the latest image that we get from registry.fp.o? > > And if we find a bug anywhere in this pipeline, we respin everything? > > Good question. I'll note that currently we do not do any specific > testing after branch point. We freeze things to get a compose to > complete, but then we move on. It's not like Beta or Final. > > > > I'd like to note that making this blocking doesn't waive any kind of > > > magic wand that makes our infrastructure more reliable. ;) > > > The container build pipeline is a long collection of fragile things. > > > It may well result in us slipping more based on things not working. ;( > > > > Hmm, quoting from https://pagure.io/releng/issue/11092: > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > > >> should probably try to do a full redeploy :-( > > > We can't upgrade it from f33 because docker is no longer in f34+ and > > > openshift origin / 3.11 doesn't support any newer either. > > > > Is this still true? I don't think we want to make the Fedora release > > process contingent on something that requires F33. > > yes, it's still true. Note thats the aarch64 osbs cluster. > The x86_64 one is rhel7. > > So, perhaps it would make sense to only make the x86_64 one blocking? Or possibly build it with osbuild, the infra there is pretty stable now for IoT, it's not perfect, but it does have x86_64/aarch64. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, 8 May 2023 at 22:11, Kevin Fenzi wrote: > On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski-Szmek > wrote: > > > > I think we need some clarity wrt. to the dependency order here. > > Let's say we: > > > In order to do this at branch point, we will need to move building this > > > image into the compose process and mark it blocking. > > OK, so we build things, but then we need to publish to registry.fp.o, > > which is asynchrounous (?). When we test the newly built ISOs, do > > No, it happens at the end of the compose (if no blocking deliverables > failed causing the compose to fail) > If we do this, we should also make the container base images release blocking since AFAIK they are currently not release blocking. > > we test them also with the latest image that we get from registry.fp.o? > > And if we find a bug anywhere in this pipeline, we respin everything? > > Good question. I'll note that currently we do not do any specific > testing after branch point. We freeze things to get a compose to > complete, but then we move on. It's not like Beta or Final. > > > > I'd like to note that making this blocking doesn't waive any kind of > > > magic wand that makes our infrastructure more reliable. ;) > > > The container build pipeline is a long collection of fragile things. > > > It may well result in us slipping more based on things not working. ;( > > > > Hmm, quoting from https://pagure.io/releng/issue/11092: > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > > >> should probably try to do a full redeploy :-( > > > We can't upgrade it from f33 because docker is no longer in f34+ and > > > openshift origin / 3.11 doesn't support any newer either. > > > > Is this still true? I don't think we want to make the Fedora release > > process contingent on something that requires F33. > > yes, it's still true. Note thats the aarch64 osbs cluster. > The x86_64 one is rhel7. > > So, perhaps it would make sense to only make the x86_64 one blocking? > kevin > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On 5/8/23 19:09, Neal Gompa wrote: > On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour > wrote: >> >> On 5/8/23 18:49, Kevin Fenzi wrote: >>> On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: Dear Kevin, >> Hmm, quoting from https://pagure.io/releng/issue/11092: Also the aarch64 cluster is running on Fedora 33 boxes, so we should probably try to do a full redeploy :-( >>> We can't upgrade it from f33 because docker is no longer in f34+ and >>> openshift origin / 3.11 doesn't support any newer either. >> >> Is this still true? I don't think we want to make the Fedora release >> process contingent on something that requires F33. > > yes, it's still true. Note thats the aarch64 osbs cluster. > The x86_64 one is rhel7. Might it be possible to replace Docker with CRI-O on the OpenShift cluster? >>> >>> Nope. openshift 3 / osbs1 uses docker. :( >>> >>> kevin >> >> alias docker=podman > > Using Podman with it through Podman's implementation of the Docker > Host protocol *may* work, but a version of Podman that has it is not > currently available for RHEL 7 (which is what OpenShift 3.11 runs on). Time for an OpenShift upgrade? -- Sincerely, Demi Marie Obenour (she/her/hers) OpenPGP_0xB288B55FFF9C22C1.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour wrote: > > On 5/8/23 18:49, Kevin Fenzi wrote: > > On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: > >> Dear Kevin, > >> > Hmm, quoting from https://pagure.io/releng/issue/11092: > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > >> should probably try to do a full redeploy :-( > > We can't upgrade it from f33 because docker is no longer in f34+ and > > openshift origin / 3.11 doesn't support any newer either. > > Is this still true? I don't think we want to make the Fedora release > process contingent on something that requires F33. > >>> > >>> yes, it's still true. Note thats the aarch64 osbs cluster. > >>> The x86_64 one is rhel7. > >> > >> Might it be possible to replace Docker with CRI-O on the OpenShift > >> cluster? > > > > Nope. openshift 3 / osbs1 uses docker. :( > > > > kevin > > alias docker=podman Using Podman with it through Podman's implementation of the Docker Host protocol *may* work, but a version of Podman that has it is not currently available for RHEL 7 (which is what OpenShift 3.11 runs on). -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On 5/8/23 18:49, Kevin Fenzi wrote: > On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: >> Dear Kevin, >> Hmm, quoting from https://pagure.io/releng/issue/11092: >> Also the aarch64 cluster is running on Fedora 33 boxes, so we >> should probably try to do a full redeploy :-( > We can't upgrade it from f33 because docker is no longer in f34+ and > openshift origin / 3.11 doesn't support any newer either. Is this still true? I don't think we want to make the Fedora release process contingent on something that requires F33. >>> >>> yes, it's still true. Note thats the aarch64 osbs cluster. >>> The x86_64 one is rhel7. >> >> Might it be possible to replace Docker with CRI-O on the OpenShift >> cluster? > > Nope. openshift 3 / osbs1 uses docker. :( > > kevin alias docker=podman -- Sincerely, Demi Marie Obenour (she/her/hers) OpenPGP_0xB288B55FFF9C22C1.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote: > Dear Kevin, > > > > Hmm, quoting from https://pagure.io/releng/issue/11092: > > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > > > >> should probably try to do a full redeploy :-( > > > > We can't upgrade it from f33 because docker is no longer in f34+ and > > > > openshift origin / 3.11 doesn't support any newer either. > > > > > > Is this still true? I don't think we want to make the Fedora release > > > process contingent on something that requires F33. > > > > yes, it's still true. Note thats the aarch64 osbs cluster. > > The x86_64 one is rhel7. > > Might it be possible to replace Docker with CRI-O on the OpenShift > cluster? Nope. openshift 3 / osbs1 uses docker. :( kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On 5/8/23 16:10, Stephen Smoogen wrote: > On Mon, 8 May 2023 at 15:58, Zbigniew Jędrzejewski-Szmek > wrote: > >> On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote: >> >>> I'd like to note that making this blocking doesn't waive any kind of >>> magic wand that makes our infrastructure more reliable. ;) >>> The container build pipeline is a long collection of fragile things. >>> It may well result in us slipping more based on things not working. ;( >> >> Hmm, quoting from https://pagure.io/releng/issue/11092: Also the aarch64 cluster is running on Fedora 33 boxes, so we should probably try to do a full redeploy :-( >>> We can't upgrade it from f33 because docker is no longer in f34+ and >>> openshift origin / 3.11 doesn't support any newer either. >> >> Is this still true? I don't think we want to make the Fedora release >> process contingent on something that requires F33. >> >> ``` > $ sudo -i ssh osbs-aarch64-node01.iad2.fedoraproject.org cat > /etc/system-release > Fedora release 33 (Thirty Three) > ``` > > My memory of this is that this is not an easy thing to 'fix'. Podman instead of Docker? -- Sincerely, Demi Marie Obenour (she/her/hers) OpenPGP_0xB288B55FFF9C22C1.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
Dear Kevin, > > Hmm, quoting from https://pagure.io/releng/issue/11092: > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > > >> should probably try to do a full redeploy :-( > > > We can't upgrade it from f33 because docker is no longer in f34+ and > > > openshift origin / 3.11 doesn't support any newer either. > > > > Is this still true? I don't think we want to make the Fedora release > > process contingent on something that requires F33. > > yes, it's still true. Note thats the aarch64 osbs cluster. > The x86_64 one is rhel7. Might it be possible to replace Docker with CRI-O on the OpenShift cluster? Best wishes, Sebastian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 08, 2023 at 07:57:30PM +, Zbigniew Jędrzejewski-Szmek wrote: > > I think we need some clarity wrt. to the dependency order here. > Let's say we: > > In order to do this at branch point, we will need to move building this > > image into the compose process and mark it blocking. > OK, so we build things, but then we need to publish to registry.fp.o, > which is asynchrounous (?). When we test the newly built ISOs, do No, it happens at the end of the compose (if no blocking deliverables failed causing the compose to fail) > we test them also with the latest image that we get from registry.fp.o? > And if we find a bug anywhere in this pipeline, we respin everything? Good question. I'll note that currently we do not do any specific testing after branch point. We freeze things to get a compose to complete, but then we move on. It's not like Beta or Final. > > I'd like to note that making this blocking doesn't waive any kind of > > magic wand that makes our infrastructure more reliable. ;) > > The container build pipeline is a long collection of fragile things. > > It may well result in us slipping more based on things not working. ;( > > Hmm, quoting from https://pagure.io/releng/issue/11092: > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > >> should probably try to do a full redeploy :-( > > We can't upgrade it from f33 because docker is no longer in f34+ and > > openshift origin / 3.11 doesn't support any newer either. > > Is this still true? I don't think we want to make the Fedora release > process contingent on something that requires F33. yes, it's still true. Note thats the aarch64 osbs cluster. The x86_64 one is rhel7. So, perhaps it would make sense to only make the x86_64 one blocking? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, 8 May 2023 at 15:58, Zbigniew Jędrzejewski-Szmek wrote: > On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote: > > > I'd like to note that making this blocking doesn't waive any kind of > > magic wand that makes our infrastructure more reliable. ;) > > The container build pipeline is a long collection of fragile things. > > It may well result in us slipping more based on things not working. ;( > > Hmm, quoting from https://pagure.io/releng/issue/11092: > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we > >> should probably try to do a full redeploy :-( > > We can't upgrade it from f33 because docker is no longer in f34+ and > > openshift origin / 3.11 doesn't support any newer either. > > Is this still true? I don't think we want to make the Fedora release > process contingent on something that requires F33. > > ``` $ sudo -i ssh osbs-aarch64-node01.iad2.fedoraproject.org cat /etc/system-release Fedora release 33 (Thirty Three) ``` My memory of this is that this is not an easy thing to 'fix'. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote: > I'm broadly in favor here, some comments in line... > > ...snip... > > First, we want to ensure that there are up to date > > [https://src.fedoraproject.org/container/fedora-toolbox > > fedora-toolbox] OCI images published on > > [https://registry.fedoraproject.org/ registry.fedoraproject.org] as > > release-blocking deliverables at critical points in the development > > schedule, just like the installation ISOs for the Editions from > > [https://download.fedoraproject.org/pub/fedora/linux/releases/ > > download.fedoraproject.org]. This must at least happen when an > > upcoming Fedora release is branched from Rawhide, and for the Beta and > > Final release candidates. If possible, they should be updated more > > frequently as part of the nightly composes. We do not expect this to > > happen after a Fedora release has gone GA. I think we need some clarity wrt. to the dependency order here. Let's say we: > In order to do this at branch point, we will need to move building this > image into the compose process and mark it blocking. OK, so we build things, but then we need to publish to registry.fp.o, which is asynchrounous (?). When we test the newly built ISOs, do we test them also with the latest image that we get from registry.fp.o? And if we find a bug anywhere in this pipeline, we respin everything? > ...snip... > > It will be beneficial to consider the > > [https://src.fedoraproject.org/container/fedora-toolbox > > fedora-toolbox] images as release-blocking deliverables because > > Fedora's [https://opencontainers.org/ OCI] infrastructure is often > > broken. Here are [https://pagure.io/releng/issue/11092 two] > > [https://pagure.io/releng/issue/11367 recent] examples of fedpkg > > container-build not working. In the second case, it was > > preventing the images from being rebuilt to pull in an > > [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important] > > bug-fix. The broken infrastructure prevents regular Fedora > > contributors from jumping in to rebuild and publish the images at > > critical points in the development schedule. Making them > > release-blocking deliverables would attract greater attention and > > scrutiny from release engineering and ensure that a Fedora development > > cycle does not proceed with broken or outdated or missing > > fedora-toolbox images. > > I'd like to note that making this blocking doesn't waive any kind of > magic wand that makes our infrastructure more reliable. ;) > The container build pipeline is a long collection of fragile things. > It may well result in us slipping more based on things not working. ;( Hmm, quoting from https://pagure.io/releng/issue/11092: >> Also the aarch64 cluster is running on Fedora 33 boxes, so we >> should probably try to do a full redeploy :-( > We can't upgrade it from f33 because docker is no longer in f34+ and > openshift origin / 3.11 doesn't support any newer either. Is this still true? I don't think we want to make the Fedora release process contingent on something that requires F33. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
On Mon, May 08, 2023 at 04:49:09PM +0100, Barry wrote: > > fedora-toolbox] [https://opencontainers.org/ OCI] images must be > This url does not exist. > > [https://containertoolbx.org/ Toolbx] to be usable when a new Fedora > Nor this one. Both work fine here. Maybe some temporary network glitch? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
I'm broadly in favor here, some comments in line... ...snip... > First, we want to ensure that there are up to date > [https://src.fedoraproject.org/container/fedora-toolbox > fedora-toolbox] OCI images published on > [https://registry.fedoraproject.org/ registry.fedoraproject.org] as > release-blocking deliverables at critical points in the development > schedule, just like the installation ISOs for the Editions from > [https://download.fedoraproject.org/pub/fedora/linux/releases/ > download.fedoraproject.org]. This must at least happen when an > upcoming Fedora release is branched from Rawhide, and for the Beta and > Final release candidates. If possible, they should be updated more > frequently as part of the nightly composes. We do not expect this to > happen after a Fedora release has gone GA. In order to do this at branch point, we will need to move building this image into the compose process and mark it blocking. Which hopefully we can do. ;) ...snip... > It will be beneficial to consider the > [https://src.fedoraproject.org/container/fedora-toolbox > fedora-toolbox] images as release-blocking deliverables because > Fedora's [https://opencontainers.org/ OCI] infrastructure is often > broken. Here are [https://pagure.io/releng/issue/11092 two] > [https://pagure.io/releng/issue/11367 recent] examples of fedpkg > container-build not working. In the second case, it was > preventing the images from being rebuilt to pull in an > [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important] > bug-fix. The broken infrastructure prevents regular Fedora > contributors from jumping in to rebuild and publish the images at > critical points in the development schedule. Making them > release-blocking deliverables would attract greater attention and > scrutiny from release engineering and ensure that a Fedora development > cycle does not proceed with broken or outdated or missing > fedora-toolbox images. I'd like to note that making this blocking doesn't waive any kind of magic wand that makes our infrastructure more reliable. ;) The container build pipeline is a long collection of fragile things. It may well result in us slipping more based on things not working. ;( kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)
> On 8 May 2023, at 14:51, Aoife Moloney wrote: > > https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker > > This document represents a proposed Change. As part of the Changes > process, proposals are publicly announced in order to receive > community feedback. This proposal will only be implemented if approved > by the Fedora Engineering Steering Committee. > > > == Summary == > > Up to date [https://src.fedoraproject.org/container/fedora-toolbox > fedora-toolbox] [https://opencontainers.org/ OCI] images must be This url does not exist. > published on [https://registry.fedoraproject.org/ > registry.fedoraproject.org] as release-blocking deliverables, and > there must be release-blocking test criteria to ensure usable > [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs. > > == Owner == > > * Name: [[User:Rishi|Debarshi Ray]], [[User:Sumantrom|Sumantro Mukherjee]] > * Email: debars...@redhat.com, sumuk...@redhat.com > > > > == Detailed Description == > Currently, there is no formal requirement for > [https://containertoolbx.org/ Toolbx] to be usable when a new Fedora Nor this one. Barry > is released. This is a problem for a tool that's so popular and > provides something as fundamental as an interactive command line > environment for software development and troubleshooting the host > operating system. This Change aims to address this. > > Toolbx has two parts — an [https://opencontainers.org/ OCI] image, > which defaults to > [https://src.fedoraproject.org/container/fedora-toolbox > fedora-toolbox] on Fedora hosts, and the > [https://src.fedoraproject.org/rpms/toolbox toolbox] RPM. The OCI > image is pulled by the RPM to set up a containerized interactive CLI > environment. > > First, we want to ensure that there are up to date > [https://src.fedoraproject.org/container/fedora-toolbox > fedora-toolbox] OCI images published on > [https://registry.fedoraproject.org/ registry.fedoraproject.org] as > release-blocking deliverables at critical points in the development > schedule, just like the installation ISOs for the Editions from > [https://download.fedoraproject.org/pub/fedora/linux/releases/ > download.fedoraproject.org]. This must at least happen when an > upcoming Fedora release is branched from Rawhide, and for the Beta and > Final release candidates. If possible, they should be updated more > frequently as part of the nightly composes. We do not expect this to > happen after a Fedora release has gone GA. > > Second, we want to have release-blocking test criteria to ensure > usable [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs at > critical points in the development schedule. This must be used to > ensure that both changes in the Toolbx stack, and future > [https://docs.fedoraproject.org/en-US/program_management/changes_policy/ > Changes] in other parts of the OS do not break Toolbx, and at least > cover the Beta and Final release candidates. > > Examples of changes in the Toolbx stack causing breakage can be > [https://bugzilla.redhat.com/show_bug.cgi?id=2183034 FUSE] preventing > RPMs with file capabilities from being installed inside Toolbx > containers, Toolbx [https://github.com/containers/toolbox/issues/643 > bind mounts] preventing RPMs with %attr() from being > installed or causing > [https://bugzilla.redhat.com/show_bug.cgi?id=2188304 > systemd-tmpfiles(8)] to throw errors, etc.. Examples of changes in > other parts of the OS can be changes to Fedora's Kerberos stack > causing Kerberos to stop working inside Toolbx containers, > [[Changes/EnableSysctlPingGroupRange|changes]] to the > sysctl(8) > [https://github.com/systemd/systemd/commit/90ce7627dfe824ff6e7c0ca5f96350fbcfec7118 > configuration] breaking ping(8), changes in > [https://github.com/containers/toolbox/issues/562 Mutter] breaking > graphical applications, etc.. > > Note that having release-blocking test criteria for the > toolbox RPM will also implicitly test the > fedora-toolbox image. > > == Feedback == > > > == Benefit to Fedora == > > This Change improves the quality of the containerized interactive > command line [https://containertoolbx.org/ Toolbx] environments on > Fedora by adding formal requirements to ensure that they are usable > when a new Fedora is released. This will bring them closer to the > reliability of similar environments running directly on the host > operating system. > > Toolbx is installed by default on CoreOS, Silverblue and Workstation. > It is indispensable for software developers using Silverblue to bypass > the difficulty of setting up a development environment in the usual > way. It is widely used, even on Workstation, by those who don't want > to pollute their host OS, or want to access a CLI environment that's > different from the host's without installing a virtual machine, or > want a pre-configured development environment. > > It will be beneficial to consider the > [https://src.fedoraproject.org/container/fedora-toolbox >
