Re: Workflow and other problems with the Fedora container infrastructure
On Tue, Jan 18, 2022 at 8:40 AM Pierre-Yves Chibon wrote: > > On Sun, Jan 16, 2022 at 08:42:30AM -0500, Josh Boyer wrote: > >On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi <[1]ke...@scrye.com> wrote: > > > > On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote: > > > > > > One of the things that has recently happened in the Koji space is the > > > addition of a kiwi-build task to build images using the KIWI tool[1]. > > > > > > KIWI supports building all kinds of operating system images, > > including > > > OCI containers. The Fedora Cloud WG is poking at the idea of using > > > KIWI for the cloud image to replace the unmaintained and brittle > > > ImageFactory, and we could also look at doing container builds with > > > KIWI to replace the OpenShift Atomic Reactor system. That would > > > drastically simplify the architecture and make container image builds > > > considerably more reasonable for the Container SIG and any other > > > stakeholders. > > > > Yeah, thats quite interesting. I would be happy to move to a pipeline > > thats less fragile here. :) > > > > There's also talk about moving things to use ImageBuilder, but I don't > > think it does containers. > > > >We can, and should, have RFEs for Image Builder to do containers. I know > >we need this internally as well. It may farm that out to buildah in the > >background or something, but that remains to be determined. > >In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I > >really think using Image Builder as the tool to build images is the best > >approach. > > The underlying tool, osbuild, can already build container. They can be made > available as tarball which one can just `podman import` afterward :) Yes. Why make people do that instead of just publishing an OCI compliant image? I know why, but that's the RFE :) josh ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Sun, Jan 16, 2022 at 08:42:30AM -0500, Josh Boyer wrote: >On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi <[1]ke...@scrye.com> wrote: > > On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote: > > > > One of the things that has recently happened in the Koji space is the > > addition of a kiwi-build task to build images using the KIWI tool[1]. > > > > KIWI supports building all kinds of operating system images, including > > OCI containers. The Fedora Cloud WG is poking at the idea of using > > KIWI for the cloud image to replace the unmaintained and brittle > > ImageFactory, and we could also look at doing container builds with > > KIWI to replace the OpenShift Atomic Reactor system. That would > > drastically simplify the architecture and make container image builds > > considerably more reasonable for the Container SIG and any other > > stakeholders. > > Yeah, thats quite interesting. I would be happy to move to a pipeline > thats less fragile here. :) > > There's also talk about moving things to use ImageBuilder, but I don't > think it does containers. > >We can, and should, have RFEs for Image Builder to do containers. I know >we need this internally as well. It may farm that out to buildah in the >background or something, but that remains to be determined. >In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I >really think using Image Builder as the tool to build images is the best >approach. The underlying tool, osbuild, can already build container. They can be made available as tarball which one can just `podman import` afterward :) Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Thu, Jan 13, 2022, at 1:48 PM, Kevin Fenzi wrote: > > > Perhaps the Fedora CoreOS folks would have some thoughts? I can't speak for the whole team, but a few points. First, the FCOS build tooling in https://github.com/coreos/coreos-assembler is designed to run as a standard container. In some cases we do run it via podman over ssh as part of multi-arch, but the main approach is to run it inside Kubernetes (OpenShift). We designed it this way because OpenShift is of great interest to at least my employer (in case anyone didn't know). That's how we run production container workloads. Until now however, we have really had a very interesting tension because the primary output of FCOS builds is not a container image, it's bootable disk images (as produced by many tools, including ImageFactory, Image Builder, the kiwi thing in this thread, and many others). However, https://fedoraproject.org/wiki/Changes/OstreeNativeContainer is going to shift our "center of gravity" much closer to a container build. I'd actually like to "decouple" the disk image builds from container builds in our pipeline more, basically so that we generate disk images using a container image as *input* - for FCOS as well as other ostree systems today, a bootable disk image is really just a platform-specific wrapper shell around that. Related to this, I am quite strongly of the opinion that the *build* system should be closely related to the *testing* system. And that relates to the "running in production" bits mentioned above. If we're building containers, then we should at least be testing them running inside a Kubernetes/OpenShift instance. And if you have that, then it just makes sense to use the same approach to run the build tooling - as a container. The build process is just another workload along with testing processes and other tools inside a production Kubernetes/OpenShift cluster. This is how it works today for the FCOS pipeline as well as downstream ones, and as mentioned above I think the ostree native container change will be a powerful incentive to "lift" the ostree side of things outside of Koji and into a container-native flow. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Sun, Jan 16, 2022 at 8:43 AM Josh Boyer wrote: > > > > On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi wrote: >> >> On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote: >> > >> > One of the things that has recently happened in the Koji space is the >> > addition of a kiwi-build task to build images using the KIWI tool[1]. >> > >> > KIWI supports building all kinds of operating system images, including >> > OCI containers. The Fedora Cloud WG is poking at the idea of using >> > KIWI for the cloud image to replace the unmaintained and brittle >> > ImageFactory, and we could also look at doing container builds with >> > KIWI to replace the OpenShift Atomic Reactor system. That would >> > drastically simplify the architecture and make container image builds >> > considerably more reasonable for the Container SIG and any other >> > stakeholders. >> >> Yeah, thats quite interesting. I would be happy to move to a pipeline >> thats less fragile here. :) >> >> There's also talk about moving things to use ImageBuilder, but I don't >> think it does containers. > > > We can, and should, have RFEs for Image Builder to do containers. I know we > need this internally as well. It may farm that out to buildah in the > background or something, but that remains to be determined. > The "Image Builder" architecture is almost as bad as OpenShift Atomic Reactor. It requires a whole separate service architecture and can't directly execute inside of a Koji mock environment. That means that it's almost as difficult and fragile to keep up with. Moreover, it can't do nearly all of the image types that Fedora needs, notably live media, install DVDs, or container images. The only thing it does that almost all other tools do not is RPM-OSTree commits. I would like OSBuild to work in Fedora the same way most of our other image build tools do: fork+exec in dynamically spun up mock roots rather than several services across multiple dedicated machines. The ones that don't follow that pattern tend to be the poorly maintained ones, such as ImageFactory and OpenShift Atomic Reactor. > In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I > really think using Image Builder as the tool to build images is the best > approach. > In the interest of the Fedora ecosystem, we should use the best tools that people can use. But also keep in mind that we don't have to use a single tool for all the Fedora's image tooling. Heck, right now we use over half a dozen different ones (including different tools for producing RPM-OSTree commits depending on the Fedora variant). Going down to 2 or 3 that are much easier for people to self-host and run is better than the current situation. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi wrote: > On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote: > > > > One of the things that has recently happened in the Koji space is the > > addition of a kiwi-build task to build images using the KIWI tool[1]. > > > > KIWI supports building all kinds of operating system images, including > > OCI containers. The Fedora Cloud WG is poking at the idea of using > > KIWI for the cloud image to replace the unmaintained and brittle > > ImageFactory, and we could also look at doing container builds with > > KIWI to replace the OpenShift Atomic Reactor system. That would > > drastically simplify the architecture and make container image builds > > considerably more reasonable for the Container SIG and any other > > stakeholders. > > Yeah, thats quite interesting. I would be happy to move to a pipeline > thats less fragile here. :) > > There's also talk about moving things to use ImageBuilder, but I don't > think it does containers. We can, and should, have RFEs for Image Builder to do containers. I know we need this internally as well. It may farm that out to buildah in the background or something, but that remains to be determined. In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I really think using Image Builder as the tool to build images is the best approach. josh ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote: > > One of the things that has recently happened in the Koji space is the > addition of a kiwi-build task to build images using the KIWI tool[1]. > > KIWI supports building all kinds of operating system images, including > OCI containers. The Fedora Cloud WG is poking at the idea of using > KIWI for the cloud image to replace the unmaintained and brittle > ImageFactory, and we could also look at doing container builds with > KIWI to replace the OpenShift Atomic Reactor system. That would > drastically simplify the architecture and make container image builds > considerably more reasonable for the Container SIG and any other > stakeholders. Yeah, thats quite interesting. I would be happy to move to a pipeline thats less fragile here. :) There's also talk about moving things to use ImageBuilder, but I don't think it does containers. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Tue, Jan 11, 2022 at 11:51 PM Kevin Fenzi wrote: > > On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote: > > Hello. > > > > My name is Lumír and I maintain a few layered container images in > > Fedora/RHEL/upstream related to Python and S2I (source to image) platform - > > namely python3, s2i-core, and s2i-base. > > > > TL;DR: We (maintainers of layered container images for languages, runtimes > > and databases) are considering migrating our images from Fedora > > infrastructure and Fedora containers registry somewhere else. Below are the > > reasons for that. If you think it’s a bad idea, let us know why. If you are > > a user of those images, let us know as well so we can stay in touch with > > you. > > > > Forgive me for my wording but I’m really sad about the current situation. > > yeah, it is sad. ;( > > > I have to honestly admit that I’m disappointed about the lack of attention > > containers in Fedora are getting. Maintaining them is painful, > > infrastructure is unstable and unreliable and there was literally no > > movement forward in recent years. Let me summarize it. > > > > Container SIG is inactive. See important issues opened for years with no > > movement further: https://pagure.io/ContainerSIG/container-sig/issues > > > > Even a simple task like merging rawhide branch to f35 for example is far > > from smooth because they are never the same, because manual changes are > > required for each branch: > > https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no > > reply) > > > > S2I container images depend on each other (python3 -> s2i-base -> s2i-core) > > but there is no automation for chain rebuilds so if I want to update > > s2i-core image (to fix a CVE or something like that) and all dependant > > images, it takes weeks because I have to rebuild them one by one and let > > them one by one pass through Bodhi updates until I can rebuild next one in > > the queue. And it’s really frustrating given the insufficient stability of > > the infra. There seems to be a way how to make this faster, but nobody > > knows, how it works: https://pagure.io/ContainerSIG/container-sig/issue/52 > > and https://github.com/containerbuildsystem/atomic-reactor/issues/1730 > > > > The same applies also if the fix is done on the RPM level because there are > > no freshmaker nor automated rebuilds. > > > > The branching structure does not really fit container needs. I maintain the > > s2i Python container and I don’t really care whether it’s based on Fedora 34 > > or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch > > for Python 3.10, one for 3.9 etc. instead of two separated branches (f34, > > f35) where the Python is the same version. > > https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old) > > Yeah, I think all the above is kind of a consequence of there not being > a active containers SIG. Infrastructure and releng doesn't really have > cycles or interest in driving containers forward, so it needs some other > folks attention. ;( > > > An example of unstable infrastructure: I’m not able to build an image for > > three weeks now and a response I got is to report the problem upstream: > > https://pagure.io/fedora-infrastructure/issue/10437 > > I know that nothing is 100% reliable but if you take a look at these issues, > > there are a lot of them: > > https://pagure.io/fedora-infrastructure/issues?search_pattern=container=Closed > > Do note that that issue was filed on December 21st. > Myself and much of the infrastructure team were off on holidays. > I only got back yesterday and there's a ton of things to do, but I did > try and look into this some. > > The reason there's so many issues I think is because the entire pipeline > is just way overkill and complex for just building containers. Basically > we have to have koji, koji builders, and 2 openshift clusters (one for > x86 and one for aarch64) to build containers, they have all kinds of > fragile handoffs between each other as well. > > > And it seems that the packages in the infrastructure are outdated a lot: > > https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuecomment-1009279161 > > To clarify, thats installed in the build container... so, _fedora_ > packages are outdated. ;( I have seen atomic-reactor being old, but I've > been afraid to update it since I don't know the pipeline. > Co-maintainers of those packages in Fedora would sure help out. > One of the things that has recently happened in the Koji space is the addition of a kiwi-build task to build images using the KIWI tool[1]. KIWI supports building all kinds of operating system images, including OCI containers. The Fedora Cloud WG is poking at the idea of using KIWI for the cloud image to replace the unmaintained and brittle ImageFactory, and we could also look at doing container builds with KIWI to replace the OpenShift Atomic Reactor system. That would drastically simplify the architecture and make
Re: Workflow and other problems with the Fedora container infrastructure
On Thu, Jan 13, 2022 at 02:43:50PM +0100, Honza Horak wrote: > On Wed, Jan 12, 2022 at 5:51 AM Kevin Fenzi wrote: > > > On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote: > > > If we migrate our container images to some other registry (e.g. a common > > > fedora space on quay.io), we’ll be able to rebuild them after every > > merged > > > PR or every week, do chain rebuilds and push them to the registry > > directly > > > from Github CI. This will make our lives significantly easier, and in > > turn > > > our images better. > > > > We are looking into dropping out registry and moving to quay.io > > ourselves. ;) > > > > > Is this something more than a thought at this point? Something where we can > follow the design and progress? We have https://pagure.io/fedora-infrastructure/issue/10386 tracking it. It's definitely something we want to do, but not sure when we will get to it. > > > Are there any benefits to using Fedora infrastructure and Fedora > > containers > > > registry which I don’t see that we should consider? > > > > I'm not sure... hopefully folks who know the pipeline will chime in > > here. > > > > If you know who the folks are, we'd be glad for a gentle ping, so they > comment here and we do not stop providing something that other would start > to miss. Perhaps the Fedora CoreOS folks would have some thoughts? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Wed, Jan 12, 2022 at 5:51 AM Kevin Fenzi wrote: > On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote: > > If we migrate our container images to some other registry (e.g. a common > > fedora space on quay.io), we’ll be able to rebuild them after every > merged > > PR or every week, do chain rebuilds and push them to the registry > directly > > from Github CI. This will make our lives significantly easier, and in > turn > > our images better. > > We are looking into dropping out registry and moving to quay.io > ourselves. ;) > > Is this something more than a thought at this point? Something where we can follow the design and progress? > > Are there any benefits to using Fedora infrastructure and Fedora > containers > > registry which I don’t see that we should consider? > > I'm not sure... hopefully folks who know the pipeline will chime in > here. > If you know who the folks are, we'd be glad for a gentle ping, so they comment here and we do not stop providing something that other would start to miss. Thanks, Honza > kevin > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Workflow and other problems with the Fedora container infrastructure
On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote: > Hello. > > My name is Lumír and I maintain a few layered container images in > Fedora/RHEL/upstream related to Python and S2I (source to image) platform - > namely python3, s2i-core, and s2i-base. > > TL;DR: We (maintainers of layered container images for languages, runtimes > and databases) are considering migrating our images from Fedora > infrastructure and Fedora containers registry somewhere else. Below are the > reasons for that. If you think it’s a bad idea, let us know why. If you are > a user of those images, let us know as well so we can stay in touch with > you. > > Forgive me for my wording but I’m really sad about the current situation. yeah, it is sad. ;( > I have to honestly admit that I’m disappointed about the lack of attention > containers in Fedora are getting. Maintaining them is painful, > infrastructure is unstable and unreliable and there was literally no > movement forward in recent years. Let me summarize it. > > Container SIG is inactive. See important issues opened for years with no > movement further: https://pagure.io/ContainerSIG/container-sig/issues > > Even a simple task like merging rawhide branch to f35 for example is far > from smooth because they are never the same, because manual changes are > required for each branch: > https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no > reply) > > S2I container images depend on each other (python3 -> s2i-base -> s2i-core) > but there is no automation for chain rebuilds so if I want to update > s2i-core image (to fix a CVE or something like that) and all dependant > images, it takes weeks because I have to rebuild them one by one and let > them one by one pass through Bodhi updates until I can rebuild next one in > the queue. And it’s really frustrating given the insufficient stability of > the infra. There seems to be a way how to make this faster, but nobody > knows, how it works: https://pagure.io/ContainerSIG/container-sig/issue/52 > and https://github.com/containerbuildsystem/atomic-reactor/issues/1730 > > The same applies also if the fix is done on the RPM level because there are > no freshmaker nor automated rebuilds. > > The branching structure does not really fit container needs. I maintain the > s2i Python container and I don’t really care whether it’s based on Fedora 34 > or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch > for Python 3.10, one for 3.9 etc. instead of two separated branches (f34, > f35) where the Python is the same version. > https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old) Yeah, I think all the above is kind of a consequence of there not being a active containers SIG. Infrastructure and releng doesn't really have cycles or interest in driving containers forward, so it needs some other folks attention. ;( > An example of unstable infrastructure: I’m not able to build an image for > three weeks now and a response I got is to report the problem upstream: > https://pagure.io/fedora-infrastructure/issue/10437 > I know that nothing is 100% reliable but if you take a look at these issues, > there are a lot of them: > https://pagure.io/fedora-infrastructure/issues?search_pattern=container=Closed Do note that that issue was filed on December 21st. Myself and much of the infrastructure team were off on holidays. I only got back yesterday and there's a ton of things to do, but I did try and look into this some. The reason there's so many issues I think is because the entire pipeline is just way overkill and complex for just building containers. Basically we have to have koji, koji builders, and 2 openshift clusters (one for x86 and one for aarch64) to build containers, they have all kinds of fragile handoffs between each other as well. > And it seems that the packages in the infrastructure are outdated a lot: > https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuecomment-1009279161 To clarify, thats installed in the build container... so, _fedora_ packages are outdated. ;( I have seen atomic-reactor being old, but I've been afraid to update it since I don't know the pipeline. Co-maintainers of those packages in Fedora would sure help out. > If we migrate our container images to some other registry (e.g. a common > fedora space on quay.io), we’ll be able to rebuild them after every merged > PR or every week, do chain rebuilds and push them to the registry directly > from Github CI. This will make our lives significantly easier, and in turn > our images better. We are looking into dropping out registry and moving to quay.io ourselves. ;) > Are there any benefits to using Fedora infrastructure and Fedora containers > registry which I don’t see that we should consider? I'm not sure... hopefully folks who know the pipeline will chime in here. kevin signature.asc Description: PGP signature ___ devel
