Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Sun, Jan 06, 2019 at 12:35:25PM -0500, Paul Frields wrote: > Update: We'll be having discussions over the next few weeks to figure > out how we can accommodate Lifecycle work without interrupting the > Fedora release cadence. That includes in Brno, where we can meet up > f2f with people working on parallel projects in automation, CI, and so > on. We may also be able to grab additional people to do work here -- > so this doesn't automatically equate to "we have to do more with > less/the same people." After talking with the Council as well as some > folks who are depending on the cadence, like IoT, it was clear we need > to look at this option. For now, I intend to remove the cadence change > from the objective requirements, unless more specific reasons why it > has to happen become clear. It's getting to be time to lay out the F31 schedule. Let's go ahead with the assumption that we'll use the regular end-of-October target. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
Update: We'll be having discussions over the next few weeks to figure out how we can accommodate Lifecycle work without interrupting the Fedora release cadence. That includes in Brno, where we can meet up f2f with people working on parallel projects in automation, CI, and so on. We may also be able to grab additional people to do work here -- so this doesn't automatically equate to "we have to do more with less/the same people." After talking with the Council as well as some folks who are depending on the cadence, like IoT, it was clear we need to look at this option. For now, I intend to remove the cadence change from the objective requirements, unless more specific reasons why it has to happen become clear. On Wed, Nov 28, 2018 at 1:05 PM Randy Barlow wrote: > > On Wed, 2018-11-28 at 18:19 +0100, Kevin Kofler wrote: > > And why do we need to stop the presses to do that? This is entirely > > orthogonal to distribution development and can be done in parallel. > > The problem is that the people who do the work to get the release done > are the same people who would need to do the retooling work, so there > is a resource conflict. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Wed, 2018-11-28 at 18:19 +0100, Kevin Kofler wrote: > And why do we need to stop the presses to do that? This is entirely > orthogonal to distribution development and can be done in parallel. The problem is that the people who do the work to get the release done are the same people who would need to do the retooling work, so there is a resource conflict. signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
Matthew Miller wrote: > I know it was a big time-off holiday week in the US, but I expected a > little more interest in this post. Perhaps it seemed like too much text to > digest along with turkey and stuffing. :) I'm highlighting it with a > subject reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) > * really actually for real gated Rawhide > * better CI pipeline tests for everything > * define a base platform -- Red Hat wants to focus resources here > * better tooling for non-base deliverables > * better metrics for everything And why do we need to stop the presses to do that? This is entirely orthogonal to distribution development and can be done in parallel. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, 2018-11-27 at 19:30 +, Peter Robinson wrote: > On Tue, Nov 27, 2018 at 6:26 PM Paul Frields wrote: > > On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore wrote: > > [...snip...] > > > > The customers RH serves have specific expectations, and in part that > > > > dictates how delivery tooling is done. Binding the community to that > > > > may be counterproductive. This is especially true now that RHEL 8 Beta > > > > is out -- it's the perfect time for us to be rethinking at that level. > > > > > > Compose tooling is largely shared, delivery tooling is not, though > > > there is room for improvement and convergence on both sides. We should > > > also consider CentOS and how we can share tooling, process and > > > resources, they still use something else entirely. Everything used in > > > Fedora is developed in the open and can be contributed to by anyone. > > > We have been talking about making the whole process flexible and > > > dynamic for years. I know internally they would like to have many of > > > the same things Fedora wants. > > > > Yeah, it sounds like we're aiming in the same direction here. This is > > part of what I mean by "decomposing the compose." We have one rather > > monolithic process right now, all or nothing. And it does so much that > > it's impossible to get any fast results from it. That doesn't mean it > > doesn't work well for what it does. But it's not very flexible, as you > > pointed out. > > Actually amusingly if you look back through the history post F-21 and > the massive delays that happened due to various non monolithic > processes one of the outcomes of the "get a release done with > Fedora.next" was to be able to do full composes every day because > prior to that we only did newRepo, a network install and live/cloud > images and often we'd get to Alpha TC1 right after branching to > discover a whole bunch of stuff was completely broken. We finally got > this with the move to "pungi 4" in Fedora 24, and in the intervening > releases we had huge amounts of complaints because rel-eng couldn't > provide all release artifacts to be pushed through CI/CD so > regressions could be fixed instantaneously. Yes, I'm aware things > change, and I often push them, but it's worth knowing the history how > we got to where we are! Sure, I know about this (I did quite a bit of work around it). But I don't see any kind of inconsistency or anything here. Yes, we needed to ability to compose the whole distro in a saner and more organized way, which is what Pungi 4 did, which was a significant improvement. But it would *also* be extremely useful to be able to do smaller, faster subsets of the compose. It's not like that's what we were doing before, after all. As you say, we still only did it *nightly*. And since it still involved a newRepo, it still took forever. > > With a more flexible compose we could choose to do smaller tasks (as > > well as capitalize on other time savings) for something like a > > speculative testing build. That kind of build could be used to perform > > integration testing as well as other tasks (automated, not manual!). > > Yes, and in some regards there's already parts of the project doing > this, the TWA release runs like this, as does the IoT release and I > mentioned above ways to break some of the bits up more. I know about this too, and mentioned it in my initial mail in this subthread. But it's all been done in a manual, ad-hoc way: we just write entirely new Pungi config files for each new compose profile we come up with. There's no notion of modularity, of being able to actually define the individual elements and easily combine and recombine them. There's no notion of inputs mapped to outputs, either, really. I don't think this approach really scales much further than we're currently doing it; I don't think it'd be practical to (for instance) manually split the big 'Fedora' compose into 'Fedora-Live', 'Fedora-ISO', 'Fedora-ARM-Disk', etc. etc. etc. Trying to do that *just along the same lines as we have created the existing 'new' composes* would probably give us an unmaintainable mess. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, Nov 27, 2018 at 6:26 PM Paul Frields wrote: > > On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore wrote: > [...snip...] > > > The customers RH serves have specific expectations, and in part that > > > dictates how delivery tooling is done. Binding the community to that > > > may be counterproductive. This is especially true now that RHEL 8 Beta > > > is out -- it's the perfect time for us to be rethinking at that level. > > > > Compose tooling is largely shared, delivery tooling is not, though > > there is room for improvement and convergence on both sides. We should > > also consider CentOS and how we can share tooling, process and > > resources, they still use something else entirely. Everything used in > > Fedora is developed in the open and can be contributed to by anyone. > > We have been talking about making the whole process flexible and > > dynamic for years. I know internally they would like to have many of > > the same things Fedora wants. > > Yeah, it sounds like we're aiming in the same direction here. This is > part of what I mean by "decomposing the compose." We have one rather > monolithic process right now, all or nothing. And it does so much that > it's impossible to get any fast results from it. That doesn't mean it > doesn't work well for what it does. But it's not very flexible, as you > pointed out. Actually amusingly if you look back through the history post F-21 and the massive delays that happened due to various non monolithic processes one of the outcomes of the "get a release done with Fedora.next" was to be able to do full composes every day because prior to that we only did newRepo, a network install and live/cloud images and often we'd get to Alpha TC1 right after branching to discover a whole bunch of stuff was completely broken. We finally got this with the move to "pungi 4" in Fedora 24, and in the intervening releases we had huge amounts of complaints because rel-eng couldn't provide all release artifacts to be pushed through CI/CD so regressions could be fixed instantaneously. Yes, I'm aware things change, and I often push them, but it's worth knowing the history how we got to where we are! > With a more flexible compose we could choose to do smaller tasks (as > well as capitalize on other time savings) for something like a > speculative testing build. That kind of build could be used to perform > integration testing as well as other tasks (automated, not manual!). Yes, and in some regards there's already parts of the project doing this, the TWA release runs like this, as does the IoT release and I mentioned above ways to break some of the bits up more. > We could even potentially test against much of the repetitive, > time-consuming matrices manually (and heroically) run by QA. Then only > choose to do more extensive tasks based on that being "green." Completely agree, but there's quite a bit of work to be done to get to that unicorn but none of the process to get there has been documented or even proposed in this thread, it's not easy (believe me a lot of people have looked over the years I've been involved in rel-eng) and most of what I've seen of actual concrete means to get there won't get us anywhere close to that, while yes it will still improve it. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, 2018-11-27 at 11:56 +0100, Vít Ondruch wrote: > Does that mean that Bodhi will be finally enabled for Rawhide or not? > Because otherwise I can't see how Bodhi is related here. Indeed, the plan is to make Bodhi manage Rawhide as well. signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore wrote: [...snip...] > > The customers RH serves have specific expectations, and in part that > > dictates how delivery tooling is done. Binding the community to that > > may be counterproductive. This is especially true now that RHEL 8 Beta > > is out -- it's the perfect time for us to be rethinking at that level. > > Compose tooling is largely shared, delivery tooling is not, though > there is room for improvement and convergence on both sides. We should > also consider CentOS and how we can share tooling, process and > resources, they still use something else entirely. Everything used in > Fedora is developed in the open and can be contributed to by anyone. > We have been talking about making the whole process flexible and > dynamic for years. I know internally they would like to have many of > the same things Fedora wants. Yeah, it sounds like we're aiming in the same direction here. This is part of what I mean by "decomposing the compose." We have one rather monolithic process right now, all or nothing. And it does so much that it's impossible to get any fast results from it. That doesn't mean it doesn't work well for what it does. But it's not very flexible, as you pointed out. With a more flexible compose we could choose to do smaller tasks (as well as capitalize on other time savings) for something like a speculative testing build. That kind of build could be used to perform integration testing as well as other tasks (automated, not manual!). We could even potentially test against much of the repetitive, time-consuming matrices manually (and heroically) run by QA. Then only choose to do more extensive tasks based on that being "green." -- Paul ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
El lun, 26-11-2018 a las 19:44 -0500, Paul Frields escribió: > On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore > wrote: > > El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió: > > > Because the people that would be tasked with doing the > > > development are > > > also tasked with cranking out the release. It's a "need more > > > people problem". > > > > This is no longer entirely true, development of the compose tools > > is > > done entirely by PNT DevOps today. Bodhi and other tools in parts > > of > > the delivery pipeline are developed by people in Fedora Engineering > > who > > also have other tasks to do. > > > > > > ready to go. Ultimately a lot of those sort of ways of > > > > optimising > > > > things are things that have been known for some time but no one > > > > has > > > > actually stepped up to do the dev work and it it into > > > > production > > > > shape. Stopping the standard distro work won't miraculously > > > > make > > > > that > > > > other work happen and appear. > > > > > > Well, I kind of disagree. Barring magical people that appear out > > > of > > > the woodwork that know how the tools work and how they need to be > > > improved, I don't see an alternative. Not doing a release to > > > focus on > > > our tech debt seems like a good tradeoff. If there are others > > > that > > > really WANT to continue cranking the release with the tools as > > > they > > > are today... that might be something that could be pursued. It > > > would > > > still require net new contributors to a critical area. > > > > We would need to engage with the team working on the compose tools > > and > > see what time they can dedicate to meeting Fedora's needs. Likely > > a > > end state needs to be defined. then the work can be scoped. > > There are a lot of assumptions wrapped up in these statements. I'm > not > sure we should be assuming that delivery mechanisms ought to all be > tied to internal Red Hat teams. This might be a good opportunity for > us (meaning people variously assigned in both places) to think about > whether we should try to decouple more of our delivery side (compose, > push, etc.), while keeping a tight bond at the build/test side. There is a lot of assumptions in your assumptions, language is hard :), I was trying to outline where things are at not at all how things should be, sorry I was not clear enough there. My main goal was pointing out changes from where things were 5 years ago. > The customers RH serves have specific expectations, and in part that > dictates how delivery tooling is done. Binding the community to that > may be counterproductive. This is especially true now that RHEL 8 > Beta > is out -- it's the perfect time for us to be rethinking at that > level. Compose tooling is largely shared, delivery tooling is not, though there is room for improvement and convergence on both sides. We should also consider CentOS and how we can share tooling, process and resources, they still use something else entirely. Everything used in Fedora is developed in the open and can be contributed to by anyone. We have been talking about making the whole process flexible and dynamic for years. I know internally they would like to have many of the same things Fedora wants. Dennis signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, Nov 27, 2018 at 9:08 AM John Florian wrote: > > On 11/26/18 3:53 PM, Kevin Fenzi wrote: > > The one issue I see off hand is that koji > > tags are kind of expensive so we can't just tag everything the way we > > may want > > Can you please elaborate a bit on this? What makes them expensive? Koji tags are representations of full repositories. They work more or less the same way that Copr projects do, it's just a lot less obvious because of the way Koji represents them in the various UIs. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On 11/26/18 3:53 PM, Kevin Fenzi wrote: The one issue I see off hand is that koji tags are kind of expensive so we can't just tag everything the way we may want Can you please elaborate a bit on this? What makes them expensive? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Tue, Nov 27, 2018 at 4:41 AM Miroslav Suchý wrote: > > Dne 26. 11. 18 v 17:03 Matthew Miller napsal(a): > > * embrace Taiga (an open source kanban tool) for project planning > > * fix the compose speed (target: one hour!) > > * really actually for real gated Rawhide > > * better CI pipeline tests for everything > > * define a base platform -- Red Hat wants to focus resources here > > * better tooling for non-base deliverables > > * better metrics for everything > > My experience is that things are fixed when we are in real time stress. When > we have more time, then people (including > me) are trying to develop/invent new things. > > I have a counter proposal: Make the release every week or month - this will > force us to automate all release processes :) It's funny you mention that, because I could foresee one outcome of focusing on a core platform being that we do a more frequent release. If we can effectively machine-test more of that platform, we could do releases on a timescale of weeks instead of only twice a year. The Atomic WG does this already. This probably makes more sense for an ostree based release than constantly spinning large, multi-GB ISOs. -- Paul ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 7:45 PM Paul Frields wrote: > > On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore wrote: > > El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió: > > > Because the people that would be tasked with doing the development are > > > also tasked with cranking out the release. It's a "need more people > > > problem". > > > > This is no longer entirely true, development of the compose tools is > > done entirely by PNT DevOps today. Bodhi and other tools in parts of > > the delivery pipeline are developed by people in Fedora Engineering who > > also have other tasks to do. > > > > > > ready to go. Ultimately a lot of those sort of ways of optimising > > > > things are things that have been known for some time but no one has > > > > actually stepped up to do the dev work and it it into production > > > > shape. Stopping the standard distro work won't miraculously make > > > > that > > > > other work happen and appear. > > > > > > Well, I kind of disagree. Barring magical people that appear out of > > > the woodwork that know how the tools work and how they need to be > > > improved, I don't see an alternative. Not doing a release to focus on > > > our tech debt seems like a good tradeoff. If there are others that > > > really WANT to continue cranking the release with the tools as they > > > are today... that might be something that could be pursued. It would > > > still require net new contributors to a critical area. > > > > We would need to engage with the team working on the compose tools and > > see what time they can dedicate to meeting Fedora's needs. Likely a > > end state needs to be defined. then the work can be scoped. > > There are a lot of assumptions wrapped up in these statements. I'm not > sure we should be assuming that delivery mechanisms ought to all be > tied to internal Red Hat teams. This might be a good opportunity for > us (meaning people variously assigned in both places) to think about > whether we should try to decouple more of our delivery side (compose, > push, etc.), while keeping a tight bond at the build/test side. Sure. We still either need more people or to drop existing things to cover that with the existing people. > The customers RH serves have specific expectations, and in part that > dictates how delivery tooling is done. Binding the community to that > may be counterproductive. This is especially true now that RHEL 8 Beta > is out -- it's the perfect time for us to be rethinking at that level. There are already a number of tools that are shared across Fedora and RHEL and I can tell you for a fact that many on the RHEL side would like to see the same or similar improvements that have been mentioned here. Let's start scoping the problems and solutions before we decide Fedora should go off and do it's own thing (again). I'm interested in evolving more than just tooling. I continue to want to see Fedora and CentOS coming together more closely, so that we can leverage both of the amazing communities these projects provide. If we do that, we should be scoping how to do release tooling effectively across both Fedora and CentOS, while still keeping RHEL in mind as well. Having 3 (or more) different implementations of compose and release tooling seems ridiculous. I'd like to get it down to a community version and then have RHEL additions for product specific things if possible, driven by RHEL stakeholders. That way we get an actual upstream/downstream relationship with our tooling in addition to our distro. josh ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
Dne 26. 11. 18 v 20:01 Randy Barlow napsal(a): > On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote: >>> * really actually for real gated Rawhide >> Is the "creating side tag for chain builds or by automated requests" >> is planned here? >> When I deal with rust packages, I often need to build some update >> which will break other packages and I need to fix them, but if >> something will prevent going that build in buildroot. > Hey Igor! > > Indeed, I do plan to have Bodhi manage side tags in order to accomplish > this. Does that mean that Bodhi will be finally enabled for Rawhide or not? Because otherwise I can't see how Bodhi is related here. Vít > There's a Bodhi kanban board that covers the various items > planned to be part of this: > > https://github.com/fedora-infra/bodhi/projects/3 > > You can see that a few of the cards in there relate to side tags. My > current thinking is that you will be able to create a side tag update, > which will work slightly different than a normal update in a few ways: > > * You create the side tag update before you make any builds, instead of > after. > * The side tag update makes, well, a side tag in Koji for you. > * You go build all the builds you need in your side tag. > * When you are done, you tell Bodhi you are ready to go to testing, and > it will work like a normal update from there on. > > Does that sound useful to you? > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org signature.asc Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
Dne 26. 11. 18 v 17:03 Matthew Miller napsal(a): > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) > * really actually for real gated Rawhide > * better CI pipeline tests for everything > * define a base platform -- Red Hat wants to focus resources here > * better tooling for non-base deliverables > * better metrics for everything My experience is that things are fixed when we are in real time stress. When we have more time, then people (including me) are trying to develop/invent new things. I have a counter proposal: Make the release every week or month - this will force us to automate all release processes :) Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Monday, November 26, 2018, Matthew Miller wrote: > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > Here's the summary from the page, which proposes we pause the release > > after F30 for these efforts: > > I know it was a big time-off holiday week in the US, but I expected a > little > more interest in this post. Perhaps it seemed like too much text to digest > along with turkey and stuffing. :) I'm highlighting it with a subject > reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) > * really actually for real gated Rawhide > * better CI pipeline tests for everything > * define a base platform -- Red Hat wants to focus resources here > * better tooling for non-base deliverables > * better metrics for everything > > > Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/ > Problem_statements > and comment in this thread. > > I am not really convinced that delaying the release for that makes sense. It's not like people will stop updating packages or that existing release tools will magically disappear (they have to work to keep existing versions running). So people can work on those improvement and bring them in when ready but stopping everything for that is overkill. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore wrote: > El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió: > > Because the people that would be tasked with doing the development are > > also tasked with cranking out the release. It's a "need more people > > problem". > > This is no longer entirely true, development of the compose tools is > done entirely by PNT DevOps today. Bodhi and other tools in parts of > the delivery pipeline are developed by people in Fedora Engineering who > also have other tasks to do. > > > > ready to go. Ultimately a lot of those sort of ways of optimising > > > things are things that have been known for some time but no one has > > > actually stepped up to do the dev work and it it into production > > > shape. Stopping the standard distro work won't miraculously make > > > that > > > other work happen and appear. > > > > Well, I kind of disagree. Barring magical people that appear out of > > the woodwork that know how the tools work and how they need to be > > improved, I don't see an alternative. Not doing a release to focus on > > our tech debt seems like a good tradeoff. If there are others that > > really WANT to continue cranking the release with the tools as they > > are today... that might be something that could be pursued. It would > > still require net new contributors to a critical area. > > We would need to engage with the team working on the compose tools and > see what time they can dedicate to meeting Fedora's needs. Likely a > end state needs to be defined. then the work can be scoped. There are a lot of assumptions wrapped up in these statements. I'm not sure we should be assuming that delivery mechanisms ought to all be tied to internal Red Hat teams. This might be a good opportunity for us (meaning people variously assigned in both places) to think about whether we should try to decouple more of our delivery side (compose, push, etc.), while keeping a tight bond at the build/test side. The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level. -- Paul ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió: > On Mon, Nov 26, 2018 at 5:01 PM Peter Robinson > wrote: > > On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller < > > mat...@fedoraproject.org> wrote: > > > On Mon, Nov 26, 2018 at 08:02:26PM +, Peter Robinson wrote: > > > > transactions is substantial. This may solve (or greatly reduce) > > > > the > > > > compose speed problem, TBD." but it doesn't report the context > > > > of the > > > > "speed increase for transactions" like which transactions? > > > > There's a lot of I/O heavy processes in the compose, like > > > > createrepo, > > > > that don't run any rpm tracsactions what so ever. > > > > > > It seems like there's a lot of room to optimize those things too. > > > For > > > example, extract the needed metadata into a database and update > > > that at > > > build time rather than compose time, and run createrepo against > > > that instead > > > of rpms directly. > > > > Completely agree there's a LOT of improvement that can be done, but > > I > > don't see why that development work cannot be done in parallel and > > put > > it into production when each of the various different components > > are > > Because the people that would be tasked with doing the development > are > also tasked with cranking out the release. It's a "need more people > problem". This is no longer entirely true, development of the compose tools is done entirely by PNT DevOps today. Bodhi and other tools in parts of the delivery pipeline are developed by people in Fedora Engineering who also have other tasks to do. > > ready to go. Ultimately a lot of those sort of ways of optimising > > things are things that have been known for some time but no one has > > actually stepped up to do the dev work and it it into production > > shape. Stopping the standard distro work won't miraculously make > > that > > other work happen and appear. > > Well, I kind of disagree. Barring magical people that appear out of > the woodwork that know how the tools work and how they need to be > improved, I don't see an alternative. Not doing a release to focus > on > our tech debt seems like a good tradeoff. If there are others that > really WANT to continue cranking the release with the tools as they > are today... that might be something that could be pursued. It would > still require net new contributors to a critical area. We would need to engage with the team working on the compose tools and see what time they can dedicate to meeting Fedora's needs. Likely a end state needs to be defined. then the work can be scoped. Dennis signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 5:01 PM Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller > wrote: > > > > On Mon, Nov 26, 2018 at 08:02:26PM +, Peter Robinson wrote: > > > transactions is substantial. This may solve (or greatly reduce) the > > > compose speed problem, TBD." but it doesn't report the context of the > > > "speed increase for transactions" like which transactions? > > > There's a lot of I/O heavy processes in the compose, like createrepo, > > > that don't run any rpm tracsactions what so ever. > > > > It seems like there's a lot of room to optimize those things too. For > > example, extract the needed metadata into a database and update that at > > build time rather than compose time, and run createrepo against that instead > > of rpms directly. > > Completely agree there's a LOT of improvement that can be done, but I > don't see why that development work cannot be done in parallel and put > it into production when each of the various different components are Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem". > ready to go. Ultimately a lot of those sort of ways of optimising > things are things that have been known for some time but no one has > actually stepped up to do the dev work and it it into production > shape. Stopping the standard distro work won't miraculously make that > other work happen and appear. Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area. josh ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller wrote: > > On Mon, Nov 26, 2018 at 08:02:26PM +, Peter Robinson wrote: > > transactions is substantial. This may solve (or greatly reduce) the > > compose speed problem, TBD." but it doesn't report the context of the > > "speed increase for transactions" like which transactions? > > There's a lot of I/O heavy processes in the compose, like createrepo, > > that don't run any rpm tracsactions what so ever. > > It seems like there's a lot of room to optimize those things too. For > example, extract the needed metadata into a database and update that at > build time rather than compose time, and run createrepo against that instead > of rpms directly. Completely agree there's a LOT of improvement that can be done, but I don't see why that development work cannot be done in parallel and put it into production when each of the various different components are ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 08:02:26PM +, Peter Robinson wrote: > transactions is substantial. This may solve (or greatly reduce) the > compose speed problem, TBD." but it doesn't report the context of the > "speed increase for transactions" like which transactions? > There's a lot of I/O heavy processes in the compose, like createrepo, > that don't run any rpm tracsactions what so ever. It seems like there's a lot of room to optimize those things too. For example, extract the needed metadata into a database and update that at build time rather than compose time, and run createrepo against that instead of rpms directly. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018, 20:09 Randy Barlow On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote: > > > * really actually for real gated Rawhide > > > > Is the "creating side tag for chain builds or by automated requests" > > is planned here? > > When I deal with rust packages, I often need to build some update > > which will break other packages and I need to fix them, but if > > something will prevent going that build in buildroot. > > Hey Igor! > Hey Randy, Indeed, I do plan to have Bodhi manage side tags in order to accomplish > this. There's a Bodhi kanban board that covers the various items > planned to be part of this: > > https://github.com/fedora-infra/bodhi/projects/3 > > You can see that a few of the cards in there relate to side tags. My > current thinking is that you will be able to create a side tag update, > which will work slightly different than a normal update in a few ways: > > * You create the side tag update before you make any builds, instead of > after. > * The side tag update makes, well, a side tag in Koji for you. > * You go build all the builds you need in your side tag. > * When you are done, you tell Bodhi you are ready to go to testing, and > it will work like a normal update from there on. > > Does that sound useful to you? > It definitely does, just wanted to ensure if it is one of things which is supposed to be worked on if we skip release. ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On 11/26/18 11:55 AM, Paul Frields wrote: > On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson >> >> While we're on the topic of unicorns, if I were the person actually in >> charge of this and had the freedom to do it how I wanted, what I'd >> really want to do is create a much more *modular* compose process, >> where inputs and outputs were defined and associated with each other, >> and you could easily define subsets to be produced at different times >> for different reasons. Not a compose, but a Compose Construction Kit. A >> big chunk of the length of "the compose" is associated with the fact >> that we define "the compose" as a single thing which produces (last >> time I counted) 70+ deliverables. If we had an easy way to say 'ok, >> right now we need a compose which produces A, B, C and D from the >> minimum necessary sets of inputs', that would be a massive improvement. >> >> This hasn't been true for a while (we actually have something like half >> a dozen or more different "composes", right now), but the system is >> still less flexible and it's still more difficult to configure >> different composes than it really ought to be. Agreed, that would be nice. The one issue I see off hand is that koji tags are kind of expensive so we can't just tag everything the way we may want to, but perhaps we can come up with some clever workflow for that. > Adam puts his finger on an important aspect of the work, which is to > "decompose the compose" (sorry). We need the ability to produce enough > A, B, C and D (abusing his example) to do integration tests for > specific inputs, such that maintainers get feedback in a reasonable > timeframe on their builds. Yeah, thats long been a desire of releng/qa... weeding out the obviously broken would be very nice. >With that, gating Rawhide can be an actual > thing. Indeed. kevin signature.asc Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 3:34 PM Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 8:22 PM Stephen Gallagher wrote: > > > > On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson wrote: > > > There's no facts or details, to quote "The reported speed increase for > > > transactions is substantial. This may solve (or greatly reduce) the > > > compose speed problem, TBD." but it doesn't report the context of the > > > "speed increase for transactions" like which transactions? > > > There's a lot of I/O heavy processes in the compose, like createrepo, > > > that don't run any rpm tracsactions what so ever. > > > > Paul was referring to image creation here. We should be able to get a > > fairly significant (at least one order of magnitude) speed-up on > > creating any image that runs an RPM installation transaction. I'm not > > sure how much that will save in the *whole* compose process, but as > > it's an identified spot where we know how to optimize it, we should do > > so. > > But ultimately removing rpm transactions is an optimisation to rpm > which, when it lands, the compose plus numerous other compoents that > use that code path will just consume and at that point get the > benefits of, it's not a reason to skip a release nor is it going to be > a revolution that gets the composes down to an hour. I don't believe > that that alone is and order of magnitude change, do you have any > actual hard figured to back the statement up? Paul overstated the general importance of the scriptlet work. As I said above, it will affect the RPM transactions. Will Woods has numbers on the actual performance gains that I'm sure he'll share on this thread at some point. I said above "I'm not sure how much that will save in the *whole* compose process" and I meant that. I'm not advocating that we postpone F31 solely for these changes. At the same time, let's be careful not to slip into a "perfect is the enemy of good" situation where we start critiquing individual solutions as not having enough of an effect. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 8:22 PM Stephen Gallagher wrote: > > On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson wrote: > > There's no facts or details, to quote "The reported speed increase for > > transactions is substantial. This may solve (or greatly reduce) the > > compose speed problem, TBD." but it doesn't report the context of the > > "speed increase for transactions" like which transactions? > > There's a lot of I/O heavy processes in the compose, like createrepo, > > that don't run any rpm tracsactions what so ever. > > Paul was referring to image creation here. We should be able to get a > fairly significant (at least one order of magnitude) speed-up on > creating any image that runs an RPM installation transaction. I'm not > sure how much that will save in the *whole* compose process, but as > it's an identified spot where we know how to optimize it, we should do > so. But ultimately removing rpm transactions is an optimisation to rpm which, when it lands, the compose plus numerous other compoents that use that code path will just consume and at that point get the benefits of, it's not a reason to skip a release nor is it going to be a revolution that gets the composes down to an hour. I don't believe that that alone is and order of magnitude change, do you have any actual hard figured to back the statement up? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson wrote: > There's no facts or details, to quote "The reported speed increase for > transactions is substantial. This may solve (or greatly reduce) the > compose speed problem, TBD." but it doesn't report the context of the > "speed increase for transactions" like which transactions? > There's a lot of I/O heavy processes in the compose, like createrepo, > that don't run any rpm tracsactions what so ever. Paul was referring to image creation here. We should be able to get a fairly significant (at least one order of magnitude) speed-up on creating any image that runs an RPM installation transaction. I'm not sure how much that will save in the *whole* compose process, but as it's an identified spot where we know how to optimize it, we should do so. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 7:52 PM Paul Frields wrote: > > On Mon, Nov 26, 2018 at 1:59 PM Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > > wrote: > > > > > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > > > > wrote: > > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > > > Here's the summary from the page, which proposes we pause the > > > > > > release > > > > > > after F30 for these efforts: > > > > > > > > > > I know it was a big time-off holiday week in the US, but I expected a > > > > > little > > > > > more interest in this post. Perhaps it seemed like too much text to > > > > > digest > > > > > along with turkey and stuffing. :) I'm highlighting it with a subject > > > > > reflecting the big, direct impact, and here's some other top-level > > > > > proposals: > > > > > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > > > * fix the compose speed (target: one hour!) > > > > > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > > > done analysis. > > > > > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > > > have both looked into it before. > > > > Yes, I mean recently, I was involved in the last time we attempted > > this, and there was a number of things identified but quite a bit has > > changed since that last happened. > > One of the large factors -- which I think was noted in the page -- was > RPM scriptlets. Stephen Gallagher met with WIll Woods and James Antill > just recently to talk about this. The time savings was substantial. > Combining that with reducing the size of "speculative composes" that There's no facts or details, to quote "The reported speed increase for transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 11:39:52AM -0800, Adam Williamson wrote: > While we're on the topic of unicorns, if I were the person actually in > charge of this and had the freedom to do it how I wanted, what I'd > really want to do is create a much more *modular* compose process, > where inputs and outputs were defined and associated with each other, > and you could easily define subsets to be produced at different times > for different reasons. Not a compose, but a Compose Construction Kit. A > big chunk of the length of "the compose" is associated with the fact > that we define "the compose" as a single thing which produces (last > time I counted) 70+ deliverables. If we had an easy way to say 'ok, > right now we need a compose which produces A, B, C and D from the > minimum necessary sets of inputs', that would be a massive improvement. +1000 -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson wrote: > On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote: > > Yeah, I think our goal should be 1 minute... just as realistic as one > > hour. ;) > > > > I have not looked into things recently. However: > > > > * The mock fsync change (https://pagure.io/releng/issue/7909) may help > > some. > > > > * We have plans to redo the setup on the s390x builders, which should > > result in them being much faster. That should help. > > > > * I know pungi maintaienrs have been doing some work to make things > > faster (even in the last release out this morning). > > > > All that said, I am not sure there's going to be a way to get it down to > > an hour. I think getting it down to 3-4hours may be very helpful tho and > > might be possible. > > While we're on the topic of unicorns, if I were the person actually in > charge of this and had the freedom to do it how I wanted, what I'd > really want to do is create a much more *modular* compose process, > where inputs and outputs were defined and associated with each other, > and you could easily define subsets to be produced at different times > for different reasons. Not a compose, but a Compose Construction Kit. A > big chunk of the length of "the compose" is associated with the fact > that we define "the compose" as a single thing which produces (last > time I counted) 70+ deliverables. If we had an easy way to say 'ok, > right now we need a compose which produces A, B, C and D from the > minimum necessary sets of inputs', that would be a massive improvement. > > This hasn't been true for a while (we actually have something like half > a dozen or more different "composes", right now), but the system is > still less flexible and it's still more difficult to configure > different composes than it really ought to be. Adam puts his finger on an important aspect of the work, which is to "decompose the compose" (sorry). We need the ability to produce enough A, B, C and D (abusing his example) to do integration tests for specific inputs, such that maintainers get feedback in a reasonable timeframe on their builds. With that, gating Rawhide can be an actual thing. -- Paul ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 1:59 PM Peter Robinson wrote: > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > wrote: > > > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > > > wrote: > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > > Here's the summary from the page, which proposes we pause the release > > > > > after F30 for these efforts: > > > > > > > > I know it was a big time-off holiday week in the US, but I expected a > > > > little > > > > more interest in this post. Perhaps it seemed like too much text to > > > > digest > > > > along with turkey and stuffing. :) I'm highlighting it with a subject > > > > reflecting the big, direct impact, and here's some other top-level > > > > proposals: > > > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > > * fix the compose speed (target: one hour!) > > > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > > done analysis. > > > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > > have both looked into it before. > > Yes, I mean recently, I was involved in the last time we attempted > this, and there was a number of things identified but quite a bit has > changed since that last happened. One of the large factors -- which I think was noted in the page -- was RPM scriptlets. Stephen Gallagher met with WIll Woods and James Antill just recently to talk about this. The time savings was substantial. Combining that with reducing the size of "speculative composes" that could be used for CI and other pre-ship purposes could get us to where we want to be. Stephen's carrying a heavy duty project at the moment but when that's done in a few weeks, this will be getting more attention. > > > The fact of the matter is that the compose has a LOT of > > > I/O and I/O is slow and the cost to upgrade the infrastructure to get > > > high through put I/O is expensive and no one has ever offered the > > > budget to resolve that problem. The fact of the matter with this is > > > that we now produce a lot of release artifacts and that takes time, > > > some of it can probably be run more in parallel, and possibly some > > > things run at different times but without a combination of investment > > > in infrastructure, and maybe even hacking and slashing of components > > > this is not simply a check box. > > > > That's why the proposal is to skip an entire release to give us time to > > work on it. If it was easy, that wouldn't be necessary. > > > > There are of course at least *two* possible solutions to "there's lots > > of I/O". One is "throw money at making the I/O faster", the other is > > "figure out how to do less I/O". > > > > One hour is a *target*, not a *requirement*. I don't think anyone's > > going to consider the time wasted if we wind up getting it down to two > > hours instead of one. > > > > (Though personally I'd like ten minutes. And two unicorns.) Just so. -- Paul ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson wrote: > On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote: > > On 11/26/18 10:58 AM, Peter Robinson wrote: > > > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > > > wrote: > > > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > > > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller < > mat...@fedoraproject.org> wrote: > > > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > > > > Here's the summary from the page, which proposes we pause the > release > > > > > > > after F30 for these efforts: > > > > > > > > > > > > I know it was a big time-off holiday week in the US, but I > expected a little > > > > > > more interest in this post. Perhaps it seemed like too much text > to digest > > > > > > along with turkey and stuffing. :) I'm highlighting it with a > subject > > > > > > reflecting the big, direct impact, and here's some other > top-level > > > > > > proposals: > > > > > > > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > > > > * fix the compose speed (target: one hour!) > > > > > > > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > > > > done analysis. > > > > > > > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > > > > have both looked into it before. > > > > > > Yes, I mean recently, I was involved in the last time we attempted > > > this, and there was a number of things identified but quite a bit has > > > changed since that last happened. > > > > Yeah, I think our goal should be 1 minute... just as realistic as one > > hour. ;) > > > > I have not looked into things recently. However: > > > > * The mock fsync change (https://pagure.io/releng/issue/7909) may help > > some. > > > > * We have plans to redo the setup on the s390x builders, which should > > result in them being much faster. That should help. > > > > * I know pungi maintaienrs have been doing some work to make things > > faster (even in the last release out this morning). > > > > All that said, I am not sure there's going to be a way to get it down to > > an hour. I think getting it down to 3-4hours may be very helpful tho and > > might be possible. > > While we're on the topic of unicorns, if I were the person actually in > charge of this and had the freedom to do it how I wanted, what I'd > really want to do is create a much more *modular* compose process, > where inputs and outputs were defined and associated with each other, > and you could easily define subsets to be produced at different times > for different reasons. Not a compose, but a Compose Construction Kit. A > big chunk of the length of "the compose" is associated with the fact > that we define "the compose" as a single thing which produces (last > time I counted) 70+ deliverables. If we had an easy way to say 'ok, > right now we need a compose which produces A, B, C and D from the > minimum necessary sets of inputs', that would be a massive improvement. > > This hasn't been true for a while (we actually have something like half > a dozen or more different "composes", right now), but the system is > still less flexible and it's still more difficult to configure > different composes than it really ought to be. > This would be great. I want to get to a state where different WG's can run their own composes by calling a service. They might actually release on their own schedules as well. (Not all, but something like Labs or Spins) . > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 7:47 PM Mohan Boddu wrote: > > > > On Mon, Nov 26, 2018 at 2:30 PM Kevin Fenzi wrote: >> >> On 11/26/18 10:58 AM, Peter Robinson wrote: >> > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson >> > wrote: >> >> >> >> On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: >> >>> On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller >> >>> wrote: >> On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: >> > Here's the summary from the page, which proposes we pause the release >> > after F30 for these efforts: >> >> I know it was a big time-off holiday week in the US, but I expected a >> little >> more interest in this post. Perhaps it seemed like too much text to >> digest >> along with turkey and stuffing. :) I'm highlighting it with a subject >> reflecting the big, direct impact, and here's some other top-level >> proposals: >> >> * embrace Taiga (an open source kanban tool) for project planning >> * fix the compose speed (target: one hour!) >> >>> >> >>> Can I have a unicorn? Everyone wants this bug absolutely no one has >> >>> done analysis. >> >> >> >> I just don't believe this is true. I'm pretty sure Dennis and Kevin >> >> have both looked into it before. >> > >> > Yes, I mean recently, I was involved in the last time we attempted >> > this, and there was a number of things identified but quite a bit has >> > changed since that last happened. >> >> Yeah, I think our goal should be 1 minute... just as realistic as one >> hour. ;) >> >> I have not looked into things recently. However: >> >> * The mock fsync change (https://pagure.io/releng/issue/7909) may help >> some. >> >> * We have plans to redo the setup on the s390x builders, which should >> result in them being much faster. That should help. >> >> * I know pungi maintaienrs have been doing some work to make things >> faster (even in the last release out this morning). >> >> All that said, I am not sure there's going to be a way to get it down to >> an hour. I think getting it down to 3-4hours may be very helpful tho and >> might be possible. > > I totally agree, also as Peter mentioned, IOT composes are just taking 1 hour > as its > just consuming the repo rather than generating newRepo as the normal > nightly composes does. If we can generate the newRepo on the fly whenever > a package gets build and just use the repo to generate the artifacts either > individually (splitting the compose) or parallely (as we are doing right now > mostly) > that would really help. Unfortunately the newRepo/installer bit is likely the most I/O intensive, but I think it we split it out into it's own process, see mail I just sent, it would be a good start, and then also a good separate standalone focus for optimizations. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 7:42 PM Adam Williamson wrote: > > On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote: > > On 11/26/18 10:58 AM, Peter Robinson wrote: > > > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > > > wrote: > > > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > > > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > > > > > wrote: > > > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > > > > Here's the summary from the page, which proposes we pause the > > > > > > > release > > > > > > > after F30 for these efforts: > > > > > > > > > > > > I know it was a big time-off holiday week in the US, but I expected > > > > > > a little > > > > > > more interest in this post. Perhaps it seemed like too much text to > > > > > > digest > > > > > > along with turkey and stuffing. :) I'm highlighting it with a > > > > > > subject > > > > > > reflecting the big, direct impact, and here's some other top-level > > > > > > proposals: > > > > > > > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > > > > * fix the compose speed (target: one hour!) > > > > > > > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > > > > done analysis. > > > > > > > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > > > > have both looked into it before. > > > > > > Yes, I mean recently, I was involved in the last time we attempted > > > this, and there was a number of things identified but quite a bit has > > > changed since that last happened. > > > > Yeah, I think our goal should be 1 minute... just as realistic as one > > hour. ;) > > > > I have not looked into things recently. However: > > > > * The mock fsync change (https://pagure.io/releng/issue/7909) may help > > some. > > > > * We have plans to redo the setup on the s390x builders, which should > > result in them being much faster. That should help. > > > > * I know pungi maintaienrs have been doing some work to make things > > faster (even in the last release out this morning). > > > > All that said, I am not sure there's going to be a way to get it down to > > an hour. I think getting it down to 3-4hours may be very helpful tho and > > might be possible. > > While we're on the topic of unicorns, if I were the person actually in > charge of this and had the freedom to do it how I wanted, what I'd > really want to do is create a much more *modular* compose process, > where inputs and outputs were defined and associated with each other, > and you could easily define subsets to be produced at different times > for different reasons. Not a compose, but a Compose Construction Kit. A > big chunk of the length of "the compose" is associated with the fact > that we define "the compose" as a single thing which produces (last > time I counted) 70+ deliverables. If we had an easy way to say 'ok, > right now we need a compose which produces A, B, C and D from the > minimum necessary sets of inputs', that would be a massive improvement. > > This hasn't been true for a while (we actually have something like half > a dozen or more different "composes", right now), but the system is > still less flexible and it's still more difficult to configure > different composes than it really ought to be. Yes, that was one of the ideas I had. Basically do the "newRepo" part of the compose that generates the new set of tagged in and signed rpms and base network isntallers 3-4 times a day and then run all the other components separately (live/arm images, CoreOS, IoT, cloud images) on their own schedule in separate components consuming the updates. This is basically what some of the bits do now as you'd indicated. It then allows focus to be put on each component to speed each piece up independently rather than as a whole. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 2:30 PM Kevin Fenzi wrote: > On 11/26/18 10:58 AM, Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > > wrote: > >> > >> On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > >>> On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller < > mat...@fedoraproject.org> wrote: > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > Here's the summary from the page, which proposes we pause the release > > after F30 for these efforts: > > I know it was a big time-off holiday week in the US, but I expected a > little > more interest in this post. Perhaps it seemed like too much text to > digest > along with turkey and stuffing. :) I'm highlighting it with a subject > reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) > >>> > >>> Can I have a unicorn? Everyone wants this bug absolutely no one has > >>> done analysis. > >> > >> I just don't believe this is true. I'm pretty sure Dennis and Kevin > >> have both looked into it before. > > > > Yes, I mean recently, I was involved in the last time we attempted > > this, and there was a number of things identified but quite a bit has > > changed since that last happened. > > Yeah, I think our goal should be 1 minute... just as realistic as one > hour. ;) > > I have not looked into things recently. However: > > * The mock fsync change (https://pagure.io/releng/issue/7909) may help > some. > > * We have plans to redo the setup on the s390x builders, which should > result in them being much faster. That should help. > > * I know pungi maintaienrs have been doing some work to make things > faster (even in the last release out this morning). > > All that said, I am not sure there's going to be a way to get it down to > an hour. I think getting it down to 3-4hours may be very helpful tho and > might be possible. > I totally agree, also as Peter mentioned, IOT composes are just taking 1 hour as its just consuming the repo rather than generating newRepo as the normal nightly composes does. If we can generate the newRepo on the fly whenever a package gets build and just use the repo to generate the artifacts either individually (splitting the compose) or parallely (as we are doing right now mostly) that would really help. > > kevin > > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote: > On 11/26/18 10:58 AM, Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > > wrote: > > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > > > > wrote: > > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > > > Here's the summary from the page, which proposes we pause the > > > > > > release > > > > > > after F30 for these efforts: > > > > > > > > > > I know it was a big time-off holiday week in the US, but I expected a > > > > > little > > > > > more interest in this post. Perhaps it seemed like too much text to > > > > > digest > > > > > along with turkey and stuffing. :) I'm highlighting it with a subject > > > > > reflecting the big, direct impact, and here's some other top-level > > > > > proposals: > > > > > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > > > * fix the compose speed (target: one hour!) > > > > > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > > > done analysis. > > > > > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > > > have both looked into it before. > > > > Yes, I mean recently, I was involved in the last time we attempted > > this, and there was a number of things identified but quite a bit has > > changed since that last happened. > > Yeah, I think our goal should be 1 minute... just as realistic as one > hour. ;) > > I have not looked into things recently. However: > > * The mock fsync change (https://pagure.io/releng/issue/7909) may help > some. > > * We have plans to redo the setup on the s390x builders, which should > result in them being much faster. That should help. > > * I know pungi maintaienrs have been doing some work to make things > faster (even in the last release out this morning). > > All that said, I am not sure there's going to be a way to get it down to > an hour. I think getting it down to 3-4hours may be very helpful tho and > might be possible. While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement. This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On 11/26/18 10:58 AM, Peter Robinson wrote: > On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson > wrote: >> >> On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: >>> On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller >>> wrote: On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > Here's the summary from the page, which proposes we pause the release > after F30 for these efforts: I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals: * embrace Taiga (an open source kanban tool) for project planning * fix the compose speed (target: one hour!) >>> >>> Can I have a unicorn? Everyone wants this bug absolutely no one has >>> done analysis. >> >> I just don't believe this is true. I'm pretty sure Dennis and Kevin >> have both looked into it before. > > Yes, I mean recently, I was involved in the last time we attempted > this, and there was a number of things identified but quite a bit has > changed since that last happened. Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;) I have not looked into things recently. However: * The mock fsync change (https://pagure.io/releng/issue/7909) may help some. * We have plans to redo the setup on the s390x builders, which should result in them being much faster. That should help. * I know pungi maintaienrs have been doing some work to make things faster (even in the last release out this morning). All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible. kevin signature.asc Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Release Cadence WAS: Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On 11/26/18 8:21 AM, Brian (bex) Exelbierd wrote: > On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller > wrote: >> >> On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: >>> Here's the summary from the page, which proposes we pause the release >>> after F30 for these efforts: >> >> I know it was a big time-off holiday week in the US, but I expected a little >> more interest in this post. Perhaps it seemed like too much text to digest >> along with turkey and stuffing. :) I'm highlighting it with a subject >> reflecting the big, direct impact, and here's some other top-level >> proposals: >> >> * embrace Taiga (an open source kanban tool) for project planning >> * fix the compose speed (target: one hour!) >> * really actually for real gated Rawhide >> * better CI pipeline tests for everything >> * define a base platform -- Red Hat wants to focus resources here >> * better tooling for non-base deliverables >> * better metrics for everything > > I am +100 to these changes. > > The idea that we can essentially not release for a year to do this > raises the interesting point that perhaps our release cadence is not > as rigid as we think it may be. I don't think we are going to skip > Gnome and other major updates, are we? If not, then this also runs > into the conversation about longer lifecycles and whether we need > releases, imho. Indeed. If we push as updates things we normally wait for releases for, would that disrupt our users? Is there a cutoff of things we would push in updates? For example, we usually do mass rebuilds of the entire collection of packages when new gcc/etc are ready, would we do that as a update in a stable release? We could solve the rolling changes issue by requiring all major changes to be modular and not enable the new module (ie, gnome 3.32 releases in f30, in 6 mmonths 3.34 comes out, we enable it as a new module, users can choose to switch to if they want, or stay on 3.32 if they ignore it or don't want to right then). That won't help for tool changes tho. kevin signature.asc Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, 2018-11-26 at 20:03 +0100, Fabio Valentini wrote: > > Also, I find it funny that - somehow - tons of resources would be > available for "better CI pipeline tests for everything" (!), but we'd > have to "focus resources" when it comes to the fedora "platform" > itself. These two sort of go together, I think: we don't truly expect to have 'better CI pipeline tests for EVERYTHING', as in *every package in the distro*, but rather one advantage of somehow defining the 'Platform' is that it gives us a *smaller* set of bits to focus the CI work on. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller wrote:
>
> On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
> > Here's the summary from the page, which proposes we pause the release
> > after F30 for these efforts:
>
> I know it was a big time-off holiday week in the US, but I expected a little
> more interest in this post.
Hi Matthew,
I initially basically ignored the previous E-Mail, because it didn't
come from a person I knew, and - from quickly glancing over the
contents - read like marketing fluff. I should have dug deeper there,
but we all only have limited time.
> Perhaps it seemed like too much text to digest
> along with turkey and stuffing. :) I'm highlighting it with a subject
> reflecting the big, direct impact, and here's some other top-level
> proposals:
This is definitely getting more attention. Maybe the initial E-Mail
should have had a less bland Subject line, too ;)
> * embrace Taiga (an open source kanban tool) for project planning
> * fix the compose speed (target: one hour!)
> * really actually for real gated Rawhide
> * better CI pipeline tests for everything
> * better tooling for non-base deliverables
> * better metrics for everything
Those all sound like good things to work on.
> * define a base platform -- Red Hat wants to focus resources here
This point though ... hmm. Also kind of mentioned on the linked wiki page:
> #3 (...) Define a small, draft set of content for the Platform which can
> iterate over time
What exactly does that mean? Moving to something two-tiered like the
ubuntu model (main / universe) - which I think is a terrible solution?
(And what would be the names for this in fedora - "core" and "extras"?
(pun definitely intended))
Also, I find it funny that - somehow - tons of resources would be
available for "better CI pipeline tests for everything" (!), but we'd
have to "focus resources" when it comes to the fedora "platform"
itself.
I hope my response doesn't read overly negative or critical - but I
think some fedora contributors (including me) have been weary of
something like this ("focusing resources") since the IBM news got
announced.
Fabio
> Please read
> https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements
> and comment in this thread.
>
> --
> Matthew Miller
>
> Fedora Project Leader
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote: > > * really actually for real gated Rawhide > > Is the "creating side tag for chain builds or by automated requests" > is planned here? > When I deal with rust packages, I often need to build some update > which will break other packages and I need to fix them, but if > something will prevent going that build in buildroot. Hey Igor! Indeed, I do plan to have Bodhi manage side tags in order to accomplish this. There's a Bodhi kanban board that covers the various items planned to be part of this: https://github.com/fedora-infra/bodhi/projects/3 You can see that a few of the cards in there relate to side tags. My current thinking is that you will be able to create a side tag update, which will work slightly different than a normal update in a few ways: * You create the side tag update before you make any builds, instead of after. * The side tag update makes, well, a side tag in Koji for you. * You go build all the builds you need in your side tag. * When you are done, you tell Bodhi you are ready to go to testing, and it will work like a normal update from there on. Does that sound useful to you? signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson wrote: > > On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > > wrote: > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > > Here's the summary from the page, which proposes we pause the release > > > > after F30 for these efforts: > > > > > > I know it was a big time-off holiday week in the US, but I expected a > > > little > > > more interest in this post. Perhaps it seemed like too much text to digest > > > along with turkey and stuffing. :) I'm highlighting it with a subject > > > reflecting the big, direct impact, and here's some other top-level > > > proposals: > > > > > > * embrace Taiga (an open source kanban tool) for project planning > > > * fix the compose speed (target: one hour!) > > > > Can I have a unicorn? Everyone wants this bug absolutely no one has > > done analysis. > > I just don't believe this is true. I'm pretty sure Dennis and Kevin > have both looked into it before. Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened. > > The fact of the matter is that the compose has a LOT of > > I/O and I/O is slow and the cost to upgrade the infrastructure to get > > high through put I/O is expensive and no one has ever offered the > > budget to resolve that problem. The fact of the matter with this is > > that we now produce a lot of release artifacts and that takes time, > > some of it can probably be run more in parallel, and possibly some > > things run at different times but without a combination of investment > > in infrastructure, and maybe even hacking and slashing of components > > this is not simply a check box. > > That's why the proposal is to skip an entire release to give us time to > work on it. If it was easy, that wouldn't be necessary. > > There are of course at least *two* possible solutions to "there's lots > of I/O". One is "throw money at making the I/O faster", the other is > "figure out how to do less I/O". > > One hour is a *target*, not a *requirement*. I don't think anyone's > going to consider the time wasted if we wind up getting it down to two > hours instead of one. > > (Though personally I'd like ten minutes. And two unicorns.) > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, 2018-11-26 at 18:44 +, Peter Robinson wrote: > On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller > wrote: > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > Here's the summary from the page, which proposes we pause the release > > > after F30 for these efforts: > > > > I know it was a big time-off holiday week in the US, but I expected a little > > more interest in this post. Perhaps it seemed like too much text to digest > > along with turkey and stuffing. :) I'm highlighting it with a subject > > reflecting the big, direct impact, and here's some other top-level > > proposals: > > > > * embrace Taiga (an open source kanban tool) for project planning > > * fix the compose speed (target: one hour!) > > Can I have a unicorn? Everyone wants this bug absolutely no one has > done analysis. I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before. > The fact of the matter is that the compose has a LOT of > I/O and I/O is slow and the cost to upgrade the infrastructure to get > high through put I/O is expensive and no one has ever offered the > budget to resolve that problem. The fact of the matter with this is > that we now produce a lot of release artifacts and that takes time, > some of it can probably be run more in parallel, and possibly some > things run at different times but without a combination of investment > in infrastructure, and maybe even hacking and slashing of components > this is not simply a check box. That's why the proposal is to skip an entire release to give us time to work on it. If it was easy, that wouldn't be necessary. There are of course at least *two* possible solutions to "there's lots of I/O". One is "throw money at making the I/O faster", the other is "figure out how to do less I/O". One hour is a *target*, not a *requirement*. I don't think anyone's going to consider the time wasted if we wind up getting it down to two hours instead of one. (Though personally I'd like ten minutes. And two unicorns.) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Release Cadence WAS: Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 4:23 PM Brian (bex) Exelbierd wrote: > > On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller > wrote: > > > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > > Here's the summary from the page, which proposes we pause the release > > > after F30 for these efforts: > > > > I know it was a big time-off holiday week in the US, but I expected a little > > more interest in this post. Perhaps it seemed like too much text to digest > > along with turkey and stuffing. :) I'm highlighting it with a subject > > reflecting the big, direct impact, and here's some other top-level > > proposals: > > > > * embrace Taiga (an open source kanban tool) for project planning > > * fix the compose speed (target: one hour!) > > * really actually for real gated Rawhide > > * better CI pipeline tests for everything > > * define a base platform -- Red Hat wants to focus resources here > > * better tooling for non-base deliverables > > * better metrics for everything > > I am +100 to these changes. > > The idea that we can essentially not release for a year to do this > raises the interesting point that perhaps our release cadence is not > as rigid as we think it may be. I don't think we are going to skip > Gnome and other major updates, are we? If not, then this also runs > into the conversation about longer lifecycles and whether we need > releases, imho. I like some of the ideas here but I feel we need to have an extremely well defined set of problems to be solved else we'll delay the release, achieve very little, and either never release again or not have anything achieved except a waste of time. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller wrote: > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > Here's the summary from the page, which proposes we pause the release > > after F30 for these efforts: > > I know it was a big time-off holiday week in the US, but I expected a little > more interest in this post. Perhaps it seemed like too much text to digest > along with turkey and stuffing. :) I'm highlighting it with a subject > reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis. The fact of the matter is that the compose has a LOT of I/O and I/O is slow and the cost to upgrade the infrastructure to get high through put I/O is expensive and no one has ever offered the budget to resolve that problem. The fact of the matter with this is that we now produce a lot of release artifacts and that takes time, some of it can probably be run more in parallel, and possibly some things run at different times but without a combination of investment in infrastructure, and maybe even hacking and slashing of components this is not simply a check box. For example the IoT compose takes around an hour but ATM we currently produce 5 artifacts, mostly in parallel, and don't do things like newRepo for the base rpms because we consume that from the main compose. > * really actually for real gated Rawhide > * better CI pipeline tests for everything > * define a base platform -- Red Hat wants to focus resources here > * better tooling for non-base deliverables > * better metrics for everything They are very nice top level ideas, but each of those components need to have in depth analysis and not just the hand wavy overview in the link below. > Please read > https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements > and comment in this thread. > > -- > Matthew Miller > > Fedora Project Leader > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 10:03 AM, Matthew Miller wrote: I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals: Can we use this as an opportunity to work out how to use the blocker bugs process for a megaupdate? Our reputation for providing an up-to-date desktop took a hit last time we did this, so we'll want to provide a GNOME 3.34 megaupdate in October 2019 to compensate for the missing Fedora release there. We'll need a monthslong testing process for this to go smoothly mid-release. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 5:14 PM Matthew Miller wrote: > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > Here's the summary from the page, which proposes we pause the release > > after F30 for these efforts: > > I know it was a big time-off holiday week in the US, but I expected a little > more interest in this post. Perhaps it seemed like too much text to digest > along with turkey and stuffing. :) I'm highlighting it with a subject > reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning That would be good, however I don't know how it is related to lifecycle of distro. P.S. would be great if someone finishes packaging of it (I did a lot of work here, but not everything). > * fix the compose speed (target: one hour!) Do we have analysis of what is taking how much time? > * really actually for real gated Rawhide Is the "creating side tag for chain builds or by automated requests" is planned here? When I deal with rust packages, I often need to build some update which will break other packages and I need to fix them, but if something will prevent going that build in buildroot. > * better CI pipeline tests for everything Does it mean that finally someone will make it usable? Do we have definition of "better"? I've tried to use CI for all rust-* packages, but it is basically unusable. * No way to re-trigger tests * Koji repo is broken even is enabled (https://pagure.io/fedora-ci/general/issue/14, no single comment for > 0.5 month) * Doesn't re-trigger after dependency changes * No way to tell that set of tests should be run on some file pattern (copy-cating same tests folder to 480 packages is not much fun) > * define a base platform -- Red Hat wants to focus resources here Can you elaborate? > * better tooling for non-base deliverables Are Ursa Major and Freshmaker included in here? > * better metrics for everything > > > Please read > https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements > and comment in this thread. I would appreciate having this thread on discourse because since introduction it is my main place where I try to have such conversations. > > -- > Matthew Miller > > Fedora Project Leader > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Release Cadence WAS: Re: Proposal: delay F31 release to work out infrastructure and lifecycle challenges
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller wrote: > > On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > > Here's the summary from the page, which proposes we pause the release > > after F30 for these efforts: > > I know it was a big time-off holiday week in the US, but I expected a little > more interest in this post. Perhaps it seemed like too much text to digest > along with turkey and stuffing. :) I'm highlighting it with a subject > reflecting the big, direct impact, and here's some other top-level > proposals: > > * embrace Taiga (an open source kanban tool) for project planning > * fix the compose speed (target: one hour!) > * really actually for real gated Rawhide > * better CI pipeline tests for everything > * define a base platform -- Red Hat wants to focus resources here > * better tooling for non-base deliverables > * better metrics for everything I am +100 to these changes. The idea that we can essentially not release for a year to do this raises the interesting point that perhaps our release cadence is not as rigid as we think it may be. I don't think we are going to skip Gnome and other major updates, are we? If not, then this also runs into the conversation about longer lifecycles and whether we need releases, imho. regards, bex > > > Please read > https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements > and comment in this thread. > > -- > Matthew Miller > > Fedora Project Leader > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- Brian (bex) Exelbierd | bexel...@redhat.com | b...@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
