Re: Rawhide plans
On Wed, 19 Aug 2015 11:52:25 -0400 Josh Boyer wrote: > I see issues that come as a side effect of _after_ you get rawhide > installed. With it being a rolling release, you theoretically never > have to do a fresh install again. Which means that the install > environment (both live and boot.iso) that are produced in rawhide get > very little testing. This in turn leads to hero testing around the > milestones for the subsequent Branched release. Yeah, that was actually mentioned in the workshop by someone (kalev?). We want people to test branched so it's stable and we can release it. If all those people are using rawhide, branched gets no testing. > I have no solution for this at the moment. I simply thought that it's > worth pointing out that if we convert lots of tester type people to > use rawhide on a daily basis, we're going to limit our tester pool > significantly. Right. I don't know that I have much solution either other than automated testing for those times that there's a branched. > This would help with the above issue I mentioned, but I'm not sure how > well it deals with non-KS installs? It boots the images in vms and screen scrapes I think... > > * Hook up taskotron to run checks on packages as they are built and > > send at least to the maintainer about the broken deps. If the > > maintainer sees their build causing lots of broken deps they can > > untag it or get someone to rebuild all the other packages. > > Why not take a step further and automatically untag it if broken deps > are found? Yeah, That was in the 'medium term' section with gating, because we need a way to override it. ...snip... > If you take the above, and then extend it to "automatically rebuild > any packages that depend on the package you just built" you are > getting pretty close to all the things OBS already does. Thats Open Suse Build service? (sorry, there's a ton of similar acronyms around that. OSBS, OBS, etc). > > * Possibly drop daily rawhide composes in favor of just rebuilding > > things when something in them changes. ie, new kernel build? then > > rebuild images/trees, new cowsay, just rebuild the repo. > > We build a new kernel every day with few exceptions. True. So, perhaps we should still do daily, but wait for some specific packages to build? Or just keep doing daily. kevin pgpnGclZWq7bK.pgp Description: OpenPGP digital signature -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Rawhide plans
On 20/08/15 03:52, Josh Boyer wrote: On Wed, Aug 19, 2015 at 11:18 AM, Kevin Fenzi wrote: [...] I think everyone also agreed that the pain points were around getting a rawhide install in the first place: 1. boot/netinstall iso is often broken or not produced. Live media likewise. 2. Upgrading from a stable release is also often hard due to broken deps, resulting in having to remove packages just to get upgraded. [...] I see issues that come as a side effect of _after_ you get rawhide installed. With it being a rolling release, you theoretically never have to do a fresh install again. Which means that the install environment (both live and boot.iso) that are produced in rawhide get very little testing. This in turn leads to hero testing around the milestones for the subsequent Branched release. I have no solution for this at the moment. I simply thought that it's worth pointing out that if we convert lots of tester type people to use rawhide on a daily basis, we're going to limit our tester pool significantly. [...] * Matt opened a thread on the marketing list about renaming rawhide. It sounds like most people would prefer us to make the changes first, then and only then look at renaming. This would make me very very sad. I suggest 'rawhide' NOT be renamed. As it improves, so will its reputation. However: I suggest that there should be a 'tannedhide', which is simply a copy of the last rawhide that passed all the checking (or at least, had nothing that seriously broke things - so things like wrong backgrounds & spelling mistakes would be okay). So a lot of people would test tannedhide, and the seriously hardcore would test rawhide. And possible a 'fashionhide', from time to time derived from 'tannedhide', that would be downloadable as a tentative set of images (including live media). [...] Cheers, Gavin -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Rawhide plans
On Wed, Aug 19, 2015 at 11:18 AM, Kevin Fenzi wrote: > Greetings. > > We had some nice discussions about rawhide in my friday workshop at > flock. I thought I would post here to get input from folks not there, > and also allow people who were there to comment more now that it's not > after 5pm on a friday of the 3rd day of flock. ;) > > * The problems/issues: > > First we talked about current issues we see. > > I think most everyone was in agreement that once you had a rawhide > install and were just applying updates, you were in pretty good shape. > Much better than you would have been a few years ago. There are bugs, > but they are often easy to workaround with downgrades until they are > fixed, etc. Also dnf tries very hard to not install anything thats part > of a set of broken deps, so while it might take longer to get some > packages you won't break your local install. > > I think everyone also agreed that the pain points were around getting a > rawhide install in the first place: > > 1. boot/netinstall iso is often broken or not produced. Live media > likewise. > > 2. Upgrading from a stable release is also often hard due to broken > deps, resulting in having to remove packages just to get upgraded. > > There was some mention of rawhide not being fully signed (which we > cannot do until we have gating of some kind and the sigul batch signing > bug fixed). > > There was mention of larger rebuilds that didn't use side tags. It's > pretty easy to request a side tag from releng and use it to do all your > rebuilds and then get it tagged back in. When larger stacks don't do > this, they break things for a few days while they sort out all the > packages. > > Finally there was mention of the baggage associated with the name > "rawhide". To this day I see people telling others that "rawhide will > eat your babies" or "rawhide is a methlab and will blow up in your face > every day". > > Do you all see other issues ? I see issues that come as a side effect of _after_ you get rawhide installed. With it being a rolling release, you theoretically never have to do a fresh install again. Which means that the install environment (both live and boot.iso) that are produced in rawhide get very little testing. This in turn leads to hero testing around the milestones for the subsequent Branched release. I have no solution for this at the moment. I simply thought that it's worth pointing out that if we convert lots of tester type people to use rawhide on a daily basis, we're going to limit our tester pool significantly. > Next I divided things up into 3 time periods. What could we do to make > things better short term (in the next few weeks), medium term (next few > months), longer term (next year). > > Short term: > > * Hook up openqa to run on rawhide every night and give us data on how > often things are broken on the install path. This could be a seperate > report, or we could hook it up to the rawhide compose. This would help with the above issue I mentioned, but I'm not sure how well it deals with non-KS installs? > * Hook up taskotron to run checks on packages as they are built and > send at least to the maintainer about the broken deps. If the > maintainer sees their build causing lots of broken deps they can > untag it or get someone to rebuild all the other packages. Why not take a step further and automatically untag it if broken deps are found? > * pungi4 landing. pungi4 is a big re-write of our pungi tool. When we > enable this in rawhide it's going to make rawhide look a lot more > like release trees. Instead of having a rawhide/ dir with just trees > and boot isos, we will have Everything tree like we do at release > time, and server/cloud/workstation trees each with their own branded > netinstall isos, the server dvd and such. Also, pungi4 emits json > logs with exactly what was produced, so we can easily parse from day > to day and know what broke, etc. Yay. > * Matt opened a thread on the marketing list about renaming rawhide. It > sounds like most people would prefer us to make the changes first, > then and only then look at renaming. This would make me very very sad. > Medium term: > > * Setup some gating. I think it should work like this: > > today you build a rawhide build and it tags into f24 tag, which is used > by the daily compose. > > I'd like to change that to build into a f24-candidate tag. At that > point taskotron or other tests are run. If the package passes it goes > to f24-pending tag. When a compose happens we run taskotron or other > tests on the f24-pending packages and if they pass, they tag into f24 > and the compose happens. > > This gives us two points to check/test things. If someone needs to > override due to a check thats wrong or some other need, they can simply > use the existing koji command line client to tag their package over and > it will go out in the compose anyhow. Note that we will then have a log > of who did this. ;) > >
Rawhide plans
Greetings. We had some nice discussions about rawhide in my friday workshop at flock. I thought I would post here to get input from folks not there, and also allow people who were there to comment more now that it's not after 5pm on a friday of the 3rd day of flock. ;) * The problems/issues: First we talked about current issues we see. I think most everyone was in agreement that once you had a rawhide install and were just applying updates, you were in pretty good shape. Much better than you would have been a few years ago. There are bugs, but they are often easy to workaround with downgrades until they are fixed, etc. Also dnf tries very hard to not install anything thats part of a set of broken deps, so while it might take longer to get some packages you won't break your local install. I think everyone also agreed that the pain points were around getting a rawhide install in the first place: 1. boot/netinstall iso is often broken or not produced. Live media likewise. 2. Upgrading from a stable release is also often hard due to broken deps, resulting in having to remove packages just to get upgraded. There was some mention of rawhide not being fully signed (which we cannot do until we have gating of some kind and the sigul batch signing bug fixed). There was mention of larger rebuilds that didn't use side tags. It's pretty easy to request a side tag from releng and use it to do all your rebuilds and then get it tagged back in. When larger stacks don't do this, they break things for a few days while they sort out all the packages. Finally there was mention of the baggage associated with the name "rawhide". To this day I see people telling others that "rawhide will eat your babies" or "rawhide is a methlab and will blow up in your face every day". Do you all see other issues ? Next I divided things up into 3 time periods. What could we do to make things better short term (in the next few weeks), medium term (next few months), longer term (next year). Short term: * Hook up openqa to run on rawhide every night and give us data on how often things are broken on the install path. This could be a seperate report, or we could hook it up to the rawhide compose. * Hook up taskotron to run checks on packages as they are built and send at least to the maintainer about the broken deps. If the maintainer sees their build causing lots of broken deps they can untag it or get someone to rebuild all the other packages. * pungi4 landing. pungi4 is a big re-write of our pungi tool. When we enable this in rawhide it's going to make rawhide look a lot more like release trees. Instead of having a rawhide/ dir with just trees and boot isos, we will have Everything tree like we do at release time, and server/cloud/workstation trees each with their own branded netinstall isos, the server dvd and such. Also, pungi4 emits json logs with exactly what was produced, so we can easily parse from day to day and know what broke, etc. * Matt opened a thread on the marketing list about renaming rawhide. It sounds like most people would prefer us to make the changes first, then and only then look at renaming. Medium term: * Setup some gating. I think it should work like this: today you build a rawhide build and it tags into f24 tag, which is used by the daily compose. I'd like to change that to build into a f24-candidate tag. At that point taskotron or other tests are run. If the package passes it goes to f24-pending tag. When a compose happens we run taskotron or other tests on the f24-pending packages and if they pass, they tag into f24 and the compose happens. This gives us two points to check/test things. If someone needs to override due to a check thats wrong or some other need, they can simply use the existing koji command line client to tag their package over and it will go out in the compose anyhow. Note that we will then have a log of who did this. ;) This will also hopefully allow us to enable autosign for rawhide packages (if we can get the batch signing bug fixed) since we can sign them at one of the above points. Longer term: * Put in place something so we can tell if a just rebuilt package is: - in the mock init root - in the deps for pungi4/mock/etc - on the boot/netinstall isos - on live media/images - not on/used by anything in composes If it is, then fire off a test rebuild of that thing, test it and only let the new thing in if it doesn't fail. We can also then relax requirements for leaf node type packages if we wish. * Possibly drop daily rawhide composes in favor of just rebuilding things when something in them changes. ie, new kernel build? then rebuild images/trees, new cowsay, just rebuild the repo. Feedback welcome. kevin pgpv383Ms0Vfz.pgp Description: OpenPGP digital signature -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
