Re: Re-evaluating Ubuntu's Milestones
On 09/04/18 02:46, Simon Quigley wrote: Hello, This past cycle was interesting, with Spectre/Meltdown complications among other things causing the first two Alphas to be cancelled, and on the Lubuntu side of things, we missed Beta 1 because of a critical bug that wasn't noticed due to lack of testing. This cycle made me question the purpose and usefulness of the milestones in a release cycle. If we do e.g. Alpha 1 as a community, these images are frozen in time and are said to be "safe to install" when in reality, further bugfixes can be spun in the daily ISO less than 24 hours later, making that milestone and the several days leading up to it something which will just be paved over the next day. In reality, due to the Proposed Migration procedures we have set in place[1], the release pocket should *always* be installable and working. Effort could be better spent making sure that the tests for these packages are better and that continuous integration is improved all around, rather than just hard freezing the archive and doing manual tests for some flavors, which often times blocks the work of others. I wanted to see if my thoughts were shared, and after speaking to several people from different flavors (albeit fairly informally, so these should not be treated as final opinions, but from Xubuntu, Ubuntu MATE, Kubuntu, and Ubuntu Budgie) regarding the state of milestones in the archive, it seemed like we shared common points, and it was clear to me that it would be a good idea to formally propose some change in these procedures. I am proposing that we get rid of the Alpha and Beta 1 milestones entirely, and we organize a monthly testing "week" (Tuesday through Thursday), which involves no archive freezes, no formally released ISOs, but testing of every product (community and Canonical) that is typically shipped as part of a final release (so including Desktop and Server). The goal would be to ensure that any issues are ironed out early on, to foster collaboration between people (Canonical, community, and everyone in between) regarding what's working and what isn't thus far in the cycle, and plans for the rest of the cycle (possibly via a short mailing list thread, something on the community hub, or whatever happens to work). So a (tentative) schedule would look like this for 18.10: 1. At the beginning of the cycle, the usual toolchain uploads are done, the autosyncer is turned on, and the "floodgates" are opened. This could potentially take us to the end of May, depending on how many transitions are started, what kind of changes are coming in from Debian, how quickly people can get things organized/migrated, and when Mark decides to announce the Calculating Camel. If the archive is consistent enough by the last weekend of May (26th/27th), we could do an organized testing session from then (so, the 29th through the 31st). This makes sure that once all of the transitions are finalized "for now" that nothing broke between the Bionic DIF[2] and that point. 2. By the latter half of June, we have hit the Feat. Def. Freeze[3] for some packages in Main, and the transitions which started from the floodgates being opened should be calmed down. The last week of June (26th through the 28th) could be another coordinated testing period. 3. In comes July, which is really the last full month you can land features that should be included in the 18.10 release, according to the usual timing of the release cycle. We would do coordinated testing that last week as well (24th through the 26th). Any last transitions could be discussed which affect everyone and could be landed in time for Feature Freeze. 4. Feature Freeze comes in on the week of August 23rd, ish. That would put coordinated testing right after that (28th through the 30th), and makes it important for there to be testing, because we can assume that features are no longer the primary focus of the flavors for the upcoming release. If upstream release cycles for common components happen to line up in a way which makes it desirable to ship in a release, that could be discussed. Otherwise, deep testing of edge cases (unusual install methods come to mind right away) would become a more specific goal, so we can ensure that the final release is as bug-free as possible. 5. In between that and the latter end of September would be UIF[4] and the Documentation String Freeze[5]. Additionally, this is also the typical spot of Final Beta, and the goal (since we didn't release a "Beta 1") would be to release an image, but call it "18.10 Beta" (thus removing versioned betas and just declaring it to be a "beta"). The archive would freeze as normal, Beta images would be released, and the archive would enter the usual semi-frozen state where seeded packages need manual approval. From this point on, the release cycle would proceed as it always has. Thoughts? [1] https://wiki.ubuntu.com/ProposedMigration [2] https://wiki.ubuntu.com/DebianImportFreeze [3] https://wiki.ubun
Re: Re-evaluating Ubuntu's Milestones
Hi Simon, On 22/04/18 17:08, Simon Quigley wrote: >> First my gut feeling is your going to see less community participation >> because there is no tangible outcome, obviously this will vary depending >> on the flavours community, some my do better, some may do worse. > The goal is to have established testing weeks increase the number for > *everyone* while lessening the burden for everyone. I really hope is does work out that way for all you flavours >> From a technical perspective having the archive frozen was quite useful, >> it allows you to focus on a fixed target, rather than getting distracted >> by a moving target that may well introduce further bugs. > Actually, I have typically seen more bugs *fixed* immediately after a > milestone release than introduced. Also, if we didn't have to spend our > time working hard to release a product (Alphas and Beta 1), we could > focus on improving and refining QA (both automated and manual) and spend > time on that instead. I think that has largely been because the freeze's have been a little to restrictive, Unless it was a installer bug or super critical flavour bug, it just would get put on hold. That is why I suggested relaxing somehow the restrictions during freeze, to allow flavours to control movement of packages during the archive freeze, The images wont necessarily be stale, because your team will upload what you need and then re-spin, without having to worry about random universe uploads etc. > >> LIkewise for >> giving the flavour leads control over re-spins rather than depending on >> daily builds. > Flavor leads do have control over respinning images, but they don't have > control over stopping them (maybe just pressing the "disable" button?); > we have yet to hear back from the Ubuntu Release Team on that. I am fully aware of that, more meant as an example, even if the resulting image does not become an official milestone release, there are many advantages to your dev teams, having the archive semi-frozen so you have control (mostly, there is obviously some overlap between flavours) of the resulting images. Flavours could leave there cron jobs running or choose to stop them and manually re-spin. Reducing that moving target to something more controlled is only going to benifet the testing week. >> I think >> traditionally there have been too many milestones in a cycle (perhaps >> somewhat biased by GNOME's late release cycle), however I still think >> the milestones serve an important purpose. If Ubuntu GNOME were still >> spinning ISO's, I'd probably advocate for a more hybrid model, use the >> more informal testing 'weeks' early in the cycle, then one beta and the >> RC Milestones. > This is precisely what I'm proposing. Get rid of Alpha 1, Alpha 2, and > Beta 1, then rename "Final Beta" to just "Beta". Then we have the RCs, > which aren't really a milestone. Good! > >> As for the automated testing, I think is important, however my >> recollections of so many milestone releases dealing with somewhat corner >> case installer bugs, wonder how you will get 100% test coverage. Also >> for some flavours the work maintaining these test cases may end up being >> as much work as co-ordinating milestone releases. I would probably >> recommend getting the automated testing in place before changing things >> too drastically. > I disagree; while putting in automated testing is definitely something > that I believe in making a priority, we don't need to block on change > for that. The goal is to "set it and forget it" (which, with the nature > of how these tests are designed, can be done in that way), so while it > may involve some initial investment, I think the return far exceeds the > time spent. > Please do not underestimate that, it will never be set and forget, as things evolve there are going to be test regressions, which won't necessarily be bugs the software but bugs in the assumptions made by tests. Just like we see now in test suites and autopkgtests across every cycle. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hello, On 04/22/2018 01:40 AM, Tim wrote: > Obviously Ubuntu GNOME no longer builds ISO's however I would like to > chip in on this discussion with a few thoughts based on past experience. > Feel free to do with them what you wish, its not meant to be criticism > in any way of the current proposal, but more just something to think > about. Sorry I didnt reply earlier, just haven't had the spare time! Thanks for taking the time to reply. > First my gut feeling is your going to see less community participation > because there is no tangible outcome, obviously this will vary depending > on the flavours community, some my do better, some may do worse. The goal is to have established testing weeks increase the number for *everyone* while lessening the burden for everyone. > From a technical perspective having the archive frozen was quite useful, > it allows you to focus on a fixed target, rather than getting distracted > by a moving target that may well introduce further bugs. Actually, I have typically seen more bugs *fixed* immediately after a milestone release than introduced. Also, if we didn't have to spend our time working hard to release a product (Alphas and Beta 1), we could focus on improving and refining QA (both automated and manual) and spend time on that instead. > LIkewise for > giving the flavour leads control over re-spins rather than depending on > daily builds. Flavor leads do have control over respinning images, but they don't have control over stopping them (maybe just pressing the "disable" button?); we have yet to hear back from the Ubuntu Release Team on that. > I would also agree at times, that is was somewhat > restrictive at times, but a semi-frozen archive where flavours had more > control over the flow of packages, could lessen that (auto-accept > flavour uploads perhaps, that don't overlap other flavours?). This would make the ISOs stale, thus proving the point of the images being "stale on arrival". ;) > I think > traditionally there have been too many milestones in a cycle (perhaps > somewhat biased by GNOME's late release cycle), however I still think > the milestones serve an important purpose. If Ubuntu GNOME were still > spinning ISO's, I'd probably advocate for a more hybrid model, use the > more informal testing 'weeks' early in the cycle, then one beta and the > RC Milestones. This is precisely what I'm proposing. Get rid of Alpha 1, Alpha 2, and Beta 1, then rename "Final Beta" to just "Beta". Then we have the RCs, which aren't really a milestone. > As for the automated testing, I think is important, however my > recollections of so many milestone releases dealing with somewhat corner > case installer bugs, wonder how you will get 100% test coverage. Also > for some flavours the work maintaining these test cases may end up being > as much work as co-ordinating milestone releases. I would probably > recommend getting the automated testing in place before changing things > too drastically. I disagree; while putting in automated testing is definitely something that I believe in making a priority, we don't need to block on change for that. The goal is to "set it and forget it" (which, with the nature of how these tests are designed, can be done in that way), so while it may involve some initial investment, I think the return far exceeds the time spent. -- Simon Quigley tsimo...@ubuntu.com tsimonq2 on freenode and OFTC 5C7A BEA2 0F86 3045 9CC8 C8B5 E27F 2CF8 458C 2FA4 signature.asc Description: OpenPGP digital signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hello, On 04/21/2018 04:15 AM, flocculant wrote: > One quick question here - imagine that :) > > Given that we all (apparently) find it hard to get people testing during > the 3 days we currently get at milestone's - why are we carrying on with > that same tight schedule? You'll know the way this happens - it's the > end of the testing session and suddenly someone is asking for help > looking at their images for some reason. > > If we are going to just organise sections of time amongst ourselves > during these new periods - why not do away with "weeks" and actually > have a week - a real week. If we can't manage to organise amongst > ourselves for longer than a couple of days then I think we should all > pack up and do something else ;) That works for me. We can work out the fine specifics when we get closer (I have a couple of things in mind that I'd like to iron out first), but I'm not opposed to making it weeks instead of "weeks". > If not, then all that's really been accomplished here is making life > easier for Canonical (not that I disagree with us doing that I hasten to > add), we as flavours are gaining nothing. This isn't really making things easier for just the Ubuntu Release Team. Even if we still kept to a strict schedule, we don't have to worry about people testing with old installs (see my point to Oliver about the "pristine" ISOs). Thanks for the email. -- Simon Quigley tsimo...@ubuntu.com tsimonq2 on freenode and OFTC 5C7A BEA2 0F86 3045 9CC8 C8B5 E27F 2CF8 458C 2FA4 signature.asc Description: OpenPGP digital signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hey Flavours, Obviously Ubuntu GNOME no longer builds ISO's however I would like to chip in on this discussion with a few thoughts based on past experience. Feel free to do with them what you wish, its not meant to be criticism in any way of the current proposal, but more just something to think about. Sorry I didnt reply earlier, just haven't had the spare time! First my gut feeling is your going to see less community participation because there is no tangible outcome, obviously this will vary depending on the flavours community, some my do better, some may do worse. From a technical perspective having the archive frozen was quite useful, it allows you to focus on a fixed target, rather than getting distracted by a moving target that may well introduce further bugs. LIkewise for giving the flavour leads control over re-spins rather than depending on daily builds. I would also agree at times, that is was somewhat restrictive at times, but a semi-frozen archive where flavours had more control over the flow of packages, could lessen that (auto-accept flavour uploads perhaps, that don't overlap other flavours?). I think traditionally there have been too many milestones in a cycle (perhaps somewhat biased by GNOME's late release cycle), however I still think the milestones serve an important purpose. If Ubuntu GNOME were still spinning ISO's, I'd probably advocate for a more hybrid model, use the more informal testing 'weeks' early in the cycle, then one beta and the RC Milestones. As for the automated testing, I think is important, however my recollections of so many milestone releases dealing with somewhat corner case installer bugs, wonder how you will get 100% test coverage. Also for some flavours the work maintaining these test cases may end up being as much work as co-ordinating milestone releases. I would probably recommend getting the automated testing in place before changing things too drastically. Regards Tim On 21/04/18 19:15, flocculant wrote: > On 21/04/18 02:35, Simon Quigley wrote: >> Happy Release Week! >> >> I do not believe there have been any -1s to this proposal from any >> flavor, nor from the Release Team, so I think it's time to move forward >> with it. >> >> In summary, what will now happen from here on out is that opt-in >> milestones will be discontinued in favor of testing "weeks" (Tuesday >> through Thursday). I can organize the testing weeks for the 18.10 cycle >> (so we can get a process going), but from the 19.04 cycle and on, >> representatives (probably Release Managers) from any active flavor can >> (and should!) organize these testing weeks. >> >> Additionally, I will look into the automated testing Steve brought up >> shortly after the 18.04 release, with the goal being to adopt that >> sooner rather than later. I'll write a follow-up email to ubuntu-release >> once I have something to show for that. >> >> Thanks everyone! >> >> >> > One quick question here - imagine that :) > > Given that we all (apparently) find it hard to get people testing > during the 3 days we currently get at milestone's - why are we > carrying on with that same tight schedule? You'll know the way this > happens - it's the end of the testing session and suddenly someone is > asking for help looking at their images for some reason. > > If we are going to just organise sections of time amongst ourselves > during these new periods - why not do away with "weeks" and actually > have a week - a real week. If we can't manage to organise amongst > ourselves for longer than a couple of days then I think we should all > pack up and do something else ;) > > If not, then all that's really been accomplished here is making life > easier for Canonical (not that I disagree with us doing that I hasten > to add), we as flavours are gaining nothing. > > We - as flavours - still have 3 days only to try and get people to > find 30 minutes in their life. > > cheers - have good weekends > > Kev > > NB - personally I'd still prefer a system where I could stop my image > building - and then do whatever testing I wanted to - then turn the > build on again. > > > > -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On 21/04/18 02:35, Simon Quigley wrote: Happy Release Week! I do not believe there have been any -1s to this proposal from any flavor, nor from the Release Team, so I think it's time to move forward with it. In summary, what will now happen from here on out is that opt-in milestones will be discontinued in favor of testing "weeks" (Tuesday through Thursday). I can organize the testing weeks for the 18.10 cycle (so we can get a process going), but from the 19.04 cycle and on, representatives (probably Release Managers) from any active flavor can (and should!) organize these testing weeks. Additionally, I will look into the automated testing Steve brought up shortly after the 18.04 release, with the goal being to adopt that sooner rather than later. I'll write a follow-up email to ubuntu-release once I have something to show for that. Thanks everyone! One quick question here - imagine that :) Given that we all (apparently) find it hard to get people testing during the 3 days we currently get at milestone's - why are we carrying on with that same tight schedule? You'll know the way this happens - it's the end of the testing session and suddenly someone is asking for help looking at their images for some reason. If we are going to just organise sections of time amongst ourselves during these new periods - why not do away with "weeks" and actually have a week - a real week. If we can't manage to organise amongst ourselves for longer than a couple of days then I think we should all pack up and do something else ;) If not, then all that's really been accomplished here is making life easier for Canonical (not that I disagree with us doing that I hasten to add), we as flavours are gaining nothing. We - as flavours - still have 3 days only to try and get people to find 30 minutes in their life. cheers - have good weekends Kev NB - personally I'd still prefer a system where I could stop my image building - and then do whatever testing I wanted to - then turn the build on again. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Happy Release Week! I do not believe there have been any -1s to this proposal from any flavor, nor from the Release Team, so I think it's time to move forward with it. In summary, what will now happen from here on out is that opt-in milestones will be discontinued in favor of testing "weeks" (Tuesday through Thursday). I can organize the testing weeks for the 18.10 cycle (so we can get a process going), but from the 19.04 cycle and on, representatives (probably Release Managers) from any active flavor can (and should!) organize these testing weeks. Additionally, I will look into the automated testing Steve brought up shortly after the 18.04 release, with the goal being to adopt that sooner rather than later. I'll write a follow-up email to ubuntu-release once I have something to show for that. Thanks everyone! -- Simon Quigley tsimo...@ubuntu.com tsimonq2 on freenode and OFTC 5C7A BEA2 0F86 3045 9CC8 C8B5 E27F 2CF8 458C 2FA4 signature.asc Description: OpenPGP digital signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
I follow suit with Steve here with regards to the automated testing (the referenced gate). If there are toolsets that already exist - even better! And from the sounds of it, the process has already been vetted. Dustin On Mon, Apr 9, 2018, 16:13 Steve Langasek, wrote: > On Mon, Apr 09, 2018 at 02:31:31PM -0500, Simon Quigley wrote: > > On 2018-04-09 03:30, Oliver Grawert wrote: > > > well, apart from actual installer fixes, your users should get all > > > these fixes through package updates anyway ... > > > Right, which is another point for getting rid of these extra milestones, > in > > my opinion. > > > > One thing that the other pro/con responses did not cover yet but that > > > should not be underestimated is the promotional aspect of milestones > > > ... > > > > You typically get press coverage for such pre-releases and will likely > > > attract more testers. > > > Not really, actually. In my experience, testers are present when there > is an > > occasion to test them, regardless of putting a name to it or releasing an > > ISO. I could see your point if my proposal was to get rid of the > milestones > > entirely with no replacements, but in this case, having the testing week > > once a month would attract press coverage as well. > > > Why? Because milestones in all reality are just a fancy name to slap on > an > > ISO that will most likely be stale the next day. You then get people > > installing from these ISOs and potentially even reporting old bugs. The > > unfortunate reality of this press coverage is that you could pick an ISO > any > > day of the month and call it "beta," and just because it has that name > on it > > means that people will install it because of the appeal of the name. > Despite > > the positive press that comes from the associated announcements (that can > > always be made regardless, which is what Lubuntu has started doing[1]), > in a > > technical sense, I would even consider it *bad* for people to install > using > > these ISOs. > > > The coordinated testing weeks would allow for there still to be positive > > press coverage (and maybe announcements resulting from cross-team > > collaboration during these times) while not having the downsides of a > > blessed image when the archive isn't in a decently stable state. > > I agree (as you know). > > The one other value milestone images provide is to give a "known good" > image > to install the development release from. We have solved this for Ubuntu > Desktop and Server by having automated tests that gate the promotion of an > image build to "current". I would strongly encourage flavors to > collaborate > around this automation, instead of continuing to rely on a heavy-weight > manual test process that leaves the "known good" image stale for weeks at a > time. > > Code for this automation lives here: > > > https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop > > If there is interest from flavors in having this image gate, I would be > willing to argue for Canonical hosting of the test infrastructure, as > necessary. > > And if there *isn't* interest from flavors in doing this, well, I also > don't > think that should block on that as a reason to carry on with the existing > milestone process, which I think is a very inefficient use of everyone's > time. > > So to summarize, I think the right path forward is: > > - discontinue all opt-in milestones for 18.10 and beyond >- implicitly discontinuing the matching milestone freezes > - coordinate a cadence of "testing weeks", organized by the flavor leads >(i.e.: requires no involvement from ubuntu-cdimage or ubuntu-release in >order to drive to success) > - at the flavor teams' discretion, implement automated QA gating of daily >image promotions > > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developerhttp://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release > -- Dustin Krysak -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
We mentioned this to Simon privately ... but saying things publicly is also useful... Ubuntu Budgie fully agrees with this approach. If the "advertisement - promotion" aspect is useful to prevent the flavours being drowned out by plain old ubuntu (!) then we could just call the post feature freeze test event "beta 1". It will give those friendly blog sites something to latch onto. David On 10 April 2018 at 16:47, Martin Wimpress wrote: > Hi all, > > > On 10 April 2018 at 03:58, Simon Quigley wrote: >> Hello, >> >> On 04/09/2018 03:13 PM, Steve Langasek wrote: >> >>> The one other value milestone images provide is to give a "known good" image >>> to install the development release from. We have solved this for Ubuntu >>> Desktop and Server by having automated tests that gate the promotion of an >>> image build to "current". I would strongly encourage flavors to collaborate >>> around this automation, instead of continuing to rely on a heavy-weight >>> manual test process that leaves the "known good" image stale for weeks at a >>> time. >>> >>> Code for this automation lives here: >>> >>> https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop >>> >>> If there is interest from flavors in having this image gate, I would be >>> willing to argue for Canonical hosting of the test infrastructure, as >>> necessary. >>> >>> And if there *isn't* interest from flavors in doing this, well, I also don't >>> think that should block on that as a reason to carry on with the existing >>> milestone process, which I think is a very inefficient use of everyone's >>> time. >> >> I think this is an excellent idea which would be great to look into for >> the 18.10 cycle. >> >> Count Lubuntu in. >> > > Absolutely count Ubuntu MATE in too. > >>> So to summarize, I think the right path forward is: >>> >>> - discontinue all opt-in milestones for 18.10 and beyond >>>- implicitly discontinuing the matching milestone freezes >>> - coordinate a cadence of "testing weeks", organized by the flavor leads >>>(i.e.: requires no involvement from ubuntu-cdimage or ubuntu-release in >>>order to drive to success) >>> - at the flavor teams' discretion, implement automated QA gating of daily >>>image promotions >> >> Great, I think we're on the same page here. :) > > If the above can happen, that would be fantastic. Automating the heavy > lifting of testing basic installer capabilities is great. In my > experience the people interested in contributing time to QA are most > interested in testing how new features look/work than ensuring the > various install options are well tested and working. > > Regards, Martin. > > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hi all, On 10 April 2018 at 03:58, Simon Quigley wrote: > Hello, > > On 04/09/2018 03:13 PM, Steve Langasek wrote: > >> The one other value milestone images provide is to give a "known good" image >> to install the development release from. We have solved this for Ubuntu >> Desktop and Server by having automated tests that gate the promotion of an >> image build to "current". I would strongly encourage flavors to collaborate >> around this automation, instead of continuing to rely on a heavy-weight >> manual test process that leaves the "known good" image stale for weeks at a >> time. >> >> Code for this automation lives here: >> >> https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop >> >> If there is interest from flavors in having this image gate, I would be >> willing to argue for Canonical hosting of the test infrastructure, as >> necessary. >> >> And if there *isn't* interest from flavors in doing this, well, I also don't >> think that should block on that as a reason to carry on with the existing >> milestone process, which I think is a very inefficient use of everyone's >> time. > > I think this is an excellent idea which would be great to look into for > the 18.10 cycle. > > Count Lubuntu in. > Absolutely count Ubuntu MATE in too. >> So to summarize, I think the right path forward is: >> >> - discontinue all opt-in milestones for 18.10 and beyond >>- implicitly discontinuing the matching milestone freezes >> - coordinate a cadence of "testing weeks", organized by the flavor leads >>(i.e.: requires no involvement from ubuntu-cdimage or ubuntu-release in >>order to drive to success) >> - at the flavor teams' discretion, implement automated QA gating of daily >>image promotions > > Great, I think we're on the same page here. :) If the above can happen, that would be fantastic. Automating the heavy lifting of testing basic installer capabilities is great. In my experience the people interested in contributing time to QA are most interested in testing how new features look/work than ensuring the various install options are well tested and working. Regards, Martin. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hello, On 04/09/2018 03:13 PM, Steve Langasek wrote: > The one other value milestone images provide is to give a "known good" image > to install the development release from. We have solved this for Ubuntu > Desktop and Server by having automated tests that gate the promotion of an > image build to "current". I would strongly encourage flavors to collaborate > around this automation, instead of continuing to rely on a heavy-weight > manual test process that leaves the "known good" image stale for weeks at a > time. > > Code for this automation lives here: > > https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop > > If there is interest from flavors in having this image gate, I would be > willing to argue for Canonical hosting of the test infrastructure, as > necessary. > > And if there *isn't* interest from flavors in doing this, well, I also don't > think that should block on that as a reason to carry on with the existing > milestone process, which I think is a very inefficient use of everyone's > time. I think this is an excellent idea which would be great to look into for the 18.10 cycle. Count Lubuntu in. > So to summarize, I think the right path forward is: > > - discontinue all opt-in milestones for 18.10 and beyond >- implicitly discontinuing the matching milestone freezes > - coordinate a cadence of "testing weeks", organized by the flavor leads >(i.e.: requires no involvement from ubuntu-cdimage or ubuntu-release in >order to drive to success) > - at the flavor teams' discretion, implement automated QA gating of daily >image promotions Great, I think we're on the same page here. :) -- Simon Quigley tsimo...@ubuntu.com tsimonq2 on freenode and OFTC 5C7A BEA2 0F86 3045 9CC8 C8B5 E27F 2CF8 458C 2FA4 signature.asc Description: OpenPGP digital signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On Mon, Apr 9, 2018 at 4:54 PM, Valorie Zimmerman wrote: > I was waiting for buy-in from the Release team and now that we have it I think Canonical's (and the Release Team's) position on this has been clear for years, but they wisely allowed the community to do their own analysis and make their own decisions. Some irrelevant commentary: Fedora now has the same general release process with only one milestone, a Beta. The Fedora 28 Beta came out last week, the same week as our (Final) Beta. Thanks, Jeremy Bicha -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On Mon, Apr 9, 2018 at 1:35 PM, flocculant wrote: > On 09/04/18 21:13, Steve Langasek wrote: > > ... > > The one other value milestone images provide is to give a "known good" image > to install the development release from. We have solved this for Ubuntu > Desktop and Server by having automated tests that gate the promotion of an > image build to "current". I would strongly encourage flavors to collaborate > around this automation, instead of continuing to rely on a heavy-weight > manual test process that leaves the "known good" image stale for weeks at a > time. > > Code for this automation lives here: > > https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop > > If there is interest from flavors in having this image gate, I would be > willing to argue for Canonical hosting of the test infrastructure, as... > > > I am certainly interested in that from Xubuntu point of view - I do try to > at least check that we built overnight. And time willing zsync and quick > boot to desktop. > > So that would make my life simpler for sure. I was waiting for buy-in from the Release team and now that we have it, I heartily endorse this. In Kubuntu we put a lot of work into our CI now, and further automated testing of the accepted packageset would be very welcome. I won't deny that human testing is very valuable, but much moreso when the testers are testing *pre-tested* packages. I think we flavors can collaborate and come up with some excellent promotion material so that all of us aren't trying to do it on our own. When the virtuous circle of better promotion and better testing gets going, all of us will have better software to offer to more users. Valorie -- http://about.me/valoriez -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On 09/04/18 21:13, Steve Langasek wrote: ... The one other value milestone images provide is to give a "known good" image to install the development release from. We have solved this for Ubuntu Desktop and Server by having automated tests that gate the promotion of an image build to "current". I would strongly encourage flavors to collaborate around this automation, instead of continuing to rely on a heavy-weight manual test process that leaves the "known good" image stale for weeks at a time. Code for this automation lives here: https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop If there is interest from flavors in having this image gate, I would be willing to argue for Canonical hosting of the test infrastructure, as... I am certainly interested in that from Xubuntu point of view - I do try to at least check that we built overnight. And time willing zsync and quick boot to desktop. So that would make my life simpler for sure. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On Mon, Apr 09, 2018 at 02:31:31PM -0500, Simon Quigley wrote: > On 2018-04-09 03:30, Oliver Grawert wrote: > > well, apart from actual installer fixes, your users should get all > > these fixes through package updates anyway ... > Right, which is another point for getting rid of these extra milestones, in > my opinion. > > One thing that the other pro/con responses did not cover yet but that > > should not be underestimated is the promotional aspect of milestones > > ... > > You typically get press coverage for such pre-releases and will likely > > attract more testers. > Not really, actually. In my experience, testers are present when there is an > occasion to test them, regardless of putting a name to it or releasing an > ISO. I could see your point if my proposal was to get rid of the milestones > entirely with no replacements, but in this case, having the testing week > once a month would attract press coverage as well. > Why? Because milestones in all reality are just a fancy name to slap on an > ISO that will most likely be stale the next day. You then get people > installing from these ISOs and potentially even reporting old bugs. The > unfortunate reality of this press coverage is that you could pick an ISO any > day of the month and call it "beta," and just because it has that name on it > means that people will install it because of the appeal of the name. Despite > the positive press that comes from the associated announcements (that can > always be made regardless, which is what Lubuntu has started doing[1]), in a > technical sense, I would even consider it *bad* for people to install using > these ISOs. > The coordinated testing weeks would allow for there still to be positive > press coverage (and maybe announcements resulting from cross-team > collaboration during these times) while not having the downsides of a > blessed image when the archive isn't in a decently stable state. I agree (as you know). The one other value milestone images provide is to give a "known good" image to install the development release from. We have solved this for Ubuntu Desktop and Server by having automated tests that gate the promotion of an image build to "current". I would strongly encourage flavors to collaborate around this automation, instead of continuing to rely on a heavy-weight manual test process that leaves the "known good" image stale for weeks at a time. Code for this automation lives here: https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/desktop If there is interest from flavors in having this image gate, I would be willing to argue for Canonical hosting of the test infrastructure, as necessary. And if there *isn't* interest from flavors in doing this, well, I also don't think that should block on that as a reason to carry on with the existing milestone process, which I think is a very inefficient use of everyone's time. So to summarize, I think the right path forward is: - discontinue all opt-in milestones for 18.10 and beyond - implicitly discontinuing the matching milestone freezes - coordinate a cadence of "testing weeks", organized by the flavor leads (i.e.: requires no involvement from ubuntu-cdimage or ubuntu-release in order to drive to success) - at the flavor teams' discretion, implement automated QA gating of daily image promotions -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Generally in favour here, and i saw no significant opposite opinions when we've informally discussed this in Kubuntu. Rik Mills Kubuntu Devel Kubuntu Council On 09/04/18 02:46, Simon Quigley wrote: > Hello, > > This past cycle was interesting, with Spectre/Meltdown complications > among other things causing the first two Alphas to be cancelled, and on > the Lubuntu side of things, we missed Beta 1 because of a critical bug > that wasn't noticed due to lack of testing. This cycle made me question > the purpose and usefulness of the milestones in a release cycle. > > If we do e.g. Alpha 1 as a community, these images are frozen in time > and are said to be "safe to install" when in reality, further bugfixes > can be spun in the daily ISO less than 24 hours later, making that > milestone and the several days leading up to it something which will > just be paved over the next day. In reality, due to the Proposed > Migration procedures we have set in place[1], the release pocket should > *always* be installable and working. Effort could be better spent making > sure that the tests for these packages are better and that continuous > integration is improved all around, rather than just hard freezing the > archive and doing manual tests for some flavors, which often times > blocks the work of others. > > I wanted to see if my thoughts were shared, and after speaking to > several people from different flavors (albeit fairly informally, so > these should not be treated as final opinions, but from Xubuntu, Ubuntu > MATE, Kubuntu, and Ubuntu Budgie) regarding the state of milestones in > the archive, it seemed like we shared common points, and it was clear to > me that it would be a good idea to formally propose some change in these > procedures. > > I am proposing that we get rid of the Alpha and Beta 1 milestones > entirely, and we organize a monthly testing "week" (Tuesday through > Thursday), which involves no archive freezes, no formally released ISOs, > but testing of every product (community and Canonical) that is typically > shipped as part of a final release (so including Desktop and Server). > The goal would be to ensure that any issues are ironed out early on, to > foster collaboration between people (Canonical, community, and everyone > in between) regarding what's working and what isn't thus far in the > cycle, and plans for the rest of the cycle (possibly via a short mailing > list thread, something on the community hub, or whatever happens to work). > > So a (tentative) schedule would look like this for 18.10: > 1. At the beginning of the cycle, the usual toolchain uploads are done, > the autosyncer is turned on, and the "floodgates" are opened. This could > potentially take us to the end of May, depending on how many transitions > are started, what kind of changes are coming in from Debian, how quickly > people can get things organized/migrated, and when Mark decides to > announce the Calculating Camel. If the archive is consistent enough by > the last weekend of May (26th/27th), we could do an organized testing > session from then (so, the 29th through the 31st). This makes sure that > once all of the transitions are finalized "for now" that nothing broke > between the Bionic DIF[2] and that point. > 2. By the latter half of June, we have hit the Feat. Def. Freeze[3] for > some packages in Main, and the transitions which started from the > floodgates being opened should be calmed down. The last week of June > (26th through the 28th) could be another coordinated testing period. > 3. In comes July, which is really the last full month you can land > features that should be included in the 18.10 release, according to the > usual timing of the release cycle. We would do coordinated testing that > last week as well (24th through the 26th). Any last transitions could be > discussed which affect everyone and could be landed in time for Feature > Freeze. > 4. Feature Freeze comes in on the week of August 23rd, ish. That would > put coordinated testing right after that (28th through the 30th), and > makes it important for there to be testing, because we can assume that > features are no longer the primary focus of the flavors for the upcoming > release. If upstream release cycles for common components happen to line > up in a way which makes it desirable to ship in a release, that could be > discussed. Otherwise, deep testing of edge cases (unusual install > methods come to mind right away) would become a more specific goal, so > we can ensure that the final release is as bug-free as possible. > 5. In between that and the latter end of September would be UIF[4] and > the Documentation String Freeze[5]. Additionally, this is also the > typical spot of Final Beta, and the goal (since we didn't release a > "Beta 1") would be to release an image, but call it "18.10 Beta" (thus > removing versioned betas and just declaring it to be a "beta"). The > archive would freeze as normal, Beta images would be rel
Re: Re-evaluating Ubuntu's Milestones
Hello Oliver, On 2018-04-09 03:30, Oliver Grawert wrote: well, apart from actual installer fixes, your users should get all these fixes through package updates anyway ... Right, which is another point for getting rid of these extra milestones, in my opinion. One thing that the other pro/con responses did not cover yet but that should not be underestimated is the promotional aspect of milestones ... You typically get press coverage for such pre-releases and will likely attract more testers. Not really, actually. In my experience, testers are present when there is an occasion to test them, regardless of putting a name to it or releasing an ISO. I could see your point if my proposal was to get rid of the milestones entirely with no replacements, but in this case, having the testing week once a month would attract press coverage as well. Why? Because milestones in all reality are just a fancy name to slap on an ISO that will most likely be stale the next day. You then get people installing from these ISOs and potentially even reporting old bugs. The unfortunate reality of this press coverage is that you could pick an ISO any day of the month and call it "beta," and just because it has that name on it means that people will install it because of the appeal of the name. Despite the positive press that comes from the associated announcements (that can always be made regardless, which is what Lubuntu has started doing[1]), in a technical sense, I would even consider it *bad* for people to install using these ISOs. The coordinated testing weeks would allow for there still to be positive press coverage (and maybe announcements resulting from cross-team collaboration during these times) while not having the downsides of a blessed image when the archive isn't in a decently stable state. [1] https://lubuntu.me/category/newsletter/ -- Simon Quigley tsimo...@ubuntu.com tsimonq2 on freenode and OFTC 5C7A BEA2 0F86 3045 9CC8 C8B5 E27F 2CF8 458C 2FA4 -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
hi, Am Sonntag, den 08.04.2018, 20:46 -0500 schrieb Simon Quigley: > If we do e.g. Alpha 1 as a community, these images are frozen in time > and are said to be "safe to install" when in reality, further > bugfixes > can be spun in the daily ISO less than 24 hours later well, apart from actual installer fixes, your users should get all these fixes through package updates anyway ... One thing that the other pro/con responses did not cover yet but that should not be underestimated is the promotional aspect of milestones ... You typically get press coverage for such pre-releases and will likely attract more testers. The install process is the place where you hit the corner-case hardware setups first, a press-announced milestone might make people test it that would typically only run into issues when they pick the release image where you do not have any chance any more to re-spin. ciao oli signature.asc Description: This is a digitally signed message part -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hi Simon, On 9 April 2018 at 02:46, Simon Quigley wrote: > I am proposing that we get rid of the Alpha and Beta 1 milestones > entirely, and we organize a monthly testing "week" (Tuesday through > Thursday), which involves no archive freezes, no formally released ISOs, > but testing of every product (community and Canonical) that is typically > shipped as part of a final release (so including Desktop and Server). > The goal would be to ensure that any issues are ironed out early on, to > foster collaboration between people (Canonical, community, and everyone > in between) regarding what's working and what isn't thus far in the > cycle, and plans for the rest of the cycle (possibly via a short mailing > list thread, something on the community hub, or whatever happens to work). > 1. Knowing in advance when lubuntu testing is desired is very helpful to me. 2. Having a testing "week" is also very helpful because I do have other demands on my time and this would make it easier for me to set aside time for testing. BW, Ian -- -- ACCU - Professionalism in programming - http://www.accu.org -- My writing - https://sites.google.com/site/ianbruntlett/ -- Free Software page - https://sites.google.com/site/ianbruntlett/home/free-software -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
On 09/04/18 02:46, Simon Quigley wrote: Hello, [snip] Thoughts? [1] https://wiki.ubuntu.com/ProposedMigration [2] https://wiki.ubuntu.com/DebianImportFreeze [3] https://wiki.ubuntu.com/FeatureDefinitionFreeze [4] https://wiki.ubuntu.com/UserInterfaceFreeze [5] https://wiki.ubuntu.com/DocumentationStringFreeze I think you know where I stand on this. That said I'll need the approval of the Xubuntu Team as a whole to make it an official point. Personally however I would prefer a system where: 1. Flavour decides their iso is in a state suitable for testing. 2. Flavour has ability to turn it's iso build off. 3. Flavour does it's testing 4. Flavour turns it's iso build on. That however obviously would need input from the Canonical Release Team etc. Cheers Kev -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
Hey Simon, We had spoken offline about this, plus I had spoken to the or project lead, and our team's (Ubuntu Budgie) general feel is that identifying issues earlier, plus bringing the flavors closer (collaboration) where there is cross over just makes sense. Throw our hat in the ring (for support). Dustin Krysak On Sun, Apr 8, 2018, 20:18 handsome_feng, wrote: > > > I wanted to see if my thoughts were shared, and after speaking to > > several people from different flavors (albeit fairly informally, so > > these should not be treated as final opinions, but from Xubuntu, Ubuntu > > MATE, Kubuntu, and Ubuntu Budgie) regarding the state of milestones in > > the archive, it seemed like we shared common points, and it was clear to > > me that it would be a good idea to formally propose some change in these > > procedures. > > Ubuntu Kylin also takes it as a good move to do the modification. Please > feel > free to add us in the supporting list. > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release > -- Dustin Krysak -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Re-evaluating Ubuntu's Milestones
> I wanted to see if my thoughts were shared, and after speaking to > several people from different flavors (albeit fairly informally, so > these should not be treated as final opinions, but from Xubuntu, Ubuntu > MATE, Kubuntu, and Ubuntu Budgie) regarding the state of milestones in > the archive, it seemed like we shared common points, and it was clear to > me that it would be a good idea to formally propose some change in these > procedures. Ubuntu Kylin also takes it as a good move to do the modification. Please feel free to add us in the supporting list. 0xB43C0E3B.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re-evaluating Ubuntu's Milestones
Hello, This past cycle was interesting, with Spectre/Meltdown complications among other things causing the first two Alphas to be cancelled, and on the Lubuntu side of things, we missed Beta 1 because of a critical bug that wasn't noticed due to lack of testing. This cycle made me question the purpose and usefulness of the milestones in a release cycle. If we do e.g. Alpha 1 as a community, these images are frozen in time and are said to be "safe to install" when in reality, further bugfixes can be spun in the daily ISO less than 24 hours later, making that milestone and the several days leading up to it something which will just be paved over the next day. In reality, due to the Proposed Migration procedures we have set in place[1], the release pocket should *always* be installable and working. Effort could be better spent making sure that the tests for these packages are better and that continuous integration is improved all around, rather than just hard freezing the archive and doing manual tests for some flavors, which often times blocks the work of others. I wanted to see if my thoughts were shared, and after speaking to several people from different flavors (albeit fairly informally, so these should not be treated as final opinions, but from Xubuntu, Ubuntu MATE, Kubuntu, and Ubuntu Budgie) regarding the state of milestones in the archive, it seemed like we shared common points, and it was clear to me that it would be a good idea to formally propose some change in these procedures. I am proposing that we get rid of the Alpha and Beta 1 milestones entirely, and we organize a monthly testing "week" (Tuesday through Thursday), which involves no archive freezes, no formally released ISOs, but testing of every product (community and Canonical) that is typically shipped as part of a final release (so including Desktop and Server). The goal would be to ensure that any issues are ironed out early on, to foster collaboration between people (Canonical, community, and everyone in between) regarding what's working and what isn't thus far in the cycle, and plans for the rest of the cycle (possibly via a short mailing list thread, something on the community hub, or whatever happens to work). So a (tentative) schedule would look like this for 18.10: 1. At the beginning of the cycle, the usual toolchain uploads are done, the autosyncer is turned on, and the "floodgates" are opened. This could potentially take us to the end of May, depending on how many transitions are started, what kind of changes are coming in from Debian, how quickly people can get things organized/migrated, and when Mark decides to announce the Calculating Camel. If the archive is consistent enough by the last weekend of May (26th/27th), we could do an organized testing session from then (so, the 29th through the 31st). This makes sure that once all of the transitions are finalized "for now" that nothing broke between the Bionic DIF[2] and that point. 2. By the latter half of June, we have hit the Feat. Def. Freeze[3] for some packages in Main, and the transitions which started from the floodgates being opened should be calmed down. The last week of June (26th through the 28th) could be another coordinated testing period. 3. In comes July, which is really the last full month you can land features that should be included in the 18.10 release, according to the usual timing of the release cycle. We would do coordinated testing that last week as well (24th through the 26th). Any last transitions could be discussed which affect everyone and could be landed in time for Feature Freeze. 4. Feature Freeze comes in on the week of August 23rd, ish. That would put coordinated testing right after that (28th through the 30th), and makes it important for there to be testing, because we can assume that features are no longer the primary focus of the flavors for the upcoming release. If upstream release cycles for common components happen to line up in a way which makes it desirable to ship in a release, that could be discussed. Otherwise, deep testing of edge cases (unusual install methods come to mind right away) would become a more specific goal, so we can ensure that the final release is as bug-free as possible. 5. In between that and the latter end of September would be UIF[4] and the Documentation String Freeze[5]. Additionally, this is also the typical spot of Final Beta, and the goal (since we didn't release a "Beta 1") would be to release an image, but call it "18.10 Beta" (thus removing versioned betas and just declaring it to be a "beta"). The archive would freeze as normal, Beta images would be released, and the archive would enter the usual semi-frozen state where seeded packages need manual approval. From this point on, the release cycle would proceed as it always has. Thoughts? [1] https://wiki.ubuntu.com/ProposedMigration [2] https://wiki.ubuntu.com/DebianImportFreeze [3] https://wiki.ubuntu.com/FeatureDefinitionFreeze [4] https://wiki
