Re: The slip down memory lane
Toshio Kuratomi wrote: So, on the whole, I agree with you. My only question is whether we're in a transitional period or if the culture is changing so slowly that we'll never get out of this treatment of our time resources. I think it's neither. It's that the new way of working being suggested in this thread is entirely unrealistic. We just cannot at the same time focus on fixing release showstoppers AND doing active development for the next release. It is nice to have Rawhide open so early, and some of us do take advantage of it (for example, the KDE SIG imported kdepim 4.5 into Rawhide (kdepim 4.5 is going to release separately from KDE 4.5 and so it didn't make F14)), but you cannot expect everyone to do Rawhide development now; if we all did, the release would end up really sucking. In addition, the way those freezes are handled also makes fixing bugs hard. While preparing for the Alpha, stable pushes for F14 updates have now been halted for 2 or 3 weeks! In addition, the new update policies, which are also being applied to the F14 branch (IMHO way too early – before NFR, builds would still enter the pending release directly until Preview (now Beta), these days we freeze at Alpha, i.e. one milestone earlier!), also slow down bugfixes. Our culture is the way it is for a reason, it cannot ever be changed. The stricter freezes just make it a PITA to do development. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. -- Peter 01234567890123456789012345678901234567890123456789012345678901234567890123456789 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Mon, 16 Aug 2010, Peter Jones wrote: On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. How long is that window anyway? -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Mon, Aug 16, 2010 at 01:06:46PM -0500, Mike McGrath wrote: On Mon, 16 Aug 2010, Peter Jones wrote: On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. How long is that window anyway? It should be about 6 months, but from F13 branch to F14 branch it was only 5 months and one week. Two of these months were after the F13 final release. F15 is not yet scheduled afaics, so it is unknown how long the window for F15 will stay open. References: https://fedoraproject.org/wiki/Releases/13/Schedule https://fedoraproject.org/wiki/Releases/14/Schedule https://fedoraproject.org/wiki/Releases/15/Schedule Regards Till pgpxe0KULM0lR.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Mon, 16 Aug 2010, Peter Jones wrote: On 08/16/2010 02:06 PM, Mike McGrath wrote: On Mon, 16 Aug 2010, Peter Jones wrote: On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. How long is that window anyway? Depends on how you count. If we count development start to feature freeze: F12: 48 days (including july 4th) F13: 53 days (including christmas and the US thanksgiving holiday) F14: 63 days (including july 4th) Or maybe development start to alpha freeze: F12: 76 days (including july 4th) F13: 84 days (including christmas and the US thanksgiving holiday) F14: 70 days (including july 4th) Of course, some people would like to count from the previous Final Development Freeze (or even earlier) to, say, feature freeze, even though this is wildly unrealistic for many of us: F12: 105 days (including july 4th) F13: 133 days (including christmas and the US thanksgiving holiday) F14: 115 days (including july 4th) this basically assumes nobody has to do any work on the previous release after the final development freeze, which isn't really true. (I realize there are other important holidays in other countries, but I figure this is a reasonable enough sample for exemplary purposes) Actually, from computing these numbers I think the best lesson is that our schedules have been so completely volatile that it's very difficult to claim they support any reasonable conclusions. I think this is worth further discussion. If the number is towards the 48-63 days level and that's what window people are actually doing development that may be a problem because it is an extremely short time period. It's also interesting that with all the freezes, deadlines, etc we have firm explicit set dates. While active development is implicit. it might be worth it to set active deployment as an explicit time period just as another reminder to everyone about when major changes are going on vs when they aren't. -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Mon, 2010-08-16 at 14:18 -0500, Mike McGrath wrote: I think this is worth further discussion. If the number is towards the 48-63 days level and that's what window people are actually doing development that may be a problem because it is an extremely short time period. It's also interesting that with all the freezes, deadlines, etc we have firm explicit set dates. While active development is implicit. it might be worth it to set active deployment as an explicit time period just as another reminder to everyone about when major changes are going on vs when they aren't. As I mentioned briefly on IRC, I think the problem is that we're kinda stuck between two models: we're trying to move to a model where development starts when N-1 branches and finishes when N Alpha hits freeze, and from then on, there's only bugfixes to N. But I think to an extent we've partially achieved the stricter freezes which restrict development to 'until Alpha freeze', but we haven't really successfully moved all our processes and conventions so that people start development for N+1 while N is still going on. Just look at the queue of updates to go into F14 after the Alpha releases, for instance; lots of that stuff is stuff that shouldn't strictly happen under the ideal of the current model. So practically speaking, most teams are starting major development from 'N-1 release' - probably minus a week for the post-release lull when everyone takes a breather. It's very unlikely that everything can actually get done between then and Alpha freeze, so stuff is running over. We even schedule it, in some cases - GNOME 2.32 clearly isn't close to done and is still going through API changes (though that's partly complicated by going to GTK+ 3 and back again). systemd is still very early. The Python 2.7 migration isn't really complete yet. These are just examples, and I'm not suggesting anyone involved with those projects is doing anything 'wrong', just observing how things are actually currently happening. For instance, right now, according to the Ideal Plan, everyone should have started on their Big Plans for F15 in Rawhide and should be committing the really big changes from now forward. Is anyone actually at that point? If not, then we're just going to go through the same cycle for F15-F16 because people will start their work for F15 after F14 is done, realize there isn't enough time to get it done before Alpha freeze, keep working on it through Alpha freeze and Beta even, and not have time to start their big changes for F16 before F15 is nearly done...and so on ad infinitum. It may be a bit of a tough cycle to break. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Mon, Aug 16, 2010 at 02:10:16PM -0700, Adam Williamson wrote: As I mentioned briefly on IRC, I think the problem is that we're kinda stuck between two models: [snip] For instance, right now, according to the Ideal Plan, everyone should have started on their Big Plans for F15 in Rawhide and should be committing the really big changes from now forward. Is anyone actually at that point? If not, then we're just going to go through the same cycle for F15-F16 because people will start their work for F15 after F14 is done, realize there isn't enough time to get it done before Alpha freeze, keep working on it through Alpha freeze and Beta even, and not have time to start their big changes for F16 before F15 is nearly done...and so on ad infinitum. It may be a bit of a tough cycle to break. I've been trying to get python3 --with-computed-gotos working in rawhide-only. But I'm hampered by the fact that gcc with fixes for an ICE hasn't been built for F15 (and hasn't been pushed to stable for F14 so it's not inherited). dmalcolm is working on getting python-3.2 into rawhide but I don't think it's been pushed yet. So, on the whole, I agree with you. My only question is whether we're in a transitional period or if the culture is changing so slowly that we'll never get out of this treatment of our time resources. -Toshio pgpaQBQxmAxfe.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Adam Williamson said the following on 08/13/2010 05:21 PM Pacific Time: On Thu, 2010-08-12 at 16:11 -0600, Linuxguy123 wrote: On the data side, it would be very interesting to go back to each one of those slips and identify the component(s) that caused the slip and then question the individuals behind them to find out what happened. Then take that information (share it with the list ?) and see what can be concluded (as a group ?) This would be incomplete data. For instance, the bug we decided to slip Alpha for was the 'basic video driver' menu entry in the installer not working. But that's not necessarily the entire cause of the slip. If the entire release process had run on time, and we'd had really testable candidate images earlier than we did, it's entirely possible this bug would have been identified and fixed in time for the Alpha to be shipped. Just looking at the specific bugs that end up blocking the release does not give a complete picture of why the release slipped. Well put. All the incremental steps along the road are documented here for Fedora 14. https://fedoraproject.org/wiki/Fedora_14_Schedule_Retrospective Each piece adds up to the point were we just can't finish in the allotted time. Anecdotally, what happened in this release is not unusual. Each release we have these this shouldn't ever happen again items and while they have different names the reasons are almost always the same--landing changes to packages or infrastructure right on the deadline or not budgeting enough time to recover before the deadline if something goes wrong. It would be great to get more entries from other people on this wiki page. Please don't send them to the list, but put them on the wiki page. John -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Mike McGrath said the following on 08/16/2010 12:18 PM Pacific Time: On Mon, 16 Aug 2010, Peter Jones wrote: On 08/16/2010 02:06 PM, Mike McGrath wrote: On Mon, 16 Aug 2010, Peter Jones wrote: On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottinghamnott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. How long is that window anyway? Depends on how you count. If we count development start to feature freeze: F12: 48 days (including july 4th) F13: 53 days (including christmas and the US thanksgiving holiday) F14: 63 days (including july 4th) Or maybe development start to alpha freeze: F12: 76 days (including july 4th) F13: 84 days (including christmas and the US thanksgiving holiday) F14: 70 days (including july 4th) Of course, some people would like to count from the previous Final Development Freeze (or even earlier) to, say, feature freeze, even though this is wildly unrealistic for many of us: F12: 105 days (including july 4th) F13: 133 days (including christmas and the US thanksgiving holiday) F14: 115 days (including july 4th) this basically assumes nobody has to do any work on the previous release after the final development freeze, which isn't really true. (I realize there are other important holidays in other countries, but I figure this is a reasonable enough sample for exemplary purposes) Actually, from computing these numbers I think the best lesson is that our schedules have been so completely volatile that it's very difficult to claim they support any reasonable conclusions. I think this is worth further discussion. If the number is towards the 48-63 days level and that's what window people are actually doing development that may be a problem because it is an extremely short time period. It's also interesting that with all the freezes, deadlines, etc we have firm explicit set dates. While active development is implicit. it might be worth it to set active deployment as an explicit time period just as another reminder to everyone about when major changes are going on vs when they aren't. -Mike It is implicit here: https://fedoraproject.org/wiki/Releases/14/Schedule It is more explicit here: http://poelstra.fedorapeople.org/schedules/f-14/f-14-devel-tasks.html -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Peter Jones said the following on 08/16/2010 11:50 AM Pacific Time: On 08/16/2010 02:06 PM, Mike McGrath wrote: On Mon, 16 Aug 2010, Peter Jones wrote: On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottinghamnott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? The window for changes is already far too short. How long is that window anyway? Depends on how you count. If we count development start to feature freeze: F12: 48 days (including july 4th) F13: 53 days (including christmas and the US thanksgiving holiday) F14: 63 days (including july 4th) Or maybe development start to alpha freeze: F12: 76 days (including july 4th) F13: 84 days (including christmas and the US thanksgiving holiday) F14: 70 days (including july 4th) Of course, some people would like to count from the previous Final Development Freeze (or even earlier) to, say, feature freeze, even though this is wildly unrealistic for many of us: F12: 105 days (including july 4th) F13: 133 days (including christmas and the US thanksgiving holiday) F14: 115 days (including july 4th) this basically assumes nobody has to do any work on the previous release after the final development freeze, which isn't really true. (I realize there are other important holidays in other countries, but I figure this is a reasonable enough sample for exemplary purposes) Actually, from computing these numbers I think the best lesson is that our schedules have been so completely volatile that it's very difficult to claim they support any reasonable conclusions. I agree. Here's the historical data I've tracked: http://poelstra.fedorapeople.org/schedules/f-14/f9-f14-schedule-analysis-v4.ods Because we are date-based and it has historically been that assumed people love Fedora so much they don't stop working on it over holidays, we've never considered the impact of holidays. Our schedules have changed a lot up until Fedora 13 and 14. This was the first time we did not try something new (after feature freeze) with the schedule. I agree it will take a few release cycles to figure out what is working and what isn't. This was my primary reason for arguing it was time to stop trying something new each release with the schedule. Because of our current scheduling methodology the development length varies between releases, but for Fedora 13 and 14 the freezes and testing durations are set the same (except for the week slip of the Fedora 13 Alpha that we absorbed which I believe contributed to slipping the Beta and Final). http://fedoraproject.org/wiki/Releases/Schedule (scheduling methodology) John -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 13:39 -0500, Mike McGrath wrote: Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? Personally, I'd like to see a 9-12 month cycle. Much as I'm sure everyone out there instantly upgrades every system the moment a new release comes out, and doesn't mind a year of overall updates, something - just something - suggests to me that this might, not, quite be true...I'd love it if we'd actually ask our users what they want. Jon. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thursday, August 12, 2010 09:33:17 pm Lennart Poettering wrote: On Thu, 12.08.10 13:19, Mike McGrath (mmcgr...@redhat.com) wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. While I side with mclasen here and believe that it is a strength of Fedora that we take the liberty to let cycles slip rather then compromise quality, I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? Agreed. On both. Slips are not problems (if it's not 6 months slip :D). Slips happens and are regular solution. But for core system freeze - it's actually THE MUST! For us, booting system with working Xorg is the point where we can start working on our packages and testing!!! Jaroslav Lennart -- Jaroslav Řezník jrez...@redhat.com Software Engineer - Base Operating Systems Brno Office: +420 532 294 275 Mobile: +420 602 797 774 Red Hat, Inc. http://cz.redhat.com/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Fri, 13 Aug 2010, Jaroslav Reznik wrote: On Thursday, August 12, 2010 09:33:17 pm Lennart Poettering wrote: On Thu, 12.08.10 13:19, Mike McGrath (mmcgr...@redhat.com) wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. While I side with mclasen here and believe that it is a strength of Fedora that we take the liberty to let cycles slip rather then compromise quality, I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? Agreed. On both. Slips are not problems (if it's not 6 months slip :D). Slips happens and are regular solution. But for core system freeze - it's actually THE MUST! For us, booting system with working Xorg is the point where we can start working on our packages and testing!!! I'll admit, this is a convenient view to have. The problem is we're not in high school anymore. We're professionals. We're expected to set and keep schedules because people besides ourselves rely on those schedules. There are other distros that set and keep schedules better then we do.. probably all of them. I'm just saying with proper planning it's possible. -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 9:34 PM, Nicolas Mailhot nicolas.mail...@laposte.net wrote: Le jeudi 12 août 2010 à 13:51 -0500, Jason L Tibbitts III a écrit : I guess I'm just saying that, if we had the developer time to do it, it would be super nice if we could get the pre-F15 rawhide is useless bit over and done with by the time F15 branches. But back in reality, I know that's a tough thing to ask for. Actually, rawhide is not useless and instable most of the year. Ironically, it is most instable at branch time when people wake up and try to cram as many new features as possible before freeze date (sadly, too few start their invasive changes after branch time as they should). So perhaps the delay between invasive features autorized and alpha is too short. Another big cause of pre-alpha instability is people who let packages rot in rawhide (because it is socially accepted to say rawhide eats babies, so why bother), and try to fix them at the last minute and branch point when it is way too late to sanely test the changes. Very well worded. Looking at the daily rawhide report makes me think that some maintainers need to be educated to fix/rebuild stuff faster. Since i'm curious, i will collect a list over the next weeks to see who takes a long time there. Maybe that's good to find out if some people need help (for whatever reason). If it's really the late new features that causes slips, we could move the feature freeze to an earlier point (two weeks maybe?). Having FESCo to reject features not ready at this point with no exception could help as well. Or maybe with exceptions if the risk of breakage/slip is low. Probably together with having an eye on it. But the latter is most likely a problem due to not enough manpower. Critpath feature freeze should be anyways a lot earlier than the rest of the packages/system. -- LG Thomas Dubium sapientiae initium -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Mike McGrath wrote: I'll admit, this is a convenient view to have. The problem is we're not in high school anymore. We're professionals. We're expected to set and keep schedules because people besides ourselves rely on those schedules. There are other distros that set and keep schedules better then we do.. probably all of them. I'm just saying with proper planning it's possible. Huh? Name a distro which never has 1-2 week slips. Even Ubuntu with all its reliable schedules talk sometimes slips. The reason you don't notice is that they schedule for early in the month, so when they slip, it's still the same month and their y.mm versioning scheme still works. But one LTS release (Dapper Drake in 2006) has been made a .06 release rather than .04, that's 2 months added to a 6 months schedule, and that was not the original schedule! So in some sense it was a 2-month slip! And Debian even routinely slips for months. As for RHEL, RH keeps its schedules secret until the very last moment, and rumors are the original schedule for RHEL 6 was already not met and it's still not out (but since I don't work for RH, I can't attest to the truthfulness of those rumors, and I guess those who theoretically could aren't allowed to comment on it). You have to choose between timeliness or quality. I'll take quality any day (as long as it doesn't get ridiculous like Debian's ages-long slips), thank you very much! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Fri, 13 Aug 2010, Kevin Kofler wrote: Mike McGrath wrote: I'll admit, this is a convenient view to have. The problem is we're not in high school anymore. We're professionals. We're expected to set and keep schedules because people besides ourselves rely on those schedules. There are other distros that set and keep schedules better then we do.. probably all of them. I'm just saying with proper planning it's possible. Huh? Name a distro which never has 1-2 week slips. Even Ubuntu with all its reliable schedules talk sometimes slips. The reason you don't notice is that they schedule for early in the month, so when they slip, it's still the same month and their y.mm versioning scheme still works. But one LTS release (Dapper Drake in 2006) has been made a .06 release rather than .04, that's 2 months added to a 6 months schedule, and that was not the original schedule! So in some sense it was a 2-month slip! And Debian even routinely slips for months. As for RHEL, RH keeps its schedules secret until the very last moment, and rumors are the original schedule for RHEL 6 was already not met and it's still not out (but since I don't work for RH, I can't attest to the truthfulness of those rumors, and I guess those who theoretically could aren't allowed to comment on it). You have to choose between timeliness or quality. I'll take quality any day (as long as it doesn't get ridiculous like Debian's ages-long slips), thank you very much! :( I'm saddened you think so little of us Kevin. I'd have thought we could do both. -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Just an idea, without _fully_ understanding the infrastructure, man power etc ramifications. Since the move to git, would it not be easier to allow features to branch rawhide, get their individual bits together (syncing with 'trunk' periodically)... Then like the kernel does, merge back the working bits to rawhide for the alpha. Which would essentially then be making sure that the features that were merged in play nice together? Each approved feature gets a branch of what would be rawhide, they can do their thing insulated from others till they are ready, like the python stuff, having all the packages built aside till they are all ready, and then being merged into rawhide when they are ready and tested... Just a thought. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Mike McGrath wrote: :( I'm saddened you think so little of us Kevin. I'd have thought we could do both. And you think Santa Claus exists, too? ;-) Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/13/2010 09:22 PM, Kevin Kofler wrote: Mike McGrath wrote: :( I'm saddened you think so little of us Kevin. I'd have thought we could do both. And you think Santa Claus exists, too? ;-) No, his goal is certainly achievable and worth trying. The frequent slips are a problem and we should fix them without compromising on the quality. Other projects are sometimes doing things better and we can always learn. Rawhide is often more broken than development branch of other distros for example. We can improve even if you don't believe in it. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 22:22:18 -0700, Jesse Keating jkeat...@redhat.com wrote: How do you suggest we be more conservative? If you expect the developers to do this on their own, good luck. If you want there to be some sort of enforcement I welcome suggestions. My suggestion would be to ask developers not to move stuff from testing to stable unless it was a significant bug fix update, during that period. How much effect just asking would have, I don't know. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Friday, August 13, 2010, 11:52:33 AM, Kevin wrote: Mike McGrath wrote: :( I'm saddened you think so little of us Kevin. I'd have thought we could do both. And you think Santa Claus exists, too? ;-) Kevin Kofler http://www.snopes.com/holidays/christmas/photos/badsanta.asp -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Fri, Aug 13, 2010 at 09:49:42 -0600, Nathanael D. Noblet nathan...@gnat.ca wrote: Since the move to git, would it not be easier to allow features to branch rawhide, get their individual bits together (syncing with 'trunk' periodically)... Then like the kernel does, merge back the working bits to rawhide for the alpha. Which would essentially then be making sure that the features that were merged in play nice together? With no frozen rawhide and early branching, I don't think this is really necessary, you can do development in rawhide and go back to the branch when ready. Most features are fairly independent and don't cause problems when they run late or have problems, outside of that feature. Some are somewhat disruptive and can make it hard to test other things while they are having their kinks worked out or just waiting for rebuilds of dependencies to be completed. Those can cause a problem if they happen too close to a freeze since they result in work needing to be done in a very short time frame. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Fri, 13 Aug 2010, Bruno Wolff III wrote: On Fri, Aug 13, 2010 at 09:49:42 -0600, Nathanael D. Noblet nathan...@gnat.ca wrote: Since the move to git, would it not be easier to allow features to branch rawhide, get their individual bits together (syncing with 'trunk' periodically)... Then like the kernel does, merge back the working bits to rawhide for the alpha. Which would essentially then be making sure that the features that were merged in play nice together? With no frozen rawhide and early branching, I don't think this is really necessary, you can do development in rawhide and go back to the branch when ready. Do we know for sure that people aren't treating the branch like rawhide? -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Nathanael D. Noblet wrote: Just an idea, without _fully_ understanding the infrastructure, man power etc ramifications. Since the move to git, would it not be easier to allow features to branch rawhide, get their individual bits together (syncing with 'trunk' periodically)... Then like the kernel does, merge back the working bits to rawhide for the alpha. Which would essentially then be making sure that the features that were merged in play nice together? Each approved feature gets a branch of what would be rawhide, they can do their thing insulated from others till they are ready, like the python stuff, having all the packages built aside till they are all ready, and then being merged into rawhide when they are ready and tested... The problem is, if they're more than one change requiring a partial or total mass rebuild (e.g. Python and Boost which both required many rebuilds for dependencies), the sets of packages to rebuild often intersect (and for Python and Boost, they did, in fact even Boost itself needed to be rebuilt for Python), complicating that process. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Bruno Wolff III wrote: Most features are fairly independent and don't cause problems when they run late or have problems, outside of that feature. Some are somewhat disruptive and can make it hard to test other things while they are having their kinks worked out or just waiting for rebuilds of dependencies to be completed. Those can cause a problem if they happen too close to a freeze since they result in work needing to be done in a very short time frame. The problem is that those are the very features that are hard to stage, because the sets of packages to rebuild often intersect. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Fri, Aug 13, 2010 at 19:07:57 +0200, Kevin Kofler kevin.kof...@chello.at wrote: Bruno Wolff III wrote: Most features are fairly independent and don't cause problems when they run late or have problems, outside of that feature. Some are somewhat disruptive and can make it hard to test other things while they are having their kinks worked out or just waiting for rebuilds of dependencies to be completed. Those can cause a problem if they happen too close to a freeze since they result in work needing to be done in a very short time frame. The problem is that those are the very features that are hard to stage, because the sets of packages to rebuild often intersect. I agree. My wish is that these be done a bit earlier so that they don't cause problems when other packagers (and people working on spins) are up against deadlines. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 16:11 -0600, Linuxguy123 wrote: On the data side, it would be very interesting to go back to each one of those slips and identify the component(s) that caused the slip and then question the individuals behind them to find out what happened. Then take that information (share it with the list ?) and see what can be concluded (as a group ?) This would be incomplete data. For instance, the bug we decided to slip Alpha for was the 'basic video driver' menu entry in the installer not working. But that's not necessarily the entire cause of the slip. If the entire release process had run on time, and we'd had really testable candidate images earlier than we did, it's entirely possible this bug would have been identified and fixed in time for the Alpha to be shipped. Just looking at the specific bugs that end up blocking the release does not give a complete picture of why the release slipped. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
The slip down memory lane
Oct 6 2006: Fedora Core 6 release date slip - http://lists.fedoraproject.org/pipermail/announce/2006-October/002243.html Oct 16 2006: Another slip in the FC6 schedule - http://lists.fedoraproject.org/pipermail/announce/2006-October/002248.html Jul 11 2006: FC6 test2 freeze slipping by a week - http://lists.fedoraproject.org/pipermail/announce/2006-July/002215.html Jul 19 2006: FC6 Test2 Freeze Slip - http://lists.fedoraproject.org/pipermail/announce/2006-July/002219.html Jul 31 2007: Fedora 8 Test 1 slipping - http://lists.fedoraproject.org/pipermail/devel-announce/2007-July/27.html Sep 4 2007: Fedora 8 Test 2 slipping - http://lists.fedoraproject.org/pipermail/devel-announce/2007-September/46.html Jan 14 2008: Slip of Alpha - http://lists.fedoraproject.org/pipermail/devel-announce/2008-January/000125.html Mar 3 2008: Beta freeze/release slipping by a week - http://lists.fedoraproject.org/pipermail/devel-announce/2008-March/000159.html Mar 20 2008: Fedora 9 Beta slipped a few days - http://lists.fedoraproject.org/pipermail/announce/2008-March/002426.html Apr 17 2009: Fedora 9 Release date slipping by two weeks (new date, May 13) - http://lists.fedoraproject.org/pipermail/announce/2008-April/002445.html Feb 3 2009: Fedora 11 Alpha slip - http://lists.fedoraproject.org/pipermail/announce/2009-February/002605.html Mar 19 2009: Fedora 11 Beta slip - http://lists.fedoraproject.org/pipermail/devel-announce/2009-March/000393.html May 19 2009: One week slip of Fedora 11 Release - http://lists.fedoraproject.org/pipermail/announce/2009-May/002648.html May 28 2009: One (more) week slip of Fedora 11 Release - http://lists.fedoraproject.org/pipermail/announce/2009-May/002652.html Aug 10 2010: Slip of Fedora 12 Alpha by one week - http://lists.fedoraproject.org/pipermail/devel-announce/2009-August/000480.html Feb 25 2010: Fedora 13 Alpha slip by one week - http://lists.fedoraproject.org/pipermail/announce/2010-February/002772.html Apr 1 2010: Fedora 13 Beta Slip 1 week - http://lists.fedoraproject.org/pipermail/devel/2010-April/134324.html May 12 2010: One week slip of Fedora 13 Release - http://lists.fedoraproject.org/pipermail/announce/2010-May/002806.html Aug 12 2010: One week slip of Fedora 14 schedule - http://lists.fedoraproject.org/pipermail/announce/2010-August/002849.html Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. Thoughts? -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. - J -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 2:19 PM, Mike McGrath mmcgr...@redhat.com wrote: ... snip ... Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. Actually, I don't think that the slips in the releases have _accumulated_ to be 'half' of a full release cycle' because aren't the target dates always at the same spot on the calendar? When it comes down to it... the very first slip caused a delay, and that delay may have propagated itself into the future by delaying the start of each subsequent release. (But even that isn't true because Rawhide keeps progressing even during a freeze.) In the end... we still have two releases a year, its just that each release takes 7 months to do. I applaud the Fedora release team for meeting their schedules as closely as they do! Fulko -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 8:19 PM, Mike McGrath mmcgr...@redhat.com wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It isn't broken so there is nothing to fix; slipping to fix issues found is a feature not a bug. We don't have any reason to rush. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? It's hard to test a moving target. Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? -Mike [1] Just picked some number slightly longer then the current cycle for purposes of discussion, not suggesting it. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 12/08/10 19:19, Mike McGrath wrote: snip How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. snip I don't think it's any failure, just that more ppl are finding problems across a greater variety of both hard\virtual-ware. Which could be inverted as it shows an increase in the continued interest in the development of Fedora. If it just dates is the worry, start testing for seven instead of six. -- Regards, Frank Murphy UTF_8 Encoded Friend of Fedora -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 12/08/10 19:19, Mike McGrath wrote: snip How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. snip I don't think it's any failure, just that more ppl are finding problems across a greater variety of both hard\virtual-ware. Which could be inverted as it shows an increase in the continued interest in the development of Fedora. If it just dates is the worry, start testing for seven instead of six. -- Regards, Frank Murphy UTF_8 Encoded Friend of Fedora -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 12 Aug 2010, Bill Nottingham wrote: Matthias Clasen (mcla...@redhat.com) said: This is a collective failure. I'd like to question that premise. Why is it a failure if we adjust our release schedule to meet our release criteria ? Well, ideally we'd be able to schedule such that we can accomplish our release criteria within the defined schedule without having to slip. I can't help but note that the slips have become more frequent as we started to actually *have* release criteria to test against. We didn't slip nearly as much when we weren't testing it. (Whether that's a good or bad thing is left as an exercise for the reader.) Any of the QA guys have any way to measure the the most common cause of our slips? Is it usually stuff we're our own upstream for? Is it integration? Is it bugs that were introduced months ago but only recently found or bugs that were just introduces in the couple of weeks before release? -Mike -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 02:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottingham nott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? It's hard to test a moving target. Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? I'm not sure that an increased cycle would really help. One thing I am curious about is why, when slipping for an Alpha target, the whole schedule slips. Can't we just take a week out of the Beta cycle? The amount of testing time is roughly the same. The other general thing is to utilize http://repos.fedorapeople.org to get wider testing for new features before merging them. Perhaps we could have a 6mo release schedule but a 12mo feature schedule. Thus, I would propose features *now* for F15, get them conditionally accepted and work on them in repos.fp.org (or rawhide if that is not possible). Then, we fork F15 from F14+updates and only merge in features that have proven stable enough for wider testing. The only way to make schedules more predictable is to keep trunk from breaking as much as possible. Breakage should occur as much as possible in private repos. Just some thoughts, hope they're helpful. Nathaniel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
MM == Mike McGrath mmcgr...@redhat.com writes: MM Possibly also stop changing earlier? Not necessarily. We should certainly try to get the earth shattering changes done as early as possible (i.e. soon after branch) but I recognize that there isn't sufficient developer time available to both stabilize one release and push all of the new stuff through rawhide at the same time. MM It's hard to test a moving target. Well, you test what you have at the time. That may not be what you could test tomorrow, but the testing is still equally valid. MM Would an 8[1] month cycle cause fewer slips per release? Fewer MM bugs? Well, don't forget that since we aren't freezing rawhide, we essentially have that long now. F15 branched, what, a few weeks ago and isn't due to be out until six months after whenever F14 ends up coming out. I guess I'm just saying that, if we had the developer time to do it, it would be super nice if we could get the pre-F15 rawhide is useless bit over and done with by the time F15 branches. But back in reality, I know that's a tough thing to ask for. - J -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 14:50:38 -0400, Nathaniel McCallum nathan...@natemccallum.com wrote: One thing I am curious about is why, when slipping for an Alpha target, the whole schedule slips. Can't we just take a week out of the Beta cycle? The amount of testing time is roughly the same. We've tried that in the past and it didn't work. Slipping the whole schedule right away is better than slipping piecewise when it comes to planning. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 13:50 -0500, Mike McGrath wrote: Any of the QA guys have any way to measure the the most common cause of our slips? Is it usually stuff we're our own upstream for? Is it integration? Is it bugs that were introduced months ago but only recently found or bugs that were just introduces in the couple of weeks before release? Not a super convenient way, but you can do it by going back and looking at previous blocker meetings and go/no-go meetings. I've done this once when there was some debate about how early the blocker issues were identified. Most blocker bugs are in anaconda. This is simply because it's the component most *likely* to have blocker bugs, because of the function it performs. We usually catch most initial blockers for any given release at the first TC stage. Bugs we slip for are usually ones identified at that stage that we couldn't fix in time, bugs introduced between TC and RC by non-critical changes to critical components (this is something that could bear to be tightened down), and bugs introduced or exposed by other blocker fixes. A common case is, say, we identify a bug in TC#1 that completely breaks FTP install; then the anaconda team fixes that for RC#1, but we discover that the same FTP install path is broken at a later point. We couldn't have found that at TC#1 time, because it's hidden by the earlier breakage. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 02:39 PM, drago01 wrote: On Thu, Aug 12, 2010 at 8:19 PM, Mike McGrath mmcgr...@redhat.com wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It isn't broken so there is nothing to fix; slipping to fix issues found is a feature not a bug. We don't have any reason to rush. I disagree, the feature is shipping on time. Shipping on time enables others in the Fedora community (people who build on, deploy, etc) know with some assurance what their schedules will look like. If I were a project manager looking at using a Linux OS in my project, a demonstrated lack of ability to ship on time is a *huge* mark against using that OS. If our schedules aren't reasonably fixed, than others have a hard time working with us. Loosing users (especially companies with resources to invest in Fedora, even if just testing) make our quality go down. Thus, in the long run, continual slips actually contribute to lack of quality. I think my point from my previous email is worth repeating: we need wider testing of new features outside rawhide/N+1 before they are merged. This avoids upheaval and we can find bugs earlier. Nathaniel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 13:19:29 -0500, Mike McGrath mmcgr...@redhat.com wrote: Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. Thoughts? One thing I have noticed is people landing big changes (such as python and systemd) that break things for a while, delay a lot of other testing. So that when the bigger changes get fixed up, other bugs get unhidden with little time to react. I'd like to see the big changes land a lot earlier, maybe a month before the branch, so that by the branch most things should be easily testable. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 12:00:29 -0700, Adam Williamson awill...@redhat.com wrote: We usually catch most initial blockers for any given release at the first TC stage. Bugs we slip for are usually ones identified at that stage that we couldn't fix in time, bugs introduced between TC and RC by This is another place we could change things. We could build the TC a bit earlier (say a week) and be more conservative about what changes go in until the final RC is approved. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 01:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottinghamnott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? It's hard to test a moving target. Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? -Mike [1] Just picked some number slightly longer then the current cycle for purposes of discussion, not suggesting it. I think that will turn into 10 quickly. I advocate rigorous testing, and sticking as close to 6 as we can. I mean, if we have to slip because of a nasty blocker, yeah, slip, of course. But I don't see how a slip decreases the user experience. Quite the opposite. Plus, I love the comment that was made, about always doing 2 releases a year, and that they each take 7 months. That makes my brain giggle. :) And the thing is, it's not wrong. :) -J -- - in your fear, speak only peace in your fear, seek only love -d. bowie -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 01:51 PM, Jason L Tibbitts III wrote: MM == Mike McGrathmmcgr...@redhat.com writes: MM Possibly also stop changing earlier? Not necessarily. We should certainly try to get the earth shattering changes done as early as possible (i.e. soon after branch) but I recognize that there isn't sufficient developer time available to both stabilize one release and push all of the new stuff through rawhide at the same time. MM It's hard to test a moving target. Well, you test what you have at the time. That may not be what you could test tomorrow, but the testing is still equally valid. MM Would an 8[1] month cycle cause fewer slips per release? Fewer MM bugs? Well, don't forget that since we aren't freezing rawhide, we essentially have that long now. F15 branched, what, a few weeks ago and isn't due to be out until six months after whenever F14 ends up coming out. YES. And this is where the kitten killing happens*, and I think it makes F14 that much stronger, because we have that much longer to muck about with each rawhide, and longer to smoketest F14 And it rocks. -J I guess I'm just saying that, if we had the developer time to do it, it would be super nice if we could get the pre-F15 rawhide is useless bit over and done with by the time F15 branches. But back in reality, I know that's a tough thing to ask for. - J *As God Intended -- - in your fear, speak only peace in your fear, seek only love -d. bowie -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 03:03 PM, Bruno Wolff III wrote: On Thu, Aug 12, 2010 at 13:19:29 -0500, Mike McGrath mmcgr...@redhat.com wrote: Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. Thoughts? One thing I have noticed is people landing big changes (such as python and systemd) that break things for a while, delay a lot of other testing. So that when the bigger changes get fixed up, other bugs get unhidden with little time to react. I'd like to see the big changes land a lot earlier, maybe a month before the branch, so that by the branch most things should be easily testable. +1 Perhaps our feature proposals need a better risk assessment. ie. Will this change create system-wide impact? Will reverting it be difficult? If the answer is yes to either of those questions, we should require (either): 1. Testing in an external repo until a some stability is demonstrated. 2. Early merge with an early risk assessment / rollback. Nathaniel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 03:08 PM, Jon Ciesla wrote: On 08/12/2010 01:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottinghamnott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? It's hard to test a moving target. Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? -Mike [1] Just picked some number slightly longer then the current cycle for purposes of discussion, not suggesting it. I think that will turn into 10 quickly. I advocate rigorous testing, and sticking as close to 6 as we can. I mean, if we have to slip because of a nasty blocker, yeah, slip, of course. But I don't see how a slip decreases the user experience. Quite the opposite. If slips are occasional, than this is correct. If slips are so routine that schedules become unpredictable, than you shift the schedules of everyone who builds something on top of Fedora. Doing this too much results in decreased technical users who provide development and testing. The end result is decreased quality. In short, predictable, regular releases increase quality. Occasional slips happen, regular slips should not. Nathaniel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 02:14 PM, Nathaniel McCallum wrote: On 08/12/2010 03:08 PM, Jon Ciesla wrote: On 08/12/2010 01:39 PM, Mike McGrath wrote: On Thu, 12 Aug 2010, Jason L Tibbitts III wrote: BN == Bill Nottinghamnott...@redhat.com writes: BN I can't help but note that the slips have become more frequent as we BN started to actually *have* release criteria to test against. We BN didn't slip nearly as much when we weren't testing it. To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. Possibly also stop changing earlier? It's hard to test a moving target. Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? -Mike [1] Just picked some number slightly longer then the current cycle for purposes of discussion, not suggesting it. I think that will turn into 10 quickly. I advocate rigorous testing, and sticking as close to 6 as we can. I mean, if we have to slip because of a nasty blocker, yeah, slip, of course. But I don't see how a slip decreases the user experience. Quite the opposite. If slips are occasional, than this is correct. If slips are so routine that schedules become unpredictable, than you shift the schedules of everyone who builds something on top of Fedora. Doing this too much results in decreased technical users who provide development and testing. The end result is decreased quality. I disagree that a clockwork release schedule is required for quality, or even perceived quality. If that's the sort of metric being looked at, the user is probably best suited to RHEL, CentOS, etc. On the other hand, I emphatically agree that you can't do it too much or too often. I just think that the slips we've had have for the most part been minor enough to be warranted by the huge gains they bought in terms of stability, usability, and other ilities. In short, predictable, regular releases increase quality. Occasional slips happen, regular slips should not. Nathaniel -- - in your fear, speak only peace in your fear, seek only love -d. bowie -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Jon Ciesla (l...@jcomserv.net) said: I disagree that a clockwork release schedule is required for quality, or even perceived quality. If that's the sort of metric being looked at, the user is probably best suited to RHEL, CentOS, etc. It would be interesting to look at RHEL/CentOS to see how predictable their release schedule actually is. I'd wager it's not that predictable either. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 21:33 +0200, Lennart Poettering wrote: On Thu, 12.08.10 13:19, Mike McGrath (mmcgr...@redhat.com) wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. While I side with mclasen here and believe that it is a strength of Fedora that we take the liberty to let cycles slip rather then compromise quality, I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? It's worth pointing out that it's actually quite rare for blocker bugs to be in those components. Kernel more than any of the others, but that is almost always down to the bits of graphics driver that live in the kernel these days. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, Aug 12, 2010 at 2:19 PM, Mike McGrath mmcgr...@redhat.com wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? [snip] This is a collective failure. While I agree that we've had a lot of schedule slips, and it's not ideal to have a slip in the schedule, I don't agree that a schedule slip is necessarily a failure *per se*. In the case of the slip in the Alpha, I'd even go so far as to say that we're doing something right -- The RC did not meet its release criteria, and so we did what we said we were going to do. Testing found problems, and we blocked on those problems. A lot of different individuals have put a lot of time and effort into getting the Alpha ready, and to say that the result of all their work is a failure because we slipped the schedule a week is a bit short-sighted in my opinion. To me, the important thing here is that we learn from the experience, and try to make things better. The fact that we've got a (admittedly basic and somewhat manual) test process in place shows that we really do care about the quality of the distribution we ship. So, the question comes down to this -- how do we learn from the process, and make the next release smoother than the last? I have three suggestions. First of all, take a look at John Poelstra's F14 retrospective page at https://fedoraproject.org/wiki/Fedora_14_Schedule_Retrospective. John has actively been trying to document the lessons we're (hopefully?) learning from our mistakes, so that we can improve next time. If we're not learning from our past mistakes, we're not moving forward. I'm sure John would appreciate our help in documenting the reasons we slip. My second suggestion is for FESCo to take a more active role in tracking the major changes that land in the distribution and judging the impact that they might have before freezes. While the major Python and systemd changes didn't end up blocking our release, I'm sure they had an impact on our ability to build test composes, and also our ability to thoroughly test the RCs before the go/no-go meeting. Third, let's all pitch in and help the QA team with some of their automated testing, so that we can more easily test RCs and know what shape they're in. We simply don't have the resources to do everything manually. Also, let me be clear. I'm not treating the six-month release cycle as sacred or immutable. I'm willing to work with the Board and FESCo to determinte wether or not the length of the cycle should be changed. I'm not convinced, however, that simply lengthening the cycle will solve the problem, unless we find better ways of making things happen before the last minute. For better or worse, deadlines make work happen, and from time to time deadlines get broken. Obviously, these things can and need to be discussed in a measured and appropriate way. Let me also point out that there has to be a healthy balance between the objective measures (X number of blocker bugs still open, Y number of tests failing) and the subjective measures (It *feels* like it's ready for an Alpha release), and I think the current go/no-go meeting does a fairly good job of finding that balance. In short, the process isn't perfect and has room for improvement... but following our process shouldn't be viewed as a failure either. -- Jared Smith Fedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 21:33 +0200, Lennart Poettering wrote: I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? This is a good point, and it's one of the reasons the 'critpath' stuff exists. It's the same concept, applied somewhat differently: rather than freeze the 'CoreOS' stuff earlier, we freeze it harder - we require more testing for those pieces. -w -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 13:39 -0500, Mike McGrath wrote: Would an 8[1] month cycle cause fewer slips per release? Fewer bugs? For me, one of the guiding principles for Fedora QA's work on tools and policies has been this: time, by itself, doesn't fix anything. Making the schedules longer isn't going to stop people trying to cram fixes/changes in at the last minute. Stronger policies about what's allowed after a freeze - or just longer freezes in general - might be more effective at stabilizing things post-freeze, so we don't slip so much. Developing better tools and processes to find bugs *before* deadlines is more appealing than longer freezes, at least to me - but either of those is a better idea than lengthening the release schedule. If we really want to do releases on time without compromising quality, I think we need to work on three things: 1) Get new code into rawhide, and testable, as soon as possible. - This means running rawhide though, and that's not always easy.. 2) Be more aggressive about deferring features which are incomplete. - This isn't such a huge penalty anymore, since it's getting a lot easier to keep a personal repo for your new changes. 3) Work on tools to speed up the bug lifecycle. - automated testing to catch regressions - better debugging tools / docs I mean, honestly - we started accepting rawhide/f14 builds six months ago. Surely some of this work could have been tested earlier, and the stuff that wasn't available to test earlier.. should maybe wait until next release? -w -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Le jeudi 12 août 2010 à 13:51 -0500, Jason L Tibbitts III a écrit : I guess I'm just saying that, if we had the developer time to do it, it would be super nice if we could get the pre-F15 rawhide is useless bit over and done with by the time F15 branches. But back in reality, I know that's a tough thing to ask for. Actually, rawhide is not useless and instable most of the year. Ironically, it is most instable at branch time when people wake up and try to cram as many new features as possible before freeze date (sadly, too few start their invasive changes after branch time as they should). So perhaps the delay between invasive features autorized and alpha is too short. Another big cause of pre-alpha instability is people who let packages rot in rawhide (because it is socially accepted to say rawhide eats babies, so why bother), and try to fix them at the last minute and branch point when it is way too late to sanely test the changes. Here I feel the only remedy is to tell packagers: when we look outwares, users-side, we affirm rawhide is dangerous, when we look inwards, packager-side, we treat rawhide problem problem reports the same way as any stable bug. -- Nicolas Mailhot -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 12.08.10 13:19, Mike McGrath (mmcgr...@redhat.com) wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. While I side with mclasen here and believe that it is a strength of Fedora that we take the liberty to let cycles slip rather then compromise quality, I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? Lennart -- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On 08/12/2010 02:22 PM, Bill Nottingham wrote: Jon Ciesla (l...@jcomserv.net) said: I disagree that a clockwork release schedule is required for quality, or even perceived quality. If that's the sort of metric being looked at, the user is probably best suited to RHEL, CentOS, etc. It would be interesting to look at RHEL/CentOS to see how predictable their release schedule actually is. I'd wager it's not that predictable either. Bill Good point, I seem to remember talk at one point about RHEL6 being Q2 2010. Not sure where I heard that, though, so someone with better information should weigh in. -J -- - in your fear, speak only peace in your fear, seek only love -d. bowie -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 15:02 -0400, Nathaniel McCallum wrote: If our schedules aren't reasonably fixed, than others have a hard time working with us. Loosing users (especially companies with resources to They are reasonably fixed. Please don't blow this out of proportion. I don't believe we've slipped more than two weeks for any of at least the last three releases. Two weeks on a six month cycle really is not a terrible slip. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
I'll reply here but I'm also bringing together some things in the rest of the thread... sorry about that. On Thu, Aug 12, 2010 at 01:19:29PM -0500, Mike McGrath wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. I think there's several ways to look at this: 0) Acknowledge that slips are going to happen and worry about other things. In the end, no one can beat Murphy. 1) Anticipate that slips are going to happen. Alternatives to work with this but still give ourselves a better chance of hitting an overall schedule: 1.1) If we think that any given release is going to slip by one month, then we should add a month between the time we compose the final bits and the availability of the release. Options for what we do if we don't use all the slip time: 1.1.1) As slips happen, we can eat into this time. The object of the game here is *to release early*. ie: we want to try not to eat into our one month window to slip but if we do, then we've already planned for it. 1.1.2) As slips happen, we can eat into this time. But if we get the final release done ahead of schedule we delay the release until our pre-announced release date. Option 1.2) Build the slip time into the milestones. Say we anticipate we could slip by a month. Add one week between the compose of the Alpha milestone and the release of Alpha, two weeks between the compose of the Beta and the release of the beta, and two weeks between the final compose and the release. Slips can eat into those weeks without impacting the next milestone. Slips that take up more than the allotted time for that milestone slip the whole release schedule. 2) Decide that we know better than Murphy and we can build a product without slips: 2.1) Have releng put a lock on the packageset earlier and more rigourously. For instance, move the Alpha change deadline where releng stops pushing packages unless they know what it will affect back to coincide with Feature Freeze. Move the FInal Change Deadline a week closer to the Beta release. Etc. Three notes: 1) I'm not a big fan of #2. Trying to cheat Murphy is a losing proposition. Working with Murphy to remain flexible is a much better idea. 2) Option #1 does not specify an exact time frame. We could get this by adding extra time to the release cycle. We could also get it by taking the slip time away from the other portions of the release. (ie: take a week away from development during the alpha nad a week away from development during the beta and use those as a two week slip time for the final release). 3) This comes at a cost. The cost is that the bits that we end up shipping won't be as fresh as they are now. We'd be taking time that previously we'd be able to spend introducing new features and instead pushing the time into stabilizing the release. Upstream tracking may or may not continue in rawhide (seeing as we have no frozen rawhide *but* many maintainers are not using rawhide to actively develop for the next Fedora until after the final release of the current Fedora in development). -Toshio pgpISoWWimyon.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
On Thu, 2010-08-12 at 13:19 -0500, Mike McGrath wrote: How can we fix this? Step 1 is to realize/admit there is a problem. You've tactfully done that. Step 2 is to gather data and knowledge. That doesn't appear to be happening in these posts. On the data side, it would be very interesting to go back to each one of those slips and identify the component(s) that caused the slip and then question the individuals behind them to find out what happened. Then take that information (share it with the list ?) and see what can be concluded (as a group ?) On the knowledge side, I highly recommend Code Complete and Writing Solid Code. Some of the replies in this thread demonstrate a lack of basic understanding of the issues at hand. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Bruno Wolff III wrote: We've tried that in the past and it didn't work. Slipping the whole schedule right away is better than slipping piecewise when it comes to planning. Huh? What's the worst that can happen? That we slip again, being at the same release date as with the cascading slip system. Whereas with the cascading slips, we risk accumulating ANOTHER slip on top of the one that was already there. So I think the practice of cascading slips is actually detrimental to the timeliness of our releases. We shouldn't cascade slips by default. If we then end up having to re-slip the next milestone, then well, that was kinda expected, so it shouldn't be counted as a failure. But we should at least TRY not to cascade the slips. (This is just a generalization of the principle of: slips WILL happen, so schedule for an EARLIER date than your actual target. If you schedule for a later date outright, you won't solve the problem, you'll just make it worse.) Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Jason L Tibbitts III wrote: To me this implies that we should begin testing earlier (or, perhaps, never stop testing) and treat any new failure as an event of significance. It's tough to meet a six month cycle if we spend half of it telling people to expect everything to be broken. We HAVE to accept that Rawhide is sometimes broken to be able to do active development. If we need Rawhide to be always 100% regression-free, we will never get anywhere. It's Rawhide for a reason. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Nathaniel McCallum wrote: I disagree, the feature is shipping on time. Shipping on time enables others in the Fedora community (people who build on, deploy, etc) know with some assurance what their schedules will look like. If I were a project manager looking at using a Linux OS in my project, a demonstrated lack of ability to ship on time is a *huge* mark against using that OS. If you need a schedule you can depend on, just add 2-3 weeks to the official schedule. Maybe even a month, waiting for the first sets of updates can't hurt anyway, they tend to fix a lot of bugs. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Will Woods wrote: This is a good point, and it's one of the reasons the 'critpath' stuff exists. It's the same concept, applied somewhat differently: rather than freeze the 'CoreOS' stuff earlier, we freeze it harder - we require more testing for those pieces. The problem is, freezing harder doesn't work, freezing earlier, on the other hand, MIGHT help, see e.g. the fallout from the incompatible change to ld rushed in the day of the F13 feature freeze, with both the feature owners and FESCo refusing to see any problem in that. That said, rather than a hard freeze, I'd like to see some risk-benefit analysis of the change. In the case of the incompatible ld change, the benefit was zero and the fallout was clearly visible, it's insane that this was considered a feature at all, but the ONLY time for such a feature is in Rawhide immediately after the branch (i.e. they could have put it into F14 instead of F13 at the same time, that would have been borderline acceptable, what they did was absolutely not!). Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
drago01 wrote: It isn't broken so there is nothing to fix; slipping to fix issues found is a feature not a bug. We don't have any reason to rush. +1 Slips DO and WILL happen. It's just a matter of fact. It also happens in other projects. We just need to accept this. If we really want to meet specific target dates for the release, e.g. May/Nov 1, then we need to schedule at least 2 weeks EARLIER, i.e. officially schedule for Apr/Oct 15 and compute all deadlines accordingly. Then the inevitable slips will just do the rest. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
Nicolas Mailhot wrote: So perhaps the delay between invasive features autorized and alpha is too short. It's true that sometimes very invasive features have been rushed in right before the feature freeze, often irrespective of the (lack of) benefits (at least at their state of development at the time). In particular, I'm thinking of the incompatible change to ld which redefined decades-old ELF semantics, which broke the build of half of the distro and which was rushed in the day of F13's feature freeze. That said, there must also be ample time for invasive changes in Rawhide, Fedora can't be leading without the occasional breakage in Rawhide. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/12/2010 12:05 PM, Bruno Wolff III wrote: On Thu, Aug 12, 2010 at 12:00:29 -0700, Adam Williamson awill...@redhat.com wrote: We usually catch most initial blockers for any given release at the first TC stage. Bugs we slip for are usually ones identified at that stage that we couldn't fix in time, bugs introduced between TC and RC by This is another place we could change things. We could build the TC a bit earlier (say a week) and be more conservative about what changes go in until the final RC is approved. How do you suggest we be more conservative? If you expect the developers to do this on their own, good luck. If you want there to be some sort of enforcement I welcome suggestions. - -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxk1ooACgkQ4v2HLvE71NUYIgCeM9z1ldok7clBrDljdF9v7Gcx hHMAmwRcUNLQMeUtdsqcFO5Um3hmJBRq =Q4BS -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The slip down memory lane
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/12/2010 12:33 PM, Lennart Poettering wrote: On Thu, 12.08.10 13:19, Mike McGrath (mmcgr...@redhat.com) wrote: Since 2006 I counted 18 slips (I think one or two of those may just be a single slip listed twice). Lets not yell, lets not flame war, lets not point fingers. How can we fix this? It's clearly not one group or one individual or we'd just go talk to them. This is a collective failure. Since 2006 we've slipped at least 16-18 weeks by my count. That's more than half of a full release cycle. While I side with mclasen here and believe that it is a strength of Fedora that we take the liberty to let cycles slip rather then compromise quality, I want to mention one thing: on opensuse the base system has a different schedule then the rest of the OS. i.e. the kernel, gcc, glibc and the low-level tools freeze first, while everything else may be hacked on a couple of weeks more. Maybe that's something to adopt for Fedora as well? Lennart Well we have the critical path set of packages, however I'm not sure how we can enforce a freeze on those packages before a freeze on all the other packages (ie forcing everything through bodhi like we do at the branch) We can ask and say pretty please, but I suspect as long as the buildsystem allows it, crap will still crash land at the last possible moment. - -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxk1tEACgkQ4v2HLvE71NUe2ACfSNGyIGfJDlpTEr4EXBhmR70+ Mj4An1V6hvOHOX0jhpdIG8HTgMLxbHBO =C4uB -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
