Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 2010-11-01 at 02:18 +0100, Miloslav Trmač wrote: Kevin, could you *please* not word things like that? There's just no need for it. I already wrote this to -test a couple of days ago: http://lists.fedoraproject.org/pipermail/test/2010-October/095135.html and we're discussing it there. I think the thread demonstrates things tend to go much more constructively if you avoid throwing words like 'blatant' and 'failure' and 'needlessly' around. Did we not fail our users? Was there a real need to fail them? (As a non-native speaker, I won't judge the relative merits of blatant vs. sucks.) I didn't say that what Kevin said was *wrong*, I said it wasn't the best way to word it. We designed a policy, put it into effect, now we're observing how well it works and we can modify its implementation on the fly. It doesn't need to be done in an adversarial spirit. Given that _this exact scenario_ was repeatedly brought up since the very start of the update acceptance criteria proposals, I think some frustration is quite justified. This situation is not really a surprise, and Fedora did not have to unnecessarily expose users to a vulnerability in order to relearn this lesson. On the other hand, other scenarios were also brought up, which have not come to pass - for instance, the same thing happening to Fedora 13 or Fedora 14. If we had simply accepted the predictions of doom and not implemented the policy at all, we would be without its benefits for the development of F13 and F14. In addition to being constructive about remedying the situation, shouldn't we try to be more constructive about _not introducing such situations_ in the first place? Mirek Saying 'oh dear, this might not work, we'd better not try' is rarely a good approach, IMHO. It's better to try things, with the proviso that you accept when they aren't working and withdraw or modify them. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 2010-11-01 at 03:54 +0100, Kevin Kofler wrote: There's exactly one constructive thing to do, it's repealing this set of policies (Critical Path and Update Acceptance Criteria) in its entirety. An update should go stable when the maintainer says so, karma should be purely informational feedback for the maintainer. I disagree. The evidence you cite does not support this conclusion. We implemented the policies for three releases. There are significant problems with one release. This does not justify the conclusion that the policies should be entirely repealed. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, Nov 1, 2010 at 5:08 PM, Adam Williamson awill...@redhat.com wrote: Saying 'oh dear, this might not work, we'd better not try' is rarely a good approach, IMHO. It's better to try things, with the proviso that you accept when they aren't working and withdraw or modify them. I would agree with this, if the let's try them and see what happens approach to a process makes it a point to set up a set of specific conditions before putting a new process in place which latch a withdraw of the process. Hey we are going to try this.. and if this or this other things happens..then we are going to stop doing this and put our heads together and have a little re-think about modifying the process -jef -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson píše v Po 01. 11. 2010 v 10:08 -0700: We designed a policy, put it into effect, now we're observing how well it works and we can modify its implementation on the fly. It doesn't need to be done in an adversarial spirit. Given that _this exact scenario_ was repeatedly brought up since the very start of the update acceptance criteria proposals, I think some frustration is quite justified. This situation is not really a surprise, and Fedora did not have to unnecessarily expose users to a vulnerability in order to relearn this lesson. On the other hand, other scenarios were also brought up, which have not come to pass - for instance, the same thing happening to Fedora 13 or Fedora 14. If we had simply accepted the predictions of doom and not implemented the policy at all, we would be without its benefits for the development of F13 and F14. A problem with this line of argument is that the benefits are not quite apparent to me. In addition to being constructive about remedying the situation, shouldn't we try to be more constructive about _not introducing such situations_ in the first place? Saying 'oh dear, this might not work, we'd better not try' is rarely a good approach, IMHO. That is a cost-benefit comparison. New does not imply improved. It's better to try things, with the proviso that you accept when they aren't working and withdraw or modify them. It's even better not to dismiss known problems with the policy, and to make sure the policy can handle them properly from the start. This was not a surprise, this was an unforced error. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 2010-11-01 at 18:29 +0100, Miloslav Trmač wrote: On the other hand, other scenarios were also brought up, which have not come to pass - for instance, the same thing happening to Fedora 13 or Fedora 14. If we had simply accepted the predictions of doom and not implemented the policy at all, we would be without its benefits for the development of F13 and F14. A problem with this line of argument is that the benefits are not quite apparent to me. The policies prevented us from shipping a number of completely broken updates, which is exactly what they were intended to do. I don't have a command handy to do a search for rejected proposed critpath updates for F14, but if you figure it out, you can see the precise results of the policy there. In addition to being constructive about remedying the situation, shouldn't we try to be more constructive about _not introducing such situations_ in the first place? Saying 'oh dear, this might not work, we'd better not try' is rarely a good approach, IMHO. That is a cost-benefit comparison. New does not imply improved. We had an extensive discussion about the benefits of testing important updates at the time the policy went into effect. I don't think it's really necessary to re-hash the entire thing. For the record, I did not say nor do I believe that new inevitably implies improved. It's better to try things, with the proviso that you accept when they aren't working and withdraw or modify them. It's even better not to dismiss known problems with the policy, and to make sure the policy can handle them properly from the start. This was not a surprise, this was an unforced error. Sorry, but characterizing it as a 'known problem' is misleading. It's easy to forecast failure, and you'll likely always be correct in *some* cases if you forecast enough failures. Only if you precisely forecast only the failures that actually happen, and do not forecast any failures that don't happen, can your forecast be considered truly reliable. If this had truly been a 'known problem' then those who predicted it would also have correctly chosen *not* to predict failure in the case of Fedora 13 and Fedora 14. The fact is that they did predict a failure which has not, in fact, come to pass (neither F13 nor F14 have long queues of old critpath updates). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson píše v Po 01. 11. 2010 v 10:39 -0700: On Mon, 2010-11-01 at 18:29 +0100, Miloslav Trmač wrote: It's better to try things, with the proviso that you accept when they aren't working and withdraw or modify them. It's even better not to dismiss known problems with the policy, and to make sure the policy can handle them properly from the start. This was not a surprise, this was an unforced error. Sorry, but characterizing it as a 'known problem' is misleading. It's easy to forecast failure, and you'll likely always be correct in *some* cases if you forecast enough failures. Only if you precisely forecast only the failures that actually happen, and do not forecast any failures that don't happen, can your forecast be considered truly reliable. The accuracy of prediction, and especially accuracy of the timing, is not at all relevant. In fact, it is _preciselly_ the unknown nature of risks that requires thinking about them in advance. People don't wear helmets because they know when something will hit their head, but because they don't know when, or even if, it will. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 2010-11-01 at 18:51 +0100, Miloslav Trmač wrote: Sorry, but characterizing it as a 'known problem' is misleading. It's easy to forecast failure, and you'll likely always be correct in *some* cases if you forecast enough failures. Only if you precisely forecast only the failures that actually happen, and do not forecast any failures that don't happen, can your forecast be considered truly reliable. The accuracy of prediction, and especially accuracy of the timing, is not at all relevant. In fact, it is _preciselly_ the unknown nature of risks that requires thinking about them in advance. Which rather contradicts your description of it as a 'known problem', yes? People don't wear helmets because they know when something will hit their head, but because they don't know when, or even if, it will. Mirek That's not really a relevant analogy. You can't 'wear a helmet' in this case. There's no way we could have implemented the policy and 'worn a helmet' by allowing updates to happen without the conditions of the policy being fulfilled; that would effectively negate the policy. If you want to continue with the analogy, what you seem to be saying is that we should never have implemented the policy in the first place, which is not analogous to wearing a helmet; it's analogous to never leaving the house just in case something hits you on the head. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson píše v Po 01. 11. 2010 v 10:55 -0700: On Mon, 2010-11-01 at 18:51 +0100, Miloslav Trmač wrote: Sorry, but characterizing it as a 'known problem' is misleading. It's easy to forecast failure, and you'll likely always be correct in *some* cases if you forecast enough failures. Only if you precisely forecast only the failures that actually happen, and do not forecast any failures that don't happen, can your forecast be considered truly reliable. The accuracy of prediction, and especially accuracy of the timing, is not at all relevant. In fact, it is _preciselly_ the unknown nature of risks that requires thinking about them in advance. Which rather contradicts your description of it as a 'known problem', yes? No; the existence of the problem was known, only the timing and precise extent was not. If you want to continue with the analogy, what you seem to be saying is that we should never have implemented the policy in the first place, That is one option; another would be adding a I'm the maintainer and I really mean it checkbox for security updates (with FESCo/Fedora QA/somebody else reviewing the cases retrospectively, if they feel like it); yeat another is not enforcing the policy on security updates at all, as I seem to remember was proposed (or even implemented?) at one time. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson wrote: On the other hand, other scenarios were also brought up, which have not come to pass - for instance, the same thing happening to Fedora 13 or Fedora 14. Nonsense. We just do not have enough evidence yet to show such things happening for F13 and F14. They CAN, and IMHO WILL, happen, e.g.: * Will a critical security fix for Konqueror get karma as quickly as the one for Firefox did? (This is especially relevant considering that some people want to put the whole KDE workspace into critpath. But even non-critpath updates need karma to get pushed.) * Would that Firefox update have gotten karma that quickly without the nagmail to the devel ML? Do you think the approach of sending nagmail scales? And at least for F14 development, there have been other, less critical failures, which I've already pointed out in the respective threads. If we had simply accepted the predictions of doom and not implemented the policy at all, we would be without its benefits for the development of F13 and F14. What benefits? I see only problems, in fact the very ones I've warned about right from the beginning. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson wrote: The policies prevented us from shipping a number of completely broken updates, which is exactly what they were intended to do. I don't have a command handy to do a search for rejected proposed critpath updates for F14, but if you figure it out, you can see the precise results of the policy there. They also let several completely broken updates through and then delayed the FIXES for those updates, exactly as I had been warning about all the time. For example, my firstboot update which was required to make the Xfce spin work again (there was an additional problem with the LXDE spin, but that one was present both before and after that update, and could only be noticed after that update was pushed) got delayed. The fact is that they did predict a failure which has not, in fact, come to pass (neither F13 nor F14 have long queues of old critpath updates). Even ONE old critpath update is a failure. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 01 Nov 2010 19:26:43 +0100 Kevin Kofler kevin.kof...@chello.at wrote: They also let several completely broken updates through and then delayed the FIXES for those updates, exactly as I had been warning about all the time. Cite(s)? For example, my firstboot update which was required to make the Xfce spin work again (there was an additional problem with the LXDE spin, but that one was present both before and after that update, and could only be noticed after that update was pushed) got delayed. If You mean: https://admin.fedoraproject.org/updates/firstboot-1.113-4.fc14 it didn't break the Xfce spin. Xfce spin is using gdm still, which means metacity is pulled in, which means firstboot was fine. So, this case seems to me like a poor example for your position. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
mån 2010-11-01 klockan 10:09 -0700 skrev Adam Williamson: I disagree. The evidence you cite does not support this conclusion. We implemented the policies for three releases. There are significant problems with one release. This does not justify the conclusion that the policies should be entirely repealed. I don't mind the process in general, but have some points where it can improve Very often the same update is submitted for several releases, and it's kind of pointless to require full karma in all releases (to require some in each release is ok). If one release has got full karma then it's reasonable to require less karma on other releases receiving the same update. The risk for non-obvious regression for some release only is fairly low, more likely there is very obvious release specific regressions like dependency failures when another package have been split/merged etc and related fuckups. We also need some obvious ways where users in general can subscribe to testing updates of stuff that they care about, to expand the userbase that performs testing of updates. Generally running a system with updates-testing always enabled is scary and not many want to take that leap. But I think that if we could give users the ability to subscribe to testing packages X,Y,Z of their choics and getting update testing notifications for those packages only from updates-testing would speed things up considerably. In addition the package management update request process could do with some serious makeover to streamline the process and reduce risk for error, but that's topic for another thread. Regards Henrik -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Mon, 2010-11-01 at 22:54 +0100, Henrik Nordström wrote: mån 2010-11-01 klockan 10:09 -0700 skrev Adam Williamson: I disagree. The evidence you cite does not support this conclusion. We implemented the policies for three releases. There are significant problems with one release. This does not justify the conclusion that the policies should be entirely repealed. I don't mind the process in general, but have some points where it can improve Very often the same update is submitted for several releases, and it's kind of pointless to require full karma in all releases (to require some in each release is ok). If one release has got full karma then it's reasonable to require less karma on other releases receiving the same update. The risk for non-obvious regression for some release only is fairly low, more likely there is very obvious release specific regressions like dependency failures when another package have been split/merged etc and related fuckups. This is a reasonable modification of the idea that an update should only require karma for one release (which would be nice if it were true but unfortunately isn't). In practice, though, there isn't much wiggle room for requiring 'less' karma. Non-critpath updates already require only one +1 to go through without the waiting time requirement. Critpath updates only require +1 from a proventester and +1 from any other tester (proven or not). I think I'd probably support the proposal that if a critpath update exists in identical form for multiple releases, and it has passed full critpath karma requirements for one release, it should require only +1 from any tester on the other releases to go out. We also need some obvious ways where users in general can subscribe to testing updates of stuff that they care about, to expand the userbase that performs testing of updates. Generally running a system with updates-testing always enabled is scary and not many want to take that leap. But I think that if we could give users the ability to subscribe to testing packages X,Y,Z of their choics and getting update testing notifications for those packages only from updates-testing would speed things up considerably. That's also a nice idea (though it's somewhat complex given that updates are *actually* pushed out as sets, and a given update may be affected by another given update even if they don't have an explicit relationship through dependencies). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
mån 2010-11-01 klockan 15:12 -0700 skrev Adam Williamson: This is a reasonable modification of the idea that an update should only require karma for one release (which would be nice if it were true but unfortunately isn't). In practice, though, there isn't much wiggle room for requiring 'less' karma. Non-critpath updates already require only one +1 to go through without the waiting time requirement. Critpath updates only require +1 from a proventester and +1 from any other tester (proven or not). Right. I was mostly thinking about the autokarma I think. Not normally doing pushes until after the waiting period. I think I'd probably support the proposal that if a critpath update exists in identical form for multiple releases, and it has passed full critpath karma requirements for one release, it should require only +1 from any tester on the other releases to go out. Yes. From the same reasoning explained before. If it's provenly tested in one release then chances are very high it works in the other releases as well unless it doesn't work at all. Regards Henrik -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Sun, 31 Oct 2010 04:37:38 +0100, Kevin wrote: Martin Stransky wrote: there's a new Firefox update waiting in Bodhi and we can't push it to stable because of new rules. We recommend you to update to it ASAP as it fixes a public critical 0day vulnerability (https://bugzilla.mozilla.org/show_bug.cgi?id=607222). Looks like the F13 build got karma quickly enough to land directly in stable after all, the F12 build, on the other hand, was stuck in testing for 2 days before finally making it out to stable. Yet another blatant example of failure of the Update Acceptance Criteria, needlessly exposing our users to critical vulnerabilities. (And no, by giving yet another special exception to Firefox wouldn't be a solution. ;-) This problem can hit any other app as well.) Kevin Kofler Okay, feedback time. Lately, there have been several attempts at urging proventesters (and not just testers in general) to give positive karma for aging critpath updates. It also has been decided by someone (or maybe even a comittee) to spam proventesters daily with [old_testing_critpath] messages for all three dist releases, with no day to unsubscribe from that (other than leaving proventesters group, which is what at least one person has threatened with, or filtering those messages). Dunno about other testers (and there aren't many yet), but I have abandoned F-12 long ago due to lack of time when F-13 became the one to use on a daily basis. And some time before F-14 Beta, my desktop has been switched to boot F-14 by default. That's the only opportunity to evaluate F-14 early and possibly find issues prior to its release. On the contrary, most of Fedora's users will wait for the final release, and many users will wait even longer. It's highly likely that bugzilla can confirm that. F-14 is the the only way forward, and don't like to spend time on F-13 and older anymore. That also applies to packagers I maintain or monitor. I simply don't see the user base [target group] anymore. About positive karma in bodhi, I don't feel comfortable signing off arbitrary updates just because they didn't crash for me after five minutes. With some updates, regression has slipped through already. And the more bugs an update addresses with either patches or a version upgrade, the more careful I would like to be when testing something. Also, in my book, an update working on F-14 may still malfunction on an older dist release due to differences in dependences and the core setup. I still don't understand why some non-security updates are rushed out with sometimes not even the package maintainer(s) having tested them at all. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
On Sun, 2010-10-31 at 04:37 +0100, Kevin Kofler wrote: Martin Stransky wrote: there's a new Firefox update waiting in Bodhi and we can't push it to stable because of new rules. We recommend you to update to it ASAP as it fixes a public critical 0day vulnerability (https://bugzilla.mozilla.org/show_bug.cgi?id=607222). Looks like the F13 build got karma quickly enough to land directly in stable after all, the F12 build, on the other hand, was stuck in testing for 2 days before finally making it out to stable. Yet another blatant example of failure of the Update Acceptance Criteria, needlessly exposing our users to critical vulnerabilities. Kevin, could you *please* not word things like that? There's just no need for it. I already wrote this to -test a couple of days ago: http://lists.fedoraproject.org/pipermail/test/2010-October/095135.html and we're discussing it there. I think the thread demonstrates things tend to go much more constructively if you avoid throwing words like 'blatant' and 'failure' and 'needlessly' around. We designed a policy, put it into effect, now we're observing how well it works and we can modify its implementation on the fly. It doesn't need to be done in an adversarial spirit. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson píše v Ne 31. 10. 2010 v 18:06 -0700: On Sun, 2010-10-31 at 04:37 +0100, Kevin Kofler wrote: Yet another blatant example of failure of the Update Acceptance Criteria, needlessly exposing our users to critical vulnerabilities. Kevin, could you *please* not word things like that? There's just no need for it. I already wrote this to -test a couple of days ago: http://lists.fedoraproject.org/pipermail/test/2010-October/095135.html and we're discussing it there. I think the thread demonstrates things tend to go much more constructively if you avoid throwing words like 'blatant' and 'failure' and 'needlessly' around. Did we not fail our users? Was there a real need to fail them? (As a non-native speaker, I won't judge the relative merits of blatant vs. sucks.) We designed a policy, put it into effect, now we're observing how well it works and we can modify its implementation on the fly. It doesn't need to be done in an adversarial spirit. Given that _this exact scenario_ was repeatedly brought up since the very start of the update acceptance criteria proposals, I think some frustration is quite justified. This situation is not really a surprise, and Fedora did not have to unnecessarily expose users to a vulnerability in order to relearn this lesson. In addition to being constructive about remedying the situation, shouldn't we try to be more constructive about _not introducing such situations_ in the first place? Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Adam Williamson wrote: I already wrote this to -test a couple of days ago: http://lists.fedoraproject.org/pipermail/test/2010-October/095135.html and we're discussing it there. I think the thread demonstrates things tend to go much more constructively if you avoid throwing words like 'blatant' and 'failure' and 'needlessly' around. We designed a policy, put it into effect, now we're observing how well it works and we can modify its implementation on the fly. It doesn't need to be done in an adversarial spirit. There's exactly one constructive thing to do, it's repealing this set of policies (Critical Path and Update Acceptance Criteria) in its entirety. An update should go stable when the maintainer says so, karma should be purely informational feedback for the maintainer. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)
Martin Stransky wrote: there's a new Firefox update waiting in Bodhi and we can't push it to stable because of new rules. We recommend you to update to it ASAP as it fixes a public critical 0day vulnerability (https://bugzilla.mozilla.org/show_bug.cgi?id=607222). Looks like the F13 build got karma quickly enough to land directly in stable after all, the F12 build, on the other hand, was stuck in testing for 2 days before finally making it out to stable. Yet another blatant example of failure of the Update Acceptance Criteria, needlessly exposing our users to critical vulnerabilities. (And no, by giving yet another special exception to Firefox wouldn't be a solution. ;-) This problem can hit any other app as well.) Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
