Re: Updates Criteria Summary/Brainstorming
On Tue, 23 Nov 2010 18:55:38 +0100, Kevin wrote: Mike Fedyk wrote: Install package from updates-testing, then +1 to karma after it works for you with your tests and normal workload. The average user won't even KNOW there's an update available in updates- testing before it's too late (i.e. all his/her data is gone, (s)he asks on forums or IRC what's up, and people point him/her to the testing update (which doesn't help because all the data is already corrupted/deleted!), at which point (s)he'll switch to Ubuntu or Window$ in disgust and swear to never use Fedora again). That can't be the full story. The distribution is based upon a long development period including several test releases. Are you trying to say that this data-eating-bug manages to slip through the cracks only for Fedora and not Ubuntu or Window$? I cannot believe that. This part of the thread is a lost cause already, if we want to go down that road while trying to fight for more freedom to decide on whether/when to push a stable update. :-/ Ubuntu contains plenty of packages with a large list of active bugs, which are not fixed with updates. Certainly not in a ASAP manner. No need to foist possibly broken software on everyone. That's exactly why bugs must be fixed ASAP! And a flood of rushed updates, which moves away farther from the originally tested final distribution, increases the risk that additional bugs must be fixed ASAP. That's going in circles. It's correct to pull up a fence and try to find more bugs prior to release of the dist *and* the updates. Returning to the topic, a new criterion for the updates acceptance could lower the hurdles for updates, which only patch the software (or the spec file) compared with the previous package release. That is, no attempts at hiding a version upgrade in a large scm-snapshot-patch, and no supposedly minor version bump which contains some fixes actually but also breaks something else (such as the infamous unexpected ABI/API breaks). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 10:39:27PM +0100, Björn Persson wrote: Toshio Kuratomi wrote: There's one easy but deeply flawed way to do this -- automatically create a usern...@fedoraproject.org bugzilla account for the user with the password used in FAS. Deeply flawed in this case because the passwords can get out of sync, we're creating accounts when people sometimes want to use a different address in bugzilla and fas, etc. Hmm, but it says here that we should use the same address in Bugzilla and FAS: https://fedoraproject.org/wiki/Join_the_package_collection_maintainers Yes, you should. When you don't the infrastructure admins get spammed until someone either fixes the mismatch or alerts me and I get time to put in a mapping for the bugzilla email address. We don't control RH bugzilla so changing bugzilla to be able to use fas to login would be problematic. That solution seems backwards anyway. I think most new contributors probably report bugs long before they become packagers, so they need Bugzilla accounts before they need FAS accounts. How about changing Bodhi to allow login with either a FAS account or a Bugzilla account? Would that be difficult to do? Yes. Everything in Fedora uses the FAS account. So, for instance, bodhi needs to know whether the person pushing an update has permission which means that they need to enter their fas account to match up with what's in the pkgdb. Implementing a separate login just for comments would also be a lot of extra work although it would be more segregated from the rest of the code so it might be doable. -Toshio pgpgCjFyutE0l.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Kevin Fenzi wrote: * Change FN-1 to just security and major bugfix. Nothing else allowed. So, if: 1) a package is updated because of a security problem 2) next day, FN+1 is released 3) next day, it is found that the fix in 1) has a very minor bug (e.g. typo in a string) 4) the string is not fixable anymore: no minor bugfixes for FN-1 If the regression is cosmetic/performance etc. it will be left unfixed, and we can't really describe this distro as Linux, you know, that wonderful world where problems are rapidly fixes by lots of people around the world. We should more honestly say that Fedora as two level of support: - 6 months (any bug) - additional 7 months (serious bugs only) Next, the Fn-2 - Fn upgrade process can be sacrificed. I personally like all kind of updates in a stable distro. Even major ones (such as KDE). I, as a user, will not be so stupid to update my presentation software a few minutes before the important meeting with the boss. I also trust the maintainers to not frequently push crap at me. -- Roberto Ragusamail at robertoragusa.it -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Michael Schwendt wrote: There are not enough [human] resources to update Fn-1 in any way it would be close[r] to the current release. You can observe it everywhere (even by drawing conclusions about ABRT reports) that Fn-2 is abandoned by our users months before its EOL date. Uh, we'd have the resources to push KDE upgrades to Fn-1 just fine (see 4.2 for F9, 4.3 for F10 and 4.4 for F11). It's not that much work to build the stuff we're building for Fn anyway also for Fn-1. It's the new unnecessarily paranoid QA we don't seem to have the resources for. We already test our KDE upgrades very hard. But most of that testing will be on Fn (and Fn+1). The packages we push to Fn-1 would be the exact same ones, just rebuilt! Nothing I would back up without learning about the _details_, please. How to determine when to blame the package maintainer? What would your rule set look like? Common sense! But that seems to be lacking in Fedora circles lately. :-( Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 03:04:30PM -0800, Mike Fedyk wrote: This still builds a reactive system instead of a preventative system. An only reactive system will not help prevent bad updates from getting out in the first place. That said, adding a reactive component to a preventative system would be a good thing. If a maintainer releases one package that puts regressions into the stable updates repo, then the minimum karma doubles on all of their packages doubles for 2 months or something like that. So feel free to push directly to stable as often as you want, but once you introduce one regression, you have to satisfy 10 karma on every package you update. The second time, you have to satisfy 20 karma on every package you update and so on. Fedora could as well just stop published updates at all, then no bad updates will hit stable ever. Simply because we can't trust that maintainer anymore. How is it the fault of the maintainer when ten testers certified that the update is ok even when it was not? Really, allowing regressions to make it to stable is so costly simply because it has to be fixed several magnitudes more times than if it is caught by people actually testing packages before they're released to the masses. In general I have to more often experience the same bugs that others already found because of old package than I have to fix regressions. At least during a release, when I have to update to a new Fedora release, bugs tend to come back. But to prevent this I would like to have automatic tested instead of lots of error prone manual testing. Regards Till pgpZoosXHBRO0.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/22/10 11:32 PM, Till Maas wrote: On Mon, Nov 22, 2010 at 02:33:35PM -0800, Jesse Keating wrote: On 11/22/10 1:50 PM, Till Maas wrote: On Mon, Nov 22, 2010 at 12:02:49PM -0800, Jesse Keating wrote: On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. Afaik there is currently no distinction between the requirement for non crit-path updates and the karma autopush level. Therefore by default non critpath updates require +3 karma, unless this has been changed since the beginning for the update criteria enforcement. I swear I've been able to set karma levels at 1 for non-critpath updates. But this means that the update is automatically pushed to stable, not that the update is approved and then pushed to stable when the maintainer requests it. Regards Till I'm sorry, I didn't realize that's what you were arguing about. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 01:29:45PM -0700, Kevin Fenzi wrote: Here's the latest list of ideas culled from this thread. Note: these are NOT my ideas, I am just gathering them up so fesco can discuss them. Feel free to add more concrete ideas, or let me know if I missed one you had posted. If folks could avoid me too or posts that contain no new information it would be easier for me to gather the actual ideas. ;) I've split things into General, Security, Critpath and non security/critpath to help organize them. Keep the ideas coming... Since people have been tossing around the general idea of testing being needed for a few maintainer/package combos where bad updates traditionally come from, here's a concrete proposal based on that: General: * Testing is only required for certain packages. Those packages are the packages where problems have occurred before so fesco or other maintainers affected by the changes deem it necessary to supplement the maintainer's testing with outside help. - Option: supplement this list with critpath packages where the maintainers desire extra testing. This means that we would no longer be dragging in dependencies immediately... only if updates by the dependency's maintaner to that package are breaking things. -Toshio pgpweNqg0ML4R.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Mike Fedyk wrote: So feel free to push directly to stable as often as you want, but once you introduce one regression, you have to satisfy 10 karma on every package you update. The second time, you have to satisfy 20 karma on every package you update and so on. At that point you can just ban the offender from Bodhi entirely, it'll have the same effect. :-/ Basically no update gets +10 karma out there in the real world! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Adam Williamson wrote: On Mon, 2010-11-22 at 16:01 +0100, Kevin Kofler wrote: Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. How about testing that it doesn't corrupt mail folders even worse? To the average user, more corrupt is a bit like more pregnant. The data is most likely a total loss even in the case which is technically less corrupt. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Michael Cronenworth wrote: So you'll double, triple, ultra swear that this[1] will never happen again? [1] http://lists.fedoraproject.org/pipermail/announce/2008-December/002572.html That wasn't a data corruption issue in the first place. It was a low-severity security fix, and the fix was very invasive and dangerous. The maintainer should have known this and NOT opted for a direct stable push in this case. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Mike Fedyk wrote: Install package from updates-testing, then +1 to karma after it works for you with your tests and normal workload. The average user won't even KNOW there's an update available in updates- testing before it's too late (i.e. all his/her data is gone, (s)he asks on forums or IRC what's up, and people point him/her to the testing update (which doesn't help because all the data is already corrupted/deleted!), at which point (s)he'll switch to Ubuntu or Window$ in disgust and swear to never use Fedora again). No need to foist possibly broken software on everyone. That's exactly why bugs must be fixed ASAP! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 10:39 PM, Tom Lane t...@redhat.com wrote: (And yeah, if we allow +1 autopush, we should definitely expect -1 is sufficient to unpush. Maybe bodhi should restrict the combination of those two settings, rather than either one alone?) Not as long people give -1 for random crap oh this does not fix a completely unrelated bug ... (where the update isn't any worse then the current build, but fixes issues for _others_). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Till Maas wrote: Afaik there is no need for a maintainer to set different acceptance thresholds for his updates. At least nobody ever explained to me why this would be helpful. * Upgrade paths! I DON'T want my foo-1.2.3-4.fc13 update to go out before my foo-1.2.3-4.fc14 update, even if it happens to get karma first. * Possibility to look at the feedback. E.g. if an update has -1, deletes all my e-mail, +1, can haz noo verzion? and +1, didn't test e-mail, everything else works, I'm NOT going to push the update! * Because people just make better decisions than software, period. In fact, IMHO automatic pushing is completely stupid, and the current process which heavily relies on it is just broken. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Tom Lane wrote: (And yeah, if we allow +1 autopush, we should definitely expect -1 is sufficient to unpush. Maybe bodhi should restrict the combination of those two settings, rather than either one alone?) Well, we've started to use +1/-999 for some KDE updates. :-) Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Michael Cronenworth wrote: -Installs PackageKit plugin to give karma through the gnome-packagekit GUI. (Nothing exists yet, but I'm gonna get started on one soon) What about KPackageKit? Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Tue, Nov 23, 2010 at 07:06:33PM +0100, Kevin Kofler wrote: Till Maas wrote: Afaik there is no need for a maintainer to set different acceptance thresholds for his updates. At least nobody ever explained to me why this would be helpful. * Upgrade paths! I DON'T want my foo-1.2.3-4.fc13 update to go out before my foo-1.2.3-4.fc14 update, even if it happens to get karma first. * Possibility to look at the feedback. E.g. if an update has -1, deletes all my e-mail, +1, can haz noo verzion? and +1, didn't test e-mail, everything else works, I'm NOT going to push the update! * Because people just make better decisions than software, period. In fact, IMHO automatic pushing is completely stupid, and the current process which heavily relies on it is just broken. I did not write about the automatic pushing threshold but only about the acceptance threshold. Regards Till pgpzKixEb0RTK.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Toshio Kuratomi wrote: We don't control RH bugzilla so changing bugzilla to be able to use fas to login would be problematic. The right solution would really be to have a separate Fedora Bugzilla tied in to Fedora infrastructure, with bug states which make sense for Fedora, not RHEL etc. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
sön 2010-11-21 klockan 11:00 +0100 skrev Till Maas: I guess this can be somehow automated. E.g. change Bodhi to drop the karma requirements for packages that had e.g. two subsequent updates without any Bodhi feedback and re-enable it if they get feedback. That would be somewhat counter productive for the goal of actually having testers, as it discourages maintainers looking for testers as a way to improve their release process. Imho the main concerns about the updates criteria is actually confusion between autokarma requirements and minimal karma requirements. At least it was for me when last discussing the topic. The actual requirements isn't very high or unreasonable. What I'd like to see is * aggregated karma across the releases for the same package version. Should be quite sufficient to test most updates on one of the active releases. * autoqa trapping dependency errors and other install failure errors, disallowing push of a completely borked package, including changes in provides breaking other packages. * packages failing the above should only be pushed in a group when dependencies have been satisfied (done automatically once push has been requested for all of them), or after requesting an exception if a dependent package for some reason can't be updated. * better integration of releases in the push process, enabling package maintainers a view of the package status across the releases. * automatic enforcement on order of release, preventing a push or at minimum alerting the maintainer if an earlier version is in a later fedora release (including rawhide). In addition to this the whole concept about how to enable actual users to test packages in a reasonable manner need quite a bit of love to make testing scale. The model of updates-testing and test everything is simply a too high threshold for most users and scares away most, and manually searching for and applying selected updates with yum do not scale. I think something along this model would help greatly in that area: * Keep updates-testing repository model as-is * Give users an easy option to get notified via package-kit when there is updates to selected packages available in updates-testing, enabling the user to select just package x,y,z, selected package groups, or everything of the packages they have installed, and to select if the user wants testing updates to be automatically installed as part of the update process or only notified and requiring manual selection each time. * A new notification icon, reminding users when they have packages from updates-testing installed for which they have not yet given feedback. Packages should automatically disappear from this list when they have been pushed to updates. * Give users a easy way of downgrading to current non-testing release of a package, giving them confidence that they can easily recover should they find or suspect that the updates-testing package fails. This would also need blocking that package release from automatic update from updates-testing. All of this could be combined. E.g. packages with enough testers get test cases and need to fulfill stronger criteria. Packages with not so many testers get test cases and only need to fulfil that similar updates need to receive good karma on one Fedora release. Imho this should be more based on how critical the package is for system operation and quality of past updates than amount or activity of testers. I.e. if a borked update gets pushed out by the package maintainer then that will increase the testing requirement on future updates of the same package for a number of package pushes. Also it could be made easier for maintainers to identify problematic updates, e.g. by warning that the dependencies or provides of an update changed when the update is created. +1 Regards Henrik -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Saturday, November 20, 2010 23:35:43 Kevin Fenzi wrote: ok, I dug through the devel list for the last month or two and wrote down all the various ideas folks have come up with to change/improve things. Here (in no particular order) are the ideas and some notes from me on how we could enable them. Please feel free to add new (actual/concrete ideas or notes): * Just drop all the requirements/go back to before we had any updates criteria. I like this idea, but I'm pretty sure this won't happen. I don't like the bureaucracy you can see all around you. Fixing problems caused by individual failure (or individual's failure) with new policy/law does not make happy contributors/people. This is the exact behavior of Civil Aviation Authority in my country - after 50 years of no problems there is one a***ole that kills himself because of ignoring physics and self-preservation, as result all sane normal people have to do more paperwork and are more restricted. The only result is pilots are more and more fed up every year. And know what, there are still people willingly breaking the rules so it does not solve anything. Another comment: When I started to work for Fedora, I tried to do my best. You know, there are some comparisons of OSes and distributions. One of the comparisons being How many days OS was vulnerable (with known exploitable security bug). So when there was new CVE-- bug, I tried to fix it as fast as possible, because I wanted to make Fedora The Best Distro. But what happened after I fixed this bug? It took whole week before new build was tagged. Should I work hard if there is no visible result? I was disappointed. Now, packages are tagged quickly and reliably, great improvement (thanks to everyone who helped with this). But again, after things were better we have new policy that delays all bug fixes from being delivered quickly and I'm disappointed again. * Change FN-1 to just security and major bugfix This may be hard to enforce or figure out if something is a major bugfix. This idea would make users less happy, at least from what I see. I manage a few Fedora systems for my friends/relatives who have almost no IT knowledge nor they can use English. They don't care about open-source and other freedoms. They use Linux just because it's free and usable (they always compare it with m$ windoze they used before). In real world, bugs happen, they don't care too much about bugs in sw, if there is visible progress. If they found a bug, they complain to me and I file it in bugzilla. Once the bug is fixed, I report these wonderful news to them and they are really happy. But... remove the bug fix delivered to Fn-1 and they won't be happy, in fact they would think that Linux sucks. Restriction to most critical bugs only is not enough. And no, updating all machines every 6 months is too much disturbing (for them) and time consuming (for me). * allow packages with a %check section to go direct to stable %check can be simple, %check can be complex, but where would you draw the line? Also very limited number of GUI apps has %check (only guess) * setup a remote test env that people could use to test things. I don't think this would make significant difference. People don't test packages because they don't have time, they are lazy, they don't know how to test it or they are just consumers (not enough IT/english knowledge). * require testing only for packages where people have signed up to be testers this could help a little for some cases * Ask maintainers to provide test cases / test cases in wiki for each package? this could help, but it's not always possible to add these test cases. One example: imap server package - new bug that can corrupt mail folders in some circumstances. Maintainer updates package and sets 'type=bugfix' in bodhi, but not always it's possible to write down any test case. It's still a bug fix and it's better to be delivered sooner than wait if anyone out there get's his mail folders corrupted. Actual policy does not help proactivity. * reduced karma requirement on other releases when one has gone stable better than nothing :) Other concrete ideas? let maintainer decide, punish (enforce any policy) only those maintainers who breaks something, not all innocent group -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Michal Hlavinka wrote: I like this idea, but I'm pretty sure this won't happen. I don't like the bureaucracy you can see all around you. Fixing problems caused by individual failure (or individual's failure) with new policy/law does not make happy contributors/people. This is the exact behavior of Civil Aviation Authority in my country - after 50 years of no problems there is one a***ole that kills himself because of ignoring physics and self-preservation, as result all sane normal people have to do more paperwork and are more restricted. The only result is pilots are more and more fed up every year. And know what, there are still people willingly breaking the rules so it does not solve anything. +1, so true (and sad). :-( (BTW, it's not just your country, they just get it dictated from the EU. It's the same all over Europe, and it's worse on the other side of the Atlantic.) Another comment: When I started to work for Fedora, I tried to do my best. You know, there are some comparisons of OSes and distributions. One of the comparisons being How many days OS was vulnerable (with known exploitable security bug). So when there was new CVE-- bug, I tried to fix it as fast as possible, because I wanted to make Fedora The Best Distro. But what happened after I fixed this bug? It took whole week before new build was tagged. Should I work hard if there is no visible result? I was disappointed. Now, packages are tagged quickly and reliably, great improvement (thanks to everyone who helped with this). But again, after things were better we have new policy that delays all bug fixes from being delivered quickly and I'm disappointed again. Indeed, these new PITA policies are very demotivating! This idea would make users less happy, at least from what I see. I manage a few Fedora systems for my friends/relatives who have almost no IT knowledge nor they can use English. They don't care about open-source and other freedoms. They use Linux just because it's free and usable (they always compare it with m$ windoze they used before). In real world, bugs happen, they don't care too much about bugs in sw, if there is visible progress. If they found a bug, they complain to me and I file it in bugzilla. Once the bug is fixed, I report these wonderful news to them and they are really happy. But... remove the bug fix delivered to Fn-1 and they won't be happy, in fact they would think that Linux sucks. Restriction to most critical bugs only is not enough. And no, updating all machines every 6 months is too much disturbing (for them) and time consuming (for me). Indeed. I think it's a big mistake to provide only second-class support for Fn-1. The assertion that that's what the people on Fn-1 want is just unfounded, based on a misunderstanding of why people use Fn-1. this could help, but it's not always possible to add these test cases. One example: imap server package - new bug that can corrupt mail folders in some circumstances. Maintainer updates package and sets 'type=bugfix' in bodhi, but not always it's possible to write down any test case. It's still a bug fix and it's better to be delivered sooner than wait if anyone out there get's his mail folders corrupted. Actual policy does not help proactivity. Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. Other concrete ideas? let maintainer decide, punish (enforce any policy) only those maintainers who breaks something, not all innocent group +1! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, 2010-11-22 at 16:01 +0100, Kevin Kofler wrote: Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. How about testing that it doesn't corrupt mail folders even worse? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Kevin Kofler wrote: Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. So you'll double, triple, ultra swear that this[1] will never happen again? [1] http://lists.fedoraproject.org/pipermail/announce/2008-December/002572.html -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 10:49:38AM -0600, Michael Cronenworth wrote: Kevin Kofler wrote: Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. So you'll double, triple, ultra swear that this[1] will never happen again? [1] http://lists.fedoraproject.org/pipermail/announce/2008-December/002572.html As though swearing it will never happen is even possible to deliver? Not all software is created equal, some depends on others, some is depended on by *many* others and that's just an unfortunate side effect of the game we play. Software is hard - Knuth There are going to be errors that need fixing, nothing is bullet proof and to think we're going to be able to pull off perfection at all times is just not realistic. -AdamM -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 09:57:40AM +0100, Henrik Nordström wrote: sön 2010-11-21 klockan 11:00 +0100 skrev Till Maas: I guess this can be somehow automated. E.g. change Bodhi to drop the karma requirements for packages that had e.g. two subsequent updates without any Bodhi feedback and re-enable it if they get feedback. That would be somewhat counter productive for the goal of actually having testers, as it discourages maintainers looking for testers as a way to improve their release process. I do not see how this discourages maintainers. I do not know of any package maintainer that stated that he did not want his updates tested. This would at least encourage people that want better tested updates to commit into testing them. The current problem is not that package maintainers do not want their packages tested, but that updates are delayed without any visible testing. Regards Till pgp18vmm1Pm2D.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
* Adam Miller [22/11/2010 18:03] : As though swearing it will never happen is even possible to deliver? I believe that's Michael's whole point. The whole 'push directly to stable' arguement rests heavily on the principle that an update is always better (from a QA standpoint) than whatever it's replacing. The problem is that there's no way to guarantee this, essentielly because it isn't true. Emmanuel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, 2010-11-22 at 18:09 +0100, Emmanuel Seyman wrote: * Adam Miller [22/11/2010 18:03] : As though swearing it will never happen is even possible to deliver? I believe that's Michael's whole point. The whole 'push directly to stable' arguement rests heavily on the principle that an update is always better (from a QA standpoint) than whatever it's replacing. The problem is that there's no way to guarantee this, essentielly because it isn't true. I believe Kevin would say his position is that the update is better than what's there already *sufficiently often* that allowing unrestricted updates is a net benefit (the question is whether an occasional bad update is a worse problem than some updates being delayed for a week or longer in the case of untested critpath updates). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sun, Nov 21, 2010 at 10:36:48PM -0600, Garrett Holmstrom wrote: On 11/21/2010 17:51, Björn Persson wrote: Andre Robatino wrote: My feeling is that it would be better for Bodhi to always require a login. Even Bugzilla does that. I suspect that a lot of people who give anonymous karma don't realize that it doesn't count, and would have created an account if they did. And using an account allows people to build a reputation, which is useful in case Bodhi is ever besieged by malicious comments and is forced to disable some accounts, or disable anonymous comments completely. Having a single account for all of Fedora including Bugzilla ought to help somewhat. Anyone who has taken the trouble of reporting a bug could then easily give karma in Bodhi. Separate accounts seems like an unnecessary obstacle to me. +1, though something tells me making it happen would take nothing short of a heroic effort. There's one easy but deeply flawed way to do this -- automatically create a usern...@fedoraproject.org bugzilla account for the user with the password used in FAS. Deeply flawed in this case because the passwords can get out of sync, we're creating accounts when people sometimes want to use a different address in bugzilla and fas, etc. We don't control RH bugzilla so changing bugzilla to be able to use fas to login would be problematic. -Toshio pgphv2IBisyQD.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 09:15:17AM -0800, Adam Williamson wrote: On Mon, 2010-11-22 at 18:09 +0100, Emmanuel Seyman wrote: The whole 'push directly to stable' arguement rests heavily on the principle that an update is always better (from a QA standpoint) than whatever it's replacing. The problem is that there's no way to guarantee this, essentielly because it isn't true. I believe Kevin would say his position is that the update is better than what's there already *sufficiently often* that allowing unrestricted updates is a net benefit (the question is whether an occasional bad update is a worse problem than some updates being delayed for a week or longer in the case of untested critpath updates). It is not some update that is delayed, but one that fixes a very bad bug like e.g. a remote code execution vulnerability. And the worst update is afaik that people had to use the command line to update instead of being able to use packagekit or kpackagekit. Regards Till pgp0acY83pfr3.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Kevin Fenzi wrote: Other concrete ideas? Quite frankly, I do not care for any of the ideas you mentioned. Here's some of my own: * Reduce quarantine time from 7 days to 3 days Reasoning: Mirror syncing. I'm not going to actively seek out and install 30 packages from koji every single day. I'd rather do a yum update and catch everything in updates-testing because those are what are needing testing. Mirrors typically need 24 hours to sync so 3 days will actually allow 2 days of testing. One day for me to install and an extra day just in case I don't update that day (weekend, etc). This just fits my needs though, so feel free to tell us why 7 days was chosen. * Allow direct-to-stable If Signed-off bodhi checkbox (web) or command option (tui) is provided by the maintainer certifying that the maintainer believes the update will work. The check should default to off, and always off, to require human intervention. If the update fails to work then the maintainer should be punished. Said punishment should be written up (yay another policy) and might consist of losing the ability to push direct-to-stable or loss of package ownership. I would prefer this option to be *retroactive* in that the old offenders (dbus, firefox, thunderbird, PackageKit, etc.) already lose their option to be d-t-s. You might put in a clause to allow a time-out period where packages could be allowed to be d-t-s again. In retrospect, having direct-to-stable be opt-out, instead of opt-in as it is now, might cool the waters for some of the more noisy maintainers. I'd be happy with this as a compromise to be a protection feature and still allow liberal Fedora updating. * Implement fedora-qa meta-package -Installs fedora-easy-karma for karma through TUI. -Installs PackageKit plugin to give karma through the gnome-packagekit GUI. (Nothing exists yet, but I'm gonna get started on one soon) -Auto-enable updates-testing. -Provide a program to generate AutoQA scripts (in the future?) This could be a checkbox in anaconda during the software selection. If we want to make QA a serious feature of Fedora it needs more exposure! Making QA easier and highly advertised would only help our cause. Leave everything else as it is. There are only a handful of complainers. Fedora, for the most part, has become update error free. Yes, there are still kinks in the process. Let's iron them out. Also, once AutoQA starts churning against our packages then more of this becomes moot. Let's not forget about it. Once we have enough manpower some of the other ideas about creating tester groups might be feasible but as it is today they would be more harmful than helpful to implement. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/22/2010 11:42 AM, Michael Cronenworth wrote: * Allow direct-to-stable If Signed-off bodhi checkbox (web) or command option (tui) is provided by the maintainer certifying that the maintainer believes the update will work. The check should default to off, and always off, to require human intervention. If the update fails to work then the maintainer should be punished. Said punishment should be written up (yay another policy) and might consist of losing the ability to push direct-to-stable or loss of package ownership. I would prefer this option to be *retroactive* in that the old offenders (dbus, firefox, thunderbird, PackageKit, etc.) already lose their option to be d-t-s. You might put in a clause to allow a time-out period where packages could be allowed to be d-t-s again. In retrospect, having direct-to-stable be opt-out, instead of opt-in as it is now, might cool the waters for some of the more noisy maintainers. I'd be happy with this as a compromise to be a protection feature and still allow liberal Fedora updating. You should clarify this. It is possible for an update to go direct to stable. If it gets sufficient karma between the time the update is submitted (when people on bugs will get a ping) and when the next releng push occurs (onceish a week dayish), the update will by-pass updates-testing and go directly to stable. For an update with an autopush karma level of 1, that could easily happen, particularly if it's an update to fix something people actively care about (read filed a bug about) It sounds like what you're asking for is the ability to have a 0 karma autopush limit. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Jesse Keating wrote: Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We/default/ the karma autopush level at +3/-3, but that's just a suggestion. Argh. Yes. Thanks for correcting me. I'm on autopilot. There's too much time to think and not enough time to work. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, 2010-11-22 at 12:02 -0800, Jesse Keating wrote: On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. we should probably change it to +2 or even +1. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Here's the latest list of ideas culled from this thread. Note: these are NOT my ideas, I am just gathering them up so fesco can discuss them. Feel free to add more concrete ideas, or let me know if I missed one you had posted. If folks could avoid me too or posts that contain no new information it would be easier for me to gather the actual ideas. ;) I've split things into General, Security, Critpath and non security/critpath to help organize them. Keep the ideas coming... General: * Just drop all the requirements/go back to before we had any updates criteria. * back off current setup until autoqa is ready, see what we want to do after that lands. * Change FN-1 to just security and major bugfix. Nothing else allowed. * allow packages with a %check section to go direct to stable. * setup a remote test env that people could use to test things. * require testing only for packages where people have signed up to be testers. * Ask maintainers to provide test cases / test cases in wiki for each package * have a way to get interested testers notified on bodhi updates for packages they care about. * reduced karma requirement on other releases when one has gone stable * aggregated karma across the releases for the same package version. * PK updates-testing integration of some kind. * allow anon karma to count. * setup fedora-qa package or group to more easily bring up more testers. Security updates: * allow security updates to go direct to stable * ask QA to commit to testing security updates * allow timeout for security updates before going to stable. Critpath updates: * allow critpath timeout for going to stable. Non critpath/security: * reduce timeout for non critpath from 7 to 3 days. * change default autokarma to 2 or 1. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Adam Williamson awill...@redhat.com writes: On Mon, 2010-11-22 at 12:02 -0800, Jesse Keating wrote: Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. we should probably change it to +2 or even +1. Yeah. Or at least document that the maintainer is allowed to reduce it; that's entirely unclear given the current policy and bodhi UI. As things stand, there is a clear expectation embodied in the default behavior that you should be able to get +3 karma. The fundamental complaint (at least as I've been making it) is that that's unrealistic given the actual available testing manpower. It's not undesirable; it'd be great if that actually did happen. But it's just not going to happen, most of the time for most packages. (And yeah, if we allow +1 autopush, we should definitely expect -1 is sufficient to unpush. Maybe bodhi should restrict the combination of those two settings, rather than either one alone?) regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Toshio Kuratomi wrote: On Sun, Nov 21, 2010 at 10:36:48PM -0600, Garrett Holmstrom wrote: On 11/21/2010 17:51, Björn Persson wrote: Andre Robatino wrote: My feeling is that it would be better for Bodhi to always require a login. Even Bugzilla does that. I suspect that a lot of people who give anonymous karma don't realize that it doesn't count, and would have created an account if they did. And using an account allows people to build a reputation, which is useful in case Bodhi is ever besieged by malicious comments and is forced to disable some accounts, or disable anonymous comments completely. Having a single account for all of Fedora including Bugzilla ought to help somewhat. Anyone who has taken the trouble of reporting a bug could then easily give karma in Bodhi. Separate accounts seems like an unnecessary obstacle to me. +1, though something tells me making it happen would take nothing short of a heroic effort. There's one easy but deeply flawed way to do this -- automatically create a usern...@fedoraproject.org bugzilla account for the user with the password used in FAS. Deeply flawed in this case because the passwords can get out of sync, we're creating accounts when people sometimes want to use a different address in bugzilla and fas, etc. Hmm, but it says here that we should use the same address in Bugzilla and FAS: https://fedoraproject.org/wiki/Join_the_package_collection_maintainers We don't control RH bugzilla so changing bugzilla to be able to use fas to login would be problematic. That solution seems backwards anyway. I think most new contributors probably report bugs long before they become packagers, so they need Bugzilla accounts before they need FAS accounts. How about changing Bodhi to allow login with either a FAS account or a Bugzilla account? Would that be difficult to do? Björn Persson signature.asc Description: This is a digitally signed message part. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 12:02:49PM -0800, Jesse Keating wrote: On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. Afaik there is currently no distinction between the requirement for non crit-path updates and the karma autopush level. Therefore by default non critpath updates require +3 karma, unless this has been changed since the beginning for the update criteria enforcement. Regards Till pgpBsS8Oztt2X.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, 2010-11-22 at 16:39 -0500, Tom Lane wrote: Adam Williamson awill...@redhat.com writes: On Mon, 2010-11-22 at 12:02 -0800, Jesse Keating wrote: Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. we should probably change it to +2 or even +1. Yeah. Or at least document that the maintainer is allowed to reduce it; that's entirely unclear given the current policy and bodhi UI. The bodhi 2.0 solution is to decouple autopush and acceptance: they really shouldn't be paired, I think it's just an unintended consequence of the current code. You should be able to set +1 threshold for acceptance allowing the maintainer to then push manually, and +3 for autopush, for instance. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/22/10 1:50 PM, Till Maas wrote: On Mon, Nov 22, 2010 at 12:02:49PM -0800, Jesse Keating wrote: On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. Afaik there is currently no distinction between the requirement for non crit-path updates and the karma autopush level. Therefore by default non critpath updates require +3 karma, unless this has been changed since the beginning for the update criteria enforcement. Regards Till I swear I've been able to set karma levels at 1 for non-critpath updates. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 7:01 AM, Kevin Kofler kevin.kof...@chello.at wrote: Michal Hlavinka wrote: this could help, but it's not always possible to add these test cases. One example: imap server package - new bug that can corrupt mail folders in some circumstances. Maintainer updates package and sets 'type=bugfix' in bodhi, but not always it's possible to write down any test case. It's still a bug fix and it's better to be delivered sooner than wait if anyone out there get's his mail folders corrupted. Actual policy does not help proactivity. Right, and the big point there should be that a bug which can corrupt mail folders should be fixed IMMEDIATELY, i.e. with a direct stable push! ANY testing requirement there is a failure. Install package from updates-testing, then +1 to karma after it works for you with your tests and normal workload. Simple. No need to foist possibly broken software on everyone. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 9:04 AM, Till Maas opensou...@till.name wrote: On Mon, Nov 22, 2010 at 09:57:40AM +0100, Henrik Nordström wrote: sön 2010-11-21 klockan 11:00 +0100 skrev Till Maas: I guess this can be somehow automated. E.g. change Bodhi to drop the karma requirements for packages that had e.g. two subsequent updates without any Bodhi feedback and re-enable it if they get feedback. That would be somewhat counter productive for the goal of actually having testers, as it discourages maintainers looking for testers as a way to improve their release process. I do not see how this discourages maintainers. I do not know of any package maintainer that stated that he did not want his updates tested. This would at least encourage people that want better tested updates to commit into testing them. The current problem is not that package maintainers do not want their packages tested, but that updates are delayed without any visible testing. The result is that if you have several updates without any karma points (positive or negative) in bohdi, your updates get out faster. Once someone does start giving it karma (again, positive or negative) your update is not subject to getting more karma points in order to hit the updates repo. Thus you punish the maintainer by giving testing their packages. It may work better if karma enforcement only comes into effect if negative karma is given to an update. This still builds a reactive system instead of a preventative system. An only reactive system will not help prevent bad updates from getting out in the first place. That said, adding a reactive component to a preventative system would be a good thing. If a maintainer releases one package that puts regressions into the stable updates repo, then the minimum karma doubles on all of their packages doubles for 2 months or something like that. So feel free to push directly to stable as often as you want, but once you introduce one regression, you have to satisfy 10 karma on every package you update. The second time, you have to satisfy 20 karma on every package you update and so on. Simply because we can't trust that maintainer anymore. Really, allowing regressions to make it to stable is so costly simply because it has to be fixed several magnitudes more times than if it is caught by people actually testing packages before they're released to the masses. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 02:33:35PM -0800, Jesse Keating wrote: On 11/22/10 1:50 PM, Till Maas wrote: On Mon, Nov 22, 2010 at 12:02:49PM -0800, Jesse Keating wrote: On 11/22/2010 11:56 AM, Michael Cronenworth wrote: It was my understanding of reading the complaints that this is what they [complainers] desire - a reversal of what we require now (3 karma and/or proventester if critpath). Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. Afaik there is currently no distinction between the requirement for non crit-path updates and the karma autopush level. Therefore by default non critpath updates require +3 karma, unless this has been changed since the beginning for the update criteria enforcement. I swear I've been able to set karma levels at 1 for non-critpath updates. But this means that the update is automatically pushed to stable, not that the update is approved and then pushed to stable when the maintainer requests it. Regards Till pgprHIZN40iHK.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Mon, Nov 22, 2010 at 02:18:45PM -0800, Adam Williamson wrote: On Mon, 2010-11-22 at 16:39 -0500, Tom Lane wrote: Adam Williamson awill...@redhat.com writes: On Mon, 2010-11-22 at 12:02 -0800, Jesse Keating wrote: Critpath requires +1 proven tester and +1 anybody. Total of +2. Non crit-path requires a minimum of +1 anybody or a 7 day timeout I do believe. I do not believe we require +3 anywhere. We /default/ the karma autopush level at +3/-3, but that's just a suggestion. we should probably change it to +2 or even +1. Yeah. Or at least document that the maintainer is allowed to reduce it; that's entirely unclear given the current policy and bodhi UI. The bodhi 2.0 solution is to decouple autopush and acceptance: they really shouldn't be paired, I think it's just an unintended consequence of the current code. You should be able to set +1 threshold for acceptance allowing the maintainer to then push manually, and +3 for autopush, for instance. Afaik there is no need for a maintainer to set different acceptance thresholds for his updates. At least nobody ever explained to me why this would be helpful. Regards Till pgpcaHpugZR52.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sat, Nov 20, 2010 at 03:35:43PM -0700, Kevin Fenzi wrote: * require testing only for packages where people have signed up to be testers Packages without 'official' testers could bypass testing or have some lower karma requirement. We would need for this a list of packages that have had people sign up to test. I guess this can be somehow automated. E.g. change Bodhi to drop the karma requirements for packages that had e.g. two subsequent updates without any Bodhi feedback and re-enable it if they get feedback. * Ask maintainers to provide test cases / test cases in wiki for each package? Test cases are not easy to make, many maintainers won't can't do so, but it would be lovely to have even a base checklist of things that should work in the package everytime. Afaik, most of the packages won't be tested and therefore the test cases won't be read. So this should be only a requirement if there are testers for a certain package. * reduced karma requirement on other releases when one has gone stable Bodhi would need to note when a update went to stable if the exact same version (with dist tag differences) was in testing for other releases. It could then allow less karma to go stable, or add +1 from the other update going stable. Other concrete ideas? All of this could be combined. E.g. packages with enough testers get test cases and need to fulfill stronger criteria. Packages with not so many testers get test cases and only need to fulfil that similar updates need to receive good karma on one Fedora release. Off course more automated testing is missing instead of all the manual requirements. Another idea would be to add a flag in bodhi to track whether a certain testing update has been installed and then have a tool to report this automatically back to Bodhi. Then there would be some information about silent testers, which can be used as a criterion, too. Also it could be made easier for maintainers to identify problematic updates, e.g. by warning that the dependencies or provides of an update changed when the update is created. Regards Till pgpz8opfnZ2go.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Till Maas wrote: All of this could be combined. E.g. packages with enough testers get test cases and need to fulfill stronger criteria. Packages with not so many testers get test cases and only need to fulfil that similar updates need to receive good karma on one Fedora release. Off course more automated testing is missing instead of all the manual requirements. Another idea would be to add a flag in bodhi to track whether a certain testing update has been installed and then have a tool to report this automatically back to Bodhi. Then there would be some information about silent testers, which can be used as a criterion, too. All this stuff would really complicate Bodhi a lot, and make it even more bug-prone than it already is. Why do we need to code this in software? Why can't we just make use of the wonderful deduction machine called a brain, which every maintainer SHOULD have? Also it could be made easier for maintainers to identify problematic updates, e.g. by warning that the dependencies or provides of an update changed when the update is created. That's something useful, and in fact AutoQA is supposed to do that (and we're still waiting for that!). Doing repetitive consistency checks and warning the maintainer about potential or definite problems is what software is good at. Taking decisions is not! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sat, 20 Nov 2010 15:35:43 -0700, Kevin wrote: Other concrete ideas? As a beginning, let's limit this thread to at most one message per person per day. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sat, Nov 20, 2010 at 03:35:43PM -0700, Kevin Fenzi wrote: ok, I dug through the devel list for the last month or two and wrote down all the various ideas folks have come up with to change/improve things. Here (in no particular order) are the ideas and some notes from me on how we could enable them. Please feel free to add new (actual/concrete ideas or notes): * Just drop all the requirements/go back to before we had any updates criteria. * Change FN-1 to just security and major bugfix This may be hard to enforce or figure out if something is a major bugfix. * allow packages with a %check section to go direct to stable Bodhi would have to have a list or some way to note these packages, it would also need to change as they were added/removed. Perhaps it could just be an AutoQA +1 for having a check section? On the con side, some checks may be simple and may not note things that are fedora only issues. * setup a remote test env that people could use to test things. This would be lovely with some kind of cloud setup. Have a set of base kickstart files and a package/tester could request an instance, install the update, test, and then the instance would be removed. We would need a timelimit of some kind, and not sure there is any HW in infrastructure for this, but it would sure be cool. ;) Perhaps working with the cloud sig we could use EC2 instances for this? * require testing only for packages where people have signed up to be testers Packages without 'official' testers could bypass testing or have some lower karma requirement. We would need for this a list of packages that have had people sign up to test. * Ask maintainers to provide test cases / test cases in wiki for each package? Test cases are not easy to make, many maintainers won't can't do so, but it would be lovely to have even a base checklist of things that should work in the package everytime. * have a way to get interested testers notified on bodhi updates for packages they care about. We would need to add some kind of tester list to pkgdb, and bodhi would need to be able to get this to mail them when a update changed state. We may not get many people signing up for some packages, but this might be a good way to know what packages we have testers for and get them more involved in testing. Ideally it could mail them on update submission at least. * reduced karma requirement on other releases when one has gone stable Bodhi would need to note when a update went to stable if the exact same version (with dist tag differences) was in testing for other releases. It could then allow less karma to go stable, or add +1 from the other update going stable. Other concrete ideas? Security update ideas: * Security updates may push directly to stable -- this would depend on our weighing of costs: a security update may be broken. OTOH, if we don't get security updates out fast, is that a worse danger to our users? I think I'd like to try one of the other options below first but those require a bit more work/coordination so we need to think of the cost to implement as well. * Ask the QA team to commit to testing security updates. The idea is that updates flagged security have a dedicated pool of testers that will check them and get them out ASAP. * Security updates have a small timeout period -- there's two questions here: Is a week (as is currently the case for non-critpath) too long? Can wehave a timeout even for critpath packages so a security update doesn't get held up forever? Critpath ideas: * critpath package timeout -- let's have a timeout period even for critpath packages. Perhaps this could be longer than for non-critpath. The big issue that people have observed with depending on timeouts, though, is that pushing new updates in the meantime resets the time that a package needs to wait to get to stable. Could bodhi be changed to let multiple packages be in the testing repository at one time and only obsolete them when a newer package enters the stable repo? That would allow longer timeout periods to make more sense. Lack of manpower ideas: * Allow anonymous karma to count. Anonymous karma would allow more people who report bugs in bugzilla to add karma in bodhi without having to get a second account in the Fedora Account System. For critpath packages, we're already mandating that a proventester must give karma so we're making sure that someone with an account is giving feedback there. * Do we believe that there's value in silent testing (knowing that people have a package installed from testing but have not submitted karma either way?) If so, create a tool that ties into yumdb and if the user opts in, submits that data to us. Then add karma based upon that data. Variation on an option you recorded: * Drop testing requirement until AutoQA can give karma. Rework based on
Re: Updates Criteria Summary/Brainstorming
Toshio Kuratomi a.badger at gmail.com writes: Lack of manpower ideas: * Allow anonymous karma to count. Anonymous karma would allow more people who report bugs in bugzilla to add karma in bodhi without having to get a second account in the Fedora Account System. For critpath packages, we're already mandating that a proventester must give karma so we're making sure that someone with an account is giving feedback there. My feeling is that it would be better for Bodhi to always require a login. Even Bugzilla does that. I suspect that a lot of people who give anonymous karma don't realize that it doesn't count, and would have created an account if they did. And using an account allows people to build a reputation, which is useful in case Bodhi is ever besieged by malicious comments and is forced to disable some accounts, or disable anonymous comments completely. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Toshio Kuratomi wrote: packages. Perhaps this could be longer than for non-critpath. The big issue that people have observed with depending on timeouts, though, is that pushing new updates in the meantime resets the time that a package needs to wait to get to stable. Could bodhi be changed to let multiple packages be in the testing repository at one time and only obsolete them when a newer package enters the stable repo? That would allow longer timeout periods to make more sense. The problem with that is that only one of the updates is going to be actually tested at a time. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/21/10 11:00 AM, Toshio Kuratomi wrote: meantime resets the time that a package needs to wait to get to stable. Could bodhi be changed to let multiple packages be in the testing repository at one time and only obsolete them when a newer package enters the stable repo? That would allow longer timeout periods to make more sense. It would be difficult client side to select which set of packages to update to, generally people will just get the latest set, not the earlier set. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
Andre Robatino wrote: My feeling is that it would be better for Bodhi to always require a login. Even Bugzilla does that. I suspect that a lot of people who give anonymous karma don't realize that it doesn't count, and would have created an account if they did. And using an account allows people to build a reputation, which is useful in case Bodhi is ever besieged by malicious comments and is forced to disable some accounts, or disable anonymous comments completely. Having a single account for all of Fedora including Bugzilla ought to help somewhat. Anyone who has taken the trouble of reporting a bug could then easily give karma in Bodhi. Separate accounts seems like an unnecessary obstacle to me. Björn Persson signature.asc Description: This is a digitally signed message part. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On 11/21/2010 17:51, Björn Persson wrote: Andre Robatino wrote: My feeling is that it would be better for Bodhi to always require a login. Even Bugzilla does that. I suspect that a lot of people who give anonymous karma don't realize that it doesn't count, and would have created an account if they did. And using an account allows people to build a reputation, which is useful in case Bodhi is ever besieged by malicious comments and is forced to disable some accounts, or disable anonymous comments completely. Having a single account for all of Fedora including Bugzilla ought to help somewhat. Anyone who has taken the trouble of reporting a bug could then easily give karma in Bodhi. Separate accounts seems like an unnecessary obstacle to me. +1, though something tells me making it happen would take nothing short of a heroic effort. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sun, 21 Nov 2010 12:42:00 +0100 Michael Schwendt mschwe...@gmail.com wrote: On Sat, 20 Nov 2010 15:35:43 -0700, Kevin wrote: Other concrete ideas? As a beginning, let's limit this thread to at most one message per person per day. That would be lovely. I guess this would be my sunday message. ;) I was intending this thread to enumerate specific ideas or actions people would like and then can take them to fesco. Argument by repetition will make it harder to gather the real proposals from the thread. It would be nice if folks would refrain from doing so. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Updates Criteria Summary/Brainstorming
On Sat, Nov 20, 2010 at 11:35 PM, Kevin Fenzi ke...@scrye.com wrote: ok, I dug through the devel list for the last month or two and wrote down all the various ideas folks have come up with to change/improve things. Here (in no particular order) are the ideas and some notes from me on how we could enable them. Please feel free to add new (actual/concrete ideas or notes): * Just drop all the requirements/go back to before we had any updates criteria. Before we go that route, maybe we should understand why we decided on the updates criteria. If I remember correctly, this was decided because some updates introduced regressions in stable Fedora releases, upsetting users. Do we really want to go back? * Change FN-1 to just security and major bugfix This may be hard to enforce or figure out if something is a major bugfix. My vote would be security-only, because people desiring the latest upstream code apparently upgrade to the latest Fedora release as soon as it's baked (and sometimes earlier), therefore there is no need to upgrade the packages in n-1. And the main reason why a user doesn't upgrade his box seems rooted in the wish to avoid unnecessary downtime or extra work - after all, they can skip a release since n-1 is supported. So these users want *stability*, not anything disruptive, and it's OK if the software they use is a bit older than what is available in the latest Fedora. The latest software is only an upgrade away after all. Any major non-security bugfix needed in n-1 wasn't detected for 6 months or so. How major is that? Or maybe it was introduced by an update and not caught early enough because we couldn't test the update. * allow packages with a %check section to go direct to stable Bodhi would have to have a list or some way to note these packages, it would also need to change as they were added/removed. Perhaps it could just be an AutoQA +1 for having a check section? On the con side, some checks may be simple and may not note things that are fedora only issues. More test automation would be an excellent way to validate updates. It's not applicable everywhere, but it should help a lot. * require testing only for packages where people have signed up to be testers Packages without 'official' testers could bypass testing or have some lower karma requirement. We would need for this a list of packages that have had people sign up to test. I'm not too keen on that one, although I don't see a way out. I don't have a big enough network at home to extensively test Nagios or 389-DS, for instance. Sanity tests of these are possible, but I won't easily (with my current resources) catch performance regressions or some kind of instability under load. * Ask maintainers to provide test cases / test cases in wiki for each package? Test cases are not easy to make, many maintainers won't can't do so, but it would be lovely to have even a base checklist of things that should work in the package everytime. Especially since test cases could form the basis of automated tests later on. And I'm pretty sure any time spent doing test automation or writing test cases would be worth it down the road, as we'd have more confidence in subsequent updates. * have a way to get interested testers notified on bodhi updates for packages they care about. We would need to add some kind of tester list to pkgdb, and bodhi would need to be able to get this to mail them when a update changed state. We may not get many people signing up for some packages, but this might be a good way to know what packages we have testers for and get them more involved in testing. Ideally it could mail them on update submission at least. * reduced karma requirement on other releases when one has gone stable Bodhi would need to note when a update went to stable if the exact same version (with dist tag differences) was in testing for other releases. It could then allow less karma to go stable, or add +1 from the other update going stable. This sounds good on paper, yet we'll have to remember it cannot eliminate testing altogether due to packages using shared libraries that might not be of the same version across releases. Still, given we're short-handed, that's a good solution. Other concrete ideas? These are all technical ideas. We need to know what experience we want to provide users with in both Fedora n and n-1 before deciding which technical idea(s) to implement. My own vision is that: * updates should not be less tested than what's in an actual release - and the QA/RE teams do an amazing amount of testing prior to releases. I would love to see more maintainers at the Go/NoGo meetings for instance. * updates should not disrupt user experience. Fedora, being community-supported, is probably unsuitable for many business tasks. Let's not invalidate Fedora for less technically minded people. * Fedora releases do have bugs. After any release, we should strive to lower the total bug count,
Re: Updates Criteria Summary/Brainstorming
Kevin Fenzi wrote: * Just drop all the requirements/go back to before we had any updates criteria. That's really the only way to go. The policy failed, it's time to withdraw it. All the other proposed solutions require even more complexity in the software and policies, for little to no gain. * Change FN-1 to just security and major bugfix This may be hard to enforce or figure out if something is a major bugfix. Indeed, this obviously doesn't work. * allow packages with a %check section to go direct to stable %check echo 'Success!' Is that OK? If not, what is? How much testing do you want to require? And most importantly, it doesn't solve the problems for those many packages for which automated testing is not feasible and/or not useful. * setup a remote test env that people could use to test things. That doesn't solve the time issue, only the I don't have a Fedora n environment issue, and not everything can be tested properly in such a setup. (Hardware-specific issues, latency-critical software etc.) * require testing only for packages where people have signed up to be testers The maintainers know best whether their packages have sufficient testers or not, just let them decide on how much feedback to wait for before going stable! A boolean is not sufficient to accurately describe the situation, e.g. requiring a karma of 5 may make sense for something like the kernel, but not for a package with only 4 testers in total, and also the testers available for a given Fedora release matter (a number which changes over time, and you can't really rely on testers updating their data each time they upgrade their system(s)). * Ask maintainers to provide test cases / test cases in wiki for each package? There are many packages where that's just not feasible. (Good luck trying to provide an exhaustive set of test cases for e.g. kdebase-workspace!) It's also a lot of extra work for the maintainers. * have a way to get interested testers notified on bodhi updates for packages they care about. That doesn't solve the problem of there not being interested testers in the first place. * reduced karma requirement on other releases when one has gone stable In principle, that makes sense. It might solve part of the issues if the reduced karma requirement is zero. (Otherwise it's just useless, since we can already set it to 1.) But you'd have to allow the maintainer to tell Bodhi what 2 updates are the same. Same EVR minus disttags as you propose has both false positives and false negatives. And why not avoid all this complexity by just always letting the maintainer decide? They know best how much value to attribute to feedback from identical or similar updates for other releases in the specific case at hand. In short, my proposal: 1. discontinue the current update acceptance enforcement (in particular, reenable direct stable pushes, and of course allow pushes from testing to stable at any moment at the maintainer's discretion), 2. drop the aggregated numeric karma score, which is devoid of any actual significance, and the autokarma (mis)feature that goes with it (keep only the +1/0/-1 emoticons on the individual comments), 3. write some recommendations which should GUIDE the maintainers on how to handle updates, but NOT FORCE anything on them (experienced maintainers follow many unwritten rules, writing them down can certainly help guiding less experienced maintainers towards doing the right thing), 4. TRUST maintainers to make the right decisions on when an update is stable enough to be pushed, also considering the impact of NOT pushing the update immediately (which has worked very well, despite some claims to the contrary based on isolated incidents blown way out of proportion). Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
