Re: Follow up from QA meeting on IRC
On Thu, Nov 12, 2009 at 03:06, Jeremiah Foster wrote: > > This email is a set of action points that were brought up during > the QA meeting on IRC which Valerio suggested. I felt the meeting was > quite productive, we have some more things to discuss and we have some > 'actionable items' as it were. To join up the community, there's now a thread on tmo about this: http://talk.maemo.org/showthread.php?t=41179 Sorry to resurrect an old thread. Cheers, Andrew -- Andrew Flegg -- mailto:and...@bleb.org | http://www.bleb.org/ ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
On Nov 12, 2009, at 3:17 AM, Dave Neary wrote: > So - how can we make giving the feedback and voting on applications > easier? Oh, this one's easy. Get some servers so rating a package isn't either impossible or a 20-minute endeavor. Even motivated testers like myself don't have much interest in getting on the website to vote when submitting one takes 3 attempts and 10 minutes of your time. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
Jeremiah Foster wrote: > Developers should not be able to promote their own applications, it > is assumed that they believe the app is production ready by > submitting it to testing. Perhaps if there were a different button > for developers, i.e. 'Demote this package', then the app can be > removed from testing? If a user finds a bug, and the dev wants to > take that version out of the repos, a developer can remove the app > entirely. As an addition, I would like to see a button to remove the app altogether, i.e. also from devel. It happened to me that I reported a bug on an application that was removed from testing as requested by its maintainer but not from devel. Another idea that comes to my mind is the possibility of subscribing to package events: build (ok, failed, etc.), entering devel/testing/extras, demotion and so on. For example, I'd like to be notified when some of my favourite packages move into testing so that I can retest them. A. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
On Nov 12, 2009, at 9:17, Dave Neary wrote: > Hi, > > Edward Page wrote: >> To help remind people that there is a checklist and whats on it, >> should the rating page link to or include the criteria? Yes - that makes sense. >> >> I see there were no notes on the algorithm. A threshold of 10 was >> annoying as a developer. As a tester, a threshold of 10 made me feel >> more comfortable not doing a full blown /opt check or power management >> check because of 10 people I could hope someone else would do it and I >> could worry about other issues like application stability. With a >> smaller threshold I would feel more of a burden to do all of the steps >> which would discourage me. > > Ed's point definitely resonates with me. The great thing about QA is > that you can crowd-source it effectively if you don't require much of > the user/tester. It seems like the Maemo QA process is more > developer-focussed than user-focussed at the moment, and is as such > pushing a lot of the responsibility for the QA process to the user. QA is logically the next step after development and is often done by an engineer, at least in the corporate world. Even in maemo, those who do QA so far have also been developers; either they have done QA on their own package or they have done it informally for other's and use the extras-testing interface. In short - the expectation is that you are pretty deep into maemo. > > This seems like an ideal opportunity to lower the barrier to > participation to tiny levels, but only if it is > > 1. easy to give a +1/-1 Well, we need to keep this distinct from merely rating the package. We are not rating here, we are promoting quality packages. You may be asked to promote a package you think frivolous, but if it meets the technical criteria you are asked to promote it. > 2. We don't require intimate knowledge of the Maemo community for > feedback (I'm thinking of the checklist, what "optification" means, etc) Feedback is welcome through a variety of means - rating the package with stars on the download page is one for example. Feedback is welcome in QA too, but let's not pretend that this isn't a serious technical process requiring at least some knowledge of the underlying operating system and perhaps the package system and even how the development process works (on a high level). We need to insure that those who are doing QA, can handle the technical demands so that the software does not damage the device and maemo's reputation. This really does require a clear specification, i.e. checklist, and knowing what optification means in the maemo context. > 3. We require enough feedback that most of the code paths in the > application will be tested before we OK an application > > Lowering the threshold to 5 is implicitly saying "we're not getting > feedback quickly enough", I think it is really saying that the bar is currently set to high. > which in turn is saying "the feedback process > is overly cumbersome for a casual user". Yes and it should be. This is critically important and has serious implications and technical challenges. Anyone at all is welcome, but you may need to learn some stuff about the QA process. The QA process is hard because developing is hard, writing code is hard, integrating into an OS is hard - we need to make sure those things work. > It seems to me that that's the > problem, rather than the contents of the checklist or the threshold in > place. If giving feedback was trivially easy (as it is, for example, in > Android Market) you would be getting hundreds of votes when new versions > of applications are released, as people installed & used them. We need a separate mechanism for this then, perhaps this is similar to the app rating system we already have on downloads? > > So - how can we make giving the feedback and voting on applications easier? I don't think the idea of "Quality Software" fits with the idea of "Easy to do". Exhibit A: Windows. Many developers are familiar with this saying said to management only half in jest: Quality, Time, Cost - pick two. Seriously, we don't want people surfing by and clicking pejoratively. We need a set of serious technical criteria to assure quality software, not a warm and fuzzy interface where the hoi polloi randomly click shiny buttons. Jereamiah ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: popularity-contest (was Re: Follow up from QA meeting on IRC)
On Nov 12, 2009, at 6:15, Quim Gil wrote: > Hi, > > ext Edward Page wrote: >> Which reminds me, any reason Maemo doesn't use Debian's popularity contest? > > At least at a community level there is > https://garage.maemo.org/projects/popcon/ Bas (the founder of that project) and I have been discussing how to implement this. debian's popularity contest is definitely going to be used at the very least as inspiration, but hopefully its code as well. Right now it spends a lot of time trying to determine the last time a tool was used and that may be more relevant to a server environment than a mobile device. What I had hoped to do was to 1. Create a md5 or sha1 hash of the device's MAC addres and IMEI as unique identifier which cannot be tracked back 2. Check which repos the device is using, to see where the software comes from 3. Ask dpkg what is installed 4. Aggregate the dpkg output and rank that, roughly like popcon but perhaps doing some other cool things I want to preserve as much privacy as possible on the device so anyone who participates is not identified. I know that I can only obfuscate data, not hide it completely, so I strongly feel this project should come with the advice that you may not want to use it. It also should not be a Nokia project because it is bad if Nokia is gathering information surreptitiously on its users. So let me be clear: this is a community project that is completely voluntary! Jeremiah ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
Hi, Edward Page wrote: > To help remind people that there is a checklist and whats on it, > should the rating page link to or include the criteria? > > I see there were no notes on the algorithm. A threshold of 10 was > annoying as a developer. As a tester, a threshold of 10 made me feel > more comfortable not doing a full blown /opt check or power management > check because of 10 people I could hope someone else would do it and I > could worry about other issues like application stability. With a > smaller threshold I would feel more of a burden to do all of the steps > which would discourage me. > > So I guess I'll share my idea. To me, it seems that one tester would > probably be enough for /opt, power management, etc. If the categories > were broken out, these could just require a net of +1 karma with a > required comment to describe steps and results regardless of whether > they gave an up or down. Net +1 is in case others disagree, they can > vote it down. Required comments either way are to make people feel > comfortable that it was tested properly and not just someone saying > "it works for me" and voting it up. Ed's point definitely resonates with me. The great thing about QA is that you can crowd-source it effectively if you don't require much of the user/tester. It seems like the Maemo QA process is more developer-focussed than user-focussed at the moment, and is as such pushing a lot of the responsibility for the QA process to the user. This seems like an ideal opportunity to lower the barrier to participation to tiny levels, but only if it is 1. easy to give a +1/-1 2. We don't require intimate knowledge of the Maemo community for feedback (I'm thinking of the checklist, what "optification" means, etc) 3. We require enough feedback that most of the code paths in the application will be tested before we OK an application Lowering the threshold to 5 is implicitly saying "we're not getting feedback quickly enough", which in turn is saying "the feedback process is overly cumbersome for a casual user". It seems to me that that's the problem, rather than the contents of the checklist or the threshold in place. If giving feedback was trivially easy (as it is, for example, in Android Market) you would be getting hundreds of votes when new versions of applications are released, as people installed & used them. So - how can we make giving the feedback and voting on applications easier? Cheers, Dave. -- maemo.org docsmaster Email: dne...@maemo.org Jabber: bo...@jabber.org ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
Hello! That all sounds OK. One other point that just came into my mind. Is it possible (I havn't yet promoted something) to leave some message to the testers while promoting application to extras-testing (or even leave permantent comments regarding testing as part of the application description)? Reason with concrete example: One of the testing requirements is that more than normal power consuming applications should give a hint at first start. In this case tester need to know how first start is detected and how the application can be made to think it was first started. In my case I would realize this with a hidden configuration file (~/.blabla.xml) that has to be deleted. This must be known by the tester. -- Gruß... Tim ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
popularity-contest (was Re: Follow up from QA meeting on IRC)
Hi, ext Edward Page wrote: > Which reminds me, any reason Maemo doesn't use Debian's popularity contest? At least at a community level there is https://garage.maemo.org/projects/popcon/ -- Quim Gil open source advocate Maemo Devices @ Nokia ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Follow up from QA meeting on IRC
The results of the meeting all sound good. As part of the UX work, should we include the application ratings as part of the testing interface? We just had someone step through random areas of an app, it seems we should leverage this and remind them to rate it. Which reminds me, any reason Maemo doesn't use Debian's popularity contest? To help remind people that there is a checklist and whats on it, should the rating page link to or include the criteria? I see there were no notes on the algorithm. A threshold of 10 was annoying as a developer. As a tester, a threshold of 10 made me feel more comfortable not doing a full blown /opt check or power management check because of 10 people I could hope someone else would do it and I could worry about other issues like application stability. With a smaller threshold I would feel more of a burden to do all of the steps which would discourage me. So I guess I'll share my idea. To me, it seems that one tester would probably be enough for /opt, power management, etc. If the categories were broken out, these could just require a net of +1 karma with a required comment to describe steps and results regardless of whether they gave an up or down. Net +1 is in case others disagree, they can vote it down. Required comments either way are to make people feel comfortable that it was tested properly and not just someone saying "it works for me" and voting it up. Also, for most apps, /opt and power management are less likely to change from release to release. A package's net-karma in those categories could carry over with the attached comment being that it carried over from release X.Y. If a tester feels highly motivated or feel its been too long since these have been tested and they find an issue, their single -1 will block promotion. Ed Page (epage) ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Follow up from QA meeting on IRC
Hello! This email is a set of action points that were brought up during the QA meeting on IRC which Valerio suggested. I felt the meeting was quite productive, we have some more things to discuss and we have some 'actionable items' as it were. To begin, there is a general feeling that the QA process requires those who test apps to follow the checklist more like a checklist and less as a set of guidelines. If a user or tester has an opinion about an app, that is what the application ratings are for in the download section. The QA process is more specific. ITEM: Testers SHOULD closely follow checklist It was suggested that the promotion threshold is too high at the moment and should be reduced by half. ITEM: Promotion SHOULD occur at +5 karma Next, the interface needs some lovin', it shows a lot of info so perhaps "it could also use the hand of a designer to make sure that any useful, but possibly extranseous, information is hidden and the most important is emphasised." Perhaps sections of the interface could be 'collapsable'? ITEM: UX designer streamline interface It was felt that if a tester gives a thumbs down, they ought to leave a comment. This perhaps should be made mandatory. ITEM: Testers MUST comment on thumbs down We roughly outlined what we imagined to be a good algorithm for testing: The tester logs in, follows the checklist, install an application via .install file, tests the app, votes and comments. Install files would be available only if one was logged in, preventing drive-by downloads of potentially buggy software. Developers should not be able to promote their own applications, it is assumed that they believe the app is production ready by submitting it to testing. Perhaps if there were a different button for developers, i.e. 'Demote this package', then the app can be removed from testing? If a user finds a bug, and the dev wants to take that version out of the repos, a developer can remove the app entirely. ITEM: Developers MUST not vote up their own apps ITEM: Developer interface SHOULD have the ability to 'demote' app It was also requested that votes can be changed. If there are made permanent, then there can be mistakes. ITEM: Votes SHOULD be changeable When packaging a library, it should not be considered in the QA interface unless it has an application as well. A developer "should upload a pretty meta-package with an icon and a human-understandable description as to why they should install them." ITEM: Libraries MUST have applications or meta packages to go through Extras-testing Other topics to various and splendid to mention were also mentioned in the meeting, of note are two: Personal package repositories in garage were discussed but they idea did not get traction. At this point they are not seen as useful. Perhaps Daniel Wilms would be willing to incorporate a thumbs up / thumbs down icon to rate packages in the HAM-like Qt app he is reportedly writing? So, there you have it, please opine. :) Jeremiah ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
