Re: Verification Culture
On Sunday, August 12, 2012 5:51:43 AM UTC-7, Robert Kaiser wrote: Jason Smith schrieb: Note - I still think it's useful for a QA driver to look through a set of bugs fixed for a certain Firefox release, it's just the process would be re-purposed for flagging a bug for needing more extensive testing for X purpose (e.g. web compatibility). I think QA should do some exploratory testing of major new features as time allows, but just verifying existing test cases that often are running automatically anyhow isn't a good use of time, I guess. Robert Kaiser We do exploratory as well as test-cased testing for all features scoped for a given release. The real gap in our existing strategy lies with mostly doing blind verification of fixed bugs with the status flag set. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
On 2012-08-10 20:41:30 +, Anthony Hughes said: I, for one, support this idea in the hypothetical form. I'd like to get various peoples' perspectives on this issue (not just QA). Like Robert says elsewhere, manually running a testcase that's already in automation doesn't make a huge amount of sense. I think running a manual verification that isn't in automation does make some amount of sense. It's also probably the first thing you want to do before doing additional planning around the verification. So my take is this: Hypothetically, we have 300 bugs. Right now, we pick out 200 we think are testable in time allotted, spend all of our allotted time on them, get maybe 150 done. Instead I'd define (formally or otherwise) three tiers: 1) Critical fixes. These need verification + additional testing. 2) Untested uncritical fixes. These have no automated tests. These should get verification if time allows. 3) Tested critical fixes: These have automated tests and do not need verification. (There's an invisible fourth tier: bugs that we can't really test around because they're too internal. But those are the 100 we triaged out above.) In our hypothetical case, what that means is that of the 200 we decided were testable, maybe 20 become tier 1. Give them whatever time is needed to do a short but comprehensive test plan around them. Then give the balance of the time to tier 2. But don't block time within release for tier 2. If tier 1 takes everything, so be it. Tier 3 should be ignored. They're already being tested to the point we care about for that release. Re: necessity of verification workflow, Verifications are important. I've seen way too many fixes go in across my career that didn't really fix the bug to think that we should take the workflow out completely, and I would never call them blind if they're against a valid testcase. They might be naive, they might be shallow, but they aren't blind. That's a misnomer. The mistake is in prioritizing them above primary testing, and in binding them to a time deadline such that we prioritize them that way. Closing bugs is part of good bug maintenance. It's nice to know for sure that you don't have to look at it ever again and, unfortunately, resolved fixed doesn't mean that. But it's not important that you know that -immediately- for all bugs. It's more of an ongoing task to make sure our focus is in the right place. We should not feel the pressure to do verification by a release deadline, not for the average bug. However, we should, if we can find resources to do so, slowly chip away at the entire resolved base to eventually verify that they're resolved, either by a manual rerun or, better, by checking an automated result of the test that went in. First pass == verified, bug closed. To that end, bug verifications are something we should be throwing at community members who want to help but don't have other special skills, and at people who are onboarding. Bug verifications are a great way to ramp up on a section of the product and to give back value to the project at the same time. In the tiered plan described up above, I'd have community and newbies helping out at tier 2 in parallel with experienced QA doing tier 1. Re: QA should be expanding automation against spec instead (per Henri's reply), We're getting there, in terms of getting more involved with this. I'm leading a project to get QA more involved with WebAPI testing, particularly at the automated level. But the assumption that everyone in the QA community has or will have that skillset is a tall and potentially exclusionary one. Further, there's value in both activities; manual verification covers things that can't be easily automated and, for critical bugs, gives you results much sooner than automation typically does. Automation has the greater long-term value, of course. But there's a balance. Geo ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
On 2012-08-13 21:08:04 +, Geo Mealer said: Instead I'd define (formally or otherwise) three tiers: 1) Critical fixes. These need verification + additional testing. 2) Untested uncritical fixes. These have no automated tests. These should get verification if time allows. 3) Tested critical fixes: These have automated tests and do not need verification. Correcting my typo, 3) Tested uncritical fixes: … Geo ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
Jason Smith schrieb: Note - I still think it's useful for a QA driver to look through a set of bugs fixed for a certain Firefox release, it's just the process would be re-purposed for flagging a bug for needing more extensive testing for X purpose (e.g. web compatibility). I think QA should do some exploratory testing of major new features as time allows, but just verifying existing test cases that often are running automatically anyhow isn't a good use of time, I guess. Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
On Fri, Aug 10, 2012 at 2:04 PM, Jason Smith jsm...@mozilla.com wrote: * Flagging of a risky patch needing to be sure it works - This is where doing a deep dive generally by formulating a test plan and executing it is useful. This suggestion reminds me of the fact that members of the JS team occasionally ask Jesse/Gary/Decoder to run their JS fuzz tools over risky patches. This works well in practice -- the fuzz tools often find problems. Running an automatic tool is of course easier than formulating a human-based plan, but it's still a good precedent. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
Sorry, this should have went to dev-platform... - Original Message - From: Anthony Hughes ahug...@mozilla.com To: dev-planning dev-plann...@lists.mozilla.org Cc: dev-qual...@lists.mozilla.org Sent: Friday, August 10, 2012 1:40:15 PM Subject: Fwd: Verification Culture I started this discussion on dev-quality[1] but there has been some suggestion that the dev-planning list is more appropriate so I'm moving the discussion here. There's been a couple of great responses to the dev-quality thread so far but I won't repost them here verbatim. The general concensus in the feedback was that QA spending a lot of time simply verifying that the immediate test conditions (or test case) is not a valuable practice. It was suggested that it would be a far more valuable use of QA's time and be of greater benefit to the quality of our product if we pulled out a subset of critical issues and ran deep-diving sprints around those issues to touch on edge-cases. I, for one, support this idea in the hypothetical form. I'd like to get various peoples' perspectives on this issue (not just QA). Thank you do David Baron, Ehsan Akhgari, Jason Smith, and Boris Zbarsky for the feedback that was the catalyst for me starting this discussion. For reference, it might help to have a look at my dev-planning post[2] which spawned the dev-quality post, which in turn has spawned the post you are now reading. Anthony Hughes Mozilla Quality Engineer 1. https://groups.google.com/forum/#!topic/mozilla.dev.quality/zpK52mDE2Jg 2. https://groups.google.com/forum/#!topic/mozilla.dev.planning/15TSrCbakEc - Forwarded Message - From: Anthony Hughes ahug...@mozilla.com To: dev-qual...@lists.mozilla.org Sent: Thursday, August 9, 2012 5:14:02 PM Subject: Verification Culture Today, David Baron brought to my attention an old bugzilla comment[1] about whether or not putting as much emphasis on bug fix verification was a useful practice or not. Having read the comment for the first time, it really got me wondering whether our cultural desire to verify so many bug fixes before releasing Firefox to the public was a prudent one. Does verifying as many fixes as we do really raise the quality bar for Firefox? Could the time we spend be better used elsewhere? If I were to ballpark it, I'd guess that nearly half of the testing we do during Beta is for bug fix verifications. Now sure, we'll always want to have some level of verification (making sure security fixes and critical regressions are *truly* fixed is important); But maybe, just maybe, we're being a little too purist in our approach. What do you think? Anthony Hughes Quality Engineer Mozilla Corporation 1. https://bugzilla.mozilla.org/show_bug.cgi?id=172191#c16 ___ dev-quality mailing list dev-qual...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-quality ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Verification Culture
On 12-08-10 5:04 PM, Jason Smith wrote: Hi Everyone, Let's try posting this again. Disregard the comments I put on the other thread. I think this is a good time to re-think our process for testing for something that is fixed or not fixed. I think a better approach that maybe we need to consider is something similar to what the security reviews do. They flag certain special areas for needing attention from a security perspective to do deep dives to find out the risks of certain area and flag them as such. In the context of QA, we could easily do something like that by reusing that process with the qawanted keyword. Some suggestions to build off of this: * Continue doing the process when a try build is created and needing QA testing to make use of the qawanted keyword to call out for explicit QA help. * Add a new process where more testing is needed after a bug has landed for some purpose. We could reuse the qawanted keyword to call attention to the bug and we can take it from there. For our QA process, we might want to file bugs in our component (Mozilla QA) to track any specialized testing requests that are needed This sounds good. * Throw out the general concept of blind verifications. Maybe we should even take out the verified portion of the bug workflow to ensure this, but that may need further discussion to get agreement on that. This^^^, a thousand times! :-) Note that not everything to test is necessarily testing that X bug works as expected. There's other cases that may warrant calling out attention to certain bugs for explicit testing after the bug lands. Here are two examples: * Unprefixing of a DOM or CSS property - It's useful to call out for explicit testing here not necessarily for actually testing if the property works (as we have a lot of automation that usually does some of the testing here), but instead call out testing for web compatibility concerns to find out if there are web compatibility risks with unprefixing X DOM or CSS property. How are we planning to test this? We have seen bugs in obscure web sites which use the name of a new DOM property for example, but it seems to me like there is no easy way for somebody to verify that adding such a property doesn't break any popular website, even, since sometimes the bug needs special interactions with the website to be triggered. * Flagging of a risky patch needing to be sure it works - This is where doing a deep dive generally by formulating a test plan and executing it is useful. This is sane too. Note - I still think it's useful for a QA driver to look through a set of bugs fixed for a certain Firefox release, it's just the process would be re-purposed for flagging a bug for needing more extensive testing for X purpose (e.g. web compatibility). I'm not quite sure why that would be useful. If we believe that doing blind verification is not helpful, why is doing that on a subset of bugs fixed in a given release better? Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
