Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 12:19 PM, janI wrote: On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. +1 and I think it was the intended idea behind this, have it run on a regular basis ideally on the build bots. The work is not finished and have to be done as so many open work items. If thet run more stable
Re: Requesting privileges to the QA list
Hello Rob, I have just been to: https://issues.apache.org/ooo But I see no difference in the window with QA privileges. Where is the button to search for open reports for me to test and confirm them? Thanks! Kind regards, Marco A.G.Pinto --- On 13/08/2013 16:11, Rob Weir wrote: On Tue, Aug 13, 2013 at 5:52 AM, Marco A.G.Pinto marcoagpi...@mail.telepac.pt wrote: Hello! I want to help, give me rights, my e-mail is: marcoagpi...@mail.telepac.pt Done. Thanks! -Rob [10:36] JZA you do need to request priviledges to the qa list [10:36] JZA we have that documented on the introductory modules for QA [10:37] JZA is just saying I want to help, please give me righs, my email is [10:37] JZA u'll get it in a day, and be ready to go. Thanks! Kind regards, Marco A.G.Pinto -- --
Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 8:29 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 1:33 PM, Rob Weir wrote: On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 12:19 PM, janI wrote: On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. +1 and I think it was the intended idea behind this, have it run on a regular basis ideally on
Re: Requesting privileges to the QA list
Hello Rob, There are no links in the top left of the page (screenshot): http://i.imgur.com/2fWizPa.png I have seen Alexandro Colorado's screenshot and he has two links there. Thanks! Kind regards, Marco A.G.Pinto --- On 15/08/2013 16:40, Rob Weir wrote: On Thu, Aug 15, 2013 at 7:44 AM, Marco A.G.Pinto marcoagpi...@mail.telepac.pt wrote: Hello Rob, I have just been to: https://issues.apache.org/ooo But I see no difference in the window with QA privileges. The main difference should be that you can edit bug fields. For example you should now be able to mark a bug as confirmed. Before you could only add a comment. Where is the button to search for open reports for me to test and confirm them? You should see, when you are logged, a set of pre-defined search queries at the top of the page, under the main Bugzilla menu. One of them should be called "unconfirmed-defects". You can use that search, or define your own. For example, if you want to focus on AOO 4.0.0 Calc bugs on Windows, then you can define a search to return only those issues. You might want to familiarize yourself with the BZ search interface (https://issues.apache.org/ooo/query.cgi) and try some custom queries. -Rob --
