On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote: > Dear Rob > The 4.0 release was too ambitious - we should advance in smaller steps. > Nothing compares to general public testing - betas and release candidates > should not be avoided. > TestLink cases should be less comprehesive (in terms of feature coverage) and > more stress testing oriented.
The number to consider here is how many defects were found and fixed during the 4.0.0 testing, before the general public users had access? I assume it was quite substantial. If so, the TestLink usage was effective. In other words, we might have found fewer bugs without using it. This is important to keep in mind: we want to prevent or find more bugs, but we're not starting from zero. We're starting from a process that does a lot of things right. I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. Regards, -Rob > Regards, > Edwin > > On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: >> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact >> that we're doing this, and their are no arguments against it, shows >> that we value quality. I'd like to take this a step further, and see >> what we can learn from the defects in AOO 4.0.0 and what we can do >> going forward to improve. >> >> Quality, in the end, is a process, not a state of grace. We improve >> by working smarter, not working harder. The goal should be to learn >> and improve, as individuals and as a community. >> >> Every regression that made it into 4.0.0 was added there by a >> programmer. And the defect went undetected by testers. This is not >> to blame. It just means that we're all human. We know that. We all >> make mistakes. I make mistakes. A quality process is not about >> becoming perfect, but about acknowledging that we make mistakes and >> that certain formal and informal practices are needed to prevent and >> detect these mistakes. >> >> But enough about generalities. I'm hoping you'll join with me in >> examining the 32 confirmed 4.0.0 regression defects and answering a >> few questions: >> >> 1) What caused the bug? What was the "root cause"? Note: >> "programmer error" is not really a cause. We should ask what caused >> the error. >> >> 2) What can we do to prevent bugs like this from being checked in? >> >> 3) Why wasn't the bug found during testing? Was it not covered by any >> existing test case? Was a test case run but the defect was not >> recognized? Was the defect introduced into the software after the >> tests had already been executed? >> >> 4) What can we do to ensure that bugs like this are caught during testing? >> >> So 2 basic questions -- what went wrong and how can we prevent it in >> the future, looked at from perspective of programmers and testers. If >> we can keep these questions in mind, and try to answer them, we may be >> able to find some patterns that can lead to some process changes for >> AOO 4.1. >> >> You can find the 4.0.0 regressions in Bugzilla here: >> >> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834 >> >> >> Regards, >> >> -Rob >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org >> For additional commands, e-mail: qa-h...@openoffice.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org > For additional commands, e-mail: qa-h...@openoffice.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
