On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo) <marcus.m...@wtnet.de> wrote: > Am 08/14/2013 09:01 PM, schrieb Raphael Bircher: > >> Am 14.08.13 20:21, schrieb Rob Weir: >>> >>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote: >>>> >>>> Dear Rob >>>> The 4.0 release was too ambitious - we should advance in smaller steps. >>>> Nothing compares to general public testing - betas and release >>>> candidates should not be avoided. >>>> TestLink cases should be less comprehesive (in terms of feature >>>> coverage) and more stress testing oriented. >>> >>> The number to consider here is how many defects were found and fixed >>> during the 4.0.0 testing, before the general public users had access? >>> I assume it was quite substantial. If so, the TestLink usage was >>> effective. In other words, we might have found fewer bugs without >>> using it. > > > In general, my feeling is that it's too early to do a retrospective and ask > "what was good, what needs to be improved?" (to say it with SCRUM words ;-) > ) Just after the first major release. >
Memories are still fresh and programmers are looking at the relevant code. This is the best time to answer these questions. > We should look on the BZ query you have mentioned and see if there is one or > more hotspots that should be improved fast. That's it. > That would be fine. I'm not suggesting anything radical. Gradual, but constant improvements are the way to high quality. > >>> This is important to keep in mind: we want to prevent or find more >>> bugs, but we're not starting from zero. We're starting from a process >>> that does a lot of things right. >>> >>> I like the idea of a public beta. But consider the numbers. The 40 >>> or so regressions that were reported came from an install base (based >>> on download figures since 4.0.0 was released) of around 3 million >>> users. Realistically, can we expect anywhere near that number in a >>> public beta? Or is it more likely that a beta program has 10,000 >>> users or fewer? I don't know the answer here. But certainly a >>> well-publicized and used beta will find more than a beta used by just >>> a few hundred users. >> >> The public beta is from my point of view realy important. Even you have >> only 10'000 Downloads of a beta, you have normaly verry experianced >> Users there, like power users from Companies. They provide realy valua >> feedback. So from my point of view, this is one of the moast important >> changes we have to do. For all Feature release a beta version. And don't >> forget, people are realy happy to do beta tests. but many of them are >> maybe not willing to follow a mailing list. > > > A public beta release is of course not the golden solution but could > activate some power users that give us the feedback we want and need. > > So, +1 for going this way. > > After the 4.1 release is done we can see if this was much better - and ask > ourself why? :-) > But several of the bugs in 4.0.0 should never have made it to even a beta. For example, the very slow saving in Excel format. What happened there? Sure, this could be fixed later in the process, in a beta, or in a 4.0.1 as we're doing now. But this really should have been detected and fixed much earlier in the process. What are betas good for? Betas are good for expanding the set of configurations. Platforms, languages, extensions, etc. But the informal "tests" done by real users tend to be shallow. Also, we have no way of determining what the test coverage is. In particular we have no basis for telling the difference between the coverage of a 2 week beta versus a 4 week beta. But with out formal test cases we can easily look at how many test cases were executed. We could even look at code coverage if we wanted to. Maybe if there was a way to record what features beta testers were using... Or even have a little survey where they report what platform they ran on, etc. But very slow saving to XLS files? A defect like that should have been caught by us before a beta and before a RC. We shouldn't expect to find bugs like that in a beta. Finally, I think we can all point to a similar open source project that has numerous betas, but still suffers from poor quality. So a public beta, by itself, is not sufficient. We need some upstream improvements as well, I think. But we should do a beta as well. But aim to have the highest quality beta we can, right? Regards, -Rob > Marcus > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org > For additional commands, e-mail: dev-h...@openoffice.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
