Hi Frank,
Frank Schönheit - Sun Microsystems Germany schrieb:
Hi Joost,
Using the Testtool is very useful to find regressions by comparing
result files of eg. a CWS build with result files of the corresponding
master workspace.
That statement might be true, but not very helpful in itself.
That is: Of the 100 regressions introduced every day, how many are found
by the testtool, with which effort, and how many are found by, say,
manual testing, with which effort? Only an asnwer to questions like
those allow us to judge the efficiency of the testtool.
Here are the numbers about reporter of regressions (issues which are
marked as regression).
- ~40% submitted by Sun QA team
- ~33% of these issues are found by automated testing with
VCLTestTool, this means ~12% of all submitted regressions
(the number could be higher I think, marking of these issues isn't
done adequate)
=> most of these issues are found before integration of a CWS
- ~22% submitted by Sun Development team
=> I do not have the numbers if they are submitted before or after
integration of a CWS.
- ~32% submitted by community
The number of found regressions by automated testing should be higher.
But then the priority of fixing issues which break the tooling should
be increase and it must be a no-go to integrate CWS which break an
automated test script. This is only required for the 'required test'.
But this isn't required for Level 2-3. To fix these test scripts or
evaluating these error messages took a lot of time in the Automated
testing team.
Don't get me wrong, I am not saying that the testtool is useless, I
actually do not know enough about it. So, I just want to point out that
the efficiency of a quality assurance tool (and testtool is one of
those) is measured not only by its success, but also by the efforts
invested into it.
The effort isn't so high, when all issues (which comes up at CWS
testing) will be fixed in the CWS. Also in the time frame when we are
near before FeatureFreeze and CodeFreeze.
Thorsten
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
