On 05/31/10 10:24, Stephan Bergmann wrote:
Just a reminder. As announced
(<http://www.openoffice.org/servlets/ReadMsg?list=interface-announce&msgNo=1266>
"cwscheckapi replaced with subsequenttests"), subsequenttests is the new
tool to run all kinds of OOo developer tests (that require a complete
installation set to test against), particularly replacing cwscheckapi.
With CWS sb120 integrated in DEV300_m80, the framework and tests will
hopefully be reliable enough for actual use, see
<http://tools.openoffice.org/servlets/ReadMsg?list=tinderbox&msgNo=360>
"new step for buildbots."
Developers are encouraged to run subsequenttests now, similarly to how
they ran cwscheckapi in the past. See the first link above for details.
Due to a code change in CWS sb120, Gregor's buildbot framework will no
longer patch away the subsequenttests step in DEV300_m80, so most of the
buildbots (those running on Gregor's framework) will automatically start
to include that step. There are apparently problems with X11 on some of
the bots (see the second link above), and there might still be sporadic
failures (see
<http://wiki.services.openoffice.org/w/index.php?title=Test_Cleanup#unoapi_Tests_2>),
potentially causing buildbot builds to go red. I leave it up to Gregor
to disable any test steps again that turn out to cause trouble; please
inform him about problems you encounter. (Due to vacation schedules, we
probably won't be able to track down those X11 problems for the next two
weeks.)
Unfortunately it seems impossible to stabilize the old test code and,
even more so, OOo itself to the point that subsequenttests would be
reliable enough to routinely run during every build. Experiments (with
local machines on CWS sb123 and with buildbots at
<http://eis.services.openoffice.org/EIS2/cws.ShowCWS?Id=9754&OpenOnly=false&Section=Tests>)
show that even if individual machines manage to successfully run
subsequenttests 50, 60, times in a row, they all do fail sooner or
later, without pattern (see the lengthy lists at
<http://wiki.services.openoffice.org/wiki/Test_Cleanup#unoapi_Tests_2>).
The old, existing tests are definitely not useless (they do find errors
in new code, occasionally). But they are obviously too fragile to tie
the decision whether a build succeeded or failed to the random outcome
of whether the tests succeeded or failed.
Therefore, I propose the following two things:
1 Keep the old tests for manual developer execution. With CWS sb123,
they should be in a shape where they mostly work, so that a developer
could run them to see whether they unearth any problems in newly written
code. But, of course, this would occasionally need manual
interpretation of the test results: If they fail just once, and
re-running them succeeds, you probably hit one of those spurious
failures unrelated to your new code. If they fail repeatedly, maybe
even on multiple platforms, it smells like you broke something. There
will be a Mechanism A to (manually) execute those tests.
2 Urge developers to write new, solid tests for new code or code under
maintenance. These should typically be unit tests (i.e., should not
require a complete OOo instance), which would have (at least) three a
advantages over the old, qadevOOo based tests: They would run quickly,
they would not suffer from OOo's fragility, and they could be run
directly within the build process. There will be a Mechanism B to
(automatically) execute those tests.
For the Mechanisms A and B: As long as those new tests are indeed all
unit tests, Mechanism B is simply to build and execute the tests
unconditionally from within the build system (as is, for example,
already done in basegfx/test/). And that implies that Mechanism A can
for now simply be the existing subsequenttests tool (which should then
no longer be included in buildbot scripts, of course; and note that CWS
sb129 adds a -k switch to subsequenttests, a la make, making it more
useful with our sporadically failing tests).
If there ever crop up new tests that do require a complete OOo
installation, we could then shift things as follows: Use plain
subsequenttests as (part of) Mechanism B (and add it back to the
buildbots) and add a switch like --extra to subsequenttests as Mechanism
A, to run the old tests (whose makefiles would need to be adapted to
only trigger on something like "$(OOO_SUBSEQUENT_TESTS)"=="extra"
instead of "$(OOO_SUBSEQUENT_TESTS)"!="").
So, action items:
- Gregor, please remove the subsequenttests step from the buildbots again.
- All developers, write new tests.
-Stephan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
