Stephan Bergmann wrote:
Hi all,
I just embarked on a new project, namely to clean up and consolidate the
various test frameworks and corresponding tests available in the OOo
build environment. These include at least:
- C++ unit tests, based on CppUnit and/or testshl2, and Java unit tests,
based on JUnit and/or qadevOOo. These are scattered across the code
base (*/qa, */test, */workben, testtools/, ...), some are executed
during a regular build (o3tl/qa, basegfx/test, basebmp/test) but most
are not even compiled during a regular build (and thus rot over time).
Some of the tests need no special environment, while others require a
working UNO runtime environment.
- The so-called "UNO-API" and "Complex" tests. These are located in
*/qa/unoapi and */qa/complex, use OOoRunner from qadevOOo, and can be
executed via cwscheckapi or checkapi. They require an installed OOo,
which cwscheckapi takes care of. They are not compiled or executed
during a regular build (they clearly cannot be executed, as they require
an installed OOo), but it is expected that cwscheckapi is manually
executed for each CWS.
- The smoke test in smoketestoo_native. It requires an installed OOo,
which smoketest.pl takes care of. It is executed at the end of a
regular build.
- The ConvWatch and Performance tests, that can be started from the EIS
page of a CWS. They require an installed OOo (and also the installation
of a corresponding master-workspace OOo, for result comparison), which
they take care of. They are not executed during a regular build, but it
is rather expected that they are manually triggered from EIS for each
CWS (where they are executed asynchronously on dedicated machines, and
their results made available in EIS).
What is *not* covered (for now?) are the automatic QA tests based on
testtool, as well as the "portability tests" (so to speak) of building
OOo on a wide range of platforms via buildbots and tinderboxes.
The two main problems with the tests listed above appear to be that (a)
many of them require an OOo installation, and they all invented their
own ways of providing one, and all those ways are brittle and start to
fail sooner or later, and (b) the tests that are not compiled or
executed during each build (CWS as well as master) start to rot sooner
or later. A third problem probably is that the tests and test
frameworks are often poorly documented and do things in non-standard
ways (e.g., testshl2 vs. plain CppUnit), so that it is not easy to
maintain existing tests and write additional ones.
I would like to address these problems. My "guiding vision" in doing so
is the following "perfect world:" There is one OOo installation in the
solver. (Ideally, it would automatically emerge from delivering the
various files directly to the appropriate locations in the solver.) All
the tests that require an OOo installation use that one installation.
(They do not modify it. Each test probably has its own, throw-away
UserInstallation directory, and soffice is started with appropriate
switches to not show unwanted first start wizards etc.) All the tests
are written using standard tools (the xUnit framework: CppUnit resp.
JUnit). For tests that have specific requirements on their environment
(i.e., require a working UNO runtime environment, or an OOo
installation), there are library routines available to set up/tear down
such environments, to be called from the xUnit setup/tearDown methods.
Generally, tests are compiled and executed during every regular build.
For tests which absolutely cannot be executed during every regular build
(maybe because they are too expensive, or require a dedicated machine,
like could be the case for the performance test), the main way to
execute them is still to have some (manual) makefile target for them.
(There may be additional convenience mechanisms, like buttons in EIS,
but they are strictly secondary.)
I know that this picture is not perfectly realistic, and that there will
be obstacles along the way that require pragmatic workarounds. Still, I
think it is important to know what the ideal should look like, even if
you have to deviate from it.
thanks for sharing your vision of the ideal world, i agree that it is
important and of course very useful to know where to go ...
As a first step, I set up CWS sb118 to experiment, as a showcase and to
gain further insight, with turning smoketestoo_native into such an ideal
test. As it turned out, the first thing I had to do on that CWS was to
replace the heavily modified CppUnit 1.8 currently used by OOo with a
plain unmodified latest version of CppUnit 1.12.1.
Comments on all of this are, of course, very welcome.
a cleanup and a consolidation of the different available and used test
frameworks etc. sounds very useful. And once we have reached a state
where we have working tests, a working framework and some documentation
in place that describes how to write new tests i am confident that we
will improve the overall quality in the long term.
Especially in case of API tests i am looking forward to have working,
maintained and regular running tests in place.
It sounds very promising and i will support your project where i can.
And to add one item to your perfect world picture. Somewhere in the
future i would like to reach a state where we introduce new APIs only
if a complex test for the API is in place also ;-)
Juergen
-Stephan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
