Hello,
see my 2 cents below.
Stephan Bergmann wrote:
Hi all,
I just embarked on a new project, namely to clean up and consolidate the
various test frameworks and corresponding tests available in the OOo
build environment. These include at least:
- C++ unit tests, based on CppUnit and/or testshl2, and Java unit tests,
based on JUnit and/or qadevOOo. These are scattered across the code
base (*/qa, */test, */workben, testtools/, ...), some are executed
during a regular build (o3tl/qa, basegfx/test, basebmp/test) but most
are not even compiled during a regular build (and thus rot over time).
Some of the tests need no special environment, while others require a
working UNO runtime environment.
- The so-called "UNO-API" and "Complex" tests. These are located in
*/qa/unoapi and */qa/complex, use OOoRunner from qadevOOo, and can be
executed via cwscheckapi or checkapi. They require an installed OOo,
which cwscheckapi takes care of. They are not compiled or executed
during a regular build (they clearly cannot be executed, as they require
an installed OOo), but it is expected that cwscheckapi is manually
executed for each CWS.
Not connected, I would like to change something: as far as it is
possible (from build order perspective), I plan to include all Java
tests into the build, where this has not been done already. Tests can
then be executed with dmake or cwscheckapi.
Execution while building is not planned - nearly all tests need an
installed runnable office to execute.
When no Java environment is set, tests will not be compiled, of course.
-Steffen
- The smoke test in smoketestoo_native. It requires an installed OOo,
which smoketest.pl takes care of. It is executed at the end of a
regular build.
- The ConvWatch and Performance tests, that can be started from the EIS
page of a CWS. They require an installed OOo (and also the installation
of a corresponding master-workspace OOo, for result comparison), which
they take care of. They are not executed during a regular build, but it
is rather expected that they are manually triggered from EIS for each
CWS (where they are executed asynchronously on dedicated machines, and
their results made available in EIS).
What is *not* covered (for now?) are the automatic QA tests based on
testtool, as well as the "portability tests" (so to speak) of building
OOo on a wide range of platforms via buildbots and tinderboxes.
The two main problems with the tests listed above appear to be that (a)
many of them require an OOo installation, and they all invented their
own ways of providing one, and all those ways are brittle and start to
fail sooner or later, and (b) the tests that are not compiled or
executed during each build (CWS as well as master) start to rot sooner
or later. A third problem probably is that the tests and test
frameworks are often poorly documented and do things in non-standard
ways (e.g., testshl2 vs. plain CppUnit), so that it is not easy to
maintain existing tests and write additional ones.
I would like to address these problems. My "guiding vision" in doing so
is the following "perfect world:" There is one OOo installation in the
solver. (Ideally, it would automatically emerge from delivering the
various files directly to the appropriate locations in the solver.) All
the tests that require an OOo installation use that one installation.
(They do not modify it. Each test probably has its own, throw-away
UserInstallation directory, and soffice is started with appropriate
switches to not show unwanted first start wizards etc.) All the tests
are written using standard tools (the xUnit framework: CppUnit resp.
JUnit). For tests that have specific requirements on their environment
(i.e., require a working UNO runtime environment, or an OOo
installation), there are library routines available to set up/tear down
such environments, to be called from the xUnit setup/tearDown methods.
Generally, tests are compiled and executed during every regular build.
For tests which absolutely cannot be executed during every regular build
(maybe because they are too expensive, or require a dedicated machine,
like could be the case for the performance test), the main way to
execute them is still to have some (manual) makefile target for them.
(There may be additional convenience mechanisms, like buttons in EIS,
but they are strictly secondary.)
I know that this picture is not perfectly realistic, and that there will
be obstacles along the way that require pragmatic workarounds. Still, I
think it is important to know what the ideal should look like, even if
you have to deviate from it.
As a first step, I set up CWS sb118 to experiment, as a showcase and to
gain further insight, with turning smoketestoo_native into such an ideal
test. As it turned out, the first thing I had to do on that CWS was to
replace the heavily modified CppUnit 1.8 currently used by OOo with a
plain unmodified latest version of CppUnit 1.12.1.
Comments on all of this are, of course, very welcome.
-Stephan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
