see my 2 cents below.

Stephan Bergmann wrote:
Hi all,

I just embarked on a new project, namely to clean up and consolidate the various test frameworks and corresponding tests available in the OOo build environment. These include at least:

- C++ unit tests, based on CppUnit and/or testshl2, and Java unit tests, based on JUnit and/or qadevOOo. These are scattered across the code base (*/qa, */test, */workben, testtools/, ...), some are executed during a regular build (o3tl/qa, basegfx/test, basebmp/test) but most are not even compiled during a regular build (and thus rot over time). Some of the tests need no special environment, while others require a working UNO runtime environment.

- The so-called "UNO-API" and "Complex" tests. These are located in */qa/unoapi and */qa/complex, use OOoRunner from qadevOOo, and can be executed via cwscheckapi or checkapi. They require an installed OOo, which cwscheckapi takes care of. They are not compiled or executed during a regular build (they clearly cannot be executed, as they require an installed OOo), but it is expected that cwscheckapi is manually executed for each CWS.

Not connected, I would like to change something: as far as it is possible (from build order perspective), I plan to include all Java tests into the build, where this has not been done already. Tests can then be executed with dmake or cwscheckapi. Execution while building is not planned - nearly all tests need an installed runnable office to execute.
When no Java environment is set, tests will not be compiled, of course.


- The smoke test in smoketestoo_native. It requires an installed OOo, which smoketest.pl takes care of. It is executed at the end of a regular build.

- The ConvWatch and Performance tests, that can be started from the EIS page of a CWS. They require an installed OOo (and also the installation of a corresponding master-workspace OOo, for result comparison), which they take care of. They are not executed during a regular build, but it is rather expected that they are manually triggered from EIS for each CWS (where they are executed asynchronously on dedicated machines, and their results made available in EIS).

What is *not* covered (for now?) are the automatic QA tests based on testtool, as well as the "portability tests" (so to speak) of building OOo on a wide range of platforms via buildbots and tinderboxes.

The two main problems with the tests listed above appear to be that (a) many of them require an OOo installation, and they all invented their own ways of providing one, and all those ways are brittle and start to fail sooner or later, and (b) the tests that are not compiled or executed during each build (CWS as well as master) start to rot sooner or later. A third problem probably is that the tests and test frameworks are often poorly documented and do things in non-standard ways (e.g., testshl2 vs. plain CppUnit), so that it is not easy to maintain existing tests and write additional ones.

I would like to address these problems. My "guiding vision" in doing so is the following "perfect world:" There is one OOo installation in the solver. (Ideally, it would automatically emerge from delivering the various files directly to the appropriate locations in the solver.) All the tests that require an OOo installation use that one installation. (They do not modify it. Each test probably has its own, throw-away UserInstallation directory, and soffice is started with appropriate switches to not show unwanted first start wizards etc.) All the tests are written using standard tools (the xUnit framework: CppUnit resp. JUnit). For tests that have specific requirements on their environment (i.e., require a working UNO runtime environment, or an OOo installation), there are library routines available to set up/tear down such environments, to be called from the xUnit setup/tearDown methods. Generally, tests are compiled and executed during every regular build. For tests which absolutely cannot be executed during every regular build (maybe because they are too expensive, or require a dedicated machine, like could be the case for the performance test), the main way to execute them is still to have some (manual) makefile target for them. (There may be additional convenience mechanisms, like buttons in EIS, but they are strictly secondary.)

I know that this picture is not perfectly realistic, and that there will be obstacles along the way that require pragmatic workarounds. Still, I think it is important to know what the ideal should look like, even if you have to deviate from it.

As a first step, I set up CWS sb118 to experiment, as a showcase and to gain further insight, with turning smoketestoo_native into such an ideal test. As it turned out, the first thing I had to do on that CWS was to replace the heavily modified CppUnit 1.8 currently used by OOo with a plain unmodified latest version of CppUnit 1.12.1.

Comments on all of this are, of course, very welcome.


To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org

To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org

Reply via email to