Stephan Bergmann wrote:
Hi all,

I just embarked on a new project, namely to clean up and consolidate the various test frameworks and corresponding tests available in the OOo build environment. These include at least:

- C++ unit tests, based on CppUnit and/or testshl2, and Java unit tests, based on JUnit and/or qadevOOo. These are scattered across the code base (*/qa, */test, */workben, testtools/, ...), some are executed during a regular build (o3tl/qa, basegfx/test, basebmp/test) but most are not even compiled during a regular build (and thus rot over time). Some of the tests need no special environment, while others require a working UNO runtime environment.

- The so-called "UNO-API" and "Complex" tests. These are located in */qa/unoapi and */qa/complex, use OOoRunner from qadevOOo, and can be executed via cwscheckapi or checkapi. They require an installed OOo, which cwscheckapi takes care of. They are not compiled or executed during a regular build (they clearly cannot be executed, as they require an installed OOo), but it is expected that cwscheckapi is manually executed for each CWS.

- The smoke test in smoketestoo_native. It requires an installed OOo, which smoketest.pl takes care of. It is executed at the end of a regular build.

- The ConvWatch and Performance tests, that can be started from the EIS page of a CWS. They require an installed OOo (and also the installation of a corresponding master-workspace OOo, for result comparison), which they take care of. They are not executed during a regular build, but it is rather expected that they are manually triggered from EIS for each CWS (where they are executed asynchronously on dedicated machines, and their results made available in EIS).

What is *not* covered (for now?) are the automatic QA tests based on testtool, as well as the "portability tests" (so to speak) of building OOo on a wide range of platforms via buildbots and tinderboxes.

The two main problems with the tests listed above appear to be that (a) many of them require an OOo installation, and they all invented their own ways of providing one, and all those ways are brittle and start to fail sooner or later, and (b) the tests that are not compiled or executed during each build (CWS as well as master) start to rot sooner or later. A third problem probably is that the tests and test frameworks are often poorly documented and do things in non-standard ways (e.g., testshl2 vs. plain CppUnit), so that it is not easy to maintain existing tests and write additional ones.

I would like to address these problems. My "guiding vision" in doing so is the following "perfect world:" There is one OOo installation in the solver. (Ideally, it would automatically emerge from delivering the various files directly to the appropriate locations in the solver.) All the tests that require an OOo installation use that one installation. (They do not modify it. Each test probably has its own, throw-away UserInstallation directory, and soffice is started with appropriate switches to not show unwanted first start wizards etc.) All the tests are written using standard tools (the xUnit framework: CppUnit resp. JUnit). For tests that have specific requirements on their environment (i.e., require a working UNO runtime environment, or an OOo installation), there are library routines available to set up/tear down such environments, to be called from the xUnit setup/tearDown methods. Generally, tests are compiled and executed during every regular build. For tests which absolutely cannot be executed during every regular build (maybe because they are too expensive, or require a dedicated machine, like could be the case for the performance test), the main way to execute them is still to have some (manual) makefile target for them. (There may be additional convenience mechanisms, like buttons in EIS, but they are strictly secondary.)

I know that this picture is not perfectly realistic, and that there will be obstacles along the way that require pragmatic workarounds. Still, I think it is important to know what the ideal should look like, even if you have to deviate from it.
thanks for sharing your vision of the ideal world, i agree that it is important and of course very useful to know where to go ...


As a first step, I set up CWS sb118 to experiment, as a showcase and to gain further insight, with turning smoketestoo_native into such an ideal test. As it turned out, the first thing I had to do on that CWS was to replace the heavily modified CppUnit 1.8 currently used by OOo with a plain unmodified latest version of CppUnit 1.12.1.

Comments on all of this are, of course, very welcome.
a cleanup and a consolidation of the different available and used test frameworks etc. sounds very useful. And once we have reached a state where we have working tests, a working framework and some documentation in place that describes how to write new tests i am confident that we will improve the overall quality in the long term.

Especially in case of API tests i am looking forward to have working, maintained and regular running tests in place.

It sounds very promising and i will support your project where i can.

And to add one item to your perfect world picture. Somewhere in the future i would like to reach a state where we introduce new APIs only if a complex test for the API is in place also ;-)

Juergen


-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org

Reply via email to