Thanks for this extremely useful information - far too useful to only
exist in a mail archive. Any objection to me creating a "Test Overview"
Wiki page and populating it initially by pasting in the body of your
message?
I am running with the unit tests disabled for two reasons. Currently,
some of them fail in my environment for what I think are test artifact
reasons rather than actual bugs. Additionally, given how slow builds
are, I am reluctant to add anything to that process.
In general, I think we really need scripts or ant targets that just run
various categories of tests.
Patricia
On 3/29/2016 5:18 AM, Damjan Jovanovic wrote:
We have many different types of tests. The documentation on them is
sparse, scattered and outdated; a Wiki page is sorely needed.
Unit tests are used by a few modules, run during the build (unless you
pass --disable-unit-tests to ./configure), cause the build to fail if
they fail (though I've noticed segmentation faults in tests are
apparently ignored), but they cannot rely on the entire AOO
environment to be available, only their module to have been built.
Previously we used cppunit for this, but have since moved to Google
Test. This is best for low-level self-contained functionality. They
are always in C/C++.
"Subsequent tests" are tests run after AOO is built. These use Junit
and are written in Java. IIRC they're off by default, you have to
define the environment variable OOO_SUBSEQUENT_TESTS when you run
"build" on the module you want to run them on, but some extra Google
Tests also run with that variable. I don't really know much about
them.
The qa tests are integration tests in the test/ directory (same
directory level as main/). They do not run during the build, and have
to be invoked separately, but can be invoked on an arbitrary AOO
instance, so you can easily run the same tests against different AOO
versions. You have to leave your computer unattended while they run,
as the tests move and click the mouse, type buttons on the keyboard,
and use the clipboard. There are several test categories: the bvt
(basic verification test) is a default set of basic tests, fvt
(functional verification test) has a lot more, pvt (performance
verification tests) measures performance, etc. Because each test opens
a new AOO instance and exits afterwards, they take ages: 10 minutes
for bvt, 1 hour for fvt. You can read more on
https://wiki.openoffice.org/wiki/QA/test_automation_guide#Getting_started_with_command_line
In main/smoketestoo_native there is a "smoketest" written in
StarBasic, that tests various functionality. It began to be refactored
as a qa test, and has been copied there now and runs as a bvt test.
We also have a spreadsheet test running as an fvt test, which can be
used to verify things work by examining a specially formatted
spreadsheet that uses spreadsheet functions or macros to obtain
results. You can see examples in
test/testuno/source/fvt/uno/sc/formula/TestFormulaDocs.java
It's worth noting that these testing frameworks need some attention.
The refactoring effort while IBM was still contributing left things
unfinished. Some test code disappeared, breaking some unit tests and
subsequent tests. Many tests (about 2 bvt tests and 25 fvt tests) have
been failing for a while. The smoketest in test/ needs to use the
document built in main/smoketestoo_native during the build instead of
its private copy (but the test/ directory is designed to be separate
from the build - should it build the document using Java functions
instead?). Also these Java qa tests seem new and some are unfinished -
were they ported from somewhere (smoketest?)?
Damjan
On Tue, Mar 29, 2016 at 5:53 AM, Patricia Shanahan <[email protected]> wrote:
I have a few simple changes I would like to check in. Before doing that, I
would normally run regression tests against my working copy. In any case, as
one of the few people who are building on Windows, I should test early, test
often.
What tests do people normally run to check that changes do not have
unintended consequences?
Thanks,
Patricia
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
