PETSc developers,
Scott Kruger has been implementing a new "test harness" for PETSc. Rather
than having tests implemented as bash lines in the makefiles, the tests will be
defined within the source code for each example with a simple format and the
actual scripts that run the tests will be automatically generated. This offers
three advantages over the current system
1) test definitions can be given in a very concise format that is independent
of how the test is run
2) multiple approaches to actually running the tests can be implemented without
requiring any changes to the test definitions for each example.
3) the tests can be compiled and run in parallel (eventually we hope to see
much faster times for testing).
Here is an example of the format that defines tests
test:
test:
suffix: yaml
requires: yaml !complex
args: -options_file_yaml bag.yml -options_view
filter: grep -v saws_port_auto_select
localrunfiles: bag.yml
The first is a test that takes no command line arguments and does no filtering
of output. The second takes command line arguments, does some filtering of the
output and this example also requires a data file associated with that example.
The expected output of tests continues to reside in the output directory with
the previously naming convention of XXX_1.out XXX_2.out where XXX is the name
of the example or XXX_suffix.out where suffix is defined in the test definition.
So far in the master branch we have only converted over two directories of
examples sys/examples/tutorials and sys/examples/tests so you can look in those
directories for other examples. Over the next few weeks we hope to convert over
all the rest of the directories.
Special thanks to Scott, Satish, and Jed for all the work they have done to
set up the new test harness.
Barry
Technical details: the new python script ./config/gmakegentest.py which is
normally automatically run at configure walks through the examples directories
looking for the formatted test definitions, for each test it generates in the
directory $PETSC_ARCH/test/src/XXX/examples/tests or tutorials/ a shell script
that can run that test. In addition config/gmakegentest.py adds dependencies
to the $PETSC_DIR/gmakefile for all the tests that it has found. The rule make
alltests then executes all those dependencies.
We will be refining both the testing harness and the test definitions as we
convert the rest of the examples over to the new approach.
