This is really great! If they don't already do so, maybe it would be simple now to modify the new example-walking scripts to ensure that the test suite covers all the examples/tutorials (maybe even nightly), noting any qualifying files that don't have a valid /*TEST .. TEST*/ block.
On Fri, Jan 13, 2017 at 11:16 PM, Barry Smith <[email protected]> wrote: > > PETSc developers, > > Scott Kruger has been implementing a new "test harness" for PETSc. > Rather than having tests implemented as bash lines in the makefiles, the > tests will be defined within the source code for each example with a simple > format and the actual scripts that run the tests will be automatically > generated. This offers three advantages over the current system > > 1) test definitions can be given in a very concise format that is independent > of how the test is run > 2) multiple approaches to actually running the tests can be implemented > without requiring any changes to the test definitions for each example. > 3) the tests can be compiled and run in parallel (eventually we hope to see > much faster times for testing). > > Here is an example of the format that defines tests > > test: > > test: > suffix: yaml > requires: yaml !complex > args: -options_file_yaml bag.yml -options_view > filter: grep -v saws_port_auto_select > localrunfiles: bag.yml > > The first is a test that takes no command line arguments and does no > filtering of output. The second takes command line arguments, does some > filtering of the output and this example also requires a data file associated > with that example. The expected output of tests continues to reside in the > output directory with the previously naming convention of XXX_1.out XXX_2.out > where XXX is the name of the example or XXX_suffix.out where suffix is > defined in the test definition. > > So far in the master branch we have only converted over two directories of > examples sys/examples/tutorials and sys/examples/tests so you can look in > those directories for other examples. Over the next few weeks we hope to > convert over all the rest of the directories. > > Special thanks to Scott, Satish, and Jed for all the work they have done to > set up the new test harness. > > Barry > > Technical details: the new python script ./config/gmakegentest.py which is > normally automatically run at configure walks through the examples > directories looking for the formatted test definitions, for each test it > generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or > tutorials/ a shell script that can run that test. In addition > config/gmakegentest.py adds dependencies to the $PETSC_DIR/gmakefile for all > the tests that it has found. The rule make alltests then executes all those > dependencies. > > We will be refining both the testing harness and the test definitions as > we convert the rest of the examples over to the new approach. > > > > >
