On Jul 24, 2014, at 10:56 AM, Dominic Meiser <[email protected]> wrote:
> Hi, > > I have a few questions regarding usage of the PETSc test suite. Some of the > questions are related to issues discussed in this thread > https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2013-January/010765.html. > > 1) Currently I manually set the DATAFILESPATH environment variable. Is this > how I'm supposed to do this or is this handled differently by the test suite? Set it when you run configure then the test suite will run those tests. —DATAFILESPATH= see datafilespath.py it also looks for a directory next to the petsc root directory called datafiles you can add any other reasonable locations to this file (for example ~/Datafiles) Barry > > 2) In the above thread the issue of size of the test suite was brought up > with the time it takes to run `make alltest` being a concern. While working > on bug fixes my workflow tends to be: Find a test case that demonstrates the > bug (usually run an existing test with different command line options); fix > the bug; move on to next related bug. This can produce several test cases > that sometimes overlap in substantial ways. For instance a bug might be > present in serial and parallel. With my workflow I would end up with a serial > and a parallel test. While this has the advantage of making it easier to > narrow down a failure it has the disadvantage of `make alltest` taking > longer. How should I balance these two aspects? > > 3) What is your workflow for examining test results? Do you scan test results > manually? Or is it mostly sufficient to only pay attention to build failures > and crashes? > > 4) Clearly it would be desirable to have tests where we can automatically > (without manual inspection) decide if the test passes or fails. This was also > discussed in the above thread. Perhaps it should be a requirement that new > tests are written in such a way that failure detection can be done > automatically? Typically these tests will then be less suitable as examples > for getting started and they should go into the test rather than tutorial > directories. Clearly scientific computing has additional challenges compared > to unit testing in "normal" software engineering when it comes to > automatically detecting test failure. Some of the techniques we use at Tech-X > include: > - hdfdiff (similar to exodiff) > - ndiff (diffing with atol and rtol for plain text) > - Decide in the driver program whether a test has passed or failed. > The last option is the most flexible. For instance one could have logic like > the following (very schematically): > if (numIters > 10 && numIters < 15 && residual < 1.0e-9) { > printf("SUCCESS\n"); > } else { > printf("FAILURE\n"); > } > Combined with a disciplined approach for the output of a test it's possible > to write strong tests with a much lower likelihood of false positives. If a > quantity is overly sensitive to valid code or configuration modifications > then don't print it. Obviously this is much easier to do if a test does not > have the dual purpose of a tutorial. > > Thanks, > Dominic > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756 > www.txcorp.com >
