Barry Smith <[email protected]> writes: > It seems to require a lot of boilerplate for each test. I sure as heck hope > we don't have to write that by hand for each test.
Here is an example: $ cat src/hpgmg/finite-element/test/t230-fmg-poisson2.sh #!/bin/sh test_description='Test Q2 Poisson solve using FMG' . ./hpgmg-sharness.sh # Error norm is converging at fourth order (superconvergent at Lagrange nodes) test_expect_stdout 'FE Poisson FMG solve fedegree=2 serial' 1 'hpgmg-fe fmg -op_type poisson2 -M 4,4,6 -smooth 4,3' ' F(4,3) 0: |e|_2/|u|_2 9.08e-03 |r|_2/|f|_2 3.35e-04 V(4,3) 1: |e|_2/|u|_2 9.17e-03 |r|_2/|f|_2 8.27e-07 V(4,3) 2: |e|_2/|u|_2 9.17e-03 |r|_2/|f|_2 5.54e-09 ' test_expect_stdout 'FE Poisson FMG solve fedegree=2 parallel' 4 'hpgmg-fe fmg -op_type poisson2 -M 4,4,6 -smooth 4,3 -p 1,2,2' ' F(4,3) 0: |e|_2/|u|_2 9.08e-03 |r|_2/|f|_2 3.35e-04 V(4,3) 1: |e|_2/|u|_2 9.17e-03 |r|_2/|f|_2 8.27e-07 V(4,3) 2: |e|_2/|u|_2 9.17e-03 |r|_2/|f|_2 5.54e-09 ' test_done Since I have quite short output here, the test output is inline. > I'm not a fan of the "parse the PETSc makefiles approach"; I think we need > to define a standard format for declaring tests that is very concise (only > exactly what is needed; I have one try in a branch somewhere). Then we can > have code that translates from the standard format to anything people want to > try with testing scripts. I am even willing to manually convert makefile crap > to this format when necessary My "parse the makefiles" was intended to automate the conversion to a more concise format that a (hopefully sane) test system can use directly. I'm not wild about introducing extra translation phases because it makes the testing system more difficult to understand and exposes more possible failure modes. FWIW, we could greatly simplify the makefiles by having each target simply call a script that runs the actual program, collects stdout and stderr and exit codes, and outputs the test result in a standard format (e.g., TAP). I.e., move the repetitive shell code we have now in each target into a single script used for all tests. This script could also have an argument containing restrictions (NoComplex, etc.) so that it can determine when to skip. Moreover, it can sandbox the execution to a temporary directory and clean up after it. As I see it, this is basically equivalent to moving the tests into a sharness-type system or similar. In my opinion, one crucial component in a revamped testing system is to enable parallel execution with a maximum number of processes (perhaps exceeded when a single job needs more than that, but definitely not otherwise). Sharness is really nice for tests that replicate a session where the state is tested as various checkpoints along the way. We have a few of these (e.g., creating and reading files), but most of our tests are independent, which maps to sharness as independent test scripts. One file per test is actually the best case for merging, but can be cumbersome otherwise.
signature.asc
Description: PGP signature
