A recent post comparing performance of some of those unit testing frameworks.
http://www.duskborn.com/posts/utest-h-performance/ My point about integration tests is that if we had more unit tests, the integration tests could have less reliance on long-form convergence logs, thus reducing the number of tests of the kind where alt files are most often needed. I'd argue it isn't orthogonal, but still independent: unit testing won't eliminate such cases. "Smith, Barry F. via petsc-dev" <[email protected]> writes: > Most packages that have good coverage with unit tests use a unit test > system to manage the unit test code. I was actually surprised how many there > are for C https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C. > But selecting evaluating and implementing a unit test environment for PETSc > is IMHO orthogonal to my question. The current question is reqarding our > current integration test environment and improving it incrementally (another > orthogonal incremental improvement might be adding the ability to do diffs on > floating point numbers in a coherent way). It would be wrong in IMHO to say > we shouldn't improve the current integration tests because it makes > developing a unit test system less likely. We'll always need a very robust > integration tests regardless of how good our future unit test system is so no > work is lost. > > Barry > > I'd be happy to see a pull request with a unit test system, but orthogonal ;) > > > >> On May 2, 2019, at 10:21 PM, Karl Rupp <[email protected]> wrote: >> >> Hi, >> >>> Scott and PETSc folks, >>> Using alt files for testing is painful. Whenever you add, for example, >>> a new variable to be output in a viewer it changes the output files and you >>> need to regenerate the alt files for all the test configurations. Even >>> though the run behavior of the code hasn't changed. >>> I'm looking for suggestions on how to handle this kind of alternative >>> output in a nicer way (alternative output usually comes from different >>> iterations counts due to different precision and often even different >>> compilers). >>> I idea I was thinking of was instead of having "alt" files we have >>> "patch" files that continue just the patch to the original output file >>> instead of a complete copy. Thus in some situations the patch file would >>> still apply even if the original output file changed thus requiring much >>> less manual work in updating alt files. Essentially the test harness would >>> test against the output file, if that fails it would apply the first patch >>> and compare again, try the second patch etc. >> >> yes, a 'patch' approach would simplify updates to the reference >> >> However: I'm not sure whether we're tackling the right problem here. Our >> diff-based testing isn't great. I'd favor more dedicated unit tests, where >> the correctness check is embedded in the test (ex*.*}) itself rather than >> determined by some text-based diff tool (which, to make matters worse, even >> filters out floating point numbers...). Not all tests can be written as such >> -- but many can and this would significantly reduce the burden on alt files. >> >> Best regards, >> Karli
