Scott and PETSc folks,
Using alt files for testing is painful. Whenever you add, for example, a
new variable to be output in a viewer it changes the output files and you need
to regenerate the alt files for all the test configurations. Even though the
run behavior of the code hasn't changed.
I'm looking for suggestions on how to handle this kind of alternative
output in a nicer way (alternative output usually comes from different
iterations counts due to different precision and often even different
compilers).
I idea I was thinking of was instead of having "alt" files we have "patch"
files that continue just the patch to the original output file instead of a
complete copy. Thus in some situations the patch file would still apply even if
the original output file changed thus requiring much less manual work in
updating alt files. Essentially the test harness would test against the output
file, if that fails it would apply the first patch and compare again, try the
second patch etc.
Scott,
What do you think? Should be an easy addition to the current model (no
need to even remove the alt testing)? Would it also be possible to add a PATCH
option to the test rule where it automatically added the new patch file?
Perhaps all the patches for a test case could all be stored in the same file
also so we don't need to manage patch_1.out patch_2.out etc? Each new patch
would just get added to the file?
Thoughts?
Barry
