Weekend project... This overhaul of the runtest.py and individual run.py files finally fixes a longstanding annoyance -- that the tests are run directly in testsuite/... and also all sorts of waste is left behind.
With this change, the testsuite is first copied to build/ARCH/testsuite, and run from there, so regular oiio/testsuite is considered read-only by the tests themselves. This means that a "make clean" clears all the cruft from the tests, nothing left behind. In the process of this change, much of the repetitive code in the individual testsuite/*/test.py files has been moved to runtest.py, thus making the run.py files in general much shorter and easier to understand. I also fixed up a few tests and removed some of the os-to-os variance, and in the process of making those things work well, I made a number of other small changes that are part of this review (but for which I will keep separate individual commits, in case we want to cherry pick them): * oiiotool -hash now prints the hash of each input image (like iinfo -hash always did) * oiiotool is better about printing error messages and exiting when it comes across an input image that it can't read. * idiff (and oiiotool --diff) no longer print NaN nonsense for 0x0 images. * testtex now supports -d (like iconvert and oiiotool) to set the desired data format for the output. * idiff and oiiotool --diff, when NOT given the -v (verbose) flag, will only print "PASS" for non-failing comparisons. Only failing images (and all images if -v is used) will print the pixel statistics. This is helpful if you want to raise the thresholds to allow a little bit of LSB disagreement from platform to platform -- the output logs will still not match if each one prints its (passing) differences. It's better for a pass-with-threshold to just print "PASS" without the stats, so that all platforms will have matching logs. I've tested this on OSX and Linux. It would be great if somebody could test on Windows and let me know if I've broken anything horribly (though, truth be told, I have no idea if the testsuite ran reliably on Windows even before this overhaul). If nobody objects, I'd like to backport this to 1.0 release as well, since it's so much nicer to have the tests execute in build (and this fixes several broken tests). For you OSL-ers out there, after this is all approved, I will do the analogous simplification and overhaul to the testsuite in OSL, it could stand to have exactly the same improvements. You can merge this Pull Request by running: git pull https://github.com/lgritz/oiio lg-testsuite Or you can view, comment on it, or merge it online at: https://github.com/OpenImageIO/oiio/pull/241 -- Commit Summary -- * oiiotool --hash prints the sha-1 hash of each input image * remove imagecache-files test, I can't figure out what the point of it is now, and it was very hard to keep its output consistent across platforms and runs * oiiotool: properly print error message and exit when an input file could not be opened. * Make diff output print 0 instead of nan for 0x0 pixel images * Add -d option to testtex to let you set the output data type * Change idiff and oiiotool -diff to only print pixel reports if the comparison fails or if verbose mode is on. That is, images that pass the comparison will not pointlessly print pixel info. * testsuite overhaul: simplify run.py files and make all temporary files write to build, not testsuite source. * Updated reference output * These .gitignore entires are no longer needed -- File Changes -- M .gitignore (122) M src/CMakeLists.txt (11) M src/cmake/oiio_macros.cmake (14) M src/doc/idiff.tex (2) M src/iconvert/iconvert.cpp (3) M src/idiff/idiff.cpp (114) M src/oiiotool/diff.cpp (108) M src/oiiotool/oiiotool.cpp (9) M src/oiiotool/oiiotool.h (4) M src/testtex/testtex.cpp (28) M testsuite/bmp/ref/out.txt (61) M testsuite/bmp/run.py (60) M testsuite/dpx/ref/out.txt (27) M testsuite/dpx/run.py (25) M testsuite/fits/ref/out.txt (229) M testsuite/fits/run.py (87) M testsuite/gpsread/ref/out.txt (4) M testsuite/gpsread/run.py (18) M testsuite/ico/ref/out.txt (77) M testsuite/ico/run.py (22) D testsuite/imagecache-files/ref/out.txt (2) D testsuite/imagecache-files/run.py (44) M testsuite/jpeg2000/ref/out.txt (131) M testsuite/jpeg2000/run.py (43) M testsuite/oiiotool-fixnan/ref/out.txt (24) M testsuite/oiiotool-fixnan/run.py (31) A testsuite/oiiotool/ref/out.txt (9) M testsuite/oiiotool/run.py (32) M testsuite/openexr-chroma/ref/out.txt (14) M testsuite/openexr-chroma/run.py (33) M testsuite/openexr-multires/ref/out.txt (1001) M testsuite/openexr-multires/run.py (57) M testsuite/openexr-suite/ref/out.txt (268) M testsuite/openexr-suite/run.py (74) M testsuite/psd/ref/out.txt (40) M testsuite/psd/run.py (30) M testsuite/rla/ref/out.txt (196) M testsuite/rla/run.py (31) D testsuite/run-template.py (18) M testsuite/runtest.py (147) M testsuite/sgi/ref/out.txt (45) M testsuite/sgi/run.py (35) M testsuite/targa-tgautils/run.py (42) D testsuite/texture-fat/.gitignore (3) M testsuite/texture-fat/run.py (22) M testsuite/texture-field3d/run.py (21) D testsuite/texture-fill/.gitignore (3) A testsuite/texture-fill/ref/out.txt (9) M testsuite/texture-fill/run.py (21) D testsuite/texture-gray/.gitignore (3) M testsuite/texture-gray/run.py (21) M testsuite/texture-grid/run.py (22) M testsuite/texture-missing/run.py (21) M testsuite/texture-overscan/run.py (24) D testsuite/texture-pointsample/.gitignore (3) M testsuite/texture-pointsample/run.py (22) M testsuite/texture-res/run.py (24) D testsuite/texture-skinny/.gitignore (3) M testsuite/texture-skinny/run.py (22) M testsuite/tiff-depths/ref/out.txt (326) M testsuite/tiff-depths/run.py (106) M testsuite/tiff-suite/ref/out.txt (139) M testsuite/tiff-suite/run.py (58) M testsuite/webp/ref/out.txt (18) M testsuite/webp/run.py (30) -- Patch Links -- https://github.com/OpenImageIO/oiio/pull/241.patch https://github.com/OpenImageIO/oiio/pull/241.diff --- Reply to this email directly or view it on GitHub: https://github.com/OpenImageIO/oiio/pull/241 _______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
