Soeren Gebbert wrote: > > message. If any such errors occured as a result of "make test", you > > would ignore all the other failures. In the same way that if error.log > > has an error for e.g. lib/gis, you wouldn't bother about all of the > > subsequent errors and focus on what was wrong with lib/gis. > > Is it possible to stop "make test" in case a library test failed or a > critical module?
Yes. Although it's not really important. The usual case is that all tests succeed. If a test fails, it's important that the failure is reported, but I wouldn't be too concerned about how we handle the fallout, given that this should be an abnormal situation. For comparison, the gcc developers have only just come around to the idea that a failure in locating a #include file shouldn't result in hundreds of additional errors. Other errors (e.g. mistyping a typedef or global variable) still cascade. > > If you're doing repeated tests on a specific module that you're > > working on, you don't want to be re-running the r.mapcalc, r.out.ascii > > etc tests every time. > > I don't know if this can be avoided in an automated test system. > Especially when each time a test case gets executed a temporary mapset > is created. Sure it can. "make test" in a module directory runs the tests for that particular module. Any tests for the test infrastructure itself would only be done when running "make test" from the top level. > > I'd export as much precision as is likely to be meaningful, erring on > > the side of slightly too much precision. The default comparison > > tolerance should be just large enough that it won't produce noise for > > the majority of modules. Modules which require more tolerance (e.g. > > due to numerical instability) should explicitly enlarge the tolerance > > and/or set an "allowed failures" limit. > > Where to set the precision in the test case? As @precision directive Yes. > which will be used for each test in the test case file You would want to be able to set it for individual commands. > > Any directory with a Makefile should support "make test" one way or > > another. Usually via appropriate rules in Lib.make, Module.make, > > Dir.make, etc. Dir.make would just run "make test" recursively (see > > the %-recursive pattern rule in Dir.make); the others would look for a > > test script then use the framework to execute it. > > > > The top-level Makefile includes Dir.make, so "make test" would use the > > recursive rule (a special rule for testing critical modules could be > > added as a prerequisite). Testing the libraries would just be > > "make -C lib test" (i.e. recursive in the "lib" directory); similarly > > for raster, vector, etc. > > Yes. Do we need special rules for critical modules or is the order of > the module directories in the Makefile's combined with a critical > annotation sufficient? The critical annotation is sufficient. "make test" from the top-level would test everything, including the critical modules. They won't necessarily be tested first, but they will get tested. It isn't necessary to optimise the process; failures in critical modules shouldn't occur often enough for it to matter. > > I'm not sure we need a separate subdirectory for the test data. > > I am sure we need several. I would suggest a separate test directory > for each test location: "test_UTM" and "test_LL". Several modules will > only work in UTM locations, other in booth. Each directory may contain > several test case files for different modules (r.univar/r3.univar) and > several .ref files. This is going to result in the test files outweighing the actual code by an order of magnitude. A typical module consists of a directory containing one source file, one Makefile and one HTML file, plus a .svn subdirectory which takes up more space than the actual module. Each subdirectory which you add will also have its own .svn subdirectory. Bear in mind that a typical Linux filesystem uses at least 4KiB (one x86 page) for each file and directory regardless of size. > > One thing that I hadn't thought about much until now is that maps can > > have a lot of different components, different modules affect different > > components, and the correct way to perform comparisons would vary > > between components. > > Do you refer to different feature types in vector maps? Point, line, > border, area, centroid and so on? No, I mean in the sense of etc/element_list. Each raster map may have cellhd, cell, cats, colr, hist, and fcell files, plus a cell_misc directory containing whatever else (f_format, f_quant, f_range and null being typical). E.g. if you're testing r.colors, you need to check the colour table, not the raster data. -- Glynn Clements <[email protected]> _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
