Hi, [snip] >> It saves us writing lots of functions that would compare datatypes; but >> OTOH it would add dependencies of other GRASS modules. Then, it will be >> necessary (or at least better) to test first the modules without >> dependencies. We would need to check if a test includes more than one >> GRASS module... or just let all tests run and fail, and then understand >> which modules are actually broken, and causing other (correct) modules' >> test to fail. > > I don't think that there's any way around this. Few GRASS modules are > useful in isolation. On the positive side, modules which are essential > for tests (e.g. g.region, *.out.ascii) etc are unlikely to be changed > regularly.
Indeed, we will need several modules for data generation, region settings and data output. The testsuite for grass 6.4 uses g.region, r/r3/v.out.ascii, r.mapcalc and several other modules. >> We chose Python, as Sören already wrote tests as shell scripts and found >> them too cumbersome and difficult to mantain. > > But is this because you're trying to perform analysis of the results > within the test scripts? In many cases there are situtation where you will need to perform several independent tests for a single module with several kinds of outputs each test: * Module output on stdout * single or multiple raster and voxel maps * single or multiple vector maps * mixed raster and vector output * and so on The framework should be able to handle such output automatically i.e: * parsing the stout and compare it with validation data (text file, MD5 checksum, ...) * Generation of ASCII output from raster, vector and voxel maps to compare with validation data (text file, MD5 checksum, ...) To make it as easy as possible the writer of the test should only provide the command line of the test and data generation and the validation data. I tried to implement this using the shell approach, but because of the limitation of the shell this was much to complex. I am not sure if this kind of data handling can be done using only the make system? The next reason is the generation of HTML representation of the test results for web access. The idea is to run tests on several different configured server after compilation. The result of these tests will be posted in the web. This is an important feature, because the developer will see what modules/libraries failed the tests on which machine configuration. Have a look at the VTK dashboard[1] and the result of the old testsuite [2]. The old testsuite has the possibility to run each module in a valgrind environment to find memory leaks. It is able to validate outputs of different kind against MD5 checksums. It logs stdout and stderr of the module and provides this data as HTML output. [3] It links to the test description [4]. Using python the implementation will be much easier as with shell and the related unix toolset. Best regards Soeren [1] http://www.cdash.org/CDash/index.php?project=VTK [2] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4/ [3] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4/log.html [4] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4//r.gwflow-test.sh.html > > -- > Glynn Clements <[email protected]> > _______________________________________________ > grass-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/grass-dev > _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
