> Hi, > > you may remember the last post I sent in January where I > proposed the way of how tests for XML-RPC will be organized. > Well, things changed and since I am going to finish my bachelor > thesis I would like to present you the result. > > After a lot of discussions with my supervisors and hours of > work I have managed to connect the code coverage tools. The > reason why there is code coverage is that documentation for both > APIs does not exist. Frontend API comes just with specification > of all calls, but it does not mention any behavior. The test > requirements therefore come from the source code. > > The "project" consists of two modules -- the tests module and > coverage module. > > The tests module contains BeakerLib-based tests. There is also > a BeakerLib plugin which utilizes common actions. These tests > basically call some XML-RPC procedure, assert the results, and > check whether there was an addition to the server log (httpd's > error_log or tomcat's catalina.out based on type of tested API > -- frontend/backend). This module works standalone and does not > depend on anything from coverage module. > > The coverage module extends the tests module by measuring the > code coverage (branch coverage, actually). It does need to be > installed and handled by a superuser. Installation instruments > the rhn.jar using Cobertura and updates the mod_wsgi/mod_python > configuration to use different handlers. These handlers just > start and stop the collecting of coverage data using coverage.py > and wrap the real handlers. After installation it is possible to > see which parts of code was executed using bundled scripts. > There might be a problem with superuser privileges (Cobertura > flushes the coverage data into file only on tomcat's shutdown > and therefore the tomcat must be restarted before evaluation of > coverage can start) but since it is reasonable to use it only > against development installation, it should not be a problem to > run few scripts as a superuser. > > The example of output (just tests module with one failing test > out of two): > > tests-module$ ./runtests.sh > FAIL of /auth/login: See /tmp/beakerlib-XR30U3q/journal.txt > Passed 1 out of 2 tests > > The example of output (the coverage module calling the same > tests as above): > > coverage-module$ sudo ./runtests.sh > Coverage data file were successfully reset > FAIL of /auth/login: See /tmp/beakerlib-eFh18KQ/journal.txt > Passed 1 out of 2 tests > Branch coverage: Java 1%, Python 3% > > If you are interested in the HTML coverage reports, see these: > Coverage.py HTML report (~ 1.5 MB): > http://vlki.cz/python_html_report.tar.gz Cobertura HTML report > (~ 5 MB): http://vlki.cz/java_html_report.tar.gz Everything was > tested using Spacewalk 1.2/Fedora 14. > > Well... Do you like it? Do you find it useful? > > Thanks for your thoughts, > Jan
Hello Jan. We had a discussion here and to summarise thoughts this would be nice: 1) Demonstrate benefit of your work to developers and testers of a Spacewalk project - would it be possible to show step by step, what would happen in this scenario: * developer adds new API call * tester notifies coverage percentage goes down * tester adds new test * coverage percentage gets fixed Would it be possible to have this demonstrated with some simple example by adding some simple API call and then testing it? 2) I know you plan to show how percentage is rising when you are adding more tests. That is nice as well. But for actual project community, above point is important as well. 3) Could you please post tests you have used to exercise code parts shown above? 4) You also might want to demonstrate, how your tool can help in a situation like: * tester believes he created test testing call X completely * then you comes and says: look, using your test you are testing only one of multiple possible code paths * (imagine some function with some "if system_is_virtual_guest" at the beginning being tested only from physical client) This should be majority of what I wanted to say. Anyway, I do like what you have done a lot. Thanks for a great work on this, Jan -- Jan Hutar Systems Management QA jhu...@redhat.com Red Hat, Inc.
pgpzxLiKWGhBq.pgp
Description: PGP signature
_______________________________________________ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
