> There are plenty of test frameworks. The question still stands: > are we allowed to run them as part of the .spec file and how do > we get test results back out of an OBS build?
The need for this kind of test automation is obvious. When a developer changes code, tests may notice that something broke up. Those tests may have be written for the modified component, or for some other component that took a hit from the change. Regardless of the problem being in the code or in the tests, as a developer, I would like to get an immediate a notification on the problem. After that, I would like to access test logs, download the image, and download the tests that were executed. Especially important, of course, are the tests that I broke. After that, I would be able to fix my code, broken tests, or both. In the system that Igor described, tests are written in separate test packages that are installed on the system that is connected to Tizen IVI HW. Note: packages are not installed on the HW, because that would limit controlling the environment, the device under test, and thus what you can test with them. On the other hand, it's not a problem to install packages and run arbitrary code on the HW from the controlling host. I believe this is convenient in the developer perspective, too. A developer can install test packages on his/her development environment, run scripts, and check out the results. We've tried to make the criteria for test packages as loose as possible, to make it easy to use any test framework and tools. For instance, packages can contain any data files and have any dependencies that can be installed on the controlling host. Test executables in packages (Python scripts in our case) are allowed to write anything under the current working directory, and everything that they leave behind is considered as outcome of the test. Non-zero exit status from an executable indicates a failing test. Outcome is stored, and it can be downloaded later on. Thus image, tests and outcome will be accessible for developers of the code and also developers of the test package. How would this sound in Moblin/MeeGo/Tizen developer perspective? Regarding fMBT, I've written test scripts to current test packages with fmbt-scripter. It's pretty handy for writing, running, debugging and even partially recording Python scripts that exercise GUI. If somebody wants to try it out, here's a link to a slide set that I was presenting on test automation day last week: http://www.cs.tut.fi/tapahtumat/testiautomaatio14/kalvot/Kervinen_fmbtgti.pdf > Ideally, I would like QA to be involved, too, but in general I > agree, it should be the developers owning that task and QA > QA engineers helping them. At minimum, I would hope that this would result in better quality images for the QA, which in turn would leave them more time to look for less obvious bugs. Yours, Antti _______________________________________________ Dev mailing list [email protected] https://lists.tizen.org/listinfo/dev
