A small update: >1st phase (these are really important, you can start with them): libmeegotouch-tests - calling with su - meego => in use with modified de-nightly- package
meegotouch-demos-tests - package should have dependency to meegotouch-demos. - test definition doesn't fetch the file produced by the test (/tmp/timedemo.csv). => Updated in nightly-de- package's xml and took into use >2nd phase (these are tested and works ok): I'm disagreeing with the 'works ok' -part contextkit-tests -does not follow test packaging rules, files missing from the test package. => unusable with automation libcontentaction-tests -sandbox.sh does not work in MeeGo => unusable. also look at http://wiki.meego.com/Quality/QA-tools/Testrunner-lite#About_process_control since the pre_step will not return tracker-tests -calling with su - meego => in use with modified package tumbler-tests -package not available libqtsparql-tests -tests.xml file in wrong location /usr/lib/libqtsparql-tests/tests.xml - should be in /usr/share/... - "/tmp/session_bus_address.user" doesn't exist - /home/user/ doesn't exist => made modified de-nightly- package and took into use meegotouch-controlpanel-tests - uses wrong directory names e.g ./usr/lib/duicontrolpanel-tests/ut_pages and ./usr/share/duicontrolpanel-tests/tests.xml => made modified de-nightly- package and took into use I'll continue going through the list but don't expect the tests to be running too soon. I'm pretty sure that I'll have to modify most of them to make then usable MeeGo. To be clear: tests themselves are mostly ok, but test test-definition xml file are not. I'll start filing bugs about these issues today. -Timo On Thu, 2011-03-31 at 11:57 +0300, Timo Härkönen wrote: > On Thu, 2011-03-31 at 11:15 +0300, miroslav.s...@tieto.com wrote: > > But why don't you use eat packages (eat-selftest)? > > http://build.meego.com/package/files?package=eat&project=Trunk > > > > here is already solved the user change: > > http://build.meego.com/package/view_file?file=eat.spec&package=eat&project=Trunk&srcmd5=377517912783c83790c1f626b0032faa > > > > # %post selftest > > # cp /etc/inittab /etc/inittab.back > > # echo "ea:5:once:su - meego -c \"eat-selftest.sh 120\"" >> /etc/inittab > > > > so the testrunner is executed as meego user.. > > > > I know. I wrote the eat-selftest thingy and maintain the eat-* packages. > The reason why I'm not using it for this is simply that the OTS (Open > Test system) [1] used with test automation doesn't work that way. If we > used the selftest thingy we would still have to write a separate test > xml and package to fetch the results from the device and would bump into > timing issues, etc. > > If it's a problem to have su - meego -c calls in the test definition I > can maintain the de-nightly-* packages that have them. > > For clarification the system is build so that a worker pc automatically > install an image to n900, fetches the test definition xml files for > named tests and executes them remotely through ssh in the device and > gets results, etc. from the tests run and uploads them into qa-reports. > This all is done fully automated without a need for human interaction > with the system. > > We'll be making a video to illustrate the process during next week. > Hopefully that will illustrate it and get rid of some confusion around > test automation. > > -Timo > > [1] http://wiki.meego.com/Quality/QA-tools/OTS > > > I would like to keep this really simple because the test xml definition > > file is changing with the library version so than when we update library we > > would have to update also this project.. > > BR > > M.Safr > > ________________________________________ > > From: Timo Härkönen [timo.harko...@digia.com] > > Sent: Thursday, March 31, 2011 10:26 AM > > To: Safr Miroslav > > Cc: meego-qa-boun...@lists.meego.com > > Subject: RE: [Meego-qa] MTF unit tests to QA > > > > On Thu, 2011-03-31 at 10:18 +0300, miroslav.s...@tieto.com wrote: > > > Thanks. This looks good. > > > I don't understand why there is that de-nightly package because for us it > > > works normally > > > testrunner + eat-device and eat-selftest and than it executes all tests > > > packages which are installed (/usr/lib/something-tests/tests.xml). > > > > > > > I needed to do the package to have it call tests with su - meego -c > > since the automation system connects to the device as root. The selftest > > thing runs as the user meego. > > > > -Timo > > > > > The results are right. > > > => Executed 190 cases. Passed 183 Failed 7 > > > in libmeegotouch 0.20.89 which is currently in trunk we did not adapted > > > all tests to meego environment. > > > In newer version the pass rate will grow because we already solved few of > > > those.. > > > > > > BR > > > Mirek Safr > > > ________________________________________ > > > From: meego-qa-boun...@lists.meego.com [meego-qa-boun...@lists.meego.com] > > > On Behalf Of Timo Härkönen [timo.harko...@digia.com] > > > Sent: Thursday, March 31, 2011 8:04 AM > > > To: Wang, Jing J > > > Cc: meego-qa@lists.meego.com > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > Results from the first nightly: > > > http://qa-reports.meego.com/1.2/Core/Nightly-Automated/N900 > > > > > > Includes: libmeegotouch-tests and tracker-tests > > > > > > I'm using modified versions of the tests.xml files to run those. > > > Basically I created new packages de-nightly-libmeegotouch-tests and > > > de-nightly-tracker-tests that just include a modified version of > > > tests.xml file. These can be found here: > > > https://build.pub.meego.com/package/files?package=de-nightly&project=home%3Atimoph > > > > > > You can also see the used test definition xmls as attachments to the > > > test run in qa-reports > > > > > > Related to how to run libmeegotouch-tests I did some experimentation > > > > > > 1) run them as root as they are > > > => Executed 190 cases. Passed 48 Failed 142 > > > 2) set xhost + to enable root x access > > > => Executed 190 cases. Passed 77 Failed 113 > > > 3) called the steps with su - meego -c > > > => Executed 190 cases. Passed 183 Failed 7 > > > > > > Number 3 is currently used in the test run > > > > > > -Timo > > > > > > On Wed, 2011-03-30 at 10:42 +0300, Timo Härkönen wrote: > > > > On Wed, 2011-03-30 at 10:18 +0300, Wang, Jing J wrote: > > > > > >> I'd prefer test steps that need to connect to X to be wrapped with > > > > > >> 'su - meego -c "/path/to/teststep"' > > > > > I can't agree with that as it looks kind of hack. Test case should > > > > > not be assumed executing as root naturally. Actually I think it is > > > > > better to assume test run as default user, and may change to root if > > > > > required. > > > > > > > > > > > > > yes. it's a bit hackish but that's the current working way to do it in > > > > full automation. I've proposed a testdefinition level fix for it in > > > > https://bugs.meego.com/show_bug.cgi?id=11276 > > > > > > > > -Timo > > > > > > > > > -----Original Message----- > > > > > From: meego-qa-boun...@lists.meego.com > > > > > [mailto:meego-qa-boun...@lists.meego.com] On Behalf Of Timo H?rk?nen > > > > > Sent: Wednesday, March 30, 2011 3:07 PM > > > > > To: miroslav.s...@tieto.com > > > > > Cc: meego-qa@lists.meego.com > > > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > > > > > Hi > > > > > > > > > > On Wed, 2011-03-30 at 09:40 +0300, miroslav.s...@tieto.com wrote: > > > > > > Hi, > > > > > > we would need to add our unit and low level funcitonal tests to QA > > > > > > asap: > > > > > > > > > > > > 1st phase (these are really important, you can start with them): > > > > > > libmeegotouch-tests > > > > > > meegotouch-demos-tests > > > > > > > > > > > > > > > > Working on this. setting dbus env is not enough. By default root > > > > > cannot > > > > > access X in MeeGo and I'm a bit worried about the effect on results > > > > > when > > > > > enabling it. I'd prefer test steps that need to connect to X to be > > > > > wrapped with 'su - meego -c "/path/to/teststep"' > > > > > > > > > > Tested two of of the packages listed below and ended up filing bugs > > > > > against both of them. please fix these issues. I'll continue checking > > > > > the other tests after I get libmeegotouch-tests running. To get them > > > > > running quickly I may have to fork the test package. > > > > > > > > > > > 2nd phase (these are tested and works ok): > > > > > > contextkit-tests > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15009 > > > > > > > > > > > libcontentaction-tests > > > > > > tracker-tests > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15006 > > > > > > > > > > > tumbler-tests > > > > > > libqtsparql-tests > > > > > > meegotouch-controlpanel-tests > > > > > > meegotouch-controlpanelapplets-tests > > > > > > meegotouch-feedback-tests > > > > > > meegotouch-feedbackreactionmaps-tests > > > > > > meegotouch-inputmethodengine-tests > > > > > > meegotouch-inputmethodframework-tests > > > > > > meegotouch-inputmethodkeyboard-tests > > > > > > meegotouchcp-datetime-tests > > > > > > meegotouchcp-language-tests > > > > > > meegotouchcp-regionformat-tests > > > > > > > > > > > > > > > > > > 3rd phase (these should work but can be added as last): > > > > > > meegotouch-benchmarks (comming soon) > > > > > > libprofile-qt-tests > > > > > > meegotouch-applauncherd-tests > > > > > > meegotouch-applifed-tests > > > > > > meegotouch-compositor-tests > > > > > > meegotouch-home-tests > > > > > > meegotouch-systemui-tests > > > > > > qmf-tests > > > > > > > > > > > > > > > > > > 4th phase: > > > > > > we have comming T-Driver functional tests soon.. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > execution: > > > > > > #we need to interact with X server > > > > > > export DISPLAY=:0 > > > > > > #we also need dbus > > > > > > cat /proc/`pidof mcompositor`/environ | tr '\000' '\n' | grep > > > > > > DBUS_SESSION_BUS_ADDRESS > > > > > > export DBUS_SESSION_BUS_ADDRESS=... > > > > > > > > > > eat-device package used in automation images sets up the dbus > > > > > environment among other things. > > > > > > > > > > > #for VKB tests we need this > > > > > > export QT_IM_MODULE=MInputContext > > > > > > > > > > > > > > > > This one I did not know about. Is it set for the user MeeGo by > > > > > default? > > > > > if so I'd would call test steps related to it with su - meego -c > > > > > "step" > > > > > > > > > > > > > > > > > #execution > > > > > > > > > > All things listed below are done automatically by ots except we don't > > > > > convert the results to html. Instead we send them to > > > > > qa-reports.meego.com. Results are not being sent there yet - I'll > > > > > inform > > > > > this list when they are. > > > > > > > > > > -Timo > > > > > > > > > > > testrunner-lite -f /usr/share/<package name>/tests.xml -v > > > > > > -o /home/meego/test_result.xml > > > > > > #for example > > > > > > testrunner-lite -f /usr/share/libmeegotouch-tests/tests.xml -v > > > > > > -o /home/meego/test_result.xml > > > > > > > > > > > > #here can be also used -L parameter to redirect output stright to > > > > > > host > > > > > > as CITA does > > > > > > > > > > > > #get the results from device > > > > > > scp /home/meego/test_result.xml user@192.168.2.14:/home/user/ > > > > > > #or from host > > > > > > scp meego@192.168.2.15:/home/meego/test_result.xml /home/user/ > > > > > > > > > > > > #get nice html > > > > > > xsltproc /home/msafr/tools/testresults.xsl ./test_result.xml > > > > > > > test_result.html > > > > > > > > > > > > #for example: > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.xml > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.html > > > > > > > > > > > > > > > > > > Please let me know if some results are available or if you have some > > > > > > problems with execution. > > > > > > > > > > > > Thanks. > > > > > > BR > > > > > > > > > > > > Miroslav Safr > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > MeeGo-qa mailing list > > > > > MeeGo-qa@lists.meego.com > > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > _______________________________________________ > > > > MeeGo-qa mailing list > > > > MeeGo-qa@lists.meego.com > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > _______________________________________________ > > > MeeGo-qa mailing list > > > MeeGo-qa@lists.meego.com > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > _______________________________________________ > MeeGo-qa mailing list > MeeGo-qa@lists.meego.com > http://lists.meego.com/listinfo/meego-qa _______________________________________________ MeeGo-qa mailing list MeeGo-qa@lists.meego.com http://lists.meego.com/listinfo/meego-qa
