Another update: After the things listed below nightly-automated test run contains 402 test cases. In addition I added features to sets so it's easier to read the results from qa-reports (the change will be visible after the next test run completes in about 2 hours).
That completes the phase 2. Please inform me if/when those test packages are fixed so I can stop using my own versions of them. meegotouch-controlpanelapplets-tests - su - meego -c => made modified de-nightly- package and took into use meegotouch-feedback-tests - tests.xml not in /usr/share/meegotouch-feedback-tests/ - su - meego (without it 1 case failed) => made modified de-nightly- package and took into use meegotouch-feedbackreactionmaps-tests - tests.xml not in /usr/share/meegotouch-feedbackreactionmaps-tests/ => made modified de-nightly- package and took into use meegotouch-inputmethodengine-tests - tests.xml not in /usr/share/meegotouch-inputmethodengine-tests/ => made modified de-nightly- package and took into use meegotouch-inputmethodframework-tests - tests.xml not in /usr/share/meegotouch-inputmethodframework-tests/ - su - meego => made modified de-nightly- package and took into use meegotouch-inputmethodkeyboard-tests - tests.xml not in /usr/share/meegotouch-inputmethodkeyboard-tests/ - su - meego => made modified de-nightly- package and took into use meegotouchcp-datetime-tests - tests.xml not in /usr/share/meegotouchcp-datetime-tests/ - su - meego => made modified de-nightly- package and took into use meegotouchcp-language-tests - tests.xml not in /usr/share/meegotouchcp-language-tests/ - su - meego => made modified de-nightly- package and took into use meegotouchcp-regionformat-tests - tests.xml not in /usr/share/meegotouchcp-regionformat-tests/ - su - meego => made modified de-nightly- package and took into use -Timo On Fri, 2011-04-01 at 07:55 +0300, Timo Härkönen wrote: > A small update: > > >1st phase (these are really important, you can start with them): > libmeegotouch-tests > - calling with su - meego > => in use with modified de-nightly- package > > meegotouch-demos-tests > - package should have dependency to meegotouch-demos. > - test definition doesn't fetch the file produced by the test > (/tmp/timedemo.csv). > => Updated in nightly-de- package's xml and took into use > > >2nd phase (these are tested and works ok): > I'm disagreeing with the 'works ok' -part > > contextkit-tests > -does not follow test packaging rules, files missing from the test > package. > => unusable with automation > > libcontentaction-tests > -sandbox.sh does not work in MeeGo > => unusable. > also look at > http://wiki.meego.com/Quality/QA-tools/Testrunner-lite#About_process_control > since the pre_step will not return > > tracker-tests > -calling with su - meego > => in use with modified package > > tumbler-tests > -package not available > > libqtsparql-tests > -tests.xml file in wrong location /usr/lib/libqtsparql-tests/tests.xml - > should be in /usr/share/... > - "/tmp/session_bus_address.user" doesn't exist > - /home/user/ doesn't exist > => made modified de-nightly- package and took into use > > meegotouch-controlpanel-tests > - uses wrong directory names > e.g ./usr/lib/duicontrolpanel-tests/ut_pages > and ./usr/share/duicontrolpanel-tests/tests.xml > => made modified de-nightly- package and took into use > > I'll continue going through the list but don't expect the tests to be > running too soon. I'm pretty sure that I'll have to modify most of them > to make then usable MeeGo. To be clear: tests themselves are mostly ok, > but test test-definition xml file are not. > > I'll start filing bugs about these issues today. > > -Timo > > On Thu, 2011-03-31 at 11:57 +0300, Timo Härkönen wrote: > > On Thu, 2011-03-31 at 11:15 +0300, miroslav.s...@tieto.com wrote: > > > But why don't you use eat packages (eat-selftest)? > > > http://build.meego.com/package/files?package=eat&project=Trunk > > > > > > here is already solved the user change: > > > http://build.meego.com/package/view_file?file=eat.spec&package=eat&project=Trunk&srcmd5=377517912783c83790c1f626b0032faa > > > > > > # %post selftest > > > # cp /etc/inittab /etc/inittab.back > > > # echo "ea:5:once:su - meego -c \"eat-selftest.sh 120\"" >> /etc/inittab > > > > > > so the testrunner is executed as meego user.. > > > > > > > I know. I wrote the eat-selftest thingy and maintain the eat-* packages. > > The reason why I'm not using it for this is simply that the OTS (Open > > Test system) [1] used with test automation doesn't work that way. If we > > used the selftest thingy we would still have to write a separate test > > xml and package to fetch the results from the device and would bump into > > timing issues, etc. > > > > If it's a problem to have su - meego -c calls in the test definition I > > can maintain the de-nightly-* packages that have them. > > > > For clarification the system is build so that a worker pc automatically > > install an image to n900, fetches the test definition xml files for > > named tests and executes them remotely through ssh in the device and > > gets results, etc. from the tests run and uploads them into qa-reports. > > This all is done fully automated without a need for human interaction > > with the system. > > > > We'll be making a video to illustrate the process during next week. > > Hopefully that will illustrate it and get rid of some confusion around > > test automation. > > > > -Timo > > > > [1] http://wiki.meego.com/Quality/QA-tools/OTS > > > > > I would like to keep this really simple because the test xml definition > > > file is changing with the library version so than when we update library > > > we would have to update also this project.. > > > BR > > > M.Safr > > > ________________________________________ > > > From: Timo Härkönen [timo.harko...@digia.com] > > > Sent: Thursday, March 31, 2011 10:26 AM > > > To: Safr Miroslav > > > Cc: meego-qa-boun...@lists.meego.com > > > Subject: RE: [Meego-qa] MTF unit tests to QA > > > > > > On Thu, 2011-03-31 at 10:18 +0300, miroslav.s...@tieto.com wrote: > > > > Thanks. This looks good. > > > > I don't understand why there is that de-nightly package because for us > > > > it works normally > > > > testrunner + eat-device and eat-selftest and than it executes all tests > > > > packages which are installed (/usr/lib/something-tests/tests.xml). > > > > > > > > > > I needed to do the package to have it call tests with su - meego -c > > > since the automation system connects to the device as root. The selftest > > > thing runs as the user meego. > > > > > > -Timo > > > > > > > The results are right. > > > > => Executed 190 cases. Passed 183 Failed 7 > > > > in libmeegotouch 0.20.89 which is currently in trunk we did not adapted > > > > all tests to meego environment. > > > > In newer version the pass rate will grow because we already solved few > > > > of those.. > > > > > > > > BR > > > > Mirek Safr > > > > ________________________________________ > > > > From: meego-qa-boun...@lists.meego.com > > > > [meego-qa-boun...@lists.meego.com] On Behalf Of Timo Härkönen > > > > [timo.harko...@digia.com] > > > > Sent: Thursday, March 31, 2011 8:04 AM > > > > To: Wang, Jing J > > > > Cc: meego-qa@lists.meego.com > > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > > > Results from the first nightly: > > > > http://qa-reports.meego.com/1.2/Core/Nightly-Automated/N900 > > > > > > > > Includes: libmeegotouch-tests and tracker-tests > > > > > > > > I'm using modified versions of the tests.xml files to run those. > > > > Basically I created new packages de-nightly-libmeegotouch-tests and > > > > de-nightly-tracker-tests that just include a modified version of > > > > tests.xml file. These can be found here: > > > > https://build.pub.meego.com/package/files?package=de-nightly&project=home%3Atimoph > > > > > > > > You can also see the used test definition xmls as attachments to the > > > > test run in qa-reports > > > > > > > > Related to how to run libmeegotouch-tests I did some experimentation > > > > > > > > 1) run them as root as they are > > > > => Executed 190 cases. Passed 48 Failed 142 > > > > 2) set xhost + to enable root x access > > > > => Executed 190 cases. Passed 77 Failed 113 > > > > 3) called the steps with su - meego -c > > > > => Executed 190 cases. Passed 183 Failed 7 > > > > > > > > Number 3 is currently used in the test run > > > > > > > > -Timo > > > > > > > > On Wed, 2011-03-30 at 10:42 +0300, Timo Härkönen wrote: > > > > > On Wed, 2011-03-30 at 10:18 +0300, Wang, Jing J wrote: > > > > > > >> I'd prefer test steps that need to connect to X to be wrapped > > > > > > >> with 'su - meego -c "/path/to/teststep"' > > > > > > I can't agree with that as it looks kind of hack. Test case should > > > > > > not be assumed executing as root naturally. Actually I think it is > > > > > > better to assume test run as default user, and may change to root > > > > > > if required. > > > > > > > > > > > > > > > > yes. it's a bit hackish but that's the current working way to do it in > > > > > full automation. I've proposed a testdefinition level fix for it in > > > > > https://bugs.meego.com/show_bug.cgi?id=11276 > > > > > > > > > > -Timo > > > > > > > > > > > -----Original Message----- > > > > > > From: meego-qa-boun...@lists.meego.com > > > > > > [mailto:meego-qa-boun...@lists.meego.com] On Behalf Of Timo H?rk?nen > > > > > > Sent: Wednesday, March 30, 2011 3:07 PM > > > > > > To: miroslav.s...@tieto.com > > > > > > Cc: meego-qa@lists.meego.com > > > > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > > > > > > > Hi > > > > > > > > > > > > On Wed, 2011-03-30 at 09:40 +0300, miroslav.s...@tieto.com wrote: > > > > > > > Hi, > > > > > > > we would need to add our unit and low level funcitonal tests to QA > > > > > > > asap: > > > > > > > > > > > > > > 1st phase (these are really important, you can start with them): > > > > > > > libmeegotouch-tests > > > > > > > meegotouch-demos-tests > > > > > > > > > > > > > > > > > > > Working on this. setting dbus env is not enough. By default root > > > > > > cannot > > > > > > access X in MeeGo and I'm a bit worried about the effect on results > > > > > > when > > > > > > enabling it. I'd prefer test steps that need to connect to X to be > > > > > > wrapped with 'su - meego -c "/path/to/teststep"' > > > > > > > > > > > > Tested two of of the packages listed below and ended up filing bugs > > > > > > against both of them. please fix these issues. I'll continue > > > > > > checking > > > > > > the other tests after I get libmeegotouch-tests running. To get them > > > > > > running quickly I may have to fork the test package. > > > > > > > > > > > > > 2nd phase (these are tested and works ok): > > > > > > > contextkit-tests > > > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15009 > > > > > > > > > > > > > libcontentaction-tests > > > > > > > tracker-tests > > > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15006 > > > > > > > > > > > > > tumbler-tests > > > > > > > libqtsparql-tests > > > > > > > meegotouch-controlpanel-tests > > > > > > > meegotouch-controlpanelapplets-tests > > > > > > > meegotouch-feedback-tests > > > > > > > meegotouch-feedbackreactionmaps-tests > > > > > > > meegotouch-inputmethodengine-tests > > > > > > > meegotouch-inputmethodframework-tests > > > > > > > meegotouch-inputmethodkeyboard-tests > > > > > > > meegotouchcp-datetime-tests > > > > > > > meegotouchcp-language-tests > > > > > > > meegotouchcp-regionformat-tests > > > > > > > > > > > > > > > > > > > > > 3rd phase (these should work but can be added as last): > > > > > > > meegotouch-benchmarks (comming soon) > > > > > > > libprofile-qt-tests > > > > > > > meegotouch-applauncherd-tests > > > > > > > meegotouch-applifed-tests > > > > > > > meegotouch-compositor-tests > > > > > > > meegotouch-home-tests > > > > > > > meegotouch-systemui-tests > > > > > > > qmf-tests > > > > > > > > > > > > > > > > > > > > > 4th phase: > > > > > > > we have comming T-Driver functional tests soon.. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > execution: > > > > > > > #we need to interact with X server > > > > > > > export DISPLAY=:0 > > > > > > > #we also need dbus > > > > > > > cat /proc/`pidof mcompositor`/environ | tr '\000' '\n' | grep > > > > > > > DBUS_SESSION_BUS_ADDRESS > > > > > > > export DBUS_SESSION_BUS_ADDRESS=... > > > > > > > > > > > > eat-device package used in automation images sets up the dbus > > > > > > environment among other things. > > > > > > > > > > > > > #for VKB tests we need this > > > > > > > export QT_IM_MODULE=MInputContext > > > > > > > > > > > > > > > > > > > This one I did not know about. Is it set for the user MeeGo by > > > > > > default? > > > > > > if so I'd would call test steps related to it with su - meego -c > > > > > > "step" > > > > > > > > > > > > > > > > > > > > #execution > > > > > > > > > > > > All things listed below are done automatically by ots except we > > > > > > don't > > > > > > convert the results to html. Instead we send them to > > > > > > qa-reports.meego.com. Results are not being sent there yet - I'll > > > > > > inform > > > > > > this list when they are. > > > > > > > > > > > > -Timo > > > > > > > > > > > > > testrunner-lite -f /usr/share/<package name>/tests.xml -v > > > > > > > -o /home/meego/test_result.xml > > > > > > > #for example > > > > > > > testrunner-lite -f /usr/share/libmeegotouch-tests/tests.xml -v > > > > > > > -o /home/meego/test_result.xml > > > > > > > > > > > > > > #here can be also used -L parameter to redirect output stright to > > > > > > > host > > > > > > > as CITA does > > > > > > > > > > > > > > #get the results from device > > > > > > > scp /home/meego/test_result.xml user@192.168.2.14:/home/user/ > > > > > > > #or from host > > > > > > > scp meego@192.168.2.15:/home/meego/test_result.xml /home/user/ > > > > > > > > > > > > > > #get nice html > > > > > > > xsltproc /home/msafr/tools/testresults.xsl ./test_result.xml > > > > > > > > test_result.html > > > > > > > > > > > > > > #for example: > > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.xml > > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.html > > > > > > > > > > > > > > > > > > > > > Please let me know if some results are available or if you have > > > > > > > some > > > > > > > problems with execution. > > > > > > > > > > > > > > Thanks. > > > > > > > BR > > > > > > > > > > > > > > Miroslav Safr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > MeeGo-qa mailing list > > > > > > MeeGo-qa@lists.meego.com > > > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > > > > _______________________________________________ > > > > > MeeGo-qa mailing list > > > > > MeeGo-qa@lists.meego.com > > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > _______________________________________________ > > > > MeeGo-qa mailing list > > > > MeeGo-qa@lists.meego.com > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > _______________________________________________ > > MeeGo-qa mailing list > > MeeGo-qa@lists.meego.com > > http://lists.meego.com/listinfo/meego-qa > > > _______________________________________________ > MeeGo-qa mailing list > MeeGo-qa@lists.meego.com > http://lists.meego.com/listinfo/meego-qa _______________________________________________ MeeGo-qa mailing list MeeGo-qa@lists.meego.com http://lists.meego.com/listinfo/meego-qa
