Thanks for the feedbacks. We will fix them asap. Regarding tumbler-tests, nokia package was replaced with upstream package that is why the tests are no longer avaiable. Let see if we can push the tests to upstream.
Br jean-luc ----- Original message ----- > A small update: > > > 1st phase (these are really important, you can start with them): > libmeegotouch-tests > - calling with su - meego > => in use with modified de-nightly- package > > meegotouch-demos-tests > - package should have dependency to meegotouch-demos. > - test definition doesn't fetch the file produced by the test > (/tmp/timedemo.csv). > => Updated in nightly-de- package's xml and took into use > > > 2nd phase (these are tested and works ok): > I'm disagreeing with the 'works ok' -part > > contextkit-tests > -does not follow test packaging rules, files missing from the test > package. > => unusable with automation > > libcontentaction-tests > -sandbox.sh does not work in MeeGo > => unusable. > also look at > http://wiki.meego.com/Quality/QA-tools/Testrunner-lite#About_process_control > since the pre_step will not return > > tracker-tests > -calling with su - meego > => in use with modified package > > tumbler-tests > -package not available > > libqtsparql-tests > -tests.xml file in wrong location /usr/lib/libqtsparql-tests/tests.xml - > should be in /usr/share/... > - "/tmp/session_bus_address.user" doesn't exist > - /home/user/ doesn't exist > => made modified de-nightly- package and took into use > > meegotouch-controlpanel-tests > - uses wrong directory names > e.g ./usr/lib/duicontrolpanel-tests/ut_pages > and ./usr/share/duicontrolpanel-tests/tests.xml > => made modified de-nightly- package and took into use > > I'll continue going through the list but don't expect the tests to be > running too soon. I'm pretty sure that I'll have to modify most of them > to make then usable MeeGo. To be clear: tests themselves are mostly ok, > but test test-definition xml file are not. > > I'll start filing bugs about these issues today. > > -Timo > > On Thu, 2011-03-31 at 11:57 +0300, Timo Härkönen wrote: > > On Thu, 2011-03-31 at 11:15 +0300, > > miroslav.s...@tieto.com<mailto:miroslav.s...@tieto.com> wrote: > > > But why don't you use eat packages (eat-selftest)? > > > http://build.meego.com/package/files?package=eat&project=Trunk > > > > > > here is already solved the user change: > > > > http://build.meego.com/package/view_file?file=eat.spec&package=eat&project=Trunk&srcmd5=377517912783c83790c1f626b0032faa > > > > > > # %post selftest > > > # cp /etc/inittab /etc/inittab.back > > > # echo "ea:5:once:su - meego -c \"eat-selftest.sh 120\"" > > > /etc/inittab > > > > > > so the testrunner is executed as meego user.. > > > > > > > I know. I wrote the eat-selftest thingy and maintain the eat-* > packages. > > The reason why I'm not using it for this is simply that the OTS (Open > > Test system) [1] used with test automation doesn't work that way. If > we > > used the selftest thingy we would still have to write a separate test > > xml and package to fetch the results from the device and would bump > into > > timing issues, etc. > > > > If it's a problem to have su - meego -c calls in the test definition I > > can maintain the de-nightly-* packages that have them. > > > > For clarification the system is build so that a worker pc > automatically > > install an image to n900, fetches the test definition xml files for > > named tests and executes them remotely through ssh in the device and > > gets results, etc. from the tests run and uploads them into > qa-reports. > > This all is done fully automated without a need for human interaction > > with the system. > > > > We'll be making a video to illustrate the process during next week. > > Hopefully that will illustrate it and get rid of some confusion around > > test automation. > > > > -Timo > > > > [1] http://wiki.meego.com/Quality/QA-tools/OTS > > > > > I would like to keep this really simple because the test xml > definition file is changing with the library version so than when we > update library we would have to update also this project.. > > > BR > > > M.Safr > > > ________________________________________ > > > From: Timo Härkönen > > > [timo.harko...@digia.com<mailto:timo.harko...@digia.com>] > > > Sent: Thursday, March 31, 2011 10:26 AM > > > To: Safr Miroslav > > > Cc: > > > meego-qa-boun...@lists.meego.com<mailto:meego-qa-boun...@lists.meego.com> > > > Subject: RE: [Meego-qa] MTF unit tests to QA > > > > > > On Thu, 2011-03-31 at 10:18 +0300, > > > miroslav.s...@tieto.com<mailto:miroslav.s...@tieto.com> wrote: > > > > Thanks. This looks good. > > > > I don't understand why there is that de-nightly package because > for us it works normally > > > > testrunner + eat-device and eat-selftest and than it executes all > tests packages which are installed (/usr/lib/something-tests/tests.xml). > > > > > > > > > > I needed to do the package to have it call tests with su - meego -c > > > since the automation system connects to the device as root. The > selftest > > > thing runs as the user meego. > > > > > > -Timo > > > > > > > The results are right. > > > > => Executed 190 cases. Passed 183 Failed 7 > > > > in libmeegotouch 0.20.89 which is currently in trunk we did not > adapted all tests to meego environment. > > > > In newer version the pass rate will grow because we already solved > few of those.. > > > > > > > > BR > > > > Mirek Safr > > > > ________________________________________ > > > > From: > > > > meego-qa-boun...@lists.meego.com<mailto:meego-qa-boun...@lists.meego.com> > [meego-qa-boun...@lists.meego.com<mailto:meego-qa-boun...@lists.meego.com>] > On Behalf Of Timo Härkönen > [timo.harko...@digia.com<mailto:timo.harko...@digia.com>] > > > > Sent: Thursday, March 31, 2011 8:04 AM > > > > To: Wang, Jing J > > > > Cc: meego-qa@lists.meego.com<mailto:meego-qa@lists.meego.com> > > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > > > Results from the first nightly: > > > > http://qa-reports.meego.com/1.2/Core/Nightly-Automated/N900 > > > > > > > > Includes: libmeegotouch-tests and tracker-tests > > > > > > > > I'm using modified versions of the tests.xml files to run those. > > > > Basically I created new packages de-nightly-libmeegotouch-tests > and > > > > de-nightly-tracker-tests that just include a modified version of > > > > tests.xml file. These can be found here: > > > > > https://build.pub.meego.com/package/files?package=de-nightly&project=home%3Atimoph > > > > > > > > You can also see the used test definition xmls as attachments to > the > > > > test run in qa-reports > > > > > > > > Related to how to run libmeegotouch-tests I did some > experimentation > > > > > > > > 1) run them as root as they are > > > > => Executed 190 cases. Passed 48 Failed 142 > > > > 2) set xhost + to enable root x access > > > > => Executed 190 cases. Passed 77 Failed 113 > > > > 3) called the steps with su - meego -c > > > > => Executed 190 cases. Passed 183 Failed 7 > > > > > > > > Number 3 is currently used in the test run > > > > > > > > -Timo > > > > > > > > On Wed, 2011-03-30 at 10:42 +0300, Timo Härkönen wrote: > > > > > On Wed, 2011-03-30 at 10:18 +0300, Wang, Jing J wrote: > > > > > > > > I'd prefer test steps that need to connect to X to be > wrapped with 'su - meego -c "/path/to/teststep"' > > > > > > I can't agree with that as it looks kind of hack. Test case > should not be assumed executing as root naturally. Actually I think it > is better to assume test run as default user, and may change to root if > required. > > > > > > > > > > > > > > > > yes. it's a bit hackish but that's the current working way to do > it in > > > > > full automation. I've proposed a testdefinition level fix for it > in > > > > > https://bugs.meego.com/show_bug.cgi?id=11276 > > > > > > > > > > -Timo > > > > > > > > > > > -----Original Message----- > > > > > > From: > > > > > > meego-qa-boun...@lists.meego.com<mailto:meego-qa-boun...@lists.meego.com> > [mailto:meego-qa-boun...@lists.meego.com] On Behalf Of Timo H?rk?nen > > > > > > Sent: Wednesday, March 30, 2011 3:07 PM > > > > > > To: miroslav.s...@tieto.com<mailto:miroslav.s...@tieto.com> > > > > > > Cc: meego-qa@lists.meego.com<mailto:meego-qa@lists.meego.com> > > > > > > Subject: Re: [Meego-qa] MTF unit tests to QA > > > > > > > > > > > > Hi > > > > > > > > > > > > On Wed, 2011-03-30 at 09:40 +0300, > > > > > > miroslav.s...@tieto.com<mailto:miroslav.s...@tieto.com> > wrote: > > > > > > > Hi, > > > > > > > we would need to add our unit and low level funcitonal tests > to QA > > > > > > > asap: > > > > > > > > > > > > > > 1st phase (these are really important, you can start with > them): > > > > > > > libmeegotouch-tests > > > > > > > meegotouch-demos-tests > > > > > > > > > > > > > > > > > > > Working on this. setting dbus env is not enough. By default > root cannot > > > > > > access X in MeeGo and I'm a bit worried about the effect on > results when > > > > > > enabling it. I'd prefer test steps that need to connect to X > to be > > > > > > wrapped with 'su - meego -c "/path/to/teststep"' > > > > > > > > > > > > Tested two of of the packages listed below and ended up filing > bugs > > > > > > against both of them. please fix these issues. I'll continue > checking > > > > > > the other tests after I get libmeegotouch-tests running. To > get them > > > > > > running quickly I may have to fork the test package. > > > > > > > > > > > > > 2nd phase (these are tested and works ok): > > > > > > > contextkit-tests > > > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15009 > > > > > > > > > > > > > libcontentaction-tests > > > > > > > tracker-tests > > > > > > > > > > > > https://bugs.meego.com/show_bug.cgi?id=15006 > > > > > > > > > > > > > tumbler-tests > > > > > > > libqtsparql-tests > > > > > > > meegotouch-controlpanel-tests > > > > > > > meegotouch-controlpanelapplets-tests > > > > > > > meegotouch-feedback-tests > > > > > > > meegotouch-feedbackreactionmaps-tests > > > > > > > meegotouch-inputmethodengine-tests > > > > > > > meegotouch-inputmethodframework-tests > > > > > > > meegotouch-inputmethodkeyboard-tests > > > > > > > meegotouchcp-datetime-tests > > > > > > > meegotouchcp-language-tests > > > > > > > meegotouchcp-regionformat-tests > > > > > > > > > > > > > > > > > > > > > 3rd phase (these should work but can be added as last): > > > > > > > meegotouch-benchmarks (comming soon) > > > > > > > libprofile-qt-tests > > > > > > > meegotouch-applauncherd-tests > > > > > > > meegotouch-applifed-tests > > > > > > > meegotouch-compositor-tests > > > > > > > meegotouch-home-tests > > > > > > > meegotouch-systemui-tests > > > > > > > qmf-tests > > > > > > > > > > > > > > > > > > > > > 4th phase: > > > > > > > we have comming T-Driver functional tests soon.. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > execution: > > > > > > > #we need to interact with X server > > > > > > > export DISPLAY=:0 > > > > > > > #we also need dbus > > > > > > > cat /proc/`pidof mcompositor`/environ | tr '\000' '\n' | > grep > > > > > > > DBUS_SESSION_BUS_ADDRESS > > > > > > > export DBUS_SESSION_BUS_ADDRESS=... > > > > > > > > > > > > eat-device package used in automation images sets up the dbus > > > > > > environment among other things. > > > > > > > > > > > > > #for VKB tests we need this > > > > > > > export QT_IM_MODULE=MInputContext > > > > > > > > > > > > > > > > > > > This one I did not know about. Is it set for the user MeeGo by > default? > > > > > > if so I'd would call test steps related to it with su - meego > -c "step" > > > > > > > > > > > > > > > > > > > > #execution > > > > > > > > > > > > All things listed below are done automatically by ots except > we don't > > > > > > convert the results to html. Instead we send them to > > > > > > qa-reports.meego.com. Results are not being sent there yet - > I'll inform > > > > > > this list when they are. > > > > > > > > > > > > -Timo > > > > > > > > > > > > > testrunner-lite -f /usr/share/<package name>/tests.xml -v > > > > > > > -o /home/meego/test_result.xml > > > > > > > #for example > > > > > > > testrunner-lite -f /usr/share/libmeegotouch-tests/tests.xml > -v > > > > > > > -o /home/meego/test_result.xml > > > > > > > > > > > > > > #here can be also used -L parameter to redirect output > stright to host > > > > > > > as CITA does > > > > > > > > > > > > > > #get the results from device > > > > > > > scp /home/meego/test_result.xml > user@192.168.2.14<mailto:user@192.168.2.14>:/home/user/ > > > > > > > #or from host > > > > > > > scp > meego@192.168.2.15<mailto:meego@192.168.2.15>:/home/meego/test_result.xml > /home/user/ > > > > > > > > > > > > > > #get nice html > > > > > > > xsltproc /home/msafr/tools/testresults.xsl ./test_result.xml > > > > > > > > > test_result.html > > > > > > > > > > > > > > #for example: > > > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.xml > > > > > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.html > > > > > > > > > > > > > > > > > > > > > Please let me know if some results are available or if you > have some > > > > > > > problems with execution. > > > > > > > > > > > > > > Thanks. > > > > > > > BR > > > > > > > > > > > > > > Miroslav Safr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > MeeGo-qa mailing list > > > > > > MeeGo-qa@lists.meego.com<mailto:MeeGo-qa@lists.meego.com> > > > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > > > > _______________________________________________ > > > > > MeeGo-qa mailing list > > > > > MeeGo-qa@lists.meego.com<mailto:MeeGo-qa@lists.meego.com> > > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > _______________________________________________ > > > > MeeGo-qa mailing list > > > > MeeGo-qa@lists.meego.com<mailto:MeeGo-qa@lists.meego.com> > > > > http://lists.meego.com/listinfo/meego-qa > > > > > > > > > > > > _______________________________________________ > > MeeGo-qa mailing list > > MeeGo-qa@lists.meego.com<mailto:MeeGo-qa@lists.meego.com> > > http://lists.meego.com/listinfo/meego-qa > > > _______________________________________________ > MeeGo-qa mailing list > MeeGo-qa@lists.meego.com<mailto:MeeGo-qa@lists.meego.com> > http://lists.meego.com/listinfo/meego-qa >
_______________________________________________ MeeGo-qa mailing list MeeGo-qa@lists.meego.com http://lists.meego.com/listinfo/meego-qa
