28. mar. 2012 20.11 skrev Esa-Pekka Miettinen <[email protected]>: > Hi, > > Lately there has been some discussion in the Mer IRC channel about the > Mer’s quality assurance (QA). The purpose of the QA is to define > processes and tools for measuring and monitoring the quality of the > Mer project and its releases. > > 1. What is needed? > > For the first we should define a testing process: when the tests are > executed and how often, where the tests are executed and what is the > test content. > > I recommend that we start the QA process with testing the prelease > image on x86 chroot environment with simple test content. After this > we can expand the test content, automate the process and support other > architectures and devices. Also in the (near) future, the test > automation should be part of the review process.
Just weighing in with what we've been thinking of early on regarding Mer QA, -- this is in no way to be considered as the chosen direction, just input to the discussion :) The direction I'll talk about originates a bit from http://wiki.merproject.org/wiki/Process -- please feel free to correct my terms to fit the industry terms for testing :) I'll just start with the general philosophy of how we were thinking to test Mer. As you know, Mer is a Core, UIs and hardware adaptations are external and delivered by our customers, "vendors" (as defined by hardware hackers, companies, projects, etc). In addition to that, there's not supposed to be any preferential treatment of vendors or devices to the project, due to the political problems that tends to cause. These principles naturally causes a bit of problems when it comes to testing as we can't have 'reference devices'. To solve this, the idea of a 'vendor check' came into existence. When having received a proposed change into Mer (through gerrit), and having done some basic core checks based on the built sources and packages, the Mer process would then signal to vendors, based on two 'subscriber lists', one for early notification, after 24 hours, it would be signalled to the rest. The idea here was that we could catch breakage issues early with vendors like virtual machine adaptations, automated tests on reference hardware people chose to contribute to early notification, before activating the full QA from everybody contributing QA work (after 24 hour), thus saving QA bandwidth of the Mer vendor community. The other side was also that vendors could very early report to the Mer project that changes causes regressions on their own combinations of Mer + hardware adaptation + potential UI, giving us the ability to move fast and release the core as 'stable' each time. It would also give maintainers the ability to quite quickly assess if a change should go in or not. As well as having the benefit that the same tools/systems that sits in the early notification lists, are the same systems that companies can use to test their Mer products well -- ie, we use same tools as vendors would, in Mer itself. Including a new device for testing would be a matter of subscribing it to the Mer notification feeds for QA. Wild dreams here could also be to have vendors contribute test coverage statistics of their own test sets, so we could assess how much of Mer gets tested through, through vendor's 'personal' test cases as well. It also makes for good marketable values to have a proven quality core. Since Mer itself, the description of the core, is change controlled, we would apply similar QA methods to new releases/package pickups (when changes are merged, they also need integration)/etc. > > 2. QA Tools and tests > > We can reuse many of tools and test packages which were used in MeeGo > project, like testrunner-lite, OTS, QA-Reports, test-definition, > testplanner, min etc. > Only issues is that we should move those tools which are not > maintained anymore from meego.gitorious to some other location. One > idea could be that we create own project for developing and > maintaining the QA tools. That project would offer released QA > toolchain to Mer project and Mer's vendors. And also for other > projects. If you'd like to utilize the Mer gerrit/hosting, this is also possible (and perhaps get automated testing of the test tools themselves..) > > 3. Infra > > For the Mer QA we need some infra: > - Test result reporting (QA-Reports will be enough, will require one > virtual machine) > - Test management (Git may be enough) > - Test farm (One server and couple workers, not urgent yet) > > I created a wiki page for QA related stuff into Mer wiki: > http://wiki.merproject.org/wiki/Quality > > I am proposing an initial QA meeting to take place in #mer-meeting on > Tuesday’s starting 11:00 UTC (before Mer release management meeting). > > Please share your ideas and thoughts (and wild dreams). > > Thanks, > -Esa-Pekka Miettinen > >
