Just an update to talk about the unity autopilot tests I included as
part of our testing this week based on feedback from the initial folks
who have attempted to run and report results.
First, please don't file bugs against unity for any test failures you
may encounter. Because the testsuite is still new, there are some tests
that may fail -- don't be alarmed if you have many failures. In order to
generate useful data, I will be manually collating the testing data from
this week and passing the information along to the unity team. I repeat,
please don't file bugs for now -- this won't be helpful to the unity
team.
Because the testsuite is so long to run in one sitting, the testcase has
been updated to explain how to run sections of the suite, as well as how
to log the results from each testcase by attaching a link to your log
via paste.ubuntu.com (or your favorite paste site :-) ). For example,
I've posted a couple results from me personally running the tests here:
http://packages.qa.ubuntu.com/qatracker/milestones/246/builds/28425/testcases/1466/results
Switcher, PASSED:
http://paste.ubuntu.com/1390078/
Quicklists, FAILED:
http://paste.ubuntu.com/1390112/
Since your not reporting any bugs, you'll have to mark everything as
'passed' (sadface :-( ), so be sure to post your failures in front of
the log when you file your results. Longer term, better reporting for
these autopilot testcases will be explored, but for now I'm enjoying
generating some logs and seeing what autopilot is capable of. Just
remember the more people who generate test results, the more work I get
to do on collating them.. haha.. Make me sweat!
Nicholas
P.S. Keep the feedback coming. I want to get people familiar with the
idea of running these autopilot testcases -- as after all, this cycle
we're going to make more :-)
On 11/25/2012 11:56 PM, Nicholas Skaggs wrote:
Ok, so we're a bit delayed in starting this because, well, there was a
Holiday in North America called Thanksgiving. Forgive me. :-p
So, this week (through Saturday) is our first cadence week. What that
means is we will be focusing testing specific packages (or isos)
during the week. What you can do to help is to run through a testcase
(or a few :-) ) at some point during the next week and report your
results. The goal here is NOT to make all the tests green everyday.
Instead, we want to test on a deeper level and really look for bugs in
the software (or even the testcases :-) ) so we can be confident in
the health of the package(s) we are testing. To that end, there is no
daily target -- rather we want to test through Saturday (whenever that
is in your localtime ;-) ). Think quality, not quantity (I know, it's
a bad pun.. quality quality...)
What this means practically is during the next week, try to execute
some testcases for our week 1 targets (which are libreoffice, unity
autopilot tests and the daily raring iso). You can find the wiki page
detailing this here:
https://wiki.ubuntu.com/QATeam/Cadence/Raring/Week1
The master schedule can be found here:
https://wiki.ubuntu.com/QATeam/Cadence/Raring
Again, I would suggest finding a day and time that works for you and
adding your results. Feel free to use IRC (#ubuntu-quality on
freenode) or this mailing list to discuss any bugs you may find. At
the end of the week, I'll recap what we did (bugs, tests ran, etc),
and we can evaluate the results to see if further followup and focus
is needed or not. Now, our testing focus's each week will change in
order to target packages that need more QA work, or new packages that
recently landed in QA.
Now, this week I will also be posting the next bits in our autopilot
series -- intending to teach you about running and writing autopilot
testcases. Our cadence testing this cycle will include running both
automated and manual tests -- we need both ;-) You'll note the
autopilot unity testsuite is included this week. I hope our subsequent
testing weeks can feature some new automated autopilot tests written
by some of you :-) If writing automated tests aren't up your alley,
don't worry we need manual testcases too!
Happy Testing everyone,
Nicholas
