Sure. Each of the test runs are a 1 to 1 copy of a manual test. So, in
the same way you would add a result, we'll have a bot account add a
result with a Pass or Fail to the daily image. It should also leave a
comment linking to the run so you can learn more if you are curious.
Simon has actually agreed to hack on this, so I hope we'll start to see
some bot results (though they will be failures!) on the tracker soon. We
could still use some help with fixing the actual tests however, so they
can provide value!
Nicholas
On 03/04/2016 11:31 PM, Istimsak Abdulbasir wrote:
Could you explain more how the tracker will be able to store test
results for each daily test?
On Mar 4, 2016 3:44 PM, "Nicholas Skaggs"
<[email protected] <mailto:[email protected]>>
wrote:
It's been an up and down cycle with mostly failing Autopilot tests
for Ubiquity automated testing. I wanted to re-iterate Max's
status updates as to what's going on so people understand what
will exist for help with testing the final images for Xenial.
What's been happening:
Max has been working on various automated tests for images all
cycle. In addition to these test for ubiquity, Max is also working
on automated smoke tests, upgrade tests, etc. The AP tests for
ubiquity are fragile and can often break, especially as Ubiquity
changes. That's been happening for some time sadly. The goal is
still to have them running for Final Beta, but time is running out.
* You can see the runs here: http://162.213.34.238:8080/. They
are not nice or pretty, but they do exist!
What we need:
* Folks to come and hack on the tests and fix them! Not just fix
them, but look after them and expand them to keep them running.
Everything you need to know is in the source tree. Branch
lp:ubiquity and look at the README in the autopilot folder. Ask
questions if needed. The tests themselves aren't too difficult to
understand, but gtk2 is painful to automate at times. Dan has done
wonderful work in writing and stepping in to fix them on occasion,
but he's also writing the wonderful dekko mail client. It's time
for others to jump in, if they are interested!
* More tests! Some have suggested simple tests such as merely
attempting to boot the image. These tests are trivial to write
since the hardwork of getting the environment ready and running
them is already in place. Using autopilot or not, if you have a
simple test idea it can probably be added and run.
* A nicer way to grok the test output. Simon has recently begun
reviving the API for the tracker, and it would be lovely to
somehow display the results on the tracker. Writing a scipt to add
a test result submission against the daily test would be simple to do!
* We've now hit issues with scalability of hardware, so I'm
looking to get a dedicated machine for this. I know we turned down
hardware in the past, but it is now becoming a problem. Obviously
it's lower priority as the reality is these tests have proven
harder to keep running than imagined. As such, folks willing to
stick with them would be wonderful!
Nicholas
--
Ubuntu-quality mailing list
[email protected]
<mailto:[email protected]>
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
--
Ubuntu-quality mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
