Hi folks! I am currently rolling out some changes to the Fedora openQA deployment which enable a new testing workflow. From now on, a subset of openQA tests should be run automatically on every critpath update, both on initial submission and on any edit of the update.
For the next little while, at least, this won't be incredibly visible. openQA sends out fedmsgs for all tests, so you can sign up for FMN notifications to learn about these results. They'll also be discoverable from the openQA web UI - https://openqa.fedoraproject.org . The results are also being forwarded to ResultsDB, so they'll be visible via ResultsDB API queries and the ResultsDB web UI. But for now, that's it...I think. Our intent is to set up the necessary bits so that these results will show up in the Bodhi web UI alongside the results for relevant Taskotron tests. There's an outside possibility that Bodhi is actually already set up to find these results in ResultsDB, in which case they'll just suddenly start showing up in Bodhi - we should know about that soon enough. :) But most likely Bodhi will need a bit of a tweak to find them. This is probably a good thing, because we need to let the tests run for a while to find out how reliable they are, and if there's an unacceptable number of false negatives/positives. Once we have some info on that and are happy that we can get things sufficiently reliable for the results to be useful, we'll hook up the Bodhi integration. The tests that are run are most of the tests that, on the 'compose test' workflow, get run on the Server DVD and Workstation Live images after installation. Between them they do a decent job of covering basic system functionality. They also cover FreeIPA server and client setup, and Workstation browser (Firefox) and terminal functionality. So hopefully, if your critpath update completely breaks one of those basic workflows, you'll find out about it before pushing it stable. At present it looks like the Workstation tests may sometimes fail simply because the base install gets stuck during boot for some reason; I'm going to look into that this week. In testing so far the Server tests seem fairly reliable, but I want to gather data from a few days worth of test runs to see how those look. Once we start sending results to Bodhi, I'll try and write up some basic instructions on how to interpret and debug openQA test results; QA folks will also be available in IRC and by email for help with this, of course. You can see sample runs on Server: https://openqa.stg.fedoraproject.org/tests/overview?groupid=1&build=FEDORA-2017-376ae2b92c&version=25&distri=fedora and Workstation: https://openqa.stg.fedoraproject.org/tests/overview?version=25&distri=fedora&build=FEDORA-2017-87896dfb59&groupid=1 the 'desktop_notifications_live' failure is a stale bit of data - that test isn't actually run any more because obviously it makes no sense in this context, but because it got run one time in early development, openQA continues to show it for that update (it won't show for any *other* update). The `desktop_update_graphical` fail is a good example of the kind of issue I'll have to look into this week: it seems to have failed because of an intermittent crasher bug in PackageKit, rather than an issue in the update. We'll have to look at skipping known- unreliable tests, or marking them somehow so you know the deal in Bodhi, or automatically re-running them, or things along those lines. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ qa-devel mailing list -- qa-devel@lists.fedoraproject.org To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
