On Mon, 2023-06-12 at 12:05 +0200, Petr Pisar wrote: > V Fri, Jun 09, 2023 at 11:38:51AM -0700, Adam Williamson napsal(a): > > On Thu, 2022-06-09 at 12:48 -0700, Adam Williamson wrote: > > > Hi folks! > > > > > > More significantly, I'd also propose that we turn on gating on openQA > > > results for Rawhide updates. This would mean Rawhide updates would be > > > held from going 'stable' (and included in the next compose) until the > > > gating openQA tests had run and passed. We may want to do this a bit > > > after turning on the tests; perhaps Fedora 37 branch point would be a > > > natural time to do it. > > > > Hi again folks! A quick update here. Now the Rawhide update testing has > > been running in production for over a year - and Kevin and I have been > > "shadow gating" Rawhide for several months, untagging updates where > > openQA tests indicate genuine bugs - I think it's time to go ahead and > > enable gating for Rawhide updates. I've worked to make sure the tests > > are reliable and failures are promptly investigated, and that Bodhi > > provides accurate information on test and gating status. I've proposed > > this as a FESCo ticket just to get some visibility and sign-off on the > > idea: > > > OpenQA update testing are the graphical tests, e.g. update.live_build test > <https://openqa.fedoraproject.org/tests/1939328>? > > I always strugle to understand the results. What does a gray frame mean? In > failed (red?) frame, how can I see the expected image? The frame is usually > split by a vertical line, so I guess one side is expected, other side is > observed. Which one is one? What does mean "Test died: no candidate needle > with tag(s) '...' matched" message?
I can give you detailed answers for all of these, but the real answer is: I or someone else from the QA team will investigate failures and process them into more useful information. We don't expect packagers to diagnose their own failures from the openQA results unless they're very keen. We see openQA testing as a service provided by the QA team, and part of the service is investigating and diagnosing (and, quite often, fixing) failures. It's not intended a self-service system like Fedora CI. One thing you should look for in the UI is the Comments tab: usually once we've looked into a failure we will drop a link there to explain it (typically a link to a bug report or pull request or issue). Short answers: * Grey frame usually just means "didn't find the expected image, but timer hasn't expired yet" - expected events in openQA have a timeout, and it logs attempts to match during that period for information. * Vertical line - yes, I think it's left side == old, right side == new, but note the diff shown is *only* the attempted match area(s) (which are outlined) by default; clicking the 'eye' icon changes this to a full image diff. * Message just means "openQA was looking for a screenshot match and didn't find it" - in openQA parlance, the expected screenshot plus metadata (what areas to match etc.) is called a "needle", and part of the metadata is the "tag(s)". This is just a standard flexibility thing, instead of being forced to match a single needle by its single name, you match a "tag", and multiple needles can provide the same tag and any single needle can provide multiple different tags, which gives several obvious benefits. > And most importantly, who should I contact to help me investigate the > failures. The QA team, however you want to do that - #fedora-qa on IRC / Matrix, test@ mailing list, #quality-team tag on Discourse, poke me on Mastodon, email, carrier pigeon...usually you can also poke me directly, if I'm not around then Lukas Ruzicka (lruzicka). I usually check openQA update test results four or five times a day on weekdays (west coast NA time) and at least once a day on weekends. > As a packager, I need to retract the update or waive it promptly. > That means either the packager needs to be able to reproduce the failure > locally (how?) This is possible, but it's not gonna be practical for most packagers. Standing up an openQA instance requires a decent amount of work and resources; you can use the official ansible plays - https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openqa plus supporting stuff in the group_vars and so on - to help, but even then it's still not trivial. FWIW, I don't even have a pet openQA instance any more (Lukas does); I just use the official staging instance for re- runs and fiddling around with stuff... > or there must be some one else to do it. We (QA team) will do this. > I would expect answers for these questions to be available in a documentation. > Yet, I can any pointer to the documentation from a page with the results. That's a good idea, I'll see if we can work in a link somewhere in Bodhi, the openQA UI, or both. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
