Hi folks! I'm hoping this new feature I've been working on since Flock is now ready for prime time.
It's pretty simple, really. Comment '/openqa test' on a dist-git PR and openQA will test it. It should report the state back to the pull request - you'll get an entry called something like "Fedora openQA tests - scratch build 147755141" that will start in 'pending' state then go to 'success', 'failure' or 'error' once all tests complete. The entry is a link to the openQA summary page where you can see the state of each test (so if you get a 'failure' result you can see specifically which test(s) failed; I don't want to report the state of each test separately to the PR because there can be dozens or hundreds). Here's a PR where you can see it "in action": https://src.fedoraproject.org/rpms/sqlite/pull-request/64 It looks for the Packit scratch build for the target branch of the PR, so if that build fails, the tests won't run. If the scratch build is not done at the time you post the comment, the tests *should* run once the scratch build succeeds (and any subsequent time a scratch build of the PR for the correct branch succeeds). If the package is in the critical path, it will run only "appropriate" tests (using the same mapping used for update tests). If the package is not in the critical path, it will just run all the update tests. There is no multi-package / multi-PR capability at this time; the tests will always run on *only* the scratch build of the PR you commented on. (with the relevant buildroot repo enabled in the system-under-test). If the change in the PR depends on a pending change to another package that has not already appeared in the buildroot, the tests will fail. Sorry. I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way, I'll try and enhance this system to work with it so related PRs are tested together. This is the primary reason this system is "on- demand" - I don't want to trigger tests automatically for all critpath package PRs, or anything like that, because we know they would sometimes fail because of interdependency issues. Right now there's no authorization check - if *anyone* comments '/openqa test' on a PR it is "enabled" for testing and tests will run every time a scratch build of the PR completes. Please don't abuse this by spamming it on PRs you're not involved in, or anything; if anyone does that I'll just have to turn the feature off unless/until I can implement some kind of "only accept comments from people with commit access to the package" mechanic, and I'd rather believe I can just trust people. Please only use this mechanism if you're actually going to find the results directly useful. The tests will trigger on both staging and production openQA instances, but only the production instance should report status back to the PR. The implementation of this is Pagure-specific, but it's not really that complicated and should hopefully be fairly easy to port to Forgejo when we move dist-git. If anyone's interested, this is the code: https://forge.fedoraproject.org/quality/fedora_openqa/commit/ac5a56f41b2a7fcf30b1105be7bdd621b1ff085d Of course, please let me know if you try it out and it doesn't work, or you notice some kind of bug. Thanks! -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @[email protected] https://www.happyassassin.net -- _______________________________________________ test mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
