Re: [Freeipa-devel] Automated Fedora update testing
On Wed, 2017-05-03 at 09:26 +0200, Petr Vobornik wrote: > > Is there a way now to check current state of current Fedoras > automatically using a script - e.g. avoid parsing mailing list or going > through runs in OpenQA? The script that generates the mails is called check-compose: https://pagure.io/fedora-qa/check-compose You can just check that out of git and run it locally. You'll need the python2-openqa_client package installed. You pass it the location of a compose, e.g.: ./check-compose https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170502.n.0/compose/ Does that help? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Automated Fedora update testing
On 04/29/2017 02:07 AM, Adam Williamson wrote: Hi folks! I thought this might be of interest to the FreeIPA community, so I thought I'd write it up here in case anyone missed it elsewhere. Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). The tests are automatically run on any critical path package, and as of today, they are also run on any update containing any of a manually-tended list of FreeIPA-related packages: 389-ds 389-ds-base bind bind-dyndb-ldap certmonger ding-libs freeipa krb5-server pki-core sssd tomcat cockpit This means that for any Fedora update containing one of these or any critical path package, Fedora's openQA FreeIPA tests should run, and you should see the results in the Fedora update system (Bodhi). You can see the results in Bodhi by clicking the Automated Updates tab for any update. For instance, here's a recent 389-ds-base update for Fedora 26: https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2 If you look at the Automated Tests tab, you can see passes for: update.server_role_deploy_domain_controller update.realmd_join_cockpit update.realmd_join_sssd indicating that this update didn't cause any problems for FreeIPA. Clicking on any test result will take you to the openQA page for the test, where you can diagnose failures and so on (explaining how to do this is a bit beyond the scope of this mail, please do ask me if you're interested!) This is really great. I hope this stuff will help us avoid shipping updates that break FreeIPA (and other key components). If you have any questions, concerns, comments, or suggestions, please do ask! . Note, if you're interested in the results for the nightly Fedora distribution composes, an email summary of the results for those is sent each time they're run to the Fedora test@ and devel@ lists, look for mails with "compose check report" in the subject. Any time any of the FreeIPA tests fails, the failure will be listed in the mail (passed tests are not specifically listed, just a count of them). I usually keep an eye on those results and analyze failures and file bugs, though. Is there a way now to check current state of current Fedoras automatically using a script - e.g. avoid parsing mailing list or going through runs in OpenQA? -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Automated Fedora update testing
On Mon, 2017-05-01 at 10:58 +0200, Lukas Slebodnik wrote:
>
> Tested with sssd and it passed as well.
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-8addfc0188
>
> freeIPA has also upstream integration tests packaged in
> python{2,3}-ipatests. They use pytest + python-pytest-multihost.
>
> Will it be possible to run some of them in openQA?
> e.g. test_installation.py (
I'd have to look into how they work in more detail (I haven't used
pytest-multihost before). There is always a question of whether it's
more appropriate to run something in openQA or Taskotron, and openQA
has some capacity limitations. How long do these tests take?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Automated Fedora update testing
Top posting FTW! (sorry) Excellent news Adam, this is awesome! Simo. On Fri, 2017-04-28 at 17:07 -0700, Adam Williamson wrote: > Hi folks! I thought this might be of interest to the FreeIPA > community, > so I thought I'd write it up here in case anyone missed it elsewhere. > > I work on the Fedora QA team, and we have been using the openQA > automated test system (developed by our friends at SUSE) to run > various > functional tests on Fedora composes for the last couple of years. > > As FreeIPA is considered a critical part of Fedora Server, we run a > few > tests that exercise FreeIPA. The tests set up a FreeIPA server, run > some basic checks on it, and also enrol two systems as clients of the > domain, one using the 'realm join' command directly, one using > Cockpit. > The client tests do some basic client functionality testing (getent, > logging in as a domain user, changing passwords, etc.) and also test > the web UI to some extent. > > Until recently we ran these tests only on Fedora's nightly > development > release distribution composes. Recently, though, we deployed some > enhancements to our openQA setup that let us run tests on Fedora > distribution updates as well, and have the results made visible > through > the Fedora update system (Bodhi). The tests are automatically run on > any critical path package, and as of today, they are also run on any > update containing any of a manually-tended list of FreeIPA-related > packages: > > 389-ds > 389-ds-base > bind > bind-dyndb-ldap > certmonger > ding-libs > freeipa > krb5-server > pki-core > sssd > tomcat > cockpit > > This means that for any Fedora update containing one of these or any > critical path package, Fedora's openQA FreeIPA tests should run, and > you should see the results in the Fedora update system (Bodhi). You > can > see the results in Bodhi by clicking the Automated Updates tab for > any > update. For instance, here's a recent 389-ds-base update for Fedora > 26: > > https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2 > > If you look at the Automated Tests tab, you can see passes for: > > update.server_role_deploy_domain_controller > update.realmd_join_cockpit > update.realmd_join_sssd > > indicating that this update didn't cause any problems for FreeIPA. > Clicking on any test result will take you to the openQA page for the > test, where you can diagnose failures and so on (explaining how to do > this is a bit beyond the scope of this mail, please do ask me if > you're > interested!) > > I hope this stuff will help us avoid shipping updates that break > FreeIPA (and other key components). If you have any questions, > concerns, comments, or suggestions, please do ask! > > To anticipate one question: you can cause *all* the tests for an > update > to be re-run by editing the update in any way (you don't have to > change > the package loadout, just changing a single character in the > description or something will do). If you think just one test result > is > bogus and want it re-run, currently, you'll have to ask someone with > the necessary power - either me or Jan Sedlak (garretraziel on IRC). > I'm in North America and he's in Europe, so we should have most > timezones covered between us. We're hoping to set up a better > mechanism > for this in future. > > Note, if you're interested in the results for the nightly Fedora > distribution composes, an email summary of the results for those is > sent each time they're run to the Fedora test@ and devel@ lists, look > for mails with "compose check report" in the subject. Any time any of > the FreeIPA tests fails, the failure will be listed in the mail > (passed > tests are not specifically listed, just a count of them). I usually > keep an eye on those results and analyze failures and file bugs, > though. > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . > net > http://www.happyassassin.net > -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Automated Fedora update testing
On (28/04/17 17:07), Adam Williamson wrote:
>Hi folks! I thought this might be of interest to the FreeIPA community,
>so I thought I'd write it up here in case anyone missed it elsewhere.
>
>I work on the Fedora QA team, and we have been using the openQA
>automated test system (developed by our friends at SUSE) to run various
>functional tests on Fedora composes for the last couple of years.
>
>As FreeIPA is considered a critical part of Fedora Server, we run a few
>tests that exercise FreeIPA. The tests set up a FreeIPA server, run
>some basic checks on it, and also enrol two systems as clients of the
>domain, one using the 'realm join' command directly, one using Cockpit.
>The client tests do some basic client functionality testing (getent,
>logging in as a domain user, changing passwords, etc.) and also test
>the web UI to some extent.
>
>Until recently we ran these tests only on Fedora's nightly development
>release distribution composes. Recently, though, we deployed some
>enhancements to our openQA setup that let us run tests on Fedora
>distribution updates as well, and have the results made visible through
>the Fedora update system (Bodhi). The tests are automatically run on
>any critical path package, and as of today, they are also run on any
>update containing any of a manually-tended list of FreeIPA-related
>packages:
>
>389-ds
>389-ds-base
>bind
>bind-dyndb-ldap
>certmonger
>ding-libs
>freeipa
>krb5-server
>pki-core
>sssd
>tomcat
>cockpit
>
>This means that for any Fedora update containing one of these or any
>critical path package, Fedora's openQA FreeIPA tests should run, and
>you should see the results in the Fedora update system (Bodhi). You can
>see the results in Bodhi by clicking the Automated Updates tab for any
>update. For instance, here's a recent 389-ds-base update for Fedora 26:
>
>https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2
>
>If you look at the Automated Tests tab, you can see passes for:
>
>update.server_role_deploy_domain_controller
>update.realmd_join_cockpit
>update.realmd_join_sssd
>
>indicating that this update didn't cause any problems for FreeIPA.
>Clicking on any test result will take you to the openQA page for the
>test, where you can diagnose failures and so on (explaining how to do
>this is a bit beyond the scope of this mail, please do ask me if you're
>interested!)
>
>I hope this stuff will help us avoid shipping updates that break
>FreeIPA (and other key components). If you have any questions,
>concerns, comments, or suggestions, please do ask!
>
>To anticipate one question: you can cause *all* the tests for an update
>to be re-run by editing the update in any way (you don't have to change
>the package loadout, just changing a single character in the
>description or something will do). If you think just one test result is
>bogus and want it re-run, currently, you'll have to ask someone with
>the necessary power - either me or Jan Sedlak (garretraziel on IRC).
>I'm in North America and he's in Europe, so we should have most
>timezones covered between us. We're hoping to set up a better mechanism
>for this in future.
>
>Note, if you're interested in the results for the nightly Fedora
>distribution composes, an email summary of the results for those is
>sent each time they're run to the Fedora test@ and devel@ lists, look
>for mails with "compose check report" in the subject. Any time any of
>the FreeIPA tests fails, the failure will be listed in the mail (passed
>tests are not specifically listed, just a count of them). I usually
>keep an eye on those results and analyze failures and file bugs,
>though.
Tested with sssd and it passed as well.
https://bodhi.fedoraproject.org/updates/FEDORA-2017-8addfc0188
freeIPA has also upstream integration tests packaged in
python{2,3}-ipatests. They use pytest + python-pytest-multihost.
Will it be possible to run some of them in openQA?
e.g. test_installation.py (
LS
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
