Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-10-18 at 09:37 -0400, pmkel...@frontier.com wrote:
> > 
> > Second, since nobody really opposed the idea of extending the critpath
> > definition slightly, here's a formal proposal to implement that. I want
> > to edit the wiki critpath page:
> > https://fedoraproject.org/wiki/Critical_path_package
> > 
> > and change it as follows:
> > 
> > 1. Under Background, change "The packages required to sustain these
> > actions make up the critical path." to:
> > 
> > "The packages required to sustain these actions initially made up the
> > critical path. Later, it was agreed that the maintainers of Editions,
> > spins, and labs can also declare packages that provide other key
> > functionality to be part of the ''critical path'' for that Edition,
> > spin or lab."
> > 
> > 2. Under Actions, change the first two sentences to read:
> > 
> > "Packages required to perform the most fundamental actions on a system
> > are always considered part of the ''critical path''. Those actions
> > include: "
> > 
> > Does anyone object to these proposed changes? Thanks!
> 
> 
> I think this is a great improvement over what we had.

So, I just realized I never actually made these changes! I've just done
so now:
https://fedoraproject.org/w/index.php?title=Critical_path_package=686714=599546
I also updated the information on how critpath is generated and how you
can view it, as that all changed as part of this too.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org
https://www.happyassassin.net



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[pmkel...@frontier.com]

On 10/17/22 8:45 PM, Adam Williamson wrote:

On Fri, 2022-09-02 at 10:26 -0700, Adam Williamson wrote:



If we don't think it's worth doing that work, then we're kinda stuck
with openQA glomming onto the critpath definition to decide which
updates to test and gate, because I don't have any other current viable
choices for that, really. And we'd have to figure out a critpath
definition that's as viable as possible for both purposes.


BTW, one other thought I've had in relation to all this is that we
could enhance the current critpath definition somewhat. Right now, it's
built out of package groups in comps which are kinda topic-separated:
there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
But the generated critical path package list is a monolith: it doesn't
distinguish between a package that's on the GNOME critpath and a
package that's on the KDE critpath, you just get a big list of all
critpath packages. It might be nice if we actually did distinguish
between those - the critpath definition could keep track of which
critpath topic(s) a package is included in, and Bodhi could display
that information in the web UI and provide it via the API. That way
manual testers could get a bit more info on why a package is critpath
and what areas to test, and openQA could potentially target its test
runs to conserve resources a bit, though this might require a bit more
coding work on the gating stuff now I think about it.


That sounds useful. We only need a volunteer to figure out the details
and do the work ;)


I actually did a huge rewrite of the thing that generates the critpath
data this week, and it probably wouldn't be to much work, honestly.
The most annoying bit would be the Bodhi frontend stuff, but that's
because I'm bad at frontend dev in general. :P But yeah, this is
definitely off in sky-castle land. I'll add it to my ever-growing list
of sky-castle projects to do when I get a couple of years of spare
time...


So, an update on this whole area!

First off, I'm actually finding the time to do the sky-castle work. The
releng critpath script now outputs critpath data by group:
https://pagure.io/releng/c/621caa542acc142d57f1247e7644846f737f8eee?branch=main

Bodhi (git HEAD Bodhi, anyway) can now read data in that format:
https://github.com/fedora-infra/bodhi/pull/4755

and when this PR is merged, it will be able to mark updates as critpath
by groups:
https://github.com/fedora-infra/bodhi/pull/4759

This has the handy bonus effect of enabling us to remove one of our
remaining uses of PDC. Once the Bodhi work is merged, I can send a PR
for the releng ansible repo to define per-group greenwave policies, run
the critpath.py script nightly on the Bodhi boxes, and switch to the
'json' critpath type in Bodhi config, and finally I can then enhance
the openQA scheduler to only run the necessary tests for each update.

Second, since nobody really opposed the idea of extending the critpath
definition slightly, here's a formal proposal to implement that. I want
to edit the wiki critpath page:
https://fedoraproject.org/wiki/Critical_path_package

and change it as follows:

1. Under Background, change "The packages required to sustain these
actions make up the critical path." to:

"The packages required to sustain these actions initially made up the
critical path. Later, it was agreed that the maintainers of Editions,
spins, and labs can also declare packages that provide other key
functionality to be part of the ''critical path'' for that Edition,
spin or lab."

2. Under Actions, change the first two sentences to read:

"Packages required to perform the most fundamental actions on a system
are always considered part of the ''critical path''. Those actions
include: "

Does anyone object to these proposed changes? Thanks!



I think this is a great improvement over what we had.

Have a Great Day!

Pat (tablepc)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Ben Cotton]

On Mon, Oct 17, 2022 at 8:54 PM Adam Williamson
 wrote:
>
> Does anyone object to these proposed changes? Thanks!

Ship it!


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Zbigniew Jędrzejewski-Szmek]

On Tue, Oct 18, 2022 at 02:45:32AM +0200, Adam Williamson wrote:
> First off, I'm actually finding the time to do the sky-castle work. The
> releng critpath script now outputs critpath data by group:
> https://pagure.io/releng/c/621caa542acc142d57f1247e7644846f737f8eee?branch=main
> 
> Bodhi (git HEAD Bodhi, anyway) can now read data in that format:
> https://github.com/fedora-infra/bodhi/pull/4755
> 
> and when this PR is merged, it will be able to mark updates as critpath
> by groups:
> https://github.com/fedora-infra/bodhi/pull/4759

Nice!

> Second, since nobody really opposed the idea of extending the critpath
> definition slightly, here's a formal proposal to implement that. I want
> to edit the wiki critpath page:
> https://fedoraproject.org/wiki/Critical_path_package
> 
> and change it as follows:
> 
> 1. Under Background, change "The packages required to sustain these
> actions make up the critical path." to:
> 
> "The packages required to sustain these actions initially made up the
> critical path. Later, it was agreed that the maintainers of Editions,
> spins, and labs can also declare packages that provide other key
> functionality to be part of the ''critical path'' for that Edition,
> spin or lab."
> 
> 2. Under Actions, change the first two sentences to read:
> 
> "Packages required to perform the most fundamental actions on a system
> are always considered part of the ''critical path''. Those actions
> include: "

+1.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Fri, 2022-09-02 at 10:26 -0700, Adam Williamson wrote:
> 
> > > If we don't think it's worth doing that work, then we're kinda stuck
> > > with openQA glomming onto the critpath definition to decide which
> > > updates to test and gate, because I don't have any other current viable
> > > choices for that, really. And we'd have to figure out a critpath
> > > definition that's as viable as possible for both purposes.
> > > 
> > > 
> > > BTW, one other thought I've had in relation to all this is that we
> > > could enhance the current critpath definition somewhat. Right now, it's
> > > built out of package groups in comps which are kinda topic-separated:
> > > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> > > But the generated critical path package list is a monolith: it doesn't
> > > distinguish between a package that's on the GNOME critpath and a
> > > package that's on the KDE critpath, you just get a big list of all
> > > critpath packages. It might be nice if we actually did distinguish
> > > between those - the critpath definition could keep track of which
> > > critpath topic(s) a package is included in, and Bodhi could display
> > > that information in the web UI and provide it via the API. That way
> > > manual testers could get a bit more info on why a package is critpath
> > > and what areas to test, and openQA could potentially target its test
> > > runs to conserve resources a bit, though this might require a bit more
> > > coding work on the gating stuff now I think about it.
> > 
> > That sounds useful. We only need a volunteer to figure out the details
> > and do the work ;)
> 
> I actually did a huge rewrite of the thing that generates the critpath
> data this week, and it probably wouldn't be to much work, honestly.
> The most annoying bit would be the Bodhi frontend stuff, but that's
> because I'm bad at frontend dev in general. :P But yeah, this is
> definitely off in sky-castle land. I'll add it to my ever-growing list
> of sky-castle projects to do when I get a couple of years of spare
> time...

So, an update on this whole area!

First off, I'm actually finding the time to do the sky-castle work. The
releng critpath script now outputs critpath data by group:
https://pagure.io/releng/c/621caa542acc142d57f1247e7644846f737f8eee?branch=main

Bodhi (git HEAD Bodhi, anyway) can now read data in that format:
https://github.com/fedora-infra/bodhi/pull/4755

and when this PR is merged, it will be able to mark updates as critpath
by groups:
https://github.com/fedora-infra/bodhi/pull/4759

This has the handy bonus effect of enabling us to remove one of our
remaining uses of PDC. Once the Bodhi work is merged, I can send a PR
for the releng ansible repo to define per-group greenwave policies, run
the critpath.py script nightly on the Bodhi boxes, and switch to the
'json' critpath type in Bodhi config, and finally I can then enhance
the openQA scheduler to only run the necessary tests for each update.

Second, since nobody really opposed the idea of extending the critpath
definition slightly, here's a formal proposal to implement that. I want
to edit the wiki critpath page:
https://fedoraproject.org/wiki/Critical_path_package

and change it as follows:

1. Under Background, change "The packages required to sustain these
actions make up the critical path." to:

"The packages required to sustain these actions initially made up the
critical path. Later, it was agreed that the maintainers of Editions,
spins, and labs can also declare packages that provide other key
functionality to be part of the ''critical path'' for that Edition,
spin or lab."

2. Under Actions, change the first two sentences to read:

"Packages required to perform the most fundamental actions on a system
are always considered part of the ''critical path''. Those actions
include: "

Does anyone object to these proposed changes? Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Fri, 2022-09-02 at 08:37 +, Zbigniew Jędrzejewski-Szmek wrote:
> > 
> > Now, because I glued openQA to the critpath because it was handy, there
> > are two sets of consequences to a package being in critical path:
> > 
> > 1. Tighter Bodhi requirements
> > 2. openQA tests are run, and results gate the update (except Rawhide)
> > 
> > So, one of the implicit questions here is, is it OK to keep twinning
> > these two sets of consequences, or should we split them up? Splitting
> > them up kinda implies answer 2) from my original mail: "Keep the
> > current "critical path" concept but define a broader group
> > of "gated packages" somewhere". Because we would then need some new
> > concept that isn't "critical path". As I said, that's more *work* -
> > it'd require us to write new code in several places[0]. Even if we
> > decide it'd be nice to do this, is it nice *enough* to be worth doing
> > that work?
> 
> I'd still vote for keeping a single critpath list and using it as
> "the list of packages that require extra care and testing".
> 
> As you describe, the original meaning of critpath has shifted, but
> it's because the way we do updates and QA has also shifted. Doing
> gating tests for a package seems much more useful than just keeping
> it longer in 'updates-testing' in hope that somebody discovers an
> important regresion in the second week.

Well, there's a caveat there - openQA doesn't test everything. On the
whole we cover quite a lot with the set of tests that gets run on
updates, but there's certainly lots of potential for there to be
important bugs it misses, that a human tester might catch. So I think
there is still a case for the higher karma requirements too.
> 
> So yeah, I don't think it makes sense to do the extra work to split
> the concepts. Also because we have way too many concepts and processes
> in Fedora already.

On the whole, though, I agree with you. I just don't trust my own
opinion because it's obviously biased by what's convenient for me. :D

> > If we don't think it's worth doing that work, then we're kinda stuck
> > with openQA glomming onto the critpath definition to decide which
> > updates to test and gate, because I don't have any other current viable
> > choices for that, really. And we'd have to figure out a critpath
> > definition that's as viable as possible for both purposes.
> > 
> > 
> > BTW, one other thought I've had in relation to all this is that we
> > could enhance the current critpath definition somewhat. Right now, it's
> > built out of package groups in comps which are kinda topic-separated:
> > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> > But the generated critical path package list is a monolith: it doesn't
> > distinguish between a package that's on the GNOME critpath and a
> > package that's on the KDE critpath, you just get a big list of all
> > critpath packages. It might be nice if we actually did distinguish
> > between those - the critpath definition could keep track of which
> > critpath topic(s) a package is included in, and Bodhi could display
> > that information in the web UI and provide it via the API. That way
> > manual testers could get a bit more info on why a package is critpath
> > and what areas to test, and openQA could potentially target its test
> > runs to conserve resources a bit, though this might require a bit more
> > coding work on the gating stuff now I think about it.
> 
> That sounds useful. We only need a volunteer to figure out the details
> and do the work ;)

I actually did a huge rewrite of the thing that generates the critpath
data this week, and it probably wouldn't be to much work, honestly.
The most annoying bit would be the Bodhi frontend stuff, but that's
because I'm bad at frontend dev in general. :P But yeah, this is
definitely off in sky-castle land. I'll add it to my ever-growing list
of sky-castle projects to do when I get a couple of years of spare
time...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Blaise Pabon]

I have a naive question...
I wonder if the folks who do a lot of zuul-ci (eg. opendev.org or
https://www.softwarefactory-project.io ) have methods to determine which
gating criteria to automate?

-Blaise

On Fri, Sep 2, 2022 at 10:39 AM Kalev Lember  wrote:

> On Fri, Sep 2, 2022 at 3:16 PM Ben Cotton  wrote:
>
>> On Wed, Aug 31, 2022 at 12:00 PM Adam Williamson
>>  wrote:
>>
>> > So, one of the implicit questions here is, is it OK to keep twinning
>> > these two sets of consequences, or should we split them up?
>>
>> Yes, it's okay to keep two sets of consequences together. In fact,
>> it's preferable. One critpath to rule them all, etc. We can always
>> tune the Bodhi requirements if we find that they're troublesome.
>>
>
> I think so too that just keeping one single list (critpath) is fine and
> avoids having to maintain two different lists with all the
> maintenance costs associated with it (the lists getting out of sync over
> time etc).
>
> Zbyzek's idea about reducing Bodhi karma requirements for critpath
> packages now that we have OpenQA gating makes sense to me as well. OpenQA
> already provides the additional testing that we originally wanted to get
> when we introduced the extra karma requirements for critpath packages.
>
> --
> Kalev
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>


-- 
LinkedIn   |  Quora
  |  Github

“If you want to go fast, go alone. If you want to go far, go
together.” --African
proverb
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Kalev Lember]

On Fri, Sep 2, 2022 at 3:16 PM Ben Cotton  wrote:

> On Wed, Aug 31, 2022 at 12:00 PM Adam Williamson
>  wrote:
>
> > So, one of the implicit questions here is, is it OK to keep twinning
> > these two sets of consequences, or should we split them up?
>
> Yes, it's okay to keep two sets of consequences together. In fact,
> it's preferable. One critpath to rule them all, etc. We can always
> tune the Bodhi requirements if we find that they're troublesome.
>

I think so too that just keeping one single list (critpath) is fine and
avoids having to maintain two different lists with all the
maintenance costs associated with it (the lists getting out of sync over
time etc).

Zbyzek's idea about reducing Bodhi karma requirements for critpath packages
now that we have OpenQA gating makes sense to me as well. OpenQA already
provides the additional testing that we originally wanted to get when we
introduced the extra karma requirements for critpath packages.

-- 
Kalev
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Ben Cotton]

On Wed, Aug 31, 2022 at 12:00 PM Adam Williamson
 wrote:

> So, one of the implicit questions here is, is it OK to keep twinning
> these two sets of consequences, or should we split them up?

Yes, it's okay to keep two sets of consequences together. In fact,
it's preferable. One critpath to rule them all, etc. We can always
tune the Bodhi requirements if we find that they're troublesome.


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Zbigniew Jędrzejewski-Szmek]

On Wed, Aug 31, 2022 at 08:59:49AM -0700, Adam Williamson wrote:
> On Wed, 2022-08-31 at 12:43 +, Zbigniew Jędrzejewski-Szmek wrote:
> > On Tue, Aug 30, 2022 at 12:10:00PM -0700, Adam Williamson wrote:
> > > On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
> > > > From my perspective, anything that blocks the release is on the
> > > > critical path. So any time there's a violation of the release criteria
> > > > and the package is not on the critical path definition, that's a bug
> > > > in the definition.
> > > > 
> > > > I recognize that this is a somewhat naïve view. For one, it may
> > > > broaden the definition beyond the current capacity of our test
> > > > infrastructure. It also may broaden the definition beyond what
> > > > maintainers are willing to put up with. These are both legitimate
> > > > problems. But the closer we can get to this ideal state, the better.
> > > > 
> > > > For anyone who is curious, I just searched for all accepted blockers
> > > > in the "Fedora" product in Bugzilla. 327 components have been a
> > > > blocker at least once. Some of those may no longer be blocking and
> > > > others will be added over time as our criteria change. The full list
> > > > with counts is at
> > > > https://bcotton.fedorapeople.org/release-blocking-components.csv if
> > > > you're interested.
> > > 
> > > Honestly, something along these lines would be my preference too, I
> > > just don't know if others would agree/support changing the critical
> > > path definition to "all release-blocking functionality" rather than
> > > "functionality needed to boot a basically-functional system".
> > 
> > I'd support increasing the scope to cover more packages in this fashion.
> > 
> > Being on critpath list is somewhat annoying because the bodhi update
> > minimum times are twice as long. For many packages this is a not a problem,
> > popular packages get enough karma within a day or two,
> > but if we expanded the list a lot, it could prove annoying to those
> > packagers. I think we could discuss lowering the minium update time
> > if this turns out to be the case.
> 
> So, that's an interesting question to consider as well, of course:
> what's the actual impact of critpath?
> 
> It'd be an interesting outcome if we broadened the critpath definition
> but diluted the Bodhi requirements, because the original purpose of
> critpath was more or less entirely the stricter Bodhi requirements.
> Until openQA came along, that was all it really meant: updates to these
> packages have stricter Bodhi requirements.
> 
> Now, because I glued openQA to the critpath because it was handy, there
> are two sets of consequences to a package being in critical path:
> 
> 1. Tighter Bodhi requirements
> 2. openQA tests are run, and results gate the update (except Rawhide)
> 
> So, one of the implicit questions here is, is it OK to keep twinning
> these two sets of consequences, or should we split them up? Splitting
> them up kinda implies answer 2) from my original mail: "Keep the
> current "critical path" concept but define a broader group
> of "gated packages" somewhere". Because we would then need some new
> concept that isn't "critical path". As I said, that's more *work* -
> it'd require us to write new code in several places[0]. Even if we
> decide it'd be nice to do this, is it nice *enough* to be worth doing
> that work?

I'd still vote for keeping a single critpath list and using it as
"the list of packages that require extra care and testing".

As you describe, the original meaning of critpath has shifted, but
it's because the way we do updates and QA has also shifted. Doing
gating tests for a package seems much more useful than just keeping
it longer in 'updates-testing' in hope that somebody discovers an
important regresion in the second week.

So yeah, I don't think it makes sense to do the extra work to split
the concepts. Also because we have way too many concepts and processes
in Fedora already.

> If we don't think it's worth doing that work, then we're kinda stuck
> with openQA glomming onto the critpath definition to decide which
> updates to test and gate, because I don't have any other current viable
> choices for that, really. And we'd have to figure out a critpath
> definition that's as viable as possible for both purposes.
> 
> 
> BTW, one other thought I've had in relation to all this is that we
> could enhance the current critpath definition somewhat. Right now, it's
> built out of package groups in comps which are kinda topic-separated:
> there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> But the generated critical path package list is a monolith: it doesn't
> distinguish between a package that's on the GNOME critpath and a
> package that's on the KDE critpath, you just get a big list of all
> critpath packages. It might be nice if we actually did distinguish
> between those - the critpath definition could keep track of which
> critpath topic(s) a package is included 

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Wed, 2022-08-31 at 12:43 +, Zbigniew Jędrzejewski-Szmek wrote:
> On Tue, Aug 30, 2022 at 12:10:00PM -0700, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
> > > From my perspective, anything that blocks the release is on the
> > > critical path. So any time there's a violation of the release criteria
> > > and the package is not on the critical path definition, that's a bug
> > > in the definition.
> > > 
> > > I recognize that this is a somewhat naïve view. For one, it may
> > > broaden the definition beyond the current capacity of our test
> > > infrastructure. It also may broaden the definition beyond what
> > > maintainers are willing to put up with. These are both legitimate
> > > problems. But the closer we can get to this ideal state, the better.
> > > 
> > > For anyone who is curious, I just searched for all accepted blockers
> > > in the "Fedora" product in Bugzilla. 327 components have been a
> > > blocker at least once. Some of those may no longer be blocking and
> > > others will be added over time as our criteria change. The full list
> > > with counts is at
> > > https://bcotton.fedorapeople.org/release-blocking-components.csv if
> > > you're interested.
> > 
> > Honestly, something along these lines would be my preference too, I
> > just don't know if others would agree/support changing the critical
> > path definition to "all release-blocking functionality" rather than
> > "functionality needed to boot a basically-functional system".
> 
> I'd support increasing the scope to cover more packages in this fashion.
> 
> Being on critpath list is somewhat annoying because the bodhi update
> minimum times are twice as long. For many packages this is a not a problem,
> popular packages get enough karma within a day or two,
> but if we expanded the list a lot, it could prove annoying to those
> packagers. I think we could discuss lowering the minium update time
> if this turns out to be the case.

So, that's an interesting question to consider as well, of course:
what's the actual impact of critpath?

It'd be an interesting outcome if we broadened the critpath definition
but diluted the Bodhi requirements, because the original purpose of
critpath was more or less entirely the stricter Bodhi requirements.
Until openQA came along, that was all it really meant: updates to these
packages have stricter Bodhi requirements.

Now, because I glued openQA to the critpath because it was handy, there
are two sets of consequences to a package being in critical path:

1. Tighter Bodhi requirements
2. openQA tests are run, and results gate the update (except Rawhide)

So, one of the implicit questions here is, is it OK to keep twinning
these two sets of consequences, or should we split them up? Splitting
them up kinda implies answer 2) from my original mail: "Keep the
current "critical path" concept but define a broader group
of "gated packages" somewhere". Because we would then need some new
concept that isn't "critical path". As I said, that's more *work* -
it'd require us to write new code in several places[0]. Even if we
decide it'd be nice to do this, is it nice *enough* to be worth doing
that work?

If we don't think it's worth doing that work, then we're kinda stuck
with openQA glomming onto the critpath definition to decide which
updates to test and gate, because I don't have any other current viable
choices for that, really. And we'd have to figure out a critpath
definition that's as viable as possible for both purposes.


BTW, one other thought I've had in relation to all this is that we
could enhance the current critpath definition somewhat. Right now, it's
built out of package groups in comps which are kinda topic-separated:
there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
But the generated critical path package list is a monolith: it doesn't
distinguish between a package that's on the GNOME critpath and a
package that's on the KDE critpath, you just get a big list of all
critpath packages. It might be nice if we actually did distinguish
between those - the critpath definition could keep track of which
critpath topic(s) a package is included in, and Bodhi could display
that information in the web UI and provide it via the API. That way
manual testers could get a bit more info on why a package is critpath
and what areas to test, and openQA could potentially target its test
runs to conserve resources a bit, though this might require a bit more
coding work on the gating stuff now I think about it.

[0]: at least: releng critpath generation scripts, bodhi, the openQA
scheduler, greenwave policies, and possibly greenwave
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Zbigniew Jędrzejewski-Szmek]

On Tue, Aug 30, 2022 at 12:10:00PM -0700, Adam Williamson wrote:
> On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
> > From my perspective, anything that blocks the release is on the
> > critical path. So any time there's a violation of the release criteria
> > and the package is not on the critical path definition, that's a bug
> > in the definition.
> > 
> > I recognize that this is a somewhat naïve view. For one, it may
> > broaden the definition beyond the current capacity of our test
> > infrastructure. It also may broaden the definition beyond what
> > maintainers are willing to put up with. These are both legitimate
> > problems. But the closer we can get to this ideal state, the better.
> > 
> > For anyone who is curious, I just searched for all accepted blockers
> > in the "Fedora" product in Bugzilla. 327 components have been a
> > blocker at least once. Some of those may no longer be blocking and
> > others will be added over time as our criteria change. The full list
> > with counts is at
> > https://bcotton.fedorapeople.org/release-blocking-components.csv if
> > you're interested.
> 
> Honestly, something along these lines would be my preference too, I
> just don't know if others would agree/support changing the critical
> path definition to "all release-blocking functionality" rather than
> "functionality needed to boot a basically-functional system".

I'd support increasing the scope to cover more packages in this fashion.

Being on critpath list is somewhat annoying because the bodhi update
minimum times are twice as long. For many packages this is a not a problem,
popular packages get enough karma within a day or two,
but if we expanded the list a lot, it could prove annoying to those
packagers. I think we could discuss lowering the minium update time
if this turns out to be the case.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Alexander Bokovoy]

On ti, 30 elo 2022, Adam Williamson wrote:

On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote:

On ma, 29 elo 2022, Adam Williamson wrote:
> On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > It sounds to me like the problem is "how do we best use the available
> > automated test resources?" so I'll answer accordingly.  Ignore me if I
> > misunderstood ;-)
>
> No, not really, sorry if I didn't explain clearly enough :D It's more
> just a 'what's the best way to work what I want to do around the
> existing definitions' question. What I want to do is have updates of
> FreeIPA-related packages gated. The awkwardness is that right now the
> definition of what gets gated is "critical path packages", and those
> clearly don't meet the current definition of "critical path".

Not all of them meet the current definition of 'critical path' but many
do. In fact, almost all crypto-implementing packages should be on that
list, if not already.


Sure, but freeipa itself, for instance, can't really be as things
stand. Nor could bind, I don't think. And several others.


Ideally, use of gating tests should be defined in the component itself.
Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
the specific component? If we do, then we really should move it there.


No. I don't think this is in-scope for gating.yaml. I *could* extend
the openQA scheduler to check some kind of per-repo config for every
package in the update, but there is nothing like this at present.

In any case, triggering the *tests* isn't the problem here; we already
do that, with the allowlist. It's doing the *gating* that's the
problem. As I said, we could add *that* to gating.yaml for each repo,
but I don't really love that idea, because you have to provide a full
list of what tests to gate on, and doing that for each repo and keeping
it updated seems like kind of a drag.

I guess in a sense it's appropriate for these packages, though, as you
could say we'd "really" want to gate on the FreeIPA-specific tests, or
at least just the server tests. But honestly, if an update to a
freeipa-related package causes GNOME to start crashing or something,
that's something I want to look into before the update goes stable,
even if it seems very weird.


From FreeIPA side I'd split this into two parts:

 - turn allowedlist into critpath list and treat all these tests as
   gating ones properly, run them for all cases

 - finally make FreeIPA CI tests to run on Fedora updates and make them
   gating too.

There are hundreds of tests we run in RHEL and in FreeIPA CI upstream
that Fedora would benefit from because they excercise more unlit corners
than OpenQA does.

I agree with your arguments below, we need to turn gating for all use
cases, especially Branched before Beta freeze, and Rawhide. Resource
wise it may be excessive but early detection of complex issues FreeIPA
testing tends to uncover is worth it.


Practically speaking, the way we do things in Fedora, there is very
little difference between "gating" and "consulting"; because of the
waiver mechanism they're effectively identical. You can always issue
waivers to override the gating. So really, gating is consulting. You
can look at the failures, decide they're bogus, and hit the waiver
button, which will immediately make the update 'pass' gating. I do
encourage people not to abuse the waiver function - if the failure
seems like a flake please use the 're-run tests' button, if you really
think it's bogus please poke me or Lukas or someone as I'm probably
working right then on dealing with it - but it is right there.


Do you have measured data for waivers? How often the waiver process is
used at all? In Branched before beta freeze?


This would be lovely, sure, but at this point we're quite far away from
the fairly concrete initial scope of my mail. I don't want to get
sidetracked into sky castle building where we file tickets and wait two
years for people to build stuff. I would like to gate updates to
FreeIPA-related packages *now*. I was all set to submit a pull request
to comps to add them to the server critpath definition, but then I
checked the official definition of "critical path" and noticed this
problem. That's where I'm coming from here: I want to make a concrete
change now, not design awesome speculative improvements on paper.


So your real issue at the moment is that 'critical path' definition is
not helping to QE. I guess, it is a worth submission to FESCO then,
right?

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
> From my perspective, anything that blocks the release is on the
> critical path. So any time there's a violation of the release criteria
> and the package is not on the critical path definition, that's a bug
> in the definition.
> 
> I recognize that this is a somewhat naïve view. For one, it may
> broaden the definition beyond the current capacity of our test
> infrastructure. It also may broaden the definition beyond what
> maintainers are willing to put up with. These are both legitimate
> problems. But the closer we can get to this ideal state, the better.
> 
> For anyone who is curious, I just searched for all accepted blockers
> in the "Fedora" product in Bugzilla. 327 components have been a
> blocker at least once. Some of those may no longer be blocking and
> others will be added over time as our criteria change. The full list
> with counts is at
> https://bcotton.fedorapeople.org/release-blocking-components.csv if
> you're interested.

Honestly, something along these lines would be my preference too, I
just don't know if others would agree/support changing the critical
path definition to "all release-blocking functionality" rather than
"functionality needed to boot a basically-functional system".

Thanks for the data! I will see if I can diff it against the current
critpath definition; that would be interesting.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-08-30 at 10:19 +0200, Petr Pisar wrote:
> V Tue, Aug 30, 2022 at 09:39:27AM +0300, Alexander Bokovoy napsal(a):
> > On ma, 29 elo 2022, Adam Williamson wrote:
> > > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > > > It sounds to me like the problem is "how do we best use the available
> > > > automated test resources?" so I'll answer accordingly.  Ignore me if I
> > > > misunderstood ;-)
> > > 
> > > No, not really, sorry if I didn't explain clearly enough :D It's more
> > > just a 'what's the best way to work what I want to do around the
> > > existing definitions' question. What I want to do is have updates of
> > > FreeIPA-related packages gated. The awkwardness is that right now the
> > > definition of what gets gated is "critical path packages", and those
> > > clearly don't meet the current definition of "critical path".
> > 
> > Not all of them meet the current definition of 'critical path' but many
> > do. In fact, almost all crypto-implementing packages should be on that
> > list, if not already.
> > 
> > Ideally, use of gating tests should be defined in the component itself.
> > Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
> > the specific component? If we do, then we really should move it there.
> > 
> > Another part of the problem is that most of these runs should probably
> > be consulting, not gating in the sense that failing them should give you
> > a clue that things might be wrong but there needs to be a human overview
> > of the failures. It is what we have right now with OpenQA-reported
> > failures for Bodhi updates in most cases, the test results aren't really
> > preventing the submissions from the going forward unless a maintainer
> > defined the update configuration to be so.
> > 
> > Ideally, we should have reverse dependency updates to trigger FreeIPA
> > tests and give their results a visibility but don't block on failures.
> 
> Exactly. If a package needs gating, it can enable it in its dist-git. At the
> same time I agree with Kevin, that if a distribution/edition needs to
> gate a set of packages, then it should also have it's own way to define the
> set.

I included this as an option in my initial mail and explained why I
don't like it much: it's a lot of maintenance, and openQA test gating
is kind of specifically conceived as being distro-wide.
> 
> If a package needs to run tests from a group of related packages, the package
> can either manually define the group, or there should be an out-of-band way of
> defining the group. (TMT calls it a test plan.)

This is not what we're talking about. openQA tests updates, not
packages. The bundling is expected to be done at the update level. If
the update does not include all the packages it should, that's a
packager error and openQA tests failing is intended, valid and useful.

> A reverse dependency is a possible, IMHO good, way of the definition. Alas,
> Fedora TMT does not support it
> .
> 
> I feel Adam is aware of it and tries to work around this TMT deficiency
> with an allowedlist.

This has nothing to do with TMT or Fedora CI in general (unless you
count gating.yaml and Greenwave as part of "Fedora CI"). The openQA
scheduler is an entirely separate codebase[0] and does not use TMT or
anything else related to Fedora CI.

In general, please remember that Fedora CI is not the only test system
using the Bodhi/greenwave industrial complex. :D

[0]: https://pagure.io/fedora-qa/fedora_openqa
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote:
> On ma, 29 elo 2022, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > > It sounds to me like the problem is "how do we best use the available
> > > automated test resources?" so I'll answer accordingly.  Ignore me if I
> > > misunderstood ;-)
> > 
> > No, not really, sorry if I didn't explain clearly enough :D It's more
> > just a 'what's the best way to work what I want to do around the
> > existing definitions' question. What I want to do is have updates of
> > FreeIPA-related packages gated. The awkwardness is that right now the
> > definition of what gets gated is "critical path packages", and those
> > clearly don't meet the current definition of "critical path".
> 
> Not all of them meet the current definition of 'critical path' but many
> do. In fact, almost all crypto-implementing packages should be on that
> list, if not already.

Sure, but freeipa itself, for instance, can't really be as things
stand. Nor could bind, I don't think. And several others.
> 
> Ideally, use of gating tests should be defined in the component itself.
> Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
> the specific component? If we do, then we really should move it there.

No. I don't think this is in-scope for gating.yaml. I *could* extend
the openQA scheduler to check some kind of per-repo config for every
package in the update, but there is nothing like this at present.

In any case, triggering the *tests* isn't the problem here; we already
do that, with the allowlist. It's doing the *gating* that's the
problem. As I said, we could add *that* to gating.yaml for each repo,
but I don't really love that idea, because you have to provide a full
list of what tests to gate on, and doing that for each repo and keeping
it updated seems like kind of a drag.

I guess in a sense it's appropriate for these packages, though, as you
could say we'd "really" want to gate on the FreeIPA-specific tests, or
at least just the server tests. But honestly, if an update to a
freeipa-related package causes GNOME to start crashing or something,
that's something I want to look into before the update goes stable,
even if it seems very weird.
> 
> Another part of the problem is that most of these runs should probably
> be consulting, not gating in the sense that failing them should give you
> a clue that things might be wrong but there needs to be a human overview
> of the failures. It is what we have right now with OpenQA-reported
> failures for Bodhi updates in most cases, the test results aren't really
> preventing the submissions from the going forward unless a maintainer
> defined the update configuration to be so.

The problem cases here are what I was thinking about. They are:
Branched before Beta freeze, and Rawhide.

For those cases, updates are automatically created on `fedpkg build`
and immediately submitted for stable. If there is no applicable gating
policy they are immediately pushed stable. There is no time for anyone
to consult.

If you use a side tag there's no automatic submission, I don't think,
but people do not use side tags as a matter of course.

Right now we don't gate Rawhide updates, but I would like to and I am
actively working on it (this would prevent me having the kind of
nightmare week I had last week where I spent dozens of hours tracking
down things that had broken in Rawhide, getting them fixed, and re-
running tests).

We *do* currently gate Branched updates before Beta freeze, and I want
FreeIPA-related packages included in that for the next cycle. I would
have liked it if, for the F37 cycle, FreeIPA-related updates that were
submitted between Branch point and Beta freeze were gated. But they
weren't.

Practically speaking, the way we do things in Fedora, there is very
little difference between "gating" and "consulting"; because of the
waiver mechanism they're effectively identical. You can always issue
waivers to override the gating. So really, gating is consulting. You
can look at the failures, decide they're bogus, and hit the waiver
button, which will immediately make the update 'pass' gating. I do
encourage people not to abuse the waiver function - if the failure
seems like a flake please use the 're-run tests' button, if you really
think it's bogus please poke me or Lukas or someone as I'm probably
working right then on dealing with it - but it is right there.
> 
> Ideally, we should have reverse dependency updates to trigger FreeIPA
> tests and give their results a visibility but don't block on failures.
> This would give an opportunnity to notice a failurer earlier but don't
> discourage maintainers from participating in a joint development.

We would like reverse dependency testing for all sorts of reasons, but
it's also all sorts of difficult (and, potentially, resource-
intensive). Of course, in openQA testing we do sort of have this
already. We run *all* tests on all critpath 

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Ben Cotton]

From my perspective, anything that blocks the release is on the
critical path. So any time there's a violation of the release criteria
and the package is not on the critical path definition, that's a bug
in the definition.

I recognize that this is a somewhat naïve view. For one, it may
broaden the definition beyond the current capacity of our test
infrastructure. It also may broaden the definition beyond what
maintainers are willing to put up with. These are both legitimate
problems. But the closer we can get to this ideal state, the better.

For anyone who is curious, I just searched for all accepted blockers
in the "Fedora" product in Bugzilla. 327 components have been a
blocker at least once. Some of those may no longer be blocking and
others will be added over time as our criteria change. The full list
with counts is at
https://bcotton.fedorapeople.org/release-blocking-components.csv if
you're interested.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Petr Pisar]

V Tue, Aug 30, 2022 at 09:39:27AM +0300, Alexander Bokovoy napsal(a):
> On ma, 29 elo 2022, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > > It sounds to me like the problem is "how do we best use the available
> > > automated test resources?" so I'll answer accordingly.  Ignore me if I
> > > misunderstood ;-)
> > 
> > No, not really, sorry if I didn't explain clearly enough :D It's more
> > just a 'what's the best way to work what I want to do around the
> > existing definitions' question. What I want to do is have updates of
> > FreeIPA-related packages gated. The awkwardness is that right now the
> > definition of what gets gated is "critical path packages", and those
> > clearly don't meet the current definition of "critical path".
> 
> Not all of them meet the current definition of 'critical path' but many
> do. In fact, almost all crypto-implementing packages should be on that
> list, if not already.
> 
> Ideally, use of gating tests should be defined in the component itself.
> Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
> the specific component? If we do, then we really should move it there.
> 
> Another part of the problem is that most of these runs should probably
> be consulting, not gating in the sense that failing them should give you
> a clue that things might be wrong but there needs to be a human overview
> of the failures. It is what we have right now with OpenQA-reported
> failures for Bodhi updates in most cases, the test results aren't really
> preventing the submissions from the going forward unless a maintainer
> defined the update configuration to be so.
> 
> Ideally, we should have reverse dependency updates to trigger FreeIPA
> tests and give their results a visibility but don't block on failures.

Exactly. If a package needs gating, it can enable it in its dist-git. At the
same time I agree with Kevin, that if a distribution/edition needs to
gate a set of packages, then it should also have it's own way to define the
set.

If a package needs to run tests from a group of related packages, the package
can either manually define the group, or there should be an out-of-band way of
defining the group. (TMT calls it a test plan.)

A reverse dependency is a possible, IMHO good, way of the definition. Alas,
Fedora TMT does not support it
.

I feel Adam is aware of it and tries to work around this TMT deficiency
with an allowedlist.

-- Petr


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Alexander Bokovoy]

On ma, 29 elo 2022, Adam Williamson wrote:

On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:

It sounds to me like the problem is "how do we best use the available
automated test resources?" so I'll answer accordingly.  Ignore me if I
misunderstood ;-)


No, not really, sorry if I didn't explain clearly enough :D It's more
just a 'what's the best way to work what I want to do around the
existing definitions' question. What I want to do is have updates of
FreeIPA-related packages gated. The awkwardness is that right now the
definition of what gets gated is "critical path packages", and those
clearly don't meet the current definition of "critical path".


Not all of them meet the current definition of 'critical path' but many
do. In fact, almost all crypto-implementing packages should be on that
list, if not already.

Ideally, use of gating tests should be defined in the component itself.
Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
the specific component? If we do, then we really should move it there.

Another part of the problem is that most of these runs should probably
be consulting, not gating in the sense that failing them should give you
a clue that things might be wrong but there needs to be a human overview
of the failures. It is what we have right now with OpenQA-reported
failures for Bodhi updates in most cases, the test results aren't really
preventing the submissions from the going forward unless a maintainer
defined the update configuration to be so.

Ideally, we should have reverse dependency updates to trigger FreeIPA
tests and give their results a visibility but don't block on failures.
This would give an opportunnity to notice a failurer earlier but don't
discourage maintainers from participating in a joint development.
Perhaps, The Fedora Packager Dashboard could be extended to pick up
results of tests relevant to your packages and display them together?
This way FreeIPA maintainers can see an overview of all tests related to
FreeIPA in one place and see if any help is needed to resolve failures.

--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> It sounds to me like the problem is "how do we best use the available
> automated test resources?" so I'll answer accordingly.  Ignore me if I
> misunderstood ;-)

No, not really, sorry if I didn't explain clearly enough :D It's more
just a 'what's the best way to work what I want to do around the
existing definitions' question. What I want to do is have updates of
FreeIPA-related packages gated. The awkwardness is that right now the
definition of what gets gated is "critical path packages", and those
clearly don't meet the current definition of "critical path".
> 
> We currently have a small list of packages that are gated behind openQA,
> and insufficient openQA resources to expand this list to all packages.
> 
> We also have a gating system based on karma, where users provide the QA
> manually.  At least, one hopes.  The karma system has some
> configurability for how much karma is needed, and for how long, etc.

Ah, no, again, I guess I should've explained that in more detail. When
I talk about "gating" I'm not talking about the manual karma mechanism.
I'm talking about the mechanism that prevents updates being pushed
stable if they fail the automated tests. That's a separate, parallel
thing which does not involve karma or manual feedback at all, it's
entirely automated.
> 
> What about a merger of those systems, where the openQA results count as
> a type of user, contributing to the status?  Give each package a "QA
> Priority" that ranges from "full gating" to "five second rule", where
> the openQA resources and the user work together to prioritize and test
> packages according to need:
> 
> * full gating requires all openQA tests to pass AND meet karma
>   requirements, openQA does these first.
> 
> * partial gating is like above, but either one (openQA or karma) can be
>   sufficient if enough time passes.
> 
> * reject only allows an openQA failure to reject an update, but the lack
>   of openQA success need not stop it if it has enough karma. (glibc uses
>   this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
>   pass adds nothing to the consensus needed)
> 
> ... and so on down the list. Each user, including openQA, can "vote" a
> pass or fail, and the rules say how all those passes and fails result in
> the update's overall pass or fail.
> 
> This would allow the openQA resources to prioritize updates that it
> knows *needs* openQA results, which allocating spare cycles to packages
> that could benefit from testing but don't require it.  It also means
> that additional openQA resources can be put to use immediately, while
> still allowing for growth in critical path size, without being wasted on
> unseen results.

This is an interesting idea, but rather complex and doesn't solve any
problem I have right now. :P Sorry for the confusion!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[DJ Delorie]

It sounds to me like the problem is "how do we best use the available
automated test resources?" so I'll answer accordingly.  Ignore me if I
misunderstood ;-)

We currently have a small list of packages that are gated behind openQA,
and insufficient openQA resources to expand this list to all packages.

We also have a gating system based on karma, where users provide the QA
manually.  At least, one hopes.  The karma system has some
configurability for how much karma is needed, and for how long, etc.

What about a merger of those systems, where the openQA results count as
a type of user, contributing to the status?  Give each package a "QA
Priority" that ranges from "full gating" to "five second rule", where
the openQA resources and the user work together to prioritize and test
packages according to need:

* full gating requires all openQA tests to pass AND meet karma
  requirements, openQA does these first.

* partial gating is like above, but either one (openQA or karma) can be
  sufficient if enough time passes.

* reject only allows an openQA failure to reject an update, but the lack
  of openQA success need not stop it if it has enough karma. (glibc uses
  this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
  pass adds nothing to the consensus needed)

... and so on down the list. Each user, including openQA, can "vote" a
pass or fail, and the rules say how all those passes and fails result in
the update's overall pass or fail.

This would allow the openQA resources to prioritize updates that it
knows *needs* openQA results, which allocating spare cycles to packages
that could benefit from testing but don't require it.  It also means
that additional openQA resources can be put to use immediately, while
still allowing for growth in critical path size, without being wasted on
unseen results.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Kevin Fenzi]

On Mon, Aug 29, 2022 at 03:50:02PM -0700, Adam Williamson wrote:
...snip...
> 
> I can think of I guess four options:
> 
> 1. Broaden the definition of the "critical path" somehow. We could just
> write in that it includes FreeIPA functionality, I guess, though that
> seems special purpose. We could broaden it to include any functionality
> covered by the release criteria, which would be quite a big increase
> but seems like a reasonable and concise definition. Or we could write
> in that it includes any functionality that is exercised by the gating
> openQA tests, though that seems a bit arbitrary - there's no
> particularly great organizing principle to the openQA tests we choose
> to run on updates, if I'm honest, it's a sort of grab bag I came up
> with.

I think this one is best... perhaps we could add something like 'and
such package groups as working groups decide are critical to their
edition' ?

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

Hi folks!

I have one of those definitional quandaries and I figured I'd throw it
at the lists for some input.

Right now, we kind of take advantage of the "critical path" concept for
automated update testing and gating via openQA.

openQA does not test all updates, only critical path updates plus
updates containing any package on a short allowlist.

Bodhi and Greenwave do not gate all updates on the openQA tests since
not all updates are tested; again we use the critpath definition. If an
update is critical path, it gets gated on the openQA tests. If it
isn't, it doesn't.

If you've been paying attention, that means there's a bit of a hole:
the packages on the 'allowlist'. These are tested, but not gated.

What's on the allowlist? Basically, FreeIPA-related packages. We have a
good set of FreeIPA tests in openQA, and both I and the FreeIPA team
wanted relevant updates to be tested with them. But those packages are
not on the critical path. So I set up this 'allowlist' mechanism.

However, the lack of gating kinda sucks. I would much prefer if we
could gate these updates as well as just testing them.

The obvious thing to do is add the FreeIPA-related packages to the
critical path, but that really is not supported by the critical path
definition:

https://fedoraproject.org/wiki/Critical_path_package

which defines it as packages required to "perform the most fundamental
actions on a system", with a list of specific areas, none of which is
"run a domain server".

So...what to do?

I can think of I guess four options:

1. Broaden the definition of the "critical path" somehow. We could just
write in that it includes FreeIPA functionality, I guess, though that
seems special purpose. We could broaden it to include any functionality
covered by the release criteria, which would be quite a big increase
but seems like a reasonable and concise definition. Or we could write
in that it includes any functionality that is exercised by the gating
openQA tests, though that seems a bit arbitrary - there's no
particularly great organizing principle to the openQA tests we choose
to run on updates, if I'm honest, it's a sort of grab bag I came up
with.

2. Keep the current "critical path" concept but define a broader group
of "gated packages" somewhere (could be comps or somewhere else). This
would require engineering work - we'd have to touch probably the openQA
scheduler, Bodhi, and greenwave configs. It's also another maintenance
burden.

3. Add gating config to each allowlisted package repo's gating.yml one
by one. I don't really like this option. It'd be a chunk of work to do
initially, and multiples the maintenance required when the list of
tests to gate on changes for some reason.

4. Do nothing, just keep only gating things that are "critical path" on
the current definition. In a utopian future, we'd manage to deploy
openQA in the cloud or get a giant pile of super fast servers so we
could test and gate *every* update, and this wouldn't be an issue any
more.

What do folks think? Any preferences? Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue