Re: Test gating enabled in Bodhi

[Randy Barlow]

On 03/01/2018 11:01 AM, mcatanz...@gnome.org wrote:
> I know Pierre-Yves warned us about the bug that the tests will be
> reported as a failure if the tests are still being run, but there was
> absolutely no indication that there were more tests that were still
> being run. So this could definitely stand to be improved.

I think you may have hit this issue:

https://github.com/fedora-infra/bodhi/issues/2124



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[mcatanzaro]

On Wed, Feb 28, 2018 at 7:49 PM, Randy Barlow 
 wrote:

It does not appear to be blocked currently.


There is a bug. Yesterday it was red and said tests failed; the result 
changed itself. I see there are many more test results now than there 
were yesterday. It looks like the tests had not finished running when I 
checked yesterday.


I know Pierre-Yves warned us about the bug that the tests will be 
reported as a failure if the tests are still being run, but there was 
absolutely no indication that there were more tests that were still 
being run. So this could definitely stand to be improved.


Michael
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Randy Barlow]

On 02/28/2018 07:05 PM, mcatanz...@gnome.org wrote:
> I have an update here:
> 
> https://bodhi.fedoraproject.org/updates/epiphany-3.26.6-1.fc27
> 
> that is blocked due to some harmless translation-related warnings from
> desktop-file-validate.

It does not appear to be blocked currently.



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[mcatanzaro]

I have an update here:

https://bodhi.fedoraproject.org/updates/epiphany-3.26.6-1.fc27

that is blocked due to some harmless translation-related warnings from 
desktop-file-validate. I'm not the translation police and do not plan 
to make any changes to the desktop file.


* Who can help unblock this update? waiverdb looks too hard for me.
* Who can fix this so that it does not cause update failures in the 
future?


Thanks,

Michael

 {
"module" : "DesktopLint",
"order" : 93,
"results" : [
   {
  "arch" : "armv7hl,i686,x86_64",
  "code" : "DesktopFileValidation",
  "context" : {
 "path" : 
"/usr/share/applications/org.gnome.Epiphany.desktop"

  },
  "diag" : "warning: value \"???-???\" for key 
\"Comment[ru]\" in group \"Desktop Entry\" looks redundant with value 
\"???-???\" of key \"Name[ru]\"",

  "subpackage" : "epiphany"
   },
   {
  "arch" : "armv7hl,i686,x86_64",
  "code" : "DesktopFileValidation",
  "context" : {
 "path" : 
"/usr/share/applications/org.gnome.Epiphany.desktop"

  },
  "diag" : "warning: value \" ???\" for key 
\"Comment[tg]\" in group \"Desktop Entry\" looks redundant with value 
\" ???\" of key \"GenericName[tg]\"",

  "subpackage" : "epiphany"
   },
   {
  "arch" : "armv7hl,i686,x86_64",
  "code" : "DesktopFileValidation",
  "context" : {
 "path" : 
"/usr/share/applications/org.gnome.Epiphany.desktop"

  },
  "diag" : "warning: value \"Veb brauzer\" for key 
\"Comment[uz]\" in group \"Desktop Entry\" looks redundant with value 
\"Veb brauzer\" of key \"GenericName[uz]\"",

  "subpackage" : "epiphany"
   },
   {
  "arch" : "armv7hl,i686,x86_64",
  "code" : "DesktopFileValidation",
  "context" : {
 "path" : 
"/usr/share/applications/org.gnome.Epiphany.desktop"

  },
  "diag" : "warning: value \"??? ???\" for key 
\"Comment[uz@cyrillic]\" in group \"Desktop Entry\" looks redundant 
with value \"??? ???\" of key \"Name[uz@cyrillic]\"",

  "subpackage" : "epiphany"
   }
],
"run_time" : 0,
"status" : "completed"
 }
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Alexander Ploumistos]

On Tue, Feb 20, 2018 at 11:10 PM, Rex Dieter  wrote:
> Alexander Ploumistos wrote:
>> I am asking because the rpm documentation leaves quite a lot to be
>> desired. If I went and changed all my "Requires: foo" to "Requires:
>> foo%{_isa}" in all my non-noarch packages, would I be plain wrong, or
>> is it justifiable - albeit an overkill?
>
> the only place I recall seeing recommendation to use %{_isa} is in subpkg
> dependencies.
>
> IMO, It's wrong to use in general, unless you have good reason to do so.  Do
> you?

Well, if I did, I'd know why it was a good reason and we wouldn't be
having this conversation :)


> Another alternative: don't make -devel depend on the main package (which is
> ok for headers-only situations like this)

That's a good point to discuss with the upstream developer as well,
because I think he intends the libraries to be shipped (and work) that
way.

Thank you for all that Rex!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rex Dieter]

Alexander Ploumistos wrote:

> Well, with some delay, the waiver worked and I was able to push the
> f26 package to batched.
> 
> 
> On Tue, Feb 20, 2018 at 8:47 PM, Rex Dieter  wrote:
>> Alexander Ploumistos wrote:
>>> OpenBabel is a runtime dependency for some optional features of
>>> Molsketch. The %{_isa} macro got added during the review
>>
>> I think the reviewer in this case was wrong to suggest that, just use
>> Requires: openbabel
> 
> I am asking because the rpm documentation leaves quite a lot to be
> desired. If I went and changed all my "Requires: foo" to "Requires:
> foo%{_isa}" in all my non-noarch packages, would I be plain wrong, or
> is it justifiable - albeit an overkill?

the only place I recall seeing recommendation to use %{_isa} is in subpkg 
dependencies.

IMO, It's wrong to use in general, unless you have good reason to do so.  Do 
you?


> We had some long discussions with the reviewer and the upstream
> developer as to what could/should be in the -devel subpackage and I
> ended up with what's there. I was wondering why the subpackage was not
> to be noarch, but then I found this in our guidelines:
> 
> Do not use noarch
> 
> It may be tempting to make the header library package noarch, since
> the header files themselves are simply text. However, a library should
> have tests which should be run on all architectures. Also, the install
> process may modify the installed headers depending on the build
> architecture. For these reasons, header-only packages must not be
> marked noarch.
> 
> 
> Upstream is working on a testsuite, so at some point down the road I
> will (probably) need it as it is.

That's fair.

Another alternative: don't make -devel depend on the main package (which is 
ok for headers-only situations like this)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Alexander Ploumistos]

Well, with some delay, the waiver worked and I was able to push the
f26 package to batched.


On Tue, Feb 20, 2018 at 8:47 PM, Rex Dieter  wrote:
> Alexander Ploumistos wrote:
>> OpenBabel is a runtime dependency for some optional features of
>> Molsketch. The %{_isa} macro got added during the review
>
> I think the reviewer in this case was wrong to suggest that, just use
> Requires: openbabel

I am asking because the rpm documentation leaves quite a lot to be
desired. If I went and changed all my "Requires: foo" to "Requires:
foo%{_isa}" in all my non-noarch packages, would I be plain wrong, or
is it justifiable - albeit an overkill?

>
> FYI, the package is multilib'd because it has a -devel subpkg, which depends
> on the main one (-devel pkgs are automatically multilib).  the -devel subpkg
> is headers only, you could consider either making it noarch, or drop it
> altogether.

We had some long discussions with the reviewer and the upstream
developer as to what could/should be in the -devel subpackage and I
ended up with what's there. I was wondering why the subpackage was not
to be noarch, but then I found this in our guidelines:

Do not use noarch

It may be tempting to make the header library package noarch, since
the header files themselves are simply text. However, a library should
have tests which should be run on all architectures. Also, the install
process may modify the installed headers depending on the build
architecture. For these reasons, header-only packages must not be
marked noarch.


Upstream is working on a testsuite, so at some point down the road I
will (probably) need it as it is.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rex Dieter]

Alexander Ploumistos wrote:


> OpenBabel is a runtime dependency for some optional features of
> Molsketch. The %{_isa} macro got added during the review

I think the reviewer in this case was wrong to suggest that, just use
Requires: openbabel

FYI, the package is multilib'd because it has a -devel subpkg, which depends 
on the main one (-devel pkgs are automatically multilib).  the -devel subpkg 
is headers only, you could consider either making it noarch, or drop it 
altogether.

-- Rex
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Adam Williamson]

On Tue, 2018-02-20 at 19:15 +0100, Alexander Ploumistos wrote:
> On Tue, Feb 20, 2018 at 5:13 PM, Rex Dieter  wrote:
> > Alexander Ploumistos wrote:
> > > First question, why "arch" and "scenario" are x86_64, but the error
> > > concerns the 32-bit build? Since OpenBabel exists on all arches, why
> > > do I get this particular error message?
> > 
> > Your package is multilib'd?
> 
> No, I don't think it is.
> 
> 
> > > I have removed and reinstalled molsketch on both i686 and x86_64 with
> > > no errors from dnf
> > 
> > and openbable is not multilib'd, that's the problem.
> > 
> > Why do you have an explicit dep?  Why is it arch'd?
> > 
> > If you sure it's still needed, a simple:
> > Requires: openbabel
> > 
> > should suffice.
> 
> OpenBabel is a runtime dependency for some optional features of
> Molsketch. The %{_isa} macro got added during the review, I had asked
> about it, but completely forgot it while we were trying to figure out
> some other stuff. Is this technically wrong or is dist.rpmdeplint
> being overzealous here?

We did have a similar case of cross-arch oddness in rpmdeplint which I
think turned out to be something extremely non-obvious:

https://pagure.io/taskotron/task-rpmdeplint/issue/9

CCing kparal.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Alexander Ploumistos]

On Tue, Feb 20, 2018 at 5:13 PM, Rex Dieter  wrote:
> Alexander Ploumistos wrote:
>> First question, why "arch" and "scenario" are x86_64, but the error
>> concerns the 32-bit build? Since OpenBabel exists on all arches, why
>> do I get this particular error message?
>
> Your package is multilib'd?

No, I don't think it is.


>> I have removed and reinstalled molsketch on both i686 and x86_64 with
>> no errors from dnf
>
> and openbable is not multilib'd, that's the problem.
>
> Why do you have an explicit dep?  Why is it arch'd?
>
> If you sure it's still needed, a simple:
> Requires: openbabel
>
> should suffice.

OpenBabel is a runtime dependency for some optional features of
Molsketch. The %{_isa} macro got added during the review, I had asked
about it, but completely forgot it while we were trying to figure out
some other stuff. Is this technically wrong or is dist.rpmdeplint
being overzealous here?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rex Dieter]

Alexander Ploumistos wrote:

> Hi,
> 
> I've been having a problem with dist.rpmdeplint checks for one of my
> packages (molsketch).
> This is a new package and it has a run-time dependency on OpenBabel:
> 
> Requires: openbabel%{?_isa}
> 
> The failure is the same on both f26 & f27:
> 
> results:
> - arch: x86_64
>   item: molsketch-0.5.1-7.fc26
>   outcome: FAILED
>   scenario: x86_64
>   type: koji_build
> 
> 
> nothing provides openbabel(x86-32) needed by molsketch-0.5.1-7.fc26.i686
> nothing provides openbabel(x86-32) needed by molsketch-0.5.1-7.fc26.i686
> 
> First question, why "arch" and "scenario" are x86_64, but the error
> concerns the 32-bit build? Since OpenBabel exists on all arches, why
> do I get this particular error message?

Your package is multilib'd? 

> I have removed and reinstalled molsketch on both i686 and x86_64 with
> no errors from dnf

and openbable is not multilib'd, that's the problem.

Why do you have an explicit dep?  Why is it arch'd?

If you sure it's still needed, a simple:
Requires: openbabel

should suffice.

-- Rex
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Alexander Ploumistos]

Hi,

I've been having a problem with dist.rpmdeplint checks for one of my
packages (molsketch).
This is a new package and it has a run-time dependency on OpenBabel:

Requires: openbabel%{?_isa}

The failure is the same on both f26 & f27:

results:
- arch: x86_64
  item: molsketch-0.5.1-7.fc26
  outcome: FAILED
  scenario: x86_64
  type: koji_build


nothing provides openbabel(x86-32) needed by molsketch-0.5.1-7.fc26.i686
nothing provides openbabel(x86-32) needed by molsketch-0.5.1-7.fc26.i686

First question, why "arch" and "scenario" are x86_64, but the error
concerns the 32-bit build? Since OpenBabel exists on all arches, why
do I get this particular error message?

I have removed and reinstalled molsketch on both i686 and x86_64 with
no errors from dnf, with the correct version of openbabel getting
pulled in every time, so I decided to submit waivers for the failing
results. That worked for the f27 package, but not for the f26 one. I
do get a "Created waiver 24 for result 19678515" from waiverdb-cli,
but even though the packages has spent a week in testing, I do not
have the option to push it to batched or stable and I do not think
this is a matter of time, since the f26 package got pushed to testing
earlier than that for f27.

I'd really appreciate any insights.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Kamil Paral]

On Tue, Jan 23, 2018 at 10:28 AM, Pierre-Yves Chibon 
wrote:

> Good Morning Fedorans!
>
> On Thursday, a new version of Bodhi was deployed that enabled Bodhi to
> gate updates based on test results. You may notice a "Test Gating
> Status" message in the right have side of the page.
>
> One thing to know about this is that there is currently a confusing
> issue where Bodhi will say that the tests have failed when the tests
> haven't finished running[0]. We are working on solving that issue, but
> for now you can just wait a while and it should report the result once
> all the tests have finished.
>
> There are three tests that must pass in order for updates to go to stable:
>
> 0. dist.depcheck - to make sure the update's dependencies are available.
>

Since I haven't seen anyone clarifying this, I'll clarify - the task is
named dist.rpmdeplint, not dist.depcheck. I fixed the wiki as well.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Kevin Kofler]

Rex Dieter wrote:
> Pierre-Yves Chibon wrote:
>> 1. dist.abicheck - to make sure the update's ABI remains stable in a
>>given Fedora release.
> 
> I think it unwise to make item 1 a mandatory item at this point.  I'd
> argue a large number of packages do not provide public api/abi that's
> worth worrying about in this regard.

+1

See:
https://pagure.io/fesco/issue/1810#comment-488673
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/UVMM7O4OEGCNMZ47HN7QYPQDIV2IJZFR/

I wrote there:
> Uh, `dist.abicheck` produces a lot of false positives on:
>
> * libraries that are internal and that nothing should depend on (e.g., in 
> QupZilla, package `qupzilla`),
> * APIs explicitly documented as "private, can change at any version", as 
> common in all Qt modules (e.g., in QtWebEngine, package
> `qt5-qtwebengine`).
>
> My packages often fail `dist.abicheck`. It is absolutely not realistic to 
> expect it to pass for all updates.

I don't think gating on this test will EVER be a reasonable thing to do. It 
has just too many false positives. There is no automated way to figure out 
which ABIs are intended to be public and which aren't. And even an API 
"public" at package level can be private at distro level. (Consider a 
library with exactly one package using it. It is always reasonable to bump 
those together in a grouped update. We have several such examples in 
Fedora.)

Adam Williamson wrote:
> Additionally, as Ralph wrote, he has removed abicheck from the list of
> gating tests for now, so you shouldn't need to worry about abicheck
> failures blocking updates, as soon as Bodhi starts applying that
> change.

FYI, it looks like this is already working. (The other thread said it'd take 
only up to 6 hours for Bodhi to pick it up, and indeed, qt5-qtwebengine is 
now showing as passing the gating even though it "fails" the bogus abicheck 
test.)

But why was this check enabled to begin with, when the issues were already 
known (see the above links)? I get the feeling that I am talking to a wall!

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Ralph Bean]

On Tue, Jan 23, 2018 at 10:33:01PM +0100, Dominik 'Rathann' Mierzejewski wrote:
> On Tuesday, 23 January 2018 at 20:16, Matthew Miller wrote:
> > On Tue, Jan 23, 2018 at 12:25:56PM -0600, Rex Dieter wrote:
> > > > There are three tests that must pass in order for updates to go to 
> > > > stable:
> > > > 0. dist.depcheck - to make sure the update's dependencies are available.
> > > > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> > > >given Fedora release.
> > > ...
> > > > Finally, if it turns out you need to push an update through despite of 
> > > > the
> > > > test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> > > > We are working on integrating this into Bodhi itself, making this 
> > > > easier.
> > > I think it unwise to make item 1 a mandatory item at this point.  I'd 
> > > argue
> > > a large number of packages do not provide public api/abi that's worth
> > > worrying about in this regard.
> >
> > I think we should definine a set of packages where we care about this,
> > and enable it on a case-by-case basis and make it advisory otherwise.
>
> That makes sense. How about we start with critical path packages?
> Alternatively, libraries which a lot of of other packages depend on
> would be good candidates.

A thought - we need a defined process for changing and proposing
changes to the greenwave policy.  Right now, it reflects a proof of
concept list that "we just made up" to get this up and running.

At the moment, anyone in the 'sysadmin-qa' group is able to make
changes to it.  Should FESCo be involved?  QA?  FPC?

I want to fix tooling issues, but don't want to be the policy arbiter.
Let's get ourselves unblocked for now and then start figuring out some
process around this stuff.


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Pierre-Yves Chibon]

On Wed, Jan 24, 2018 at 03:25:48AM -0500, Randy Barlow wrote:
> On 01/23/2018 05:12 PM, Adam Williamson wrote:
> > The initial thought we - we being Tim, Kamil, Josef, Ralph and I - had
> > is to simply invert the policy, if we can, so it becomes "the update
> > passes *unless* we can find a 'fail' for any of the required tests". So
> > all updates would be push-able (so far as this mechanism is concerned,
> > ignoring all the old ones) until one of the gating tests definitely
> > failed.
> 
> This sounds like a reasonable workaround to me.
> 
> > If we can't do that, we're going to just have to disable the gating
> > again until this is sorted out; we're definitely of the opinion that
> > Taskotron doesn't yet provide enough of a solid guarantee that all the
> > tests will be run for a policy which *assumes* that will be the case to
> > be viable.
> 
> I agree, the gating is a bit too unreliable as is to stay in its current
> state.

Well, we can remove dist.rpmdeplint but we have not had complain about the
AtomicCI results gating so far, so let's not turn off everything entirely.


Pierre


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Randy Barlow]

On 01/23/2018 05:12 PM, Adam Williamson wrote:
> The initial thought we - we being Tim, Kamil, Josef, Ralph and I - had
> is to simply invert the policy, if we can, so it becomes "the update
> passes *unless* we can find a 'fail' for any of the required tests". So
> all updates would be push-able (so far as this mechanism is concerned,
> ignoring all the old ones) until one of the gating tests definitely
> failed.

This sounds like a reasonable workaround to me.

> If we can't do that, we're going to just have to disable the gating
> again until this is sorted out; we're definitely of the opinion that
> Taskotron doesn't yet provide enough of a solid guarantee that all the
> tests will be run for a policy which *assumes* that will be the case to
> be viable.

I agree, the gating is a bit too unreliable as is to stay in its current
state.



signature.asc
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Philip Kovacs]

On Tue, 2018-01-23 at 21:25 +, Philip Kovacs wrote:
>> Can someone please elaborate on how I can control the abi tests
>> directly?Where exactly can I access these and refine them on a per-
>> package basis?

>That text isn't talking about "fixing the tests", but about fixing the
>*bugs*. It assumes that the test indicates a real issue that needs
>fixing, therefore you can fix the issue in your package and cause the
>test to pass.

OK, thanks. The abi check canvases everything in the package and is sensitive 
to changes that are internal to the package.  Thoser internal changes are not 
bugs for us to manage.Take, for example, a c/c++ package with many .so plugins 
or other internal libraries that provide services to the application, but are 
not user-facing -- the abi check sees these internal changes as noteworthy, but 
they really are not and should be filtered.  
I guess I'll have to use the waiver for now.  

On Tuesday, January 23, 2018 5:44 PM, Adam Williamson 
 wrote:
 

 On Tue, 2018-01-23 at 21:25 +, Philip Kovacs wrote:
> Can someone please elaborate on how I can control the abi tests
> directly?Where exactly can I access these and refine them on a per-
> package basis?

> How to fix the tests?

That text isn't talking about "fixing the tests", but about fixing the
*bugs*. It assumes that the test indicates a real issue that needs
fixing, therefore you can fix the issue in your package and cause the
test to pass.

It doesn't really cover the scenario where what the test is reporting
isn't really a problem and doesn't need to be fixed. You can't resolve
that from your package git repository, but you *can* submit a waiver,
as discussed upthread.

Additionally, as Ralph wrote, he has removed abicheck from the list of
gating tests for now, so you shouldn't need to worry about abicheck
failures blocking updates, as soon as Bodhi starts applying that
change.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


   ___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Tim Flink]

On Tue, 23 Jan 2018 11:06:59 +0100
Pierre-Yves Chibon  wrote:

> On Tue, Jan 23, 2018 at 09:42:50AM +, Richard W.M. Jones wrote:
> > On Tue, Jan 23, 2018 at 10:28:14AM +0100, Pierre-Yves Chibon
> > wrote:  
> > > Good Morning Fedorans!
> > > 
> > > On Thursday, a new version of Bodhi was deployed that enabled
> > > Bodhi to gate updates based on test results. You may notice a
> > > "Test Gating Status" message in the right have side of the page.
> > > 
> > > One thing to know about this is that there is currently a
> > > confusing issue where Bodhi will say that the tests have failed
> > > when the tests haven't finished running[0]. We are working on
> > > solving that issue, but for now you can just wait a while and it
> > > should report the result once all the tests have finished.
> > > 
> > > There are three tests that must pass in order for updates to go
> > > to stable:
> > > 
> > > 0. dist.depcheck - to make sure the update's dependencies are
> > > available. 1. dist.abicheck - to make sure the update's ABI
> > > remains stable in a given Fedora release.
> > > 2. AtomicCI pipeline - packages that are part of the Atomic Host
> > > *and* include in their dist-git tests, must have these tests
> > > passed in the AtomicCI pipeline  
> > 
> > This update cannot be pushed:
> > 
> >   https://bodhi.fedoraproject.org/updates/FEDORA-2018-932548462e
> > 
> > When I click on the "automated tests" it says:
> > 
> >   "Automated Test Results
> >   Failed to talk to Greenwave.
> >   The update can not be pushed: 1 of 2 required tests not found"
> > 
> > and then the only "red" test (ie. failure) is rpmlint which is just
> > a bunch of rpmlint being wrong.  Nothing about "dist.depcheck" or
> > "dist.abicheck" is mentioned anywhere.  
> 
> abicheck is the first test results shown in the automated tests tab
> but indeed, no depcheck mentioned. Maybe someone from taskotron could
> help us on this?

In this particular case, I think that the update in question went from
updates-testing-pending to updates-testing while taskotron was down for
an updgrade on the 9th. There are no rpmdeplint results in resultsdb
which is what I assume greenwave is complaining about.

To be honest, we've always assumed that this situation was unlikely,
verging on impossible due to the way that rpmdeplint is scheduled and
run. Obviously, it's more likely than we thought and is an issue that
we'll need to address going forward before gating can be enabled for
everything.

Actually, all of these issues are a bit surprising and there are
several things which have gone from "eh, nobody cares. we'll get around
to it someday" to "we should probably get that figured out now-ish" due
to the issues raised around the gating in bodhi.

Tim


pgp4aAexCmlR9.pgp
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Adam Williamson]

On Tue, 2018-01-23 at 18:12 +, Richard W.M. Jones wrote:

> The problem for the OCaml packages is missing tests or tests that
> haven't been run:
> 
> https://bodhi.fedoraproject.org/updates/FEDORA-2018-932548462e
> https://bodhi.fedoraproject.org/updates/FEDORA-2018-ecd3541af9
> 
> BTW these both really need to be pushed as soon as possible because
> they fix a license violation.

Yes, we're looking into that too - see
https://github.com/fedora-infra/bodhi/issues/2133 for some discussion.
Basically sometimes Taskotron tests for some package or other just
don't run for some transient reason - some bit of the process is down
or not working correctly, and because it's all fedmsg-driven, there's
really only one shot and if the tests don't trigger correctly at the
time they should, nothing goes back and re-triggers them later.
Improving this has never been a high priority because it hasn't really
mattered a lot to anyone...until now. The policy as currently
implemented basically means "the update *only* passes if we can find a
'pass' for each of the required tests". So if one isn't run, the update
fails.

The initial thought we - we being Tim, Kamil, Josef, Ralph and I - had
is to simply invert the policy, if we can, so it becomes "the update
passes *unless* we can find a 'fail' for any of the required tests". So
all updates would be push-able (so far as this mechanism is concerned,
ignoring all the old ones) until one of the gating tests definitely
failed.

If we can't do that, we're going to just have to disable the gating
again until this is sorted out; we're definitely of the opinion that
Taskotron doesn't yet provide enough of a solid guarantee that all the
tests will be run for a policy which *assumes* that will be the case to
be viable.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Adam Williamson]

On Tue, 2018-01-23 at 21:25 +, Philip Kovacs wrote:
> Can someone please elaborate on how I can control the abi tests
> directly?Where exactly can I access these and refine them on a per-
> package basis?

> How to fix the tests?

That text isn't talking about "fixing the tests", but about fixing the
*bugs*. It assumes that the test indicates a real issue that needs
fixing, therefore you can fix the issue in your package and cause the
test to pass.

It doesn't really cover the scenario where what the test is reporting
isn't really a problem and doesn't need to be fixed. You can't resolve
that from your package git repository, but you *can* submit a waiver,
as discussed upthread.

Additionally, as Ralph wrote, he has removed abicheck from the list of
gating tests for now, so you shouldn't need to worry about abicheck
failures blocking updates, as soon as Bodhi starts applying that
change.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Adam Williamson]

On Tue, 2018-01-23 at 14:16 -0500, Matthew Miller wrote:
> On Tue, Jan 23, 2018 at 12:25:56PM -0600, Rex Dieter wrote:
> > > There are three tests that must pass in order for updates to go to stable:
> > > 0. dist.depcheck - to make sure the update's dependencies are available.
> > > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> > >given Fedora release.
> > 
> > ...
> > > Finally, if it turns out you need to push an update through despite of the
> > > test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> > > We are working on integrating this into Bodhi itself, making this easier.
> > 
> > I think it unwise to make item 1 a mandatory item at this point.  I'd argue 
> > a large number of packages do not provide public api/abi that's worth 
> > worrying about in this regard.
> 
> I think we should definine a set of packages where we care about this,
> and enable it on a case-by-case basis and make it advisory otherwise.

I think we could probably do something a *bit* less icky than a
completely manual package list. What I thought of as a very simple
initial heuristic would be to only care about changes in "any shared
object installed to /usr/lib or /usr/lib64", optionally we could
restrict that further to "in a critpath package". That's installed
*directly to those directories*, not to any subdirectories of them.

This would certainly be less than the set of things we *should* care
about, because there are various mechanisms by which a shared object
that's not directly in %{libdir} can have a public ABI of some kind
(ldconfig snippets, etc). But it would at least be something to start
with...unless I'm missing something.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Dominik 'Rathann' Mierzejewski]

On Tuesday, 23 January 2018 at 20:16, Matthew Miller wrote:
> On Tue, Jan 23, 2018 at 12:25:56PM -0600, Rex Dieter wrote:
> > > There are three tests that must pass in order for updates to go to stable:
> > > 0. dist.depcheck - to make sure the update's dependencies are available.
> > > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> > >given Fedora release.
> > ...
> > > Finally, if it turns out you need to push an update through despite of the
> > > test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> > > We are working on integrating this into Bodhi itself, making this easier.
> > I think it unwise to make item 1 a mandatory item at this point.  I'd argue 
> > a large number of packages do not provide public api/abi that's worth 
> > worrying about in this regard.
> 
> I think we should definine a set of packages where we care about this,
> and enable it on a case-by-case basis and make it advisory otherwise.

That makes sense. How about we start with critical path packages?
Alternatively, libraries which a lot of of other packages depend on
would be good candidates.

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPMFusion   http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Philip Kovacs]

Can someone please elaborate on how I can control the abi tests directly?Where 
exactly can I access these and refine them on a per-package basis?

How to fix the tests?
The tests are all in your hand, you can fix the dist.depcheck and dist.abicheck 
by adjusting the update or the build and you can fix the package level testing 
since the tests are stored in the dist-git repository of the package itself. 
You have the control on the tests.
 

On Tuesday, January 23, 2018 2:16 PM, Matthew Miller 
 wrote:
 

 On Tue, Jan 23, 2018 at 12:25:56PM -0600, Rex Dieter wrote:
> > There are three tests that must pass in order for updates to go to stable:
> > 0. dist.depcheck - to make sure the update's dependencies are available.
> > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> >    given Fedora release.
> ...
> > Finally, if it turns out you need to push an update through despite of the
> > test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> > We are working on integrating this into Bodhi itself, making this easier.
> I think it unwise to make item 1 a mandatory item at this point.  I'd argue 
> a large number of packages do not provide public api/abi that's worth 
> worrying about in this regard.

I think we should definine a set of packages where we care about this,
and enable it on a case-by-case basis and make it advisory otherwise.

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


   ___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Matthew Miller]

On Tue, Jan 23, 2018 at 12:25:56PM -0600, Rex Dieter wrote:
> > There are three tests that must pass in order for updates to go to stable:
> > 0. dist.depcheck - to make sure the update's dependencies are available.
> > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> >given Fedora release.
> ...
> > Finally, if it turns out you need to push an update through despite of the
> > test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> > We are working on integrating this into Bodhi itself, making this easier.
> I think it unwise to make item 1 a mandatory item at this point.  I'd argue 
> a large number of packages do not provide public api/abi that's worth 
> worrying about in this regard.

I think we should definine a set of packages where we care about this,
and enable it on a case-by-case basis and make it advisory otherwise.

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rafael dos Santos]

On 23 January 2018 at 19:22, Ralph Bean  wrote:

> On Tue, Jan 23, 2018 at 06:35:45PM +0100, Rafael dos Santos wrote:
> > I get this error after clicking the authorization link:
> >
> > [16450:16491:0123/182437.407698:ERROR:browser_gpu_
> channel_host_factory.cc(120)]
> > Failed to launch GPU process.
> > Created new window in existing browser session.
> > 127.0.0.1 - - [23/Jan/2018 18:24:37] "GET
> > /?error_description=Unknown+client+ID=unauthorized_client
> HTTP/1.1"
> > 200 47
> > Traceback (most recent call last):
> >   File "/usr/bin/waiverdb-cli", line 11, in 
> > load_entry_point('waiverdb==0.4.0', 'console_scripts',
> 'waiverdb-cli')()
> >   File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in
> > __call__
> > return self.main(*args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in
> main
> > rv = self.invoke(ctx)
> >   File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in
> invoke
> > return ctx.invoke(self.callback, **ctx.params)
> >   File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in
> invoke
> > return callback(*args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/waiverdb/cli.py", line 102, in
> cli
> > if not resp.ok:
> > AttributeError: 'NoneType' object has no attribute 'ok'
>
> ACK.  Just resolved this one now.  Can you try again?
>
> (You may need to visit https://id.fedoraproject.org/logout first.)
>

Yes, it's working fine now. Thanks.


Att.
--
Rafael Fonseca
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rex Dieter]

Pierre-Yves Chibon wrote:

> On Thursday, a new version of Bodhi was deployed that enabled Bodhi to
> gate updates based on test results. You may notice a "Test Gating
> Status" message in the right have side of the page.
...
> There are three tests that must pass in order for updates to go to stable:
> 
> 0. dist.depcheck - to make sure the update's dependencies are available.
> 1. dist.abicheck - to make sure the update's ABI remains stable in a
>given Fedora release.

...
> Finally, if it turns out you need to push an update through despite of the
> test results, you can do so using waiver-cli (dnf install waiverdb-cli).
> We are working on integrating this into Bodhi itself, making this easier.


I think it unwise to make item 1 a mandatory item at this point.  I'd argue 
a large number of packages do not provide public api/abi that's worth 
worrying about in this regard.

-- Rex
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Ralph Bean]

On Tue, Jan 23, 2018 at 06:35:45PM +0100, Rafael dos Santos wrote:
> I get this error after clicking the authorization link:
>
> [16450:16491:0123/182437.407698:ERROR:browser_gpu_channel_host_factory.cc(120)]
> Failed to launch GPU process.
> Created new window in existing browser session.
> 127.0.0.1 - - [23/Jan/2018 18:24:37] "GET
> /?error_description=Unknown+client+ID=unauthorized_client HTTP/1.1"
> 200 47
> Traceback (most recent call last):
>   File "/usr/bin/waiverdb-cli", line 11, in 
> load_entry_point('waiverdb==0.4.0', 'console_scripts', 'waiverdb-cli')()
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in
> __call__
> return self.main(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
> rv = self.invoke(ctx)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
> return ctx.invoke(self.callback, **ctx.params)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
> return callback(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/waiverdb/cli.py", line 102, in cli
> if not resp.ok:
> AttributeError: 'NoneType' object has no attribute 'ok'

ACK.  Just resolved this one now.  Can you try again?

(You may need to visit https://id.fedoraproject.org/logout first.)


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Richard W.M. Jones]

On Tue, Jan 23, 2018 at 05:18:42PM +0100, Adam Williamson wrote:
> On Tue, 2018-01-23 at 08:42 -0700, Jerry James wrote:
> > Where are the instructions?  Why is informing packagers, the group
> > most affected by this change, an afterthought?  We should have been
> > told about all of this, in detail, prior to the thing being turned on,
> > *well* before it was turned on.
> 
> So, my answer to this is second-hand - apologies if any of it is wrong,
> and the folks involved will no doubt correct me. But as I understand
> it, I think they weren't really expecting the tests that were turned on
> to have false positives; the tests that were chosen were intended to be
> ones that wouldn't cause this kind of problem, so there'd be more time
> to get all the waiverdb integrations in place. I don't think they
> actually anticipated that false positives would happen and people would
> need to use waiverdb-cli like this, which is why instructions for it
> weren't part of the plan. I'm sure no-one intended to cause disruption
> and inconvenience, and we're sorry about it. I'll try to ask the
> relevant folks if perhaps we should stop gating on the abidiff test for
> now as a short-term measure, or something like that.

The problem for the OCaml packages is missing tests or tests that
haven't been run:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-932548462e
https://bodhi.fedoraproject.org/updates/FEDORA-2018-ecd3541af9

BTW these both really need to be pushed as soon as possible because
they fix a license violation.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Josh Stone]

On 01/23/2018 01:28 AM, Pierre-Yves Chibon wrote:
> There are three tests that must pass in order for updates to go to stable:
> 
> 0. dist.depcheck - to make sure the update's dependencies are available.
> 1. dist.abicheck - to make sure the update's ABI remains stable in a
>given Fedora release.
> 2. AtomicCI pipeline - packages that are part of the Atomic Host *and* include
>in their dist-git tests, must have these tests passed in the AtomicCI 
> pipeline
> 
> This last requirement only concern packages that are in the Atomic Host while
> the first two is enforced for all packages.

In my rust+cargo updates, I'm getting "1 of 4 required tests not found".

f27: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d5eb234853
f26: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ae3b642c47

It looks like it ran lots of tests for cargo, but only one on rust.
What can I do to resolve this?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Rafael dos Santos]

I get this error after clicking the authorization link:

[16450:16491:0123/182437.407698:ERROR:browser_gpu_channel_host_factory.cc(120)]
Failed to launch GPU process.
Created new window in existing browser session.
127.0.0.1 - - [23/Jan/2018 18:24:37] "GET
/?error_description=Unknown+client+ID=unauthorized_client HTTP/1.1"
200 47
Traceback (most recent call last):
  File "/usr/bin/waiverdb-cli", line 11, in 
load_entry_point('waiverdb==0.4.0', 'console_scripts', 'waiverdb-cli')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in
__call__
return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/waiverdb/cli.py", line 102, in cli
if not resp.ok:
AttributeError: 'NoneType' object has no attribute 'ok'


Att.
--
Rafael Fonseca
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Tom Hughes]

On 23/01/18 16:18, Adam Williamson wrote:

On Tue, 2018-01-23 at 08:42 -0700, Jerry James wrote:


Where are the instructions?  Why is informing packagers, the group
most affected by this change, an afterthought?  We should have been
told about all of this, in detail, prior to the thing being turned on,
*well* before it was turned on.


So, my answer to this is second-hand - apologies if any of it is wrong,
and the folks involved will no doubt correct me. But as I understand
it, I think they weren't really expecting the tests that were turned on
to have false positives; the tests that were chosen were intended to be
ones that wouldn't cause this kind of problem, so there'd be more time
to get all the waiverdb integrations in place. I don't think they
actually anticipated that false positives would happen and people would
need to use waiverdb-cli like this, which is why instructions for it
weren't part of the plan. I'm sure no-one intended to cause disruption
and inconvenience, and we're sorry about it. I'll try to ask the
relevant folks if perhaps we should stop gating on the abidiff test for
now as a short-term measure, or something like that.


There was one that came up on IRC yesterday that had failed abidiff 
entirely because of added functions as far as I could see.


I don't know if that counts as a false positive or not because I don't 
know what the design goals are - added functions should mean it's not 
going to break any existing programs but new programs built against it 
might not work against old versions of the library.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Adam Williamson]

On Tue, 2018-01-23 at 08:42 -0700, Jerry James wrote:
> 
> There are no instructions on how to use it on this wiki page.  Okay,
> we'll try using --help:
> 
> $ waiverdb-cli --help
> Usage: waiverdb-cli [OPTIONS]
> 
>   Creates new waivers against test results.
> 
>   Examples:
> 
>   waiverdb-cli -r 123 -r 456 -p "fedora-26" -c "It's dead!"
> 
> Options:
>   -C, --config-file PATH  Specify a config file to use
>   -r, --result-id INTEGER Specify one or more results to be waived
>   -p, --product-version TEXT  Specify one of PDC's product version
>   identifiers.
>   --waived / --no-waived  Whether or not the result is waived
>   -c, --comment TEXT  A comment explaining why the result is waived
>   -h, --help  Show this message and exit.
> 
> 
> Ookaay, so what is a result-id and how do I figure out which
> one I need to supply for my particular case?

It's how resultsdb identifies a single specific result. Each result in
resultsdb has one, and they're unique.

However, I see an obvious problem here: I can't see any easy way to
figure out the result ID of the failure from the Bodhi web UI. The URL
you get taken to when you click on the test result in Bodhi doesn't
include it, so far as I can see. You *can* actually find the failed
test quite 'easily' from the resultsdb web UI, but you have to know how
to do it: go to the search page - 
https://taskotron.fedoraproject.org/resultsdb/results - and enter the
package NVR (gap-pkg-io-4.5.1-1.fc27) as the 'item', and dist.abicheck
as the 'testcase', then search, and you'll find it. Then click
'details' and you can see the ID is 18913262. But that's obviously not
an ideal workflow! As Pierre said, they're working on hooking this all
up from the Bodhi web UI so waiving will be a much easier process.

>   I can guess what a
> product-version is, but why is it in a different format from every
> other Fedora tool that takes such a thing?

Because that's the form PDC uses; waiverdb works with PDC, and PDC
chose its format a while ago, so that's kind of plumbed in now.

>   How do I point this tool
> at the update of mine that is currently being blocked?

At least AIUI, you don't actually have to: all that matching logic is
performed by the various systems involved. Basically, Bodhi asks
Greenwave if the update is good to go; Greenwave asks ResultsDB for
tests associated with the update, and asks WaiverDB for waivers
associated with those tests. So its answer to Bodhi will take the
waiver into account. All you have to do is submit the waiver.

> Where are the instructions?  Why is informing packagers, the group
> most affected by this change, an afterthought?  We should have been
> told about all of this, in detail, prior to the thing being turned on,
> *well* before it was turned on.

So, my answer to this is second-hand - apologies if any of it is wrong,
and the folks involved will no doubt correct me. But as I understand
it, I think they weren't really expecting the tests that were turned on
to have false positives; the tests that were chosen were intended to be
ones that wouldn't cause this kind of problem, so there'd be more time
to get all the waiverdb integrations in place. I don't think they
actually anticipated that false positives would happen and people would
need to use waiverdb-cli like this, which is why instructions for it
weren't part of the plan. I'm sure no-one intended to cause disruption
and inconvenience, and we're sorry about it. I'll try to ask the
relevant folks if perhaps we should stop gating on the abidiff test for
now as a short-term measure, or something like that.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Jerry James]

On Tue, Jan 23, 2018 at 2:28 AM, Pierre-Yves Chibon  wrote:
> Finally, if it turns out you need to push an update through despite of the 
> test
> results, you can do so using waiver-cli (dnf install waiverdb-cli). We are
> working on integrating this into Bodhi itself, making this easier.

Good!  I need this.  So let's apply it to my update.  H, there is
no man page in the waiverdb-cli package.  There are no instructions on
how to use it in this email.

> We have started a wiki page to store all of this information and that we will
> keep up to date as improvements are done or for any frequent questions coming 
> up:
>
>   https://fedoraproject.org/wiki/CI/gating_updates

There are no instructions on how to use it on this wiki page.  Okay,
we'll try using --help:

$ waiverdb-cli --help
Usage: waiverdb-cli [OPTIONS]

  Creates new waivers against test results.

  Examples:

  waiverdb-cli -r 123 -r 456 -p "fedora-26" -c "It's dead!"

Options:
  -C, --config-file PATH  Specify a config file to use
  -r, --result-id INTEGER Specify one or more results to be waived
  -p, --product-version TEXT  Specify one of PDC's product version
  identifiers.
  --waived / --no-waived  Whether or not the result is waived
  -c, --comment TEXT  A comment explaining why the result is waived
  -h, --help  Show this message and exit.


Ookaay, so what is a result-id and how do I figure out which
one I need to supply for my particular case?  I can guess what a
product-version is, but why is it in a different format from every
other Fedora tool that takes such a thing?  How do I point this tool
at the update of mine that is currently being blocked?

Where are the instructions?  Why is informing packagers, the group
most affected by this change, an afterthought?  We should have been
told about all of this, in detail, prior to the thing being turned on,
*well* before it was turned on.
-- 
Jerry James
http://www.jamezone.org/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Richard W.M. Jones]

On Tue, Jan 23, 2018 at 09:47:44AM +, Tom Hughes wrote:
> On 23/01/18 09:42, Richard W.M. Jones wrote:
> 
> >When I click on the "automated tests" it says:
> >
> >   "Automated Test Results
> >   Failed to talk to Greenwave.
> >   The update can not be pushed: 1 of 2 required tests not found"
> >
> >and then the only "red" test (ie. failure) is rpmlint which is just a
> >bunch of rpmlint being wrong.  Nothing about "dist.depcheck" or
> >"dist.abicheck" is mentioned anywhere.
> 
> If you're using uMatrix or similar then check it isn't getting in
> the way - the "Failed to talk to Greenwave" is a message that I was
> getting yesterday until I trained uMatrix to allow it.
> 
> I'm not getting that message but I am getting the rest so it looks
> like there is an issue here.

I enabled Greenwave, but still I don't see what the error is.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Pierre-Yves Chibon]

On Tue, Jan 23, 2018 at 09:42:50AM +, Richard W.M. Jones wrote:
> On Tue, Jan 23, 2018 at 10:28:14AM +0100, Pierre-Yves Chibon wrote:
> > Good Morning Fedorans!
> > 
> > On Thursday, a new version of Bodhi was deployed that enabled Bodhi to
> > gate updates based on test results. You may notice a "Test Gating
> > Status" message in the right have side of the page.
> > 
> > One thing to know about this is that there is currently a confusing
> > issue where Bodhi will say that the tests have failed when the tests
> > haven't finished running[0]. We are working on solving that issue, but
> > for now you can just wait a while and it should report the result once
> > all the tests have finished.
> > 
> > There are three tests that must pass in order for updates to go to stable:
> > 
> > 0. dist.depcheck - to make sure the update's dependencies are available.
> > 1. dist.abicheck - to make sure the update's ABI remains stable in a
> >given Fedora release.
> > 2. AtomicCI pipeline - packages that are part of the Atomic Host *and* 
> > include
> >in their dist-git tests, must have these tests passed in the AtomicCI 
> > pipeline
> 
> This update cannot be pushed:
> 
>   https://bodhi.fedoraproject.org/updates/FEDORA-2018-932548462e
> 
> When I click on the "automated tests" it says:
> 
>   "Automated Test Results
>   Failed to talk to Greenwave.
>   The update can not be pushed: 1 of 2 required tests not found"
> 
> and then the only "red" test (ie. failure) is rpmlint which is just a
> bunch of rpmlint being wrong.  Nothing about "dist.depcheck" or
> "dist.abicheck" is mentioned anywhere.

abicheck is the first test results shown in the automated tests tab but indeed,
no depcheck mentioned. Maybe someone from taskotron could help us on this?


Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Tom Hughes]

On 23/01/18 09:42, Richard W.M. Jones wrote:


When I click on the "automated tests" it says:

   "Automated Test Results
   Failed to talk to Greenwave.
   The update can not be pushed: 1 of 2 required tests not found"

and then the only "red" test (ie. failure) is rpmlint which is just a
bunch of rpmlint being wrong.  Nothing about "dist.depcheck" or
"dist.abicheck" is mentioned anywhere.


If you're using uMatrix or similar then check it isn't getting in the 
way - the "Failed to talk to Greenwave" is a message that I was getting 
yesterday until I trained uMatrix to allow it.


I'm not getting that message but I am getting the rest so it looks like 
there is an issue here.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Test gating enabled in Bodhi

[Richard W.M. Jones]

On Tue, Jan 23, 2018 at 10:28:14AM +0100, Pierre-Yves Chibon wrote:
> Good Morning Fedorans!
> 
> On Thursday, a new version of Bodhi was deployed that enabled Bodhi to
> gate updates based on test results. You may notice a "Test Gating
> Status" message in the right have side of the page.
> 
> One thing to know about this is that there is currently a confusing
> issue where Bodhi will say that the tests have failed when the tests
> haven't finished running[0]. We are working on solving that issue, but
> for now you can just wait a while and it should report the result once
> all the tests have finished.
> 
> There are three tests that must pass in order for updates to go to stable:
> 
> 0. dist.depcheck - to make sure the update's dependencies are available.
> 1. dist.abicheck - to make sure the update's ABI remains stable in a
>given Fedora release.
> 2. AtomicCI pipeline - packages that are part of the Atomic Host *and* include
>in their dist-git tests, must have these tests passed in the AtomicCI 
> pipeline

This update cannot be pushed:

  https://bodhi.fedoraproject.org/updates/FEDORA-2018-932548462e

When I click on the "automated tests" it says:

  "Automated Test Results
  Failed to talk to Greenwave.
  The update can not be pushed: 1 of 2 required tests not found"

and then the only "red" test (ie. failure) is rpmlint which is just a
bunch of rpmlint being wrong.  Nothing about "dist.depcheck" or
"dist.abicheck" is mentioned anywhere.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org