Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure
On Tue, Jul 10, 2018 at 03:16:14PM -0400, Matthew Treinish wrote: > On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote: > > On Mon, 9 Jul 2018, Matthew Treinish wrote: > > > > > It's definitely a bug, and likely a bug in stestr (or one of the lower > > > level > > > packages like testtools or python-subunit), because that's what's > > > generating > > > the return code. Tox just looks at the return code from the commands to > > > figure > > > out if things were successful or not. I'm a bit surprised by this though I > > > thought we covered the unxsuccess and xfail cases because I would have > > > expected > > > cdent to file a bug if it didn't. Looking at the stestr tests we don't > > > have > > > coverage for the unxsuccess case so I can see how this slipped through. > > > > This was reported on testrepository some years ago and a bit of > > analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196 > > > > This actually helps a lot, because I was seeing the same issue when I tried > writing a quick patch to address this. When I manually poked the TestResult > object it didn't have anything in the unxsuccess list. So instead of relying > on that I wrote this patch: > > https://github.com/mtreinish/stestr/pull/188 > > which uses the output filter's internal function for counting results to > find unxsuccess tests. It's still not perfect though because if someone > runs with the --no-subunit-trace flag it still doesn't work (because that > call path never gets run) but it's at least a starting point. I've > marked it as WIP for now, but I'm thinking we could merge it as is and > leave the --no-subunit-trace and unxsuccess as a known issues for now, > since xfail and unxsuccess are pretty uncommon in practice. (gabbi is the > only thing I've seen really use it) > > > > > So yeah, I did file a bug but it fell off the radar during those > > dark times. > > > Just following up here, after digging some more and getting a detailed bug filed by electrofelix [1] I was able to throw together a different patch that should solve this in a better way: https://github.com/mtreinish/stestr/pull/190 Once that lands I can push a bugfix release to get it out there so people can actually use the fix. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure
On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote: > On Mon, 9 Jul 2018, Matthew Treinish wrote: > > > It's definitely a bug, and likely a bug in stestr (or one of the lower > > level > > packages like testtools or python-subunit), because that's what's generating > > the return code. Tox just looks at the return code from the commands to > > figure > > out if things were successful or not. I'm a bit surprised by this though I > > thought we covered the unxsuccess and xfail cases because I would have > > expected > > cdent to file a bug if it didn't. Looking at the stestr tests we don't have > > coverage for the unxsuccess case so I can see how this slipped through. > > This was reported on testrepository some years ago and a bit of > analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196 > This actually helps a lot, because I was seeing the same issue when I tried writing a quick patch to address this. When I manually poked the TestResult object it didn't have anything in the unxsuccess list. So instead of relying on that I wrote this patch: https://github.com/mtreinish/stestr/pull/188 which uses the output filter's internal function for counting results to find unxsuccess tests. It's still not perfect though because if someone runs with the --no-subunit-trace flag it still doesn't work (because that call path never gets run) but it's at least a starting point. I've marked it as WIP for now, but I'm thinking we could merge it as is and leave the --no-subunit-trace and unxsuccess as a known issues for now, since xfail and unxsuccess are pretty uncommon in practice. (gabbi is the only thing I've seen really use it) -Matt Treinish > So yeah, I did file a bug but it fell off the radar during those > dark times. > signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure
On Mon, 9 Jul 2018, Matthew Treinish wrote: It's definitely a bug, and likely a bug in stestr (or one of the lower level packages like testtools or python-subunit), because that's what's generating the return code. Tox just looks at the return code from the commands to figure out if things were successful or not. I'm a bit surprised by this though I thought we covered the unxsuccess and xfail cases because I would have expected cdent to file a bug if it didn't. Looking at the stestr tests we don't have coverage for the unxsuccess case so I can see how this slipped through. This was reported on testrepository some years ago and a bit of analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196 So yeah, I did file a bug but it fell off the radar during those dark times. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure
On Mon, Jul 09, 2018 at 06:59:42PM -0500, Eric Fried wrote:
> In gabbi, there's a way [1] to mark a test as an expected failure, which
> makes it show up in your stestr run thusly:
>
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.710821s] ... ok
>
> ==
> Totals
> ==
> Ran: 1 tests in 9. sec.
> - Passed: 0
> - Skipped: 0
> - Expected Fail: 1
> - Unexpected Success: 0
> - Failed: 0
>
> If I go fix the thing causing the heretofore-expected failure, but
> forget to take out the `xfail: True`, it does this:
>
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.710517s] ... FAILED
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.00s] ... ok
>
> ==
> Failed 1 tests - output below:
> ==
>
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> --
>
>
> ==
> Totals
> ==
> Ran: 2 tests in 9. sec.
> - Passed: 1
> - Skipped: 0
> - Expected Fail: 0
> - Unexpected Success: 1
> - Failed: 0
>
> BUT it does not cause the run to fail. For example, see the
> nova-tox-functional results for [2] (specifically PS4): the test appears
> twice in the middle of the run [3] and prints failure output [4] but the
> job passes [5].
>
> So I'm writing this email because I have no idea if this is expected
> behavior or a bug (I'm hoping the latter, cause it's whack, yo); and if
> a bug, I have no idea whose bug it should be. Help?
It's definitely a bug, and likely a bug in stestr (or one of the lower level
packages like testtools or python-subunit), because that's what's generating
the return code. Tox just looks at the return code from the commands to figure
out if things were successful or not. I'm a bit surprised by this though I
thought we covered the unxsuccess and xfail cases because I would have expected
cdent to file a bug if it didn't. Looking at the stestr tests we don't have
coverage for the unxsuccess case so I can see how this slipped through.
Looking at the where the return code for the output from the run command is
generated (it's a bit weird because run calls the load command internally which
handles the output generation, result storage, and final return code):
https://github.com/mtreinish/stestr/blob/master/stestr/commands/load.py#L222-L225
I'm thinking it might be an issue in testtools or python-subunit, I don't
remember
which generates the results object used there (if it is subunit it'll be a
subclass from testtools). But I'll have to trace through it to be sure. In the
mean time we can easily workaround the issue in stestr itself by just manually
checking the result status instead of relying on the existing function from the
results class.
-Matt Treinish
>
> [1] https://gabbi.readthedocs.io/en/latest/format.html?highlight=xfail
> [2] https://review.openstack.org/#/c/579921/4
> [3]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_22_11_846366
> [4]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_31_07_229271
> [5]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/testr_results.html.gz
>
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure
In gabbi, there's a way [1] to mark a test as an expected failure, which
makes it show up in your stestr run thusly:
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710821s] ... ok
==
Totals
==
Ran: 1 tests in 9. sec.
- Passed: 0
- Skipped: 0
- Expected Fail: 1
- Unexpected Success: 0
- Failed: 0
If I go fix the thing causing the heretofore-expected failure, but
forget to take out the `xfail: True`, it does this:
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710517s] ... FAILED
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.00s] ... ok
==
Failed 1 tests - output below:
==
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
--
==
Totals
==
Ran: 2 tests in 9. sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 1
- Failed: 0
BUT it does not cause the run to fail. For example, see the
nova-tox-functional results for [2] (specifically PS4): the test appears
twice in the middle of the run [3] and prints failure output [4] but the
job passes [5].
So I'm writing this email because I have no idea if this is expected
behavior or a bug (I'm hoping the latter, cause it's whack, yo); and if
a bug, I have no idea whose bug it should be. Help?
Thanks,
efried
[1] https://gabbi.readthedocs.io/en/latest/format.html?highlight=xfail
[2] https://review.openstack.org/#/c/579921/4
[3]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_22_11_846366
[4]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_31_07_229271
[5]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/testr_results.html.gz
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
