Re: [openstack-dev] [ceilometer] [gnocchi] monitoring unexpected success of xfail designated tests
On Fri, 6 Mar 2015, Chris Dent wrote: It looks like the problem is in subunit (this is with a locally modified gabbi test): $ for i in testtools subunit ; \ do python -m $i.run discover gnocchi.tests.gabbi &>/dev/null || \ echo "$i catches uxsuccess as fail" ; \ done testtools catches uxsuccess as fail I've filed a bug: https://bugs.launchpad.net/subunit/+bug/1429196 Actually I guess that's just the way subunit works. It doesn't exit with failure even when there is failure. So that suggests the problem is in testr's interpretation of the result stream. I'll update the bug. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [gnocchi] monitoring unexpected success of xfail designated tests
On Fri, 6 Mar 2015, Julien Danjou wrote: On Fri, Mar 06 2015, Chris Dent wrote: Gnocchi has already started using pretty_tox.sh, so as long as there is a recent version of subunit-trace (from tempest-lib) installed, gnocchi test runs will tell when there has been an unexpected success. Are you saying that unexpected success is not a failure? testr will announce "commands succeeded" when there is a uxsuccess in the result set. There me be some special sauce we can give it (or perhaps subunit.run) to say otherwise, but as things are currently written in the tox files, uxsuccess means pass. That sounds wrong. Where should we fix that? It looks like the problem is in subunit (this is with a locally modified gabbi test): $ for i in testtools subunit ; \ do python -m $i.run discover gnocchi.tests.gabbi &>/dev/null || \ echo "$i catches uxsuccess as fail" ; \ done testtools catches uxsuccess as fail I've filed a bug: https://bugs.launchpad.net/subunit/+bug/1429196 -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [gnocchi] monitoring unexpected success of xfail designated tests
On Fri, Mar 06 2015, Chris Dent wrote: > Gnocchi has already started using pretty_tox.sh, so as long as there > is a recent version of subunit-trace (from tempest-lib) installed, > gnocchi test runs will tell when there has been an unexpected > success. Are you saying that unexpected success is not a failure? That sounds wrong. Where should we fix that? -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
