On 02/12/2014 04:25 PM, Maru Newby wrote: > > On Feb 12, 2014, at 12:36 PM, Sean Dague <[email protected]> wrote: > >> On 02/12/2014 01:48 PM, Maru Newby wrote: >>> At the last 2 summits, I've suggested that API tests could be maintained in >>> the Neutron tree and reused by Tempest. I've finally submitted some >>> patches that demonstrate this concept: >>> >>> https://review.openstack.org/#/c/72585/ (implements a unit test for the >>> lifecycle of the network resource) >>> https://review.openstack.org/#/c/72588/ (runs the test with tempest rest >>> clients) >>> >>> My hope is to make API test maintenance a responsibility of the Neutron >>> team. The API compatibility of each Neutron plugin has to be validated by >>> Neutron tests anyway, and if the tests are structured as I am proposing, >>> Tempest can reuse those efforts rather than duplicating them. >>> >>> I've added this topic to this week's agenda, and I would really appreciate >>> it interested parties would take a look at the patches in question to >>> prepare themselves to participate in the discussion. >>> >>> >>> m. >>> _______________________________________________ >>> OpenStack-dev mailing list >>> [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> Realistically, having API tests duplicated in the Tempest tree is a >> feature, not a bug. >> >> tempest/api is there for double book keep accounting, and it has been >> really effective at preventing accidental breakage of our APIs (which >> used to happen all the time), so I don't think putting API testing in >> neutron obviates that. > > Given how limited our testing resources are, might it be worth considering > whether 'double-entry accounting' is actually the best way to preventing > accidental breakage going forward? Might reasonable alternatives exist, such > as clearly separating api tests from other tests in the neutron tree and > giving review oversight only to qualified individuals?
Our direct experience is that if we don't do this, within 2 weeks some
project will have landed API breaking changes. This approach actually
takes a lot of review load off the core reviewers, so reverting to a
model which puts more work back on the review team (given the current
review load), isn't something I think we want.
I get that there is a cost with this. But there is a cost of all of
this. And because API tests should be write once for the API (and
specifically not evolving), the upfront cost vs. the continually paid
cost of review time in tree has been the better trade off.
-Sean
--
Sean Dague
Samsung Research America
[email protected] / [email protected]
http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
