On 11/01/18 16:36, Colleen Murphy wrote: > Hi everyone, > > We have governance review under debate[1] that we need the community's help > on. > The debate is over what recommendation the TC should make to the Interop team > on where the tests it uses for the OpenStack trademark program should be > located, specifically those for the new add-on program being introduced. Let > me > badly summarize: > > A couple of years ago we issued a resolution[2] officially recommending that > the Interop team use solely tempest as its source of tests for capability > verification. The Interop team has always had the view that the developers, > being the people closest to the project they're creating, are the best people > to write tests verifying correct functionality, and so the Interop team > doesn't > maintain its own test suite, instead selecting tests from those written in > coordination between the QA team and the other project teams. These tests are > used to validate clouds applying for the OpenStack Powered tag, and since all > of the projects included in the OpenStack Powered program already had tests in > tempest, this was a natural fit. When we consider adding new trademark > programs > comprising of other projects, the test source is less obvious. Two examples > are > designate, which has never had tests in the tempest repo, and heat, which > recently had its tests removed from the tempest repo. >
<snip/> > > As I'm not deeply steeped in the history of either the Interop or QA teams I > am > sure I've misrepresented some details here, I'm sorry about that. But we'd > like > to get this resolution moving forward and we're currently stuck, so this > thread > is intended to gather enough community input to get unstuck and avoid letting > this proposal become stale. Please respond to this thread or comment on the > resolution proposal[1] if you have any thoughts. > > Colleen > > [1] https://review.openstack.org/#/c/521602 > [2] > https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html > [3] > https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html > I had hoped for more of a discussion on this before I jumped back into this debate - but it seams to be stalled still, so here it goes. I proposed this initially as we were unclear on where the tests should go - we had a resolution that said all tests go into openstack/tempest (with a list of reasons why), and the guidance and discussion that been had in various summits was that "add-ons" should stay in plugins. So right now, we (by the governance rules) should be pushing tests to tempest for the new programs. In the resolution that placed the tests in tempest there was a few reasons proposed: For example, API and behavioral changes must be carefully managed, as must mundane aspects such as test and module naming and location within the test suite. Even changes that leave tests functionally equivalent may cause unexpected consequences for their use in DefCore processes and validation. Placing the tests in a central repository will make it easier to maintain consistency and avoid breaking the trademark enforcement tool. This still applies, and even more so for teams that traditionally do not have a strong QE contributor / reviewer base (aka projects not in "core"). Centralizing the tests also makes it easier for anyone running the validation tool against their cloud or cloud distribution to use the tests. It is easier to install the test suite and its dependencies, and it is easier to read and understand a set of tests following a consistent implementation pattern. Apparently users do not need central tests anymore, feedback from RefStack is that people who run these tests are comfortable dealing with extra python packages. The point about a single set of tests, in a single location and style still stands. Finally, having the tests in a central location makes it easier to ensure that all members of the community have equal input into what the tests do and how they are implemented and maintained. Seems like a good value to strive for. One of the items that has been used to push back against adding "add-ons" to tempest has been that tempest has a defined scope, and neither of the current add-ons fit in that scope. Can someone clarify what the set of criteria is? I think it will help this discussion. Another push back is the "scaling" issue - adding more tests will overload the QA team. Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite of 353. I do not think there is many other add-ons proposed yet, and the new Vertical programs will probably mainly be re-using tests in the openstack/tempest repos as is. This is not a big tent-esque influx of programs - the only projects that can be added to the trademarks are programs in tc-approved-release [4], so I do not see scaling as a big issue, especially as these tests are such base concepts that if they need to be changed there is a completely new API, so the only overhead will be ensuring that nothing in tempest breaks the new tests (which is a good thing for trademark tests). Personally, for me, I like option 3. I did not initially add it, as I knew it would cause endless bikesheding, but I do think it fits both a technical and social model. I see 2 immediate routes forward: - Option 1, and we start adding these tests asap - Pseudo Option 2, were we delete the resolution at [2] as it clearly does not apply anymore, and abandon the review at [1]. Finally - do not conflate my actions with those of the Designate team. I have seen people talking about how this resolution was leverage the team needed to move our tests in tree. This is definitely *not* true. Having our tests in a plugin is useful to us, and if the above resolution passed, I cannot see a situation where we would try to move any tests that were not listed in the interop standards. This is something I have done as an individual in the community, not something the designate team have pushed for. [4] - https://governance.openstack.org/tc/reference/tags/tc_approved-release.html > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
signature.asc
Description: OpenPGP digital signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
