On Mon, Jan 15, 2018 at 12:59 PM Erno Kuvaja <ekuv...@redhat.com> wrote:
> On Thu, Jan 11, 2018 at 4:36 PM, Colleen Murphy <coll...@gazlene.net> > wrote: > > Hi everyone, > > > > We have governance review under debate[1] that we need the community's > help on. > > The debate is over what recommendation the TC should make to the Interop > team > > on where the tests it uses for the OpenStack trademark program should be > > located, specifically those for the new add-on program being introduced. > Let me > > badly summarize: > > > > A couple of years ago we issued a resolution[2] officially recommending > that > > the Interop team use solely tempest as its source of tests for capability > > verification. The Interop team has always had the view that the > developers, > > being the people closest to the project they're creating, are the best > people > > to write tests verifying correct functionality, and so the Interop team > doesn't > > maintain its own test suite, instead selecting tests from those written > in > > coordination between the QA team and the other project teams. These > tests are > > used to validate clouds applying for the OpenStack Powered tag, and > since all > > of the projects included in the OpenStack Powered program already had > tests in > > tempest, this was a natural fit. When we consider adding new trademark > programs > > comprising of other projects, the test source is less obvious. Two > examples are > > designate, which has never had tests in the tempest repo, and heat, which > > recently had its tests removed from the tempest repo. > > > > So far the patch proposes three options: > > > > 1) All trademark-related tests should go in the tempest repo, in > accordance > > with the original resolution. This would mean that even projects that > have > > never had tests in tempest would now have to add at least some of > their > > black-box tests to tempest. > > > > The value of this option is that centralizes tests used for the Interop > program > > in a location where interop-minded folks from the QA team can control > them. The > > downside is that projects that so far have avoided having a dependency on > > tempest will now lose some control over the black-box tests that they > use for > > functional and integration that would now also be used for trademark > > certification. > > There's also concern for the review bandwidth of the QA team - we can't > expect > > the QA team to be continually responsible for an ever-growing list of > projects > > and their trademark tests. > > > > 2) All trademark-related tests for *add-on projects* should be sourced > from > > plugins external to tempest. > > > > The value of this option is it allows project teams to retain control > over > > these tests. The potential problem with it is that individual project > teams are > > not necessarily reviewing test changes with an eye for interop concerns > and so > > could inadvertently change the behavior of the trademark-verification > tools. > > > > 3) All trademark-related tests should go in a single separate tempest > plugin. > > > > This has the value of giving the QA and Interop teams control over > > interop-related tests while also making clear the distinction between > tests > > used for trademark verification and tests used for CI. Matt's argument > against > > this is that there actually is very little distinction between those two > cases, > > and that a given test could have many different applications. > > > > Other ideas that have been thrown around are: > > > > * Maintaining a branch in the tempest repo that Interop tests are pulled > from. > > > > * Tagging Interop-related tests with decorators to make it clear that > they need > > to be handled carefully. > > > > At the heart of the issue is the perception that projects that keep their > > integration tests within the tempest tree are somehow blessed, maybe by > the QA > > team or by the TC. It would be nice to try to clarify what technical > > and political > > reasons we have for why different projects have tests in different > places - > > review bandwidth of the QA team, ownership/control by the project teams, > > technical interdependency between certain projects, or otherwise. > > > As someone who has been in middle of all that already once I'd like to > bring up bit more fundamental problem into this topic. I'm not able to > provide one size fits all solution but hopefully some insight that > would help the community to make the right decision. > > I think the biggest problem is who's fox is let to guard the chicken coop. > > By that I mean the basic problem of our testing still relies on what > is tested based on which assumptions and by whom. If the tests are > provided by the project teams, the test is more likely to cover the > intended usecase of the feature as it's implemented and if there is > bug found on that, the likelyhood that the test is altered is quite > high also the individual projects might not have the best idea what > might be the important things to the interoperability and trademark > purposes. Obviously when the test is written against intended behavior > it's less likely but also those changes might sneak in affecting the > interoprability. On the other hand if the test is written by > QA/interoperability people, is it actually testing the right thing and > is there more fundamental need to break it due to the fact that > instead of catching and reporting the bug when the test is written, we > start enforcing it. Are the tests written based on the intended > behavior, documented behavior or the current actual behavior? And the > biggest question of them all is who is going to have the bandwidth to > understand the depth of the projects and their ties between to ensure > we minimize the above? > > In perfect world all features are bug free, rational to use and well > documented so that anyone can easily write a test that can be ran > against any version to verify that we do not have regressions. We just > are not living in that perfect world and each of the options are risky > to cause conflicts. > > I think the optimal solution if we were introducing this as new fresh > concept would be using tempest as engine to run trademark test plugins > from their own repo and those plugins would be provided in > collaboration between the trademark group as what are the > functionalities tested, QA to ensure that the tests actually verify > what they should be testing and the project teams ensuring that the > tested feature is a) behaving and b) tested as it's intended to work > and documentation is aligned with that, where the faults on any 3 > could be rectified before enforcing. Unfortunately I do not see us as > the community having the resources to this "the right way" and I have > really hard time trying to decide which of the proposed options would > be least bad. > Why not? So far interoperability testing has been a shared effort between project, QA and interop team. For a project to be part of an interoperability program there must be resources allocated to writing / maintaining interop tests. If they are not avaiable the interop program won't be very successful. andreaf > > I think the worst case scenario is that we scrape together what ever > we can just to have something to say that we test it and not have > consistency nor clear responsibility of who, what and how. > (Unfortunately I think this is the current situation and I'm super > happy to hear that this is being discussed and the decision is not > made lightly.) > > Best, > Erno -jokke- Kuvaja > > > Ultimately, as Jeremy said in the comments on the resolution patch, the > > recommendation should be one that works best for the QA and Interop > teams. So > > far we've heard from Matt and Mark expressing moderate support for > option 2. > > We'd like to hear more from those teams about how they see this working, > > especially with regard to concerns about the quality and stability > standards > > that out-of-tree tests may be held to. We additionally need input from > the > > whole community on how maintaining trademark-related tests in tempest > will > > affect you if you don't already have your tests there. We'd especially > like to > > address any perceptions of favoritism or exclusionism that stem from > these > > issues. > > > > And to quickly clear up one detail before it makes it onto this thread, > the > > Queens Community Goal about splitting tempest plugins out of the main > project's > > tree[3] is entirely about addressing technical problems related to > packaging for > > existing tempest plugins, it's not a decree about what should live > > within the tempest > > repository nor does it have anything to do with the Interop program. > > > > As I'm not deeply steeped in the history of either the Interop or QA > teams I am > > sure I've misrepresented some details here, I'm sorry about that. But > we'd like > > to get this resolution moving forward and we're currently stuck, so this > thread > > is intended to gather enough community input to get unstuck and avoid > letting > > this proposal become stale. Please respond to this thread or comment on > the > > resolution proposal[1] if you have any thoughts. > > > > Colleen > > > > [1] https://review.openstack.org/#/c/521602 > > [2] > https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html > > [3] > https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev