On Fri, Jun 2, 2017, at 09:51 AM, Matthew Treinish wrote: > On Thu, Jun 01, 2017 at 11:57:00AM -0400, Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200: > > > Graham Hayes wrote: > > > > On 01/06/17 01:30, Matthew Treinish wrote: > > > >> TBH, it's a bit premature to have the discussion. These additional > > > >> programs do > > > >> not exist yet, and there is a governance road block around this. Right > > > >> now the > > > >> set of projects that can be used defcore/interopWG is limited to the > > > >> set of > > > >> projects in: > > > >> > > > >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html > > > > > > > > Sure - but that is a solved problem, when the interop committee is > > > > ready to propose them, they can add projects into that tag. Or am I > > > > misunderstanding [1] (again)? > > > > > > I think you understand it well. The Board/InteropWG should propose > > > additions/removals of this tag, which will then be approved by the TC: > > > > > > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process > > > > > > > [...] > > > >> We had a forum session on it (I can't find the etherpad for the > > > >> session) which > > > >> was pretty speculative because it was about planning the new programs. > > > >> Part of > > > >> that discussion was around the feasibility of using tests in plugins > > > >> and whether > > > >> that would be desirable. Personally, I was in favor of doing that for > > > >> some of > > > >> the proposed programs because of the way they were organized it was a > > > >> good fit. > > > >> This is because the proposed new programs were extra additions on top > > > >> of the > > > >> base existing interop program. But it was hardly a definitive > > > >> discussion. > > > > > > > > Which will create 2 classes of testing for interop programs. > > > > > > FWIW I would rather have a single way of doing "tests used in trademark > > > programs" without differentiating between old and new trademark programs. > > > > > > I fear that we are discussing solutions before defining the problem. We > > > want: > > > > > > 1- Decentralize test maintenance, through more tempest plugins, to > > > account for limited QA resources > > > 2- Additional codereview constraints and approval rules for tests that > > > happen to be used in trademark programs > > > 3- Discoverability/ease-of-install of the set of tests that happen to be > > > used in trademark programs > > > 4- A git repo layout that can be simply explained, for new teams to > > > understand > > > > > > It feels like the current git repo layout (result of that 2016-05-04 > > > resolution) optimizes for 2 and 3, which kind of works until you add > > > more trademark programs, at which point it breaks 1 and 4. > > > > > > I feel like you could get 2 and 3 without necessarily using git repo > > > boundaries (using Gerrit approval rules and some tooling to install/run > > > subset of tests across multiple git repos), which would allow you to > > > optimize git repo layout to get 1 and 4... > > > > > > Or am I missing something ? > > > > > > > Right. The point of having the trademark tests "in tempest" was not > > to have them "in the tempest repo", that was just an implementation > > detail of the policy of "put them in a repository managed by people > > who understand the expanded review rules". > > There was more to it than this, a big part was duplication of effort as > well. > Tempest itself is almost a perfect fit for the scope of the testing > defcore is > doing. While tempest does additional testing that defcore doesn't use, a > large > subset is exactly what they want. > > > > > There were a lot of unexpected issues when we started treating the > > test suite as a production tool for validating a cloud. We have > > to be careful about how we change the behavior of tests, for example, > > even if the API responses are expected to be the same. It's not > > fair to vendors or operators who get trademark approval with one > > release to have significant changes in behavior in the exact same > > tests for the next release. > > I actually find this to be kinda misleading. Tempest has always had > running on any cloud as part of it's mission. I think you're referring > to the monster defcore thread from last summer about proprietary nova > extensions > adding on to API responses. This is honestly a completely separate > problem > which is not something I want to dive into again, because that was a much > more > nuanced problem that involved much more than just code review. > > > > > At the early stage, when the DefCore team was still figuring out > > these issues, it made sense to put all of the tests in one place > > with a review team that was actively participating in establishing > > the process. If we better understand the "rules" for these tests > > now, we can document them and distribute the work of maintaining the > > test suites. > > I think you're overestimating how much work is actually being done > bidirectionally here. The interaction with defcore is more straight > consumption > then you might think. They tend to just pick and choose from what tempest > has > and don't actually identify gaps or contribute back into tempest much. > > This actually is the crux of my entire concern, that assuming we do > widely > expand the number of trademark programs there is an assumption that a > bunch of > people are going to show up, write tests, and maintain them. However, all > past > evidence shows that this just doesn't ever happen. I linked to that graph > from > one of my talks to try and help illustrate this. That was back in the > days when > tempest supported all official openstack projects and was a > **requirement** for > projects to graduate. Even then the contribution in this space was > minimal or > non-existent for newer projects. > > But that being said it's just a concern, we have yet to actually reach > this > point because of defcore requirements, and we might never actually get > there. > > > > > And yes, I agree with the argument that we should be fair and treat > > all projects the same way. If we're going to move tests out of the > > tempest repository, we should move all of them. The QA team can > > still help maintain the test suites for whatever projects they want, > > even if those tests are in plugins. > > Again, where has this been proposed? I've yet to see anything where > tests being removed from tempest has being proposed anywhere. Also, moving > tests from tempest to some other place doesn't magically solve any of the > issues. > The fundamental problem I'm concerned with is a large expansion in the number > of trademark programs overloading a small team. Just saying the QA team can > still help maintain them doesn't change the scaling problem. It just shifts > that from the tempest repo to another repo. (or multiples)
Yeah, we should define and clear the problems before discussing the solution. I think Thierry already mentioned about that like below. ----------- > > > I fear that we are discussing solutions before defining the problem. We > > > want: > > > > > > 1- Decentralize test maintenance, through more tempest plugins, to > > > account for limited QA resources > > > 2- Additional codereview constraints and approval rules for tests that > > > happen to be used in trademark programs > > > 3- Discoverability/ease-of-install of the set of tests that happen to be > > > used in trademark programs > > > 4- A git repo layout that can be simply explained, for new teams to > > > understand ----------- > > But as I've said so many times, where is there a formal proposal for the > program expansion? Where has a proposed test for defcore been rejected from > tempest? This all is still very in loose terms and seems to be a completely > hypothetical discussion. I feel like we're searching for a problem to solve > before there is actually one. I agree here, too. Of course, we can have casual conversation. But I feel this topic it too big for a *casual*. And if we should change something, I think it's better to have concrete patches, specs and/or *formal* proposal like Matthew said. Otherwise, discussion may not be productive. -- Masayuki Igawa > > -Matt Treinish > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
