Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Fri, Jan 19, 2018 at 4:23 PM, Graham Hayes wrote: > > > On 19/01/18 03:28, Ghanshyam Mann wrote: >> On Thu, Jan 11, 2018 at 10:06 PM, Colleen Murphy wrote: >>> Hi everyone, >>> >>> We have governance review under debate[1] that we need the community's help >>> on. >>> The debate is over what recommendation the TC should make to the Interop >>> team >>> on where the tests it uses for the OpenStack trademark program should be >>> located, specifically those for the new add-on program being introduced. >>> Let me >>> badly summarize: >>> >>> A couple of years ago we issued a resolution[2] officially recommending that >>> the Interop team use solely tempest as its source of tests for capability >>> verification. The Interop team has always had the view that the developers, >>> being the people closest to the project they're creating, are the best >>> people >>> to write tests verifying correct functionality, and so the Interop team >>> doesn't >>> maintain its own test suite, instead selecting tests from those written in >>> coordination between the QA team and the other project teams. These tests >>> are >>> used to validate clouds applying for the OpenStack Powered tag, and since >>> all >>> of the projects included in the OpenStack Powered program already had tests >>> in >>> tempest, this was a natural fit. When we consider adding new trademark >>> programs >>> comprising of other projects, the test source is less obvious. Two examples >>> are >>> designate, which has never had tests in the tempest repo, and heat, which >>> recently had its tests removed from the tempest repo. >>> >>> So far the patch proposes three options: >>> >>> 1) All trademark-related tests should go in the tempest repo, in accordance >>>with the original resolution. This would mean that even projects that >>> have >>>never had tests in tempest would now have to add at least some of their >>>black-box tests to tempest. >>> >>> The value of this option is that centralizes tests used for the Interop >>> program >>> in a location where interop-minded folks from the QA team can control them. >>> The >>> downside is that projects that so far have avoided having a dependency on >>> tempest will now lose some control over the black-box tests that they use >>> for >>> functional and integration that would now also be used for trademark >>> certification. >>> There's also concern for the review bandwidth of the QA team - we can't >>> expect >>> the QA team to be continually responsible for an ever-growing list of >>> projects >>> and their trademark tests. >>> >>> 2) All trademark-related tests for *add-on projects* should be sourced from >>>plugins external to tempest. >>> >>> The value of this option is it allows project teams to retain control over >>> these tests. The potential problem with it is that individual project teams >>> are >>> not necessarily reviewing test changes with an eye for interop concerns and >>> so >>> could inadvertently change the behavior of the trademark-verification tools. >>> >>> 3) All trademark-related tests should go in a single separate tempest >>> plugin. >>> >>> This has the value of giving the QA and Interop teams control over >>> interop-related tests while also making clear the distinction between tests >>> used for trademark verification and tests used for CI. Matt's argument >>> against >>> this is that there actually is very little distinction between those two >>> cases, >>> and that a given test could have many different applications. >> >> options#3 can solve centralize test location issue but there is >> another issue it leads. If we start moving all interop test to >> separate interop repo then, many of exiting tempest test (used by >> interop) also falls under this category. Which means those existing >> tempest tests need to stay in 2 location one in new interop plugin and >> second in tempest also as tempest is being used for lot other purpose >> also, gate, production Cloud testing & stability etc. Duplication >> tests in 2 location is not good option. > > We could just install the interop plugin into all the gates, and ensure > it is ran, which would mean the tests are only ever in one place. That cover gate things at some extend and with workaround but not outside gate where test are being used to test the cloud. > > >>> >>> Other ideas that have been thrown around are: >>> >>> * Maintaining a branch in the tempest repo that Interop tests are pulled >>> from. >>> >>> * Tagging Interop-related tests with decorators to make it clear that they >>> need >>> to be handled carefully. >> >> Nice and imp point. This is been take care very carefully in Tempest >> till now . While changing tests or removing test, we have a very clear >> and strict process [4] to not affect any interop tests and i think it >> is 100% success till now, i have not heard any complained that we have >> changed any test which has broken interop. Adding new decorator etc >>
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 2018-01-22 15:04:31 + (+), Graham Hayes wrote: > On 19/01/18 16:27, Andrea Frittoli wrote: [...] > > it is possible to mark tests so that team may require a +1 from > > interop/qa when specific tests are modified. > > Is there? AFAIK gerrit does not do per file path permissions, so > unless we have a job that just checks the votes on a patch, and > passes or fails if a test changes (which would be awful for the > teams) we cannot do that. [...] I read that as implying that it's possible _culturally_ to add code comments or similar to remind reviewers that they need to seek additional feedback under certain conditions. Just because a rule can't be blindly enforced with some technological solution like CI jobs or review system ACLs doesn't mean it's impossible. It does however almost certainly mean more work for (and an inevitable increase in mistakes made by) already overburdened reviewers. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Mon, Jan 22, 2018 at 3:05 PM Graham Hayes wrote: > > > On 19/01/18 16:27, Andrea Frittoli wrote: > > > > Thanks for the summary! > > > > To be honest I don't see why this decision has to be difficult to take. > > I think the confusion comes from one thing being decided already, and > now conflicting direction is being given to teams, without anyone > updating the Governance repo. > > (It is not as if there was not plenty of warning, I have been raising it > as an issue for over a year) > > > Nothing we decide today is written in stone and the main risk ahead of > > us is to take > > a decision that requires a lot of upfront work and that it ends up > > providing no > > significant benefit, or even making things worst in some aspect. So we > > may try one way > > today and if we hit some significant issue we can still change. > > > > TL;DR my preferred option would be number (2) - it's the least initial > > effort, so the > > least risk, and deciding for (2) now won't make it any difficult in the > > future to switch > > to option (1) or option (3). I'm not pushing back on (2), I just think > > (1) is more convenient. > > Details below each option. > > > > > > > > So far the patch proposes three options: > > > > 1) All trademark-related tests should go in the tempest repo, in > > accordance > >with the original resolution. This would mean that even projects > > that have > >never had tests in tempest would now have to add at least some of > > their > >black-box tests to tempest. > > > > > > This option is a valid one, but I think it introduces too much extra > > work and > > testing complications for too little benefit. > > What it does do is *guarantee* that the InterOp suite will work, as it > will be CI'd. I see these programs as important enough that we should CI > the tooling used for them, but I seem to be in a minority. > Add-ons intreoperability tests will be CI'd for every change in Tempest as long as they are executed in a job that runs on every change in Tempest. This can be achieve regardless of the location of the tests and having the tests in the Tempest tree is not by itself a guarantee that they will be executed against every change. > > > > > > > The value of this option is that centralizes tests used for the > > Interop program > > in a location where interop-minded folks from the QA team can > > control them. > > > > > > There are other ways this can be achieved - it is possible to mark tests > > so that > > team may require a +1 from interop/qa when specific tests are modified. > > Is there? AFAIK gerrit does not do per file path permissions, so unless > we have a job that just checks the votes on a patch, and passes or fails > if a test changes (which would be awful for the teams) we cannot do > that. > If we really want to enforce having a vote from someone it may be tricky, yes, but I don't think enforcement is what we need, rather awareness, For governance and project-config patches reviewed always ask for a +1 from the project PTL where relevant, and add-on project reviewers could do the same. To help building awareness we could have automation in place to post a comment to Gerrit, like we do for elastic recheck. We could do that on every change to the plugin in the beginning and include a link to the interoperability recommendation to help reviewers in their job. > > > > > > > The > > downside is that projects that so far have avoided having a > > dependency on > > tempest will now lose some control over the black-box tests that > > they use for > > functional and integration that would now also be used for trademark > > certification. > > There's also concern for the review bandwidth of the QA team - we > > can't expect > > the QA team to be continually responsible for an ever-growing list > > of projects > > and their trademark tests. > > > > > > If we restrict to interop tests, the review bandwidth issue is probably > > not so bad. > > The QA team would have to request the domain knowledge required for > proper > > review from the respective teams anyways. > > > > There are other complications introduced though: > > > > - service clients and other common bits (config and so) would have to > > move to > > Tempest since we cannot have tempest depend on plugins. But then > modifying > > those common bits on Tempest side would risk to break non-interop > tests. > > Solution for that is to make all those bits stable interfaces for > plugins > > Is this not already the case? e.g. the neutron plugin uses the nova > service client already. > What I was saying is that Tempest cannot depend on a Tempest plugin, so service clients should be in Tempest. Which is absolutely fine, I'm was just listing out the work that needs to be done if we move add-on interoperability tests into Tempest. > > This would also help for the neutron plugin which is currently importi
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 19/01/18 16:27, Andrea Frittoli wrote: > > Thanks for the summary! > > To be honest I don't see why this decision has to be difficult to take. I think the confusion comes from one thing being decided already, and now conflicting direction is being given to teams, without anyone updating the Governance repo. (It is not as if there was not plenty of warning, I have been raising it as an issue for over a year) > Nothing we decide today is written in stone and the main risk ahead of > us is to take > a decision that requires a lot of upfront work and that it ends up > providing no > significant benefit, or even making things worst in some aspect. So we > may try one way > today and if we hit some significant issue we can still change. > > TL;DR my preferred option would be number (2) - it's the least initial > effort, so the > least risk, and deciding for (2) now won't make it any difficult in the > future to switch > to option (1) or option (3). I'm not pushing back on (2), I just think > (1) is more convenient. > Details below each option. > > > > So far the patch proposes three options: > > 1) All trademark-related tests should go in the tempest repo, in > accordance > with the original resolution. This would mean that even projects > that have > never had tests in tempest would now have to add at least some of > their > black-box tests to tempest. > > > This option is a valid one, but I think it introduces too much extra > work and > testing complications for too little benefit. What it does do is *guarantee* that the InterOp suite will work, as it will be CI'd. I see these programs as important enough that we should CI the tooling used for them, but I seem to be in a minority. > > > The value of this option is that centralizes tests used for the > Interop program > in a location where interop-minded folks from the QA team can > control them. > > > There are other ways this can be achieved - it is possible to mark tests > so that > team may require a +1 from interop/qa when specific tests are modified. Is there? AFAIK gerrit does not do per file path permissions, so unless we have a job that just checks the votes on a patch, and passes or fails if a test changes (which would be awful for the teams) we cannot do that. > > > The > downside is that projects that so far have avoided having a > dependency on > tempest will now lose some control over the black-box tests that > they use for > functional and integration that would now also be used for trademark > certification. > There's also concern for the review bandwidth of the QA team - we > can't expect > the QA team to be continually responsible for an ever-growing list > of projects > and their trademark tests. > > > If we restrict to interop tests, the review bandwidth issue is probably > not so bad. > The QA team would have to request the domain knowledge required for proper > review from the respective teams anyways. > > There are other complications introduced though: > > - service clients and other common bits (config and so) would have to > move to > Tempest since we cannot have tempest depend on plugins. But then modifying > those common bits on Tempest side would risk to break non-interop tests. > Solution for that is to make all those bits stable interfaces for plugins Is this not already the case? e.g. the neutron plugin uses the nova service client already. This would also help for the neutron plugin which is currently importing DNS service clients from the designate-tempest-plugin repo - having them in the tempest repo would allow them to to be more stable, and remove the extra dependency. > > - tempest would have to add new CI jobs to run the interop tests from add-on > program on every tempest change so that the new code is tested on a > regular > basis. That is a good thing, and we should probably do that for the other 2 options as well... > > - heat tests are wrapped in a Tempest plugin but actually written in > Gabbi so we > would need to add Gabbi as a dependency to Tempest > > Nothing too terrible really, but I think it might not be worth the extra > effort, especially > now that teams available resources are getting thinner and thinner. > > > 2) All trademark-related tests for *add-on projects* should be > sourced from > plugins external to tempest. > > I wouldn't go as far as saying they "should" be sourced. I think saying > that they > *may* be sourced from a plugin is enough. Apart from that this is my > favourite > option. The only thing required really is updating the resolution and we are > ready to go. > > With all the plugins no in own branchless repositories, the usability > concern is > not so strong anymore really. > > > The value of this option is it allows project teams to retain > control over > these tests. > > > Other value is
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thu, Jan 11, 2018 at 4:36 PM Colleen Murphy wrote: > Hi everyone, > > We have governance review under debate[1] that we need the community's > help on. > The debate is over what recommendation the TC should make to the Interop > team > on where the tests it uses for the OpenStack trademark program should be > located, specifically those for the new add-on program being introduced. > Let me > badly summarize: > > A couple of years ago we issued a resolution[2] officially recommending > that > the Interop team use solely tempest as its source of tests for capability > verification. The Interop team has always had the view that the developers, > being the people closest to the project they're creating, are the best > people > to write tests verifying correct functionality, and so the Interop team > doesn't > maintain its own test suite, instead selecting tests from those written in > coordination between the QA team and the other project teams. These tests > are > used to validate clouds applying for the OpenStack Powered tag, and since > all > of the projects included in the OpenStack Powered program already had > tests in > tempest, this was a natural fit. When we consider adding new trademark > programs > comprising of other projects, the test source is less obvious. Two > examples are > designate, which has never had tests in the tempest repo, and heat, which > recently had its tests removed from the tempest repo. > Thanks for the summary! To be honest I don't see why this decision has to be difficult to take. Nothing we decide today is written in stone and the main risk ahead of us is to take a decision that requires a lot of upfront work and that it ends up providing no significant benefit, or even making things worst in some aspect. So we may try one way today and if we hit some significant issue we can still change. TL;DR my preferred option would be number (2) - it's the least initial effort, so the least risk, and deciding for (2) now won't make it any difficult in the future to switch to option (1) or option (3). I'm not pushing back on (2), I just think (1) is more convenient. Details below each option. > > So far the patch proposes three options: > > 1) All trademark-related tests should go in the tempest repo, in accordance >with the original resolution. This would mean that even projects that > have >never had tests in tempest would now have to add at least some of their >black-box tests to tempest. > > This option is a valid one, but I think it introduces too much extra work and testing complications for too little benefit. > The value of this option is that centralizes tests used for the Interop > program > in a location where interop-minded folks from the QA team can control > them. There are other ways this can be achieved - it is possible to mark tests so that team may require a +1 from interop/qa when specific tests are modified. > The > downside is that projects that so far have avoided having a dependency on > tempest will now lose some control over the black-box tests that they use > for > functional and integration that would now also be used for trademark > certification. > There's also concern for the review bandwidth of the QA team - we can't > expect > the QA team to be continually responsible for an ever-growing list of > projects > and their trademark tests. > If we restrict to interop tests, the review bandwidth issue is probably not so bad. The QA team would have to request the domain knowledge required for proper review from the respective teams anyways. There are other complications introduced though: - service clients and other common bits (config and so) would have to move to Tempest since we cannot have tempest depend on plugins. But then modifying those common bits on Tempest side would risk to break non-interop tests. Solution for that is to make all those bits stable interfaces for plugins - tempest would have to add new CI jobs to run the interop tests from add-on program on every tempest change so that the new code is tested on a regular basis. - heat tests are wrapped in a Tempest plugin but actually written in Gabbi so we would need to add Gabbi as a dependency to Tempest Nothing too terrible really, but I think it might not be worth the extra effort, especially now that teams available resources are getting thinner and thinner. > 2) All trademark-related tests for *add-on projects* should be sourced from >plugins external to tempest. > > I wouldn't go as far as saying they "should" be sourced. I think saying that they *may* be sourced from a plugin is enough. Apart from that this is my favourite option. The only thing required really is updating the resolution and we are ready to go. With all the plugins no in own branchless repositories, the usability concern is not so strong anymore really. > The value of this option is it allows project teams to retain control over > these tests. Other value is given by
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thu, Jan 18, 2018 at 3:33 PM Graham Hayes wrote: > > > On 11/01/18 16:36, Colleen Murphy wrote: > > Hi everyone, > > > > We have governance review under debate[1] that we need the community's > help on. > > The debate is over what recommendation the TC should make to the Interop > team > > on where the tests it uses for the OpenStack trademark program should be > > located, specifically those for the new add-on program being introduced. > Let me > > badly summarize: > > > > A couple of years ago we issued a resolution[2] officially recommending > that > > the Interop team use solely tempest as its source of tests for capability > > verification. The Interop team has always had the view that the > developers, > > being the people closest to the project they're creating, are the best > people > > to write tests verifying correct functionality, and so the Interop team > doesn't > > maintain its own test suite, instead selecting tests from those written > in > > coordination between the QA team and the other project teams. These > tests are > > used to validate clouds applying for the OpenStack Powered tag, and > since all > > of the projects included in the OpenStack Powered program already had > tests in > > tempest, this was a natural fit. When we consider adding new trademark > programs > > comprising of other projects, the test source is less obvious. Two > examples are > > designate, which has never had tests in the tempest repo, and heat, which > > recently had its tests removed from the tempest repo. > > > > > > > > > As I'm not deeply steeped in the history of either the Interop or QA > teams I am > > sure I've misrepresented some details here, I'm sorry about that. But > we'd like > > to get this resolution moving forward and we're currently stuck, so this > thread > > is intended to gather enough community input to get unstuck and avoid > letting > > this proposal become stale. Please respond to this thread or comment on > the > > resolution proposal[1] if you have any thoughts. > > > > Colleen > > > > [1] https://review.openstack.org/#/c/521602 > > [2] > https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html > > [3] > https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html > > > > I had hoped for more of a discussion on this before I jumped back into > this debate - but it seams to be stalled still, so here it goes. > > I proposed this initially as we were unclear on where the tests should > go - we had a resolution that said all tests go into openstack/tempest > (with a list of reasons why), and the guidance and discussion that been > had in various summits was that "add-ons" should stay in plugins. > > So right now, we (by the governance rules) should be pushing tests to > tempest for the new programs. > > In the resolution that placed the tests in tempest there was a few > reasons proposed: > > For example, API and behavioral changes must be carefully managed, as > must mundane aspects such as test and module naming and location > within the test suite. Even changes that leave tests functionally > equivalent may cause unexpected consequences for their use in DefCore > processes and validation. Placing the tests in a central repository > will make it easier to maintain consistency and avoid breaking the > trademark enforcement tool. > > This still applies, and even more so for teams that traditionally do not > have a strong QE contributor / reviewer base (aka projects not in > "core"). > > Centralizing the tests also makes it easier for anyone running the > validation tool against their cloud or cloud distribution to use the > tests. It is easier to install the test suite and its dependencies, > and it is easier to read and understand a set of tests following a > consistent implementation pattern. > > Apparently users do not need central tests anymore, feedback from > RefStack is that people who run these tests are comfortable dealing > with extra python packages. > > The point about a single set of tests, in a single location and style > still stands. > > Finally, having the tests in a central location makes it easier to > ensure that all members of the community have equal input into what > the tests do and how they are implemented and maintained. > > Seems like a good value to strive for. > > One of the items that has been used to push back against adding > "add-ons" to tempest has been that tempest has a defined scope, and > neither of the current add-ons fit in that scope. > > Can someone clarify what the set of criteria is? I think it will help > this discussion. > > Another push back is the "scaling" issue - adding more tests will > overload the QA team. > > Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite > of 353. > > I do not think there is many other add-ons proposed yet, and the new > Vertical programs will probably mainly be re-using tests in the > openstack/tempest repos as i
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Mon, Jan 15, 2018 at 12:59 PM Erno Kuvaja wrote: > On Thu, Jan 11, 2018 at 4:36 PM, Colleen Murphy > wrote: > > Hi everyone, > > > > We have governance review under debate[1] that we need the community's > help on. > > The debate is over what recommendation the TC should make to the Interop > team > > on where the tests it uses for the OpenStack trademark program should be > > located, specifically those for the new add-on program being introduced. > Let me > > badly summarize: > > > > A couple of years ago we issued a resolution[2] officially recommending > that > > the Interop team use solely tempest as its source of tests for capability > > verification. The Interop team has always had the view that the > developers, > > being the people closest to the project they're creating, are the best > people > > to write tests verifying correct functionality, and so the Interop team > doesn't > > maintain its own test suite, instead selecting tests from those written > in > > coordination between the QA team and the other project teams. These > tests are > > used to validate clouds applying for the OpenStack Powered tag, and > since all > > of the projects included in the OpenStack Powered program already had > tests in > > tempest, this was a natural fit. When we consider adding new trademark > programs > > comprising of other projects, the test source is less obvious. Two > examples are > > designate, which has never had tests in the tempest repo, and heat, which > > recently had its tests removed from the tempest repo. > > > > So far the patch proposes three options: > > > > 1) All trademark-related tests should go in the tempest repo, in > accordance > >with the original resolution. This would mean that even projects that > have > >never had tests in tempest would now have to add at least some of > their > >black-box tests to tempest. > > > > The value of this option is that centralizes tests used for the Interop > program > > in a location where interop-minded folks from the QA team can control > them. The > > downside is that projects that so far have avoided having a dependency on > > tempest will now lose some control over the black-box tests that they > use for > > functional and integration that would now also be used for trademark > > certification. > > There's also concern for the review bandwidth of the QA team - we can't > expect > > the QA team to be continually responsible for an ever-growing list of > projects > > and their trademark tests. > > > > 2) All trademark-related tests for *add-on projects* should be sourced > from > >plugins external to tempest. > > > > The value of this option is it allows project teams to retain control > over > > these tests. The potential problem with it is that individual project > teams are > > not necessarily reviewing test changes with an eye for interop concerns > and so > > could inadvertently change the behavior of the trademark-verification > tools. > > > > 3) All trademark-related tests should go in a single separate tempest > plugin. > > > > This has the value of giving the QA and Interop teams control over > > interop-related tests while also making clear the distinction between > tests > > used for trademark verification and tests used for CI. Matt's argument > against > > this is that there actually is very little distinction between those two > cases, > > and that a given test could have many different applications. > > > > Other ideas that have been thrown around are: > > > > * Maintaining a branch in the tempest repo that Interop tests are pulled > from. > > > > * Tagging Interop-related tests with decorators to make it clear that > they need > > to be handled carefully. > > > > At the heart of the issue is the perception that projects that keep their > > integration tests within the tempest tree are somehow blessed, maybe by > the QA > > team or by the TC. It would be nice to try to clarify what technical > > and political > > reasons we have for why different projects have tests in different > places - > > review bandwidth of the QA team, ownership/control by the project teams, > > technical interdependency between certain projects, or otherwise. > > > As someone who has been in middle of all that already once I'd like to > bring up bit more fundamental problem into this topic. I'm not able to > provide one size fits all solution but hopefully some insight that > would help the community to make the right decision. > > I think the biggest problem is who's fox is let to guard the chicken coop. > > By that I mean the basic problem of our testing still relies on what > is tested based on which assumptions and by whom. If the tests are > provided by the project teams, the test is more likely to cover the > intended usecase of the feature as it's implemented and if there is > bug found on that, the likelyhood that the test is altered is quite > high also the individual projects might not have the best idea what > might be the i
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Fri, Jan 12, 2018 at 9:50 AM Luigi Toscano wrote: > On Thursday, 11 January 2018 23:52:00 CET Matt Riedemann wrote: > > On 1/11/2018 10:36 AM, Colleen Murphy wrote: > > > 1) All trademark-related tests should go in the tempest repo, in > > > accordance > > > > > > with the original resolution. This would mean that even projects > that > > > have > > > never had tests in tempest would now have to add at least some of > > > their > > > black-box tests to tempest. > > > > > > The value of this option is that centralizes tests used for the Interop > > > program in a location where interop-minded folks from the QA team can > > > control them. The downside is that projects that so far have avoided > > > having a dependency on tempest will now lose some control over the > > > black-box tests that they use for functional and integration that would > > > now also be used for trademark certification. > > > There's also concern for the review bandwidth of the QA team - we can't > > > expect the QA team to be continually responsible for an ever-growing > list > > > of projects and their trademark tests. > > > > How many tests are we talking about for designate and heat? Half a > > dozen? A dozen? More? > > > > If it's just a couple of tests per project it doesn't seem terrible to > > have them live in Tempest so you get the "interop eye" on reviews, as > > noted in your email. If it's a considerable amount, then option 2 seems > > the best for the majority of parties. > > I would argue that it does not scale; what if some test is taken out from > the > interoperability, and others are added? It would mean moving tests from one > repository to another, with change of paths. I think that the solution 2, > where the repository where a test belong and the functionality of a test > are > not linked, is better. > This probably does not happen too often, but it does happen, and I agree that it would make things easier to have interop and non-iterop tests in the same repo. > > Ciao > -- > Luigi > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Colleen Murphy wrote: > On Fri, Jan 19, 2018 at 1:52 AM, Ken'ichi Ohmichi > wrote: >> 2018-01-18 12:36 GMT-08:00 Doug Hellmann : >>> >>> I feel pretty sure that was discussed in a TC meeting, but I can't >>> find that. I do find Matt and Ken'ichi voting +1 on the resolution >>> itself. https://review.openstack.org/#/c/312718/. If I remember >>> correctly, Ken'ichi was the PTL at the time. >> >> Yeah, I have still agreed with the resolution. >> When I voted +1 on that, core projects were defined as 6 projects like >> Nova, Cinder, Glance, Keystone, Neutron and Swift. >> And the project navigator also showed these 6 projects as core projects. >> Now I cannot find such definition on the project navigator[1], the >> definition has been changed? >> I just want to clarify "is it true that designate and heat become core >> projects?" >> If there is a concrete decision, I don't have any objections against >> that we have these projects tests in Tempest as the resolution. > > I think the fuzziness between what we're colloquially calling "core" > (or sometimes "integrated"), what has tests in tempest, and what is > part of the original trademark program, is part of the problem. Right. People are using "core" to mean different things. And the reason is that there is no definition for it. So it's a convenient catch-all term for "the main projects" that works in all situations, whatever you mean by that. The bylaws only define "Trademark Designated OpenStack Software", and the badly-named "Defcore" subcommittee was tasked with defining it, out of a subset of the "OpenStack Technical Committee Approved Release", which itself is a subset of the OpenStack official deliverables. The resolution above did not mention "core" at all. It only mentions the interoperability tests used by the Defcore workgroup to define "Trademark Designated OpenStack Software". The tension here is that the QA team agreed to the resolution with the feeling that "Trademark Designated OpenStack Software" would be a pretty narrow set. With the creation of add-on trademark programs, that set probably increased in size more than they expected. I agree the language in our resolution is clear (and mandates that the interop tests related to Designate or Heat are accepted in tempest). We just need to reaffirm or reconsider that resolution based on the recent evolution of "Trademark Designated OpenStack Software". -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 19/01/18 00:52, Ken'ichi Ohmichi wrote: > 2018-01-18 12:36 GMT-08:00 Doug Hellmann : >> Excerpts from Doug Hellmann's message of 2018-01-18 15:21:12 -0500: >>> Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +: On 18/01/18 18:52, Doug Hellmann wrote: > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: >> On 18/01/18 16:25, Doug Hellmann wrote: >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: >> >> >> >>> >>> In the past the QA team agreed to accept trademark-related tests from >>> all projects in the tempest repo. Has that changed? >>> >> >> There has not been an explict rejection but in all conversations the >> response has been "non core projects are outside the scope of tempest". >> >> Honestly, everytime we have tried to do something to core tempest >> we have had major pushback, and I want to clarify this before I or >> someone else put in the work of porting the base clients, getting CI >> configured*, and proposing the tests to tempest. > > OK. > > The current policy doesn't say anything about "core" or different > trademark programs or any other criteria. > > The TC therefore encourages the DefCore committee to consider it an > indication of future technical direction that we do not want tests > outside of the Tempest repository used for trademark enforcement, and > that any new or existing tests that cover capabilities they want to > consider for trademark enforcement should be placed in Tempest. > > That all seems very clear to me (setting aside some specific word > choices like "future technical direction" that tie the resolution > to language in the bylaws). Regardless of technical reasons why > it may not be necessary, we still have many social justifications > for doing it the way we originally set out to do it. Tests related > to trademark enforcement need to go into the tempest repository. > > The way I think this should work (and the way I remember us describing > it at the time the policy was established) is the Interop WG > (previously DefCore) should identify capabilities and tests, then > ask project teams to reproduce those tests in the tempest repo. > When the tests land, they can be used by the trademark program. > Teams can also, at their leisure, decide whether to remove the > original versions of the tests from whatever repo they existed in > to begin with. > > Graham, you've proposed a new resolution with several options for > where to put tests for "add-on programs." I don't think we need > that resolution if we want the tests to continue to live in tempest. > The existing resolution doesn't qualify which tests, beyond "for > trademark enforcement" and more words won't make that more clear, > IMO. > > Now if you *do* want to change the policy, we should talk about > that. But I can't tell whether you want to change it, you're worried > the policy is unclear, or it is not being followed. Can you clarify > which it is? It is not being followed. I have brought this up at every forum session on these programs, and the people in the room from QA have *always* pushed back on it. >>> >>> OK, so that's a problem. I need to hear from the QA team why they've >>> reversed that decision. >>> And, for clarity (I saw this in a few logs) QA have *never* said that they will take the interop designated tests for the DNS project into openstack/tempest. >>> >>> When we approved the resolution that describes the current policy, the >>> QA team agreed that they would take tests for trademark. There was no >>> stipulation about which projects those apply to. >> >> I feel pretty sure that was discussed in a TC meeting, but I can't >> find that. I do find Matt and Ken'ichi voting +1 on the resolution >> itself. https://review.openstack.org/#/c/312718/. If I remember >> correctly, Ken'ichi was the PTL at the time. > > Yeah, I have still agreed with the resolution. > When I voted +1 on that, core projects were defined as 6 projects like > Nova, Cinder, Glance, Keystone, Neutron and Swift. > And the project navigator also showed these 6 projects as core projects. > Now I cannot find such definition on the project navigator[1], the > definition has been changed? > I just want to clarify "is it true that designate and heat become core > projects?" > If there is a concrete decision, I don't have any objections against > that we have these projects tests in Tempest as the resolution. This seems to be the problem - there is not now, or ever been a "core" project definition that was decided by TC / community. We have a set of projects that most people will refer to as "core", but there is no way to add projects to that. What was highlighted on the project
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 19/01/18 03:28, Ghanshyam Mann wrote: > On Thu, Jan 11, 2018 at 10:06 PM, Colleen Murphy wrote: >> Hi everyone, >> >> We have governance review under debate[1] that we need the community's help >> on. >> The debate is over what recommendation the TC should make to the Interop team >> on where the tests it uses for the OpenStack trademark program should be >> located, specifically those for the new add-on program being introduced. Let >> me >> badly summarize: >> >> A couple of years ago we issued a resolution[2] officially recommending that >> the Interop team use solely tempest as its source of tests for capability >> verification. The Interop team has always had the view that the developers, >> being the people closest to the project they're creating, are the best people >> to write tests verifying correct functionality, and so the Interop team >> doesn't >> maintain its own test suite, instead selecting tests from those written in >> coordination between the QA team and the other project teams. These tests are >> used to validate clouds applying for the OpenStack Powered tag, and since all >> of the projects included in the OpenStack Powered program already had tests >> in >> tempest, this was a natural fit. When we consider adding new trademark >> programs >> comprising of other projects, the test source is less obvious. Two examples >> are >> designate, which has never had tests in the tempest repo, and heat, which >> recently had its tests removed from the tempest repo. >> >> So far the patch proposes three options: >> >> 1) All trademark-related tests should go in the tempest repo, in accordance >>with the original resolution. This would mean that even projects that have >>never had tests in tempest would now have to add at least some of their >>black-box tests to tempest. >> >> The value of this option is that centralizes tests used for the Interop >> program >> in a location where interop-minded folks from the QA team can control them. >> The >> downside is that projects that so far have avoided having a dependency on >> tempest will now lose some control over the black-box tests that they use for >> functional and integration that would now also be used for trademark >> certification. >> There's also concern for the review bandwidth of the QA team - we can't >> expect >> the QA team to be continually responsible for an ever-growing list of >> projects >> and their trademark tests. >> >> 2) All trademark-related tests for *add-on projects* should be sourced from >>plugins external to tempest. >> >> The value of this option is it allows project teams to retain control over >> these tests. The potential problem with it is that individual project teams >> are >> not necessarily reviewing test changes with an eye for interop concerns and >> so >> could inadvertently change the behavior of the trademark-verification tools. >> >> 3) All trademark-related tests should go in a single separate tempest plugin. >> >> This has the value of giving the QA and Interop teams control over >> interop-related tests while also making clear the distinction between tests >> used for trademark verification and tests used for CI. Matt's argument >> against >> this is that there actually is very little distinction between those two >> cases, >> and that a given test could have many different applications. > > options#3 can solve centralize test location issue but there is > another issue it leads. If we start moving all interop test to > separate interop repo then, many of exiting tempest test (used by > interop) also falls under this category. Which means those existing > tempest tests need to stay in 2 location one in new interop plugin and > second in tempest also as tempest is being used for lot other purpose > also, gate, production Cloud testing & stability etc. Duplication > tests in 2 location is not good option. We could just install the interop plugin into all the gates, and ensure it is ran, which would mean the tests are only ever in one place. >> >> Other ideas that have been thrown around are: >> >> * Maintaining a branch in the tempest repo that Interop tests are pulled >> from. >> >> * Tagging Interop-related tests with decorators to make it clear that they >> need >> to be handled carefully. > > Nice and imp point. This is been take care very carefully in Tempest > till now . While changing tests or removing test, we have a very clear > and strict process [4] to not affect any interop tests and i think it > is 100% success till now, i have not heard any complained that we have > changed any test which has broken interop. Adding new decorator etc > has different issues to we did not accepted but main problem is solved > by defining process.. Out of interest, what is the issue with a new test tag? it seems like it would be a good way to highlight to people what tests need extra care. > >> >> At the heart of the issue is the perception that projects that keep
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 19/01/18 04:10, Ghanshyam Mann wrote: > On Thu, Jan 18, 2018 at 11:22 PM, Graham Hayes wrote: >> On 18/01/18 16:25, Doug Hellmann wrote: >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: >> >> >> >>> >>> In the past the QA team agreed to accept trademark-related tests from >>> all projects in the tempest repo. Has that changed? >>> >> >> There has not been an explict rejection but in all conversations the >> response has been "non core projects are outside the scope of tempest". >> >> Honestly, everytime we have tried to do something to core tempest >> we have had major pushback, and I want to clarify this before I or >> someone else put in the work of porting the base clients, getting CI >> configured*, and proposing the tests to tempest. > > Yes, i do not remember that we have rejected the actual proposal or > patches or ever said that "QA team not going to accept interop needed > test inside Tempest even those are for other project than tempest > scope". Rather its been discussed that we need to check the situation > when new project going to be make in interop program. At least i > remember the discussion during Barcelona summit where there talked > about heat tests. We discussed we can check that based on when heat is > going to make in interop program. The decision seems to have been made already - people from the QA team have been pushing for Option 2 since Boston. > Anyways let's analysts the current situation and work on best possible > solution than past things. I agree with Doug point about previous > resolution was passed and then why this new resolution ? and what is > not clear in previous resolution?. I think main issue is in > understanding the difference between 'trademark' program and 'adds-on > trademark' program. Let me add the things point by point. They are both trademark programs under the By-Laws. > 1. What is difference between "Trademark" program and "Adds-on > Trademark" program from interop certification? Can new projects go > under "Trademark" program. > This will be helpful to understand the situation of keeping all > "Trademark" program tests and "Adds-on" program tests together or > separate. For example: any difference of doing their certification, > logo etc. The only difference is that an Add On needs a cloud to pass an OpenStack Powered program first. (e.g. you cannot have OpenStack DNS / Orchestration on its own) > 2. As per previous resolution, and with all point of centralized test > location, expertise review, project independent ownership etc etc i > agree with option#1 and no "NO" to that now also. Now question comes > to practice implementation of that resolution which depends on 2 > factor: > > 1. scale and number of program going to be in interop: > As per current proposal, (i think its heat and designate and > around 20-30 tests as total) there is no issue for tempest team to > add/review/maintain them. But if that grows in number of program (than > number tests for e.x. having 50 tests of designate than 10 is not much > different things) and say 10 more program then it is difficult for QA > team to maintain those. > > 2. QA team review bandwidth. > This is one of the big obstacle to extend the tempest scope. Like > other project, QA team face less contributors issues. Since 1-2 years, > I have been trying to attract new contributor in QA during upstream > training, mentorship program etc but people gets disappear after month > or so. Even all QA members are trying their best in this area but > unfortunately no success. > > With both these factor i feel we can go with current resolution > (option#1- below solution) and help QA team also if situation gets > worst (QA team also human beings and need time to sleep :)). > > 1. QA team accept all interop defined program tests (tests only needed > by interop ). > 2. Define a very clear process for collaboration between Interop, QA, > project team to help on adding/maintaining tests. Something like clear > guidelines of test req from interop, MUST +1 from interop and project > PTL. > 3. If interop program grows more which become difficult to maintain by > QA team then accept the necessary change to resolution. This sounds like a great way forward. > -gmann > >> >> - Graham >> >> >> * With zuulv3 this is *much* easier, so not as big a deal as it once was >> >> >> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > signature.asc Descr
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Fri, Jan 19, 2018 at 1:52 AM, Ken'ichi Ohmichi wrote: > 2018-01-18 12:36 GMT-08:00 Doug Hellmann : >> >> I feel pretty sure that was discussed in a TC meeting, but I can't >> find that. I do find Matt and Ken'ichi voting +1 on the resolution >> itself. https://review.openstack.org/#/c/312718/. If I remember >> correctly, Ken'ichi was the PTL at the time. > > Yeah, I have still agreed with the resolution. > When I voted +1 on that, core projects were defined as 6 projects like > Nova, Cinder, Glance, Keystone, Neutron and Swift. > And the project navigator also showed these 6 projects as core projects. > Now I cannot find such definition on the project navigator[1], the > definition has been changed? > I just want to clarify "is it true that designate and heat become core > projects?" > If there is a concrete decision, I don't have any objections against > that we have these projects tests in Tempest as the resolution. I think the fuzziness between what we're colloquially calling "core" (or sometimes "integrated"), what has tests in tempest, and what is part of the original trademark program, is part of the problem. As I understand it, designate and heat are not trying to become "core". What they are applying for is to be part of a new subgroup within the trademark program. The question at hand is, given that they are not "core" (whatever that really means), but they are going to be part of the trademark program, is there a technical reason they shouldn't have some of their tests in tempest? And if not, is there a social reason for it? Colleen > > Thanks > Ken Ohmichi > > --- > [1]: https://www.openstack.org/software/project-navigator > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thu, Jan 18, 2018 at 11:22 PM, Graham Hayes wrote: > On 18/01/18 16:25, Doug Hellmann wrote: >> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > > > >> >> In the past the QA team agreed to accept trademark-related tests from >> all projects in the tempest repo. Has that changed? >> > > There has not been an explict rejection but in all conversations the > response has been "non core projects are outside the scope of tempest". > > Honestly, everytime we have tried to do something to core tempest > we have had major pushback, and I want to clarify this before I or > someone else put in the work of porting the base clients, getting CI > configured*, and proposing the tests to tempest. Yes, i do not remember that we have rejected the actual proposal or patches or ever said that "QA team not going to accept interop needed test inside Tempest even those are for other project than tempest scope". Rather its been discussed that we need to check the situation when new project going to be make in interop program. At least i remember the discussion during Barcelona summit where there talked about heat tests. We discussed we can check that based on when heat is going to make in interop program. Anyways let's analysts the current situation and work on best possible solution than past things. I agree with Doug point about previous resolution was passed and then why this new resolution ? and what is not clear in previous resolution?. I think main issue is in understanding the difference between 'trademark' program and 'adds-on trademark' program. Let me add the things point by point. 1. What is difference between "Trademark" program and "Adds-on Trademark" program from interop certification? Can new projects go under "Trademark" program. This will be helpful to understand the situation of keeping all "Trademark" program tests and "Adds-on" program tests together or separate. For example: any difference of doing their certification, logo etc. 2. As per previous resolution, and with all point of centralized test location, expertise review, project independent ownership etc etc i agree with option#1 and no "NO" to that now also. Now question comes to practice implementation of that resolution which depends on 2 factor: 1. scale and number of program going to be in interop: As per current proposal, (i think its heat and designate and around 20-30 tests as total) there is no issue for tempest team to add/review/maintain them. But if that grows in number of program (than number tests for e.x. having 50 tests of designate than 10 is not much different things) and say 10 more program then it is difficult for QA team to maintain those. 2. QA team review bandwidth. This is one of the big obstacle to extend the tempest scope. Like other project, QA team face less contributors issues. Since 1-2 years, I have been trying to attract new contributor in QA during upstream training, mentorship program etc but people gets disappear after month or so. Even all QA members are trying their best in this area but unfortunately no success. With both these factor i feel we can go with current resolution (option#1- below solution) and help QA team also if situation gets worst (QA team also human beings and need time to sleep :)). 1. QA team accept all interop defined program tests (tests only needed by interop ). 2. Define a very clear process for collaboration between Interop, QA, project team to help on adding/maintaining tests. Something like clear guidelines of test req from interop, MUST +1 from interop and project PTL. 3. If interop program grows more which become difficult to maintain by QA team then accept the necessary change to resolution. -gmann > > - Graham > > > * With zuulv3 this is *much* easier, so not as big a deal as it once was > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thu, Jan 11, 2018 at 10:06 PM, Colleen Murphy wrote: > Hi everyone, > > We have governance review under debate[1] that we need the community's help > on. > The debate is over what recommendation the TC should make to the Interop team > on where the tests it uses for the OpenStack trademark program should be > located, specifically those for the new add-on program being introduced. Let > me > badly summarize: > > A couple of years ago we issued a resolution[2] officially recommending that > the Interop team use solely tempest as its source of tests for capability > verification. The Interop team has always had the view that the developers, > being the people closest to the project they're creating, are the best people > to write tests verifying correct functionality, and so the Interop team > doesn't > maintain its own test suite, instead selecting tests from those written in > coordination between the QA team and the other project teams. These tests are > used to validate clouds applying for the OpenStack Powered tag, and since all > of the projects included in the OpenStack Powered program already had tests in > tempest, this was a natural fit. When we consider adding new trademark > programs > comprising of other projects, the test source is less obvious. Two examples > are > designate, which has never had tests in the tempest repo, and heat, which > recently had its tests removed from the tempest repo. > > So far the patch proposes three options: > > 1) All trademark-related tests should go in the tempest repo, in accordance >with the original resolution. This would mean that even projects that have >never had tests in tempest would now have to add at least some of their >black-box tests to tempest. > > The value of this option is that centralizes tests used for the Interop > program > in a location where interop-minded folks from the QA team can control them. > The > downside is that projects that so far have avoided having a dependency on > tempest will now lose some control over the black-box tests that they use for > functional and integration that would now also be used for trademark > certification. > There's also concern for the review bandwidth of the QA team - we can't expect > the QA team to be continually responsible for an ever-growing list of projects > and their trademark tests. > > 2) All trademark-related tests for *add-on projects* should be sourced from >plugins external to tempest. > > The value of this option is it allows project teams to retain control over > these tests. The potential problem with it is that individual project teams > are > not necessarily reviewing test changes with an eye for interop concerns and so > could inadvertently change the behavior of the trademark-verification tools. > > 3) All trademark-related tests should go in a single separate tempest plugin. > > This has the value of giving the QA and Interop teams control over > interop-related tests while also making clear the distinction between tests > used for trademark verification and tests used for CI. Matt's argument against > this is that there actually is very little distinction between those two > cases, > and that a given test could have many different applications. options#3 can solve centralize test location issue but there is another issue it leads. If we start moving all interop test to separate interop repo then, many of exiting tempest test (used by interop) also falls under this category. Which means those existing tempest tests need to stay in 2 location one in new interop plugin and second in tempest also as tempest is being used for lot other purpose also, gate, production Cloud testing & stability etc. Duplication tests in 2 location is not good option. > > Other ideas that have been thrown around are: > > * Maintaining a branch in the tempest repo that Interop tests are pulled from. > > * Tagging Interop-related tests with decorators to make it clear that they > need > to be handled carefully. Nice and imp point. This is been take care very carefully in Tempest till now . While changing tests or removing test, we have a very clear and strict process [4] to not affect any interop tests and i think it is 100% success till now, i have not heard any complained that we have changed any test which has broken interop. Adding new decorator etc has different issues to we did not accepted but main problem is solved by defining process.. > > At the heart of the issue is the perception that projects that keep their > integration tests within the tempest tree are somehow blessed, maybe by the QA > team or by the TC. It would be nice to try to clarify what technical > and political > reasons we have for why different projects have tests in different places - > review bandwidth of the QA team, ownership/control by the project teams, > technical interdependency between certain projects, or otherwise. > > Ultimately, as Jeremy said in the comments on the resolution pat
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
2018-01-18 12:36 GMT-08:00 Doug Hellmann : > Excerpts from Doug Hellmann's message of 2018-01-18 15:21:12 -0500: >> Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +: >> > >> > On 18/01/18 18:52, Doug Hellmann wrote: >> > > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: >> > >> On 18/01/18 16:25, Doug Hellmann wrote: >> > >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: >> > >> >> > >> >> > >> >> > >>> >> > >>> In the past the QA team agreed to accept trademark-related tests from >> > >>> all projects in the tempest repo. Has that changed? >> > >>> >> > >> >> > >> There has not been an explict rejection but in all conversations the >> > >> response has been "non core projects are outside the scope of tempest". >> > >> >> > >> Honestly, everytime we have tried to do something to core tempest >> > >> we have had major pushback, and I want to clarify this before I or >> > >> someone else put in the work of porting the base clients, getting CI >> > >> configured*, and proposing the tests to tempest. >> > > >> > > OK. >> > > >> > > The current policy doesn't say anything about "core" or different >> > > trademark programs or any other criteria. >> > > >> > > The TC therefore encourages the DefCore committee to consider it an >> > > indication of future technical direction that we do not want tests >> > > outside of the Tempest repository used for trademark enforcement, and >> > > that any new or existing tests that cover capabilities they want to >> > > consider for trademark enforcement should be placed in Tempest. >> > > >> > > That all seems very clear to me (setting aside some specific word >> > > choices like "future technical direction" that tie the resolution >> > > to language in the bylaws). Regardless of technical reasons why >> > > it may not be necessary, we still have many social justifications >> > > for doing it the way we originally set out to do it. Tests related >> > > to trademark enforcement need to go into the tempest repository. >> > > >> > > The way I think this should work (and the way I remember us describing >> > > it at the time the policy was established) is the Interop WG >> > > (previously DefCore) should identify capabilities and tests, then >> > > ask project teams to reproduce those tests in the tempest repo. >> > > When the tests land, they can be used by the trademark program. >> > > Teams can also, at their leisure, decide whether to remove the >> > > original versions of the tests from whatever repo they existed in >> > > to begin with. >> > > >> > > Graham, you've proposed a new resolution with several options for >> > > where to put tests for "add-on programs." I don't think we need >> > > that resolution if we want the tests to continue to live in tempest. >> > > The existing resolution doesn't qualify which tests, beyond "for >> > > trademark enforcement" and more words won't make that more clear, >> > > IMO. >> > > >> > > Now if you *do* want to change the policy, we should talk about >> > > that. But I can't tell whether you want to change it, you're worried >> > > the policy is unclear, or it is not being followed. Can you clarify >> > > which it is? >> > >> > It is not being followed. >> > >> > I have brought this up at every forum session on these programs, and the >> > people in the room from QA have *always* pushed back on it. >> >> OK, so that's a problem. I need to hear from the QA team why they've >> reversed that decision. >> >> > >> > And, for clarity (I saw this in a few logs) QA have *never* said that >> > they will take the interop designated tests for the DNS project into >> > openstack/tempest. >> >> When we approved the resolution that describes the current policy, the >> QA team agreed that they would take tests for trademark. There was no >> stipulation about which projects those apply to. > > I feel pretty sure that was discussed in a TC meeting, but I can't > find that. I do find Matt and Ken'ichi voting +1 on the resolution > itself. https://review.openstack.org/#/c/312718/. If I remember > correctly, Ken'ichi was the PTL at the time. Yeah, I have still agreed with the resolution. When I voted +1 on that, core projects were defined as 6 projects like Nova, Cinder, Glance, Keystone, Neutron and Swift. And the project navigator also showed these 6 projects as core projects. Now I cannot find such definition on the project navigator[1], the definition has been changed? I just want to clarify "is it true that designate and heat become core projects?" If there is a concrete decision, I don't have any objections against that we have these projects tests in Tempest as the resolution. Thanks Ken Ohmichi --- [1]: https://www.openstack.org/software/project-navigator __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://l
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Doug Hellmann's message of 2018-01-18 15:21:12 -0500: > Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +: > > > > On 18/01/18 18:52, Doug Hellmann wrote: > > > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: > > >> On 18/01/18 16:25, Doug Hellmann wrote: > > >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > > >> > > >> > > >> > > >>> > > >>> In the past the QA team agreed to accept trademark-related tests from > > >>> all projects in the tempest repo. Has that changed? > > >>> > > >> > > >> There has not been an explict rejection but in all conversations the > > >> response has been "non core projects are outside the scope of tempest". > > >> > > >> Honestly, everytime we have tried to do something to core tempest > > >> we have had major pushback, and I want to clarify this before I or > > >> someone else put in the work of porting the base clients, getting CI > > >> configured*, and proposing the tests to tempest. > > > > > > OK. > > > > > > The current policy doesn't say anything about "core" or different > > > trademark programs or any other criteria. > > > > > > The TC therefore encourages the DefCore committee to consider it an > > > indication of future technical direction that we do not want tests > > > outside of the Tempest repository used for trademark enforcement, and > > > that any new or existing tests that cover capabilities they want to > > > consider for trademark enforcement should be placed in Tempest. > > > > > > That all seems very clear to me (setting aside some specific word > > > choices like "future technical direction" that tie the resolution > > > to language in the bylaws). Regardless of technical reasons why > > > it may not be necessary, we still have many social justifications > > > for doing it the way we originally set out to do it. Tests related > > > to trademark enforcement need to go into the tempest repository. > > > > > > The way I think this should work (and the way I remember us describing > > > it at the time the policy was established) is the Interop WG > > > (previously DefCore) should identify capabilities and tests, then > > > ask project teams to reproduce those tests in the tempest repo. > > > When the tests land, they can be used by the trademark program. > > > Teams can also, at their leisure, decide whether to remove the > > > original versions of the tests from whatever repo they existed in > > > to begin with. > > > > > > Graham, you've proposed a new resolution with several options for > > > where to put tests for "add-on programs." I don't think we need > > > that resolution if we want the tests to continue to live in tempest. > > > The existing resolution doesn't qualify which tests, beyond "for > > > trademark enforcement" and more words won't make that more clear, > > > IMO. > > > > > > Now if you *do* want to change the policy, we should talk about > > > that. But I can't tell whether you want to change it, you're worried > > > the policy is unclear, or it is not being followed. Can you clarify > > > which it is? > > > > It is not being followed. > > > > I have brought this up at every forum session on these programs, and the > > people in the room from QA have *always* pushed back on it. > > OK, so that's a problem. I need to hear from the QA team why they've > reversed that decision. > > > > > And, for clarity (I saw this in a few logs) QA have *never* said that > > they will take the interop designated tests for the DNS project into > > openstack/tempest. > > When we approved the resolution that describes the current policy, the > QA team agreed that they would take tests for trademark. There was no > stipulation about which projects those apply to. I feel pretty sure that was discussed in a TC meeting, but I can't find that. I do find Matt and Ken'ichi voting +1 on the resolution itself. https://review.openstack.org/#/c/312718/. If I remember correctly, Ken'ichi was the PTL at the time. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +: > > On 18/01/18 18:52, Doug Hellmann wrote: > > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: > >> On 18/01/18 16:25, Doug Hellmann wrote: > >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > >> > >> > >> > >>> > >>> In the past the QA team agreed to accept trademark-related tests from > >>> all projects in the tempest repo. Has that changed? > >>> > >> > >> There has not been an explict rejection but in all conversations the > >> response has been "non core projects are outside the scope of tempest". > >> > >> Honestly, everytime we have tried to do something to core tempest > >> we have had major pushback, and I want to clarify this before I or > >> someone else put in the work of porting the base clients, getting CI > >> configured*, and proposing the tests to tempest. > > > > OK. > > > > The current policy doesn't say anything about "core" or different > > trademark programs or any other criteria. > > > > The TC therefore encourages the DefCore committee to consider it an > > indication of future technical direction that we do not want tests > > outside of the Tempest repository used for trademark enforcement, and > > that any new or existing tests that cover capabilities they want to > > consider for trademark enforcement should be placed in Tempest. > > > > That all seems very clear to me (setting aside some specific word > > choices like "future technical direction" that tie the resolution > > to language in the bylaws). Regardless of technical reasons why > > it may not be necessary, we still have many social justifications > > for doing it the way we originally set out to do it. Tests related > > to trademark enforcement need to go into the tempest repository. > > > > The way I think this should work (and the way I remember us describing > > it at the time the policy was established) is the Interop WG > > (previously DefCore) should identify capabilities and tests, then > > ask project teams to reproduce those tests in the tempest repo. > > When the tests land, they can be used by the trademark program. > > Teams can also, at their leisure, decide whether to remove the > > original versions of the tests from whatever repo they existed in > > to begin with. > > > > Graham, you've proposed a new resolution with several options for > > where to put tests for "add-on programs." I don't think we need > > that resolution if we want the tests to continue to live in tempest. > > The existing resolution doesn't qualify which tests, beyond "for > > trademark enforcement" and more words won't make that more clear, > > IMO. > > > > Now if you *do* want to change the policy, we should talk about > > that. But I can't tell whether you want to change it, you're worried > > the policy is unclear, or it is not being followed. Can you clarify > > which it is? > > It is not being followed. > > I have brought this up at every forum session on these programs, and the > people in the room from QA have *always* pushed back on it. OK, so that's a problem. I need to hear from the QA team why they've reversed that decision. > > And, for clarity (I saw this in a few logs) QA have *never* said that > they will take the interop designated tests for the DNS project into > openstack/tempest. When we approved the resolution that describes the current policy, the QA team agreed that they would take tests for trademark. There was no stipulation about which projects those apply to. > > To the point that the interop tooling was developed to support plugins > (which would seem to be in breach of this resolution, but I am sure > there is reasons for this.) I can see it being useful to support plugins for evaluating tests before they are accepted. > > I do want to have option 3 (interop-tempest-plugin), but right now I > will settle for us either: > > A: Doing what we planned on before (Option 1) (Prefered) > B: Documenting the fact that things have changed (Option 2), and >articulate and record the reasoning for the change. > > I think Add Ons are going to the Board in Dublin for the change from > Advisory, in the 2018.02 standard so we need to get clarity on this. I agree. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 18/01/18 18:52, Doug Hellmann wrote: > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: >> On 18/01/18 16:25, Doug Hellmann wrote: >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: >> >> >> >>> >>> In the past the QA team agreed to accept trademark-related tests from >>> all projects in the tempest repo. Has that changed? >>> >> >> There has not been an explict rejection but in all conversations the >> response has been "non core projects are outside the scope of tempest". >> >> Honestly, everytime we have tried to do something to core tempest >> we have had major pushback, and I want to clarify this before I or >> someone else put in the work of porting the base clients, getting CI >> configured*, and proposing the tests to tempest. > > OK. > > The current policy doesn't say anything about "core" or different > trademark programs or any other criteria. > > The TC therefore encourages the DefCore committee to consider it an > indication of future technical direction that we do not want tests > outside of the Tempest repository used for trademark enforcement, and > that any new or existing tests that cover capabilities they want to > consider for trademark enforcement should be placed in Tempest. > > That all seems very clear to me (setting aside some specific word > choices like "future technical direction" that tie the resolution > to language in the bylaws). Regardless of technical reasons why > it may not be necessary, we still have many social justifications > for doing it the way we originally set out to do it. Tests related > to trademark enforcement need to go into the tempest repository. > > The way I think this should work (and the way I remember us describing > it at the time the policy was established) is the Interop WG > (previously DefCore) should identify capabilities and tests, then > ask project teams to reproduce those tests in the tempest repo. > When the tests land, they can be used by the trademark program. > Teams can also, at their leisure, decide whether to remove the > original versions of the tests from whatever repo they existed in > to begin with. > > Graham, you've proposed a new resolution with several options for > where to put tests for "add-on programs." I don't think we need > that resolution if we want the tests to continue to live in tempest. > The existing resolution doesn't qualify which tests, beyond "for > trademark enforcement" and more words won't make that more clear, > IMO. > > Now if you *do* want to change the policy, we should talk about > that. But I can't tell whether you want to change it, you're worried > the policy is unclear, or it is not being followed. Can you clarify > which it is? It is not being followed. I have brought this up at every forum session on these programs, and the people in the room from QA have *always* pushed back on it. And, for clarity (I saw this in a few logs) QA have *never* said that they will take the interop designated tests for the DNS project into openstack/tempest. To the point that the interop tooling was developed to support plugins (which would seem to be in breach of this resolution, but I am sure there is reasons for this.) I do want to have option 3 (interop-tempest-plugin), but right now I will settle for us either: A: Doing what we planned on before (Option 1) (Prefered) B: Documenting the fact that things have changed (Option 2), and articulate and record the reasoning for the change. I think Add Ons are going to the Board in Dublin for the change from Advisory, in the 2018.02 standard so we need to get clarity on this. - Graham > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +: > On 18/01/18 16:25, Doug Hellmann wrote: > > Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > > > > > > > In the past the QA team agreed to accept trademark-related tests from > > all projects in the tempest repo. Has that changed? > > > > There has not been an explict rejection but in all conversations the > response has been "non core projects are outside the scope of tempest". > > Honestly, everytime we have tried to do something to core tempest > we have had major pushback, and I want to clarify this before I or > someone else put in the work of porting the base clients, getting CI > configured*, and proposing the tests to tempest. OK. The current policy doesn't say anything about "core" or different trademark programs or any other criteria. The TC therefore encourages the DefCore committee to consider it an indication of future technical direction that we do not want tests outside of the Tempest repository used for trademark enforcement, and that any new or existing tests that cover capabilities they want to consider for trademark enforcement should be placed in Tempest. That all seems very clear to me (setting aside some specific word choices like "future technical direction" that tie the resolution to language in the bylaws). Regardless of technical reasons why it may not be necessary, we still have many social justifications for doing it the way we originally set out to do it. Tests related to trademark enforcement need to go into the tempest repository. The way I think this should work (and the way I remember us describing it at the time the policy was established) is the Interop WG (previously DefCore) should identify capabilities and tests, then ask project teams to reproduce those tests in the tempest repo. When the tests land, they can be used by the trademark program. Teams can also, at their leisure, decide whether to remove the original versions of the tests from whatever repo they existed in to begin with. Graham, you've proposed a new resolution with several options for where to put tests for "add-on programs." I don't think we need that resolution if we want the tests to continue to live in tempest. The existing resolution doesn't qualify which tests, beyond "for trademark enforcement" and more words won't make that more clear, IMO. Now if you *do* want to change the policy, we should talk about that. But I can't tell whether you want to change it, you're worried the policy is unclear, or it is not being followed. Can you clarify which it is? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 18/01/18 16:25, Doug Hellmann wrote: > Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > > In the past the QA team agreed to accept trademark-related tests from > all projects in the tempest repo. Has that changed? > There has not been an explict rejection but in all conversations the response has been "non core projects are outside the scope of tempest". Honestly, everytime we have tried to do something to core tempest we have had major pushback, and I want to clarify this before I or someone else put in the work of porting the base clients, getting CI configured*, and proposing the tests to tempest. - Graham * With zuulv3 this is *much* easier, so not as big a deal as it once was signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +: > I had hoped for more of a discussion on this before I jumped back into > this debate - but it seams to be stalled still, so here it goes. > > I proposed this initially as we were unclear on where the tests should > go - we had a resolution that said all tests go into openstack/tempest > (with a list of reasons why), and the guidance and discussion that been > had in various summits was that "add-ons" should stay in plugins. > > So right now, we (by the governance rules) should be pushing tests to > tempest for the new programs. > > In the resolution that placed the tests in tempest there was a few > reasons proposed: > > For example, API and behavioral changes must be carefully managed, as > must mundane aspects such as test and module naming and location > within the test suite. Even changes that leave tests functionally > equivalent may cause unexpected consequences for their use in DefCore > processes and validation. Placing the tests in a central repository > will make it easier to maintain consistency and avoid breaking the > trademark enforcement tool. > > This still applies, and even more so for teams that traditionally do not > have a strong QE contributor / reviewer base (aka projects not in > "core"). > > Centralizing the tests also makes it easier for anyone running the > validation tool against their cloud or cloud distribution to use the > tests. It is easier to install the test suite and its dependencies, > and it is easier to read and understand a set of tests following a > consistent implementation pattern. > > Apparently users do not need central tests anymore, feedback from > RefStack is that people who run these tests are comfortable dealing > with extra python packages. > > The point about a single set of tests, in a single location and style > still stands. > > Finally, having the tests in a central location makes it easier to > ensure that all members of the community have equal input into what > the tests do and how they are implemented and maintained. > > Seems like a good value to strive for. > > One of the items that has been used to push back against adding > "add-ons" to tempest has been that tempest has a defined scope, and > neither of the current add-ons fit in that scope. > > Can someone clarify what the set of criteria is? I think it will help > this discussion. > > Another push back is the "scaling" issue - adding more tests will > overload the QA team. In the past the QA team agreed to accept trademark-related tests from all projects in the tempest repo. Has that changed? > > Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite > of 353. > > I do not think there is many other add-ons proposed yet, and the new > Vertical programs will probably mainly be re-using tests in the > openstack/tempest repos as is. > > This is not a big tent-esque influx of programs - the only projects > that can be added to the trademarks are programs in tc-approved-release > [4], so I do not see scaling as a big issue, especially as these tests > are such base concepts that if they need to be changed there is a > completely new API, so the only overhead will be ensuring that nothing > in tempest breaks the new tests (which is a good thing for trademark > tests). > > Personally, for me, I like option 3. I did not initially add it, as I > knew it would cause endless bikesheding, but I do think it fits both > a technical and social model. > > I see 2 immediate routes forward: > > - Option 1, and we start adding these tests asap > - Pseudo Option 2, were we delete the resolution at [2] as it clearly >does not apply anymore, and abandon the review at [1]. > > Finally - do not conflate my actions with those of the Designate team. > I have seen people talking about how this resolution was leverage the > team needed to move our tests in tree. This is definitely *not* true. > Having our tests in a plugin is useful to us, and if the above > resolution passed, I cannot see a situation where we would try to > move any tests that were not listed in the interop standards. > > This is something I have done as an individual in the community, not > something the designate team have pushed for. Thanks for pushing for a clear resolution to this, Graham. > > > [4] - > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openst
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 11/01/18 16:36, Colleen Murphy wrote: > Hi everyone, > > We have governance review under debate[1] that we need the community's help > on. > The debate is over what recommendation the TC should make to the Interop team > on where the tests it uses for the OpenStack trademark program should be > located, specifically those for the new add-on program being introduced. Let > me > badly summarize: > > A couple of years ago we issued a resolution[2] officially recommending that > the Interop team use solely tempest as its source of tests for capability > verification. The Interop team has always had the view that the developers, > being the people closest to the project they're creating, are the best people > to write tests verifying correct functionality, and so the Interop team > doesn't > maintain its own test suite, instead selecting tests from those written in > coordination between the QA team and the other project teams. These tests are > used to validate clouds applying for the OpenStack Powered tag, and since all > of the projects included in the OpenStack Powered program already had tests in > tempest, this was a natural fit. When we consider adding new trademark > programs > comprising of other projects, the test source is less obvious. Two examples > are > designate, which has never had tests in the tempest repo, and heat, which > recently had its tests removed from the tempest repo. > > > As I'm not deeply steeped in the history of either the Interop or QA teams I > am > sure I've misrepresented some details here, I'm sorry about that. But we'd > like > to get this resolution moving forward and we're currently stuck, so this > thread > is intended to gather enough community input to get unstuck and avoid letting > this proposal become stale. Please respond to this thread or comment on the > resolution proposal[1] if you have any thoughts. > > Colleen > > [1] https://review.openstack.org/#/c/521602 > [2] > https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html > [3] > https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html > I had hoped for more of a discussion on this before I jumped back into this debate - but it seams to be stalled still, so here it goes. I proposed this initially as we were unclear on where the tests should go - we had a resolution that said all tests go into openstack/tempest (with a list of reasons why), and the guidance and discussion that been had in various summits was that "add-ons" should stay in plugins. So right now, we (by the governance rules) should be pushing tests to tempest for the new programs. In the resolution that placed the tests in tempest there was a few reasons proposed: For example, API and behavioral changes must be carefully managed, as must mundane aspects such as test and module naming and location within the test suite. Even changes that leave tests functionally equivalent may cause unexpected consequences for their use in DefCore processes and validation. Placing the tests in a central repository will make it easier to maintain consistency and avoid breaking the trademark enforcement tool. This still applies, and even more so for teams that traditionally do not have a strong QE contributor / reviewer base (aka projects not in "core"). Centralizing the tests also makes it easier for anyone running the validation tool against their cloud or cloud distribution to use the tests. It is easier to install the test suite and its dependencies, and it is easier to read and understand a set of tests following a consistent implementation pattern. Apparently users do not need central tests anymore, feedback from RefStack is that people who run these tests are comfortable dealing with extra python packages. The point about a single set of tests, in a single location and style still stands. Finally, having the tests in a central location makes it easier to ensure that all members of the community have equal input into what the tests do and how they are implemented and maintained. Seems like a good value to strive for. One of the items that has been used to push back against adding "add-ons" to tempest has been that tempest has a defined scope, and neither of the current add-ons fit in that scope. Can someone clarify what the set of criteria is? I think it will help this discussion. Another push back is the "scaling" issue - adding more tests will overload the QA team. Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite of 353. I do not think there is many other add-ons proposed yet, and the new Vertical programs will probably mainly be re-using tests in the openstack/tempest repos as is. This is not a big tent-esque influx of programs - the only projects that can be added to the trademarks are programs in tc-approved-release [4], so I do not see scaling as a big issue, especially as these tests are such base concepts that if
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 12/01/18 09:49, Luigi Toscano wrote: > On Thursday, 11 January 2018 23:52:00 CET Matt Riedemann wrote: >> On 1/11/2018 10:36 AM, Colleen Murphy wrote: >>> 1) All trademark-related tests should go in the tempest repo, in >>> accordance >>> >>> with the original resolution. This would mean that even projects that >>> have >>> never had tests in tempest would now have to add at least some of >>> their >>> black-box tests to tempest. >>> >>> The value of this option is that centralizes tests used for the Interop >>> program in a location where interop-minded folks from the QA team can >>> control them. The downside is that projects that so far have avoided >>> having a dependency on tempest will now lose some control over the >>> black-box tests that they use for functional and integration that would >>> now also be used for trademark certification. >>> There's also concern for the review bandwidth of the QA team - we can't >>> expect the QA team to be continually responsible for an ever-growing list >>> of projects and their trademark tests. >> >> How many tests are we talking about for designate and heat? Half a >> dozen? A dozen? More? >> >> If it's just a couple of tests per project it doesn't seem terrible to >> have them live in Tempest so you get the "interop eye" on reviews, as >> noted in your email. If it's a considerable amount, then option 2 seems >> the best for the majority of parties. > > I would argue that it does not scale; what if some test is taken out from the > interoperability, and others are added? It would mean moving tests from one > repository to another, with change of paths. I think that the solution 2, > where the repository where a test belong and the functionality of a test are > not linked, is better. > > Ciao > How is this different from the 6 programs already in tempest? signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Erno Kuvaja's message of 2018-01-15 12:59:44 +: > I think the worst case scenario is that we scrape together what ever > we can just to have something to say that we test it and not have > consistency nor clear responsibility of who, what and how. > (Unfortunately I think this is the current situation and I'm super > happy to hear that this is being discussed and the decision is not > made lightly.) That seems very far from the current situation to me. We have a large integration test suite written primarily by contributors to the projects it tests. A subset of that is used for the trademark tests. That same subset is in 1 repo, managed by the QA team, who apply the extra review criteria needed for the trademark program to be stable. The fact that so many people seem uninformed about how all of this works is exactly why I think it's a mistake to spread the tests out and have a bunch of different teams applying different review criteria to them. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thu, Jan 11, 2018 at 4:36 PM, Colleen Murphy wrote: > Hi everyone, > > We have governance review under debate[1] that we need the community's help > on. > The debate is over what recommendation the TC should make to the Interop team > on where the tests it uses for the OpenStack trademark program should be > located, specifically those for the new add-on program being introduced. Let > me > badly summarize: > > A couple of years ago we issued a resolution[2] officially recommending that > the Interop team use solely tempest as its source of tests for capability > verification. The Interop team has always had the view that the developers, > being the people closest to the project they're creating, are the best people > to write tests verifying correct functionality, and so the Interop team > doesn't > maintain its own test suite, instead selecting tests from those written in > coordination between the QA team and the other project teams. These tests are > used to validate clouds applying for the OpenStack Powered tag, and since all > of the projects included in the OpenStack Powered program already had tests in > tempest, this was a natural fit. When we consider adding new trademark > programs > comprising of other projects, the test source is less obvious. Two examples > are > designate, which has never had tests in the tempest repo, and heat, which > recently had its tests removed from the tempest repo. > > So far the patch proposes three options: > > 1) All trademark-related tests should go in the tempest repo, in accordance >with the original resolution. This would mean that even projects that have >never had tests in tempest would now have to add at least some of their >black-box tests to tempest. > > The value of this option is that centralizes tests used for the Interop > program > in a location where interop-minded folks from the QA team can control them. > The > downside is that projects that so far have avoided having a dependency on > tempest will now lose some control over the black-box tests that they use for > functional and integration that would now also be used for trademark > certification. > There's also concern for the review bandwidth of the QA team - we can't expect > the QA team to be continually responsible for an ever-growing list of projects > and their trademark tests. > > 2) All trademark-related tests for *add-on projects* should be sourced from >plugins external to tempest. > > The value of this option is it allows project teams to retain control over > these tests. The potential problem with it is that individual project teams > are > not necessarily reviewing test changes with an eye for interop concerns and so > could inadvertently change the behavior of the trademark-verification tools. > > 3) All trademark-related tests should go in a single separate tempest plugin. > > This has the value of giving the QA and Interop teams control over > interop-related tests while also making clear the distinction between tests > used for trademark verification and tests used for CI. Matt's argument against > this is that there actually is very little distinction between those two > cases, > and that a given test could have many different applications. > > Other ideas that have been thrown around are: > > * Maintaining a branch in the tempest repo that Interop tests are pulled from. > > * Tagging Interop-related tests with decorators to make it clear that they > need > to be handled carefully. > > At the heart of the issue is the perception that projects that keep their > integration tests within the tempest tree are somehow blessed, maybe by the QA > team or by the TC. It would be nice to try to clarify what technical > and political > reasons we have for why different projects have tests in different places - > review bandwidth of the QA team, ownership/control by the project teams, > technical interdependency between certain projects, or otherwise. > As someone who has been in middle of all that already once I'd like to bring up bit more fundamental problem into this topic. I'm not able to provide one size fits all solution but hopefully some insight that would help the community to make the right decision. I think the biggest problem is who's fox is let to guard the chicken coop. By that I mean the basic problem of our testing still relies on what is tested based on which assumptions and by whom. If the tests are provided by the project teams, the test is more likely to cover the intended usecase of the feature as it's implemented and if there is bug found on that, the likelyhood that the test is altered is quite high also the individual projects might not have the best idea what might be the important things to the interoperability and trademark purposes. Obviously when the test is written against intended behavior it's less likely but also those changes might sneak in affecting the interoprability. On the other hand if the test is written by QA/inte
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Excerpts from Luigi Toscano's message of 2018-01-12 10:49:55 +0100: > On Thursday, 11 January 2018 23:52:00 CET Matt Riedemann wrote: > > On 1/11/2018 10:36 AM, Colleen Murphy wrote: > > > 1) All trademark-related tests should go in the tempest repo, in > > > accordance > > > > > > with the original resolution. This would mean that even projects that > > > have > > > never had tests in tempest would now have to add at least some of > > > their > > > black-box tests to tempest. > > > > > > The value of this option is that centralizes tests used for the Interop > > > program in a location where interop-minded folks from the QA team can > > > control them. The downside is that projects that so far have avoided > > > having a dependency on tempest will now lose some control over the > > > black-box tests that they use for functional and integration that would > > > now also be used for trademark certification. > > > There's also concern for the review bandwidth of the QA team - we can't > > > expect the QA team to be continually responsible for an ever-growing list > > > of projects and their trademark tests. > > > > How many tests are we talking about for designate and heat? Half a > > dozen? A dozen? More? > > > > If it's just a couple of tests per project it doesn't seem terrible to > > have them live in Tempest so you get the "interop eye" on reviews, as > > noted in your email. If it's a considerable amount, then option 2 seems > > the best for the majority of parties. > > I would argue that it does not scale; what if some test is taken out from the > interoperability, and others are added? It would mean moving tests from one > repository to another, with change of paths. I think that the solution 2, > where the repository where a test belong and the functionality of a test are > not linked, is better. > > Ciao How often do the interop test suites change in that way? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On Thursday, 11 January 2018 23:52:00 CET Matt Riedemann wrote: > On 1/11/2018 10:36 AM, Colleen Murphy wrote: > > 1) All trademark-related tests should go in the tempest repo, in > > accordance > > > > with the original resolution. This would mean that even projects that > > have > > never had tests in tempest would now have to add at least some of > > their > > black-box tests to tempest. > > > > The value of this option is that centralizes tests used for the Interop > > program in a location where interop-minded folks from the QA team can > > control them. The downside is that projects that so far have avoided > > having a dependency on tempest will now lose some control over the > > black-box tests that they use for functional and integration that would > > now also be used for trademark certification. > > There's also concern for the review bandwidth of the QA team - we can't > > expect the QA team to be continually responsible for an ever-growing list > > of projects and their trademark tests. > > How many tests are we talking about for designate and heat? Half a > dozen? A dozen? More? > > If it's just a couple of tests per project it doesn't seem terrible to > have them live in Tempest so you get the "interop eye" on reviews, as > noted in your email. If it's a considerable amount, then option 2 seems > the best for the majority of parties. I would argue that it does not scale; what if some test is taken out from the interoperability, and others are added? It would mean moving tests from one repository to another, with change of paths. I think that the solution 2, where the repository where a test belong and the functionality of a test are not linked, is better. Ciao -- Luigi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
On 1/11/2018 10:36 AM, Colleen Murphy wrote: 1) All trademark-related tests should go in the tempest repo, in accordance with the original resolution. This would mean that even projects that have never had tests in tempest would now have to add at least some of their black-box tests to tempest. The value of this option is that centralizes tests used for the Interop program in a location where interop-minded folks from the QA team can control them. The downside is that projects that so far have avoided having a dependency on tempest will now lose some control over the black-box tests that they use for functional and integration that would now also be used for trademark certification. There's also concern for the review bandwidth of the QA team - we can't expect the QA team to be continually responsible for an ever-growing list of projects and their trademark tests. How many tests are we talking about for designate and heat? Half a dozen? A dozen? More? If it's just a couple of tests per project it doesn't seem terrible to have them live in Tempest so you get the "interop eye" on reviews, as noted in your email. If it's a considerable amount, then option 2 seems the best for the majority of parties. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][tc] Clarifying testing recommendation for interop programs
Hi everyone, We have governance review under debate[1] that we need the community's help on. The debate is over what recommendation the TC should make to the Interop team on where the tests it uses for the OpenStack trademark program should be located, specifically those for the new add-on program being introduced. Let me badly summarize: A couple of years ago we issued a resolution[2] officially recommending that the Interop team use solely tempest as its source of tests for capability verification. The Interop team has always had the view that the developers, being the people closest to the project they're creating, are the best people to write tests verifying correct functionality, and so the Interop team doesn't maintain its own test suite, instead selecting tests from those written in coordination between the QA team and the other project teams. These tests are used to validate clouds applying for the OpenStack Powered tag, and since all of the projects included in the OpenStack Powered program already had tests in tempest, this was a natural fit. When we consider adding new trademark programs comprising of other projects, the test source is less obvious. Two examples are designate, which has never had tests in the tempest repo, and heat, which recently had its tests removed from the tempest repo. So far the patch proposes three options: 1) All trademark-related tests should go in the tempest repo, in accordance with the original resolution. This would mean that even projects that have never had tests in tempest would now have to add at least some of their black-box tests to tempest. The value of this option is that centralizes tests used for the Interop program in a location where interop-minded folks from the QA team can control them. The downside is that projects that so far have avoided having a dependency on tempest will now lose some control over the black-box tests that they use for functional and integration that would now also be used for trademark certification. There's also concern for the review bandwidth of the QA team - we can't expect the QA team to be continually responsible for an ever-growing list of projects and their trademark tests. 2) All trademark-related tests for *add-on projects* should be sourced from plugins external to tempest. The value of this option is it allows project teams to retain control over these tests. The potential problem with it is that individual project teams are not necessarily reviewing test changes with an eye for interop concerns and so could inadvertently change the behavior of the trademark-verification tools. 3) All trademark-related tests should go in a single separate tempest plugin. This has the value of giving the QA and Interop teams control over interop-related tests while also making clear the distinction between tests used for trademark verification and tests used for CI. Matt's argument against this is that there actually is very little distinction between those two cases, and that a given test could have many different applications. Other ideas that have been thrown around are: * Maintaining a branch in the tempest repo that Interop tests are pulled from. * Tagging Interop-related tests with decorators to make it clear that they need to be handled carefully. At the heart of the issue is the perception that projects that keep their integration tests within the tempest tree are somehow blessed, maybe by the QA team or by the TC. It would be nice to try to clarify what technical and political reasons we have for why different projects have tests in different places - review bandwidth of the QA team, ownership/control by the project teams, technical interdependency between certain projects, or otherwise. Ultimately, as Jeremy said in the comments on the resolution patch, the recommendation should be one that works best for the QA and Interop teams. So far we've heard from Matt and Mark expressing moderate support for option 2. We'd like to hear more from those teams about how they see this working, especially with regard to concerns about the quality and stability standards that out-of-tree tests may be held to. We additionally need input from the whole community on how maintaining trademark-related tests in tempest will affect you if you don't already have your tests there. We'd especially like to address any perceptions of favoritism or exclusionism that stem from these issues. And to quickly clear up one detail before it makes it onto this thread, the Queens Community Goal about splitting tempest plugins out of the main project's tree[3] is entirely about addressing technical problems related to packaging for existing tempest plugins, it's not a decree about what should live within the tempest repository nor does it have anything to do with the Interop program. As I'm not deeply steeped in the history of either the Interop or QA teams I am sure I've misrepresented some details here, I'm sorry about that. But we
