Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On 05/06/17 09:43, Sean Dague wrote: On 06/01/2017 06:09 AM, Chris Dent wrote: It's clear from this thread and other conversations that the management of tempest plugins is creating a multiplicity of issues and confusions: * Some projects are required to use plugins and some are not. This creates classes of projects. While this is true, there are also reasons for that. We decided to break up the compute service into distinct parts years ago to help let each part grow dedicated expertise (images, networking, block storage). However, there is a ton of coupling here even though these are broken up. My continued resistance to decomposing the QA side of those projects is getting that integration testing right, and debugging it is hard, because there are so many interactions required to have a working server started. And Nova, Neutron, Cinder are the top three most active projects in OpenStack. So the rate of change individually is quite high. Forcing those services out into plugins because of the feeling that Presumably that could be addressed by splitting the Nova/Neutron/Cinder/Glance tests not used by the OpenStack Powered trademark program (aka DefCore) into a combined base-compute Tempest plugin though? Or are you saying there's coupling between the Defcore and non-Defcore tests that isn't mediated by tempest-lib? If that's the case then I'd be concerned that we might run into this problem again in the future. something doesn't look fair on paper is just generating more work to create spherical elephants, instead of acknowledging the amount of work the QA team has on it's shoulders, and letting it optimize for a better experience by OpenStack users. Especially given limited resources. -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On 06/01/2017 06:09 AM, Chris Dent wrote: > It's clear from this thread and other conversations that the > management of tempest plugins is creating a multiplicity of issues > and confusions: > > * Some projects are required to use plugins and some are not. This > creates classes of projects. While this is true, there are also reasons for that. We decided to break up the compute service into distinct parts years ago to help let each part grow dedicated expertise (images, networking, block storage). However, there is a ton of coupling here even though these are broken up. My continued resistance to decomposing the QA side of those projects is getting that integration testing right, and debugging it is hard, because there are so many interactions required to have a working server started. And Nova, Neutron, Cinder are the top three most active projects in OpenStack. So the rate of change individually is quite high. Forcing those services out into plugins because of the feeling that something doesn't look fair on paper is just generating more work to create spherical elephants, instead of acknowledging the amount of work the QA team has on it's shoulders, and letting it optimize for a better experience by OpenStack users. Especially given limited resources. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Thu, 1 Jun 2017, Matthew Treinish wrote: On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote: A lot of this results, in part, from there being no single guiding pattern and principle for how (and where) the tests are to be managed. It sounds like you want to write a general testing guide for openstack. Have you started this effort anywhere? I don't think anyone would be opposed to starting a document for that, it seems like a reasonable thing to have. But, I think you'll find there is not a one size fits all solution though, because every project has their own requirements and needs for testing. No, I haven't made any decisions about what ought to happen. I'm still trying to figure out if there is a problem, a suite of problems, or everything is great. Knowing what the problems are tends to be a reasonable thing to do before proposing or implementing solutions, especially if we want those solutions to be most correct. So have you read the documentation: https://docs.openstack.org/developer/tempest/ (or any of the other relevant documentation and filed bugs about where you think there are gaps? This is something that really bugs me sometimes (yes the pun is intended) just like anything else this is all about iterative improvements. These broad trends are things tempest and (every project hopefully) have been working on. But improvements don't just magically occur overnight it takes time to implement them. This is a huge part of the colllaboration issues I was identifying in my previous message. Somebody says "there seems to be some confusion here" and somebody else comes along and asks "have you filed bugs?" or "have you proposed a solution?". Well, "no" because like I said above I don't know what (or even _if_) there's something to fix or the relevant foundations of the confusion. I have some suspicions or concerns that the implicit hierarchy of some tempests tests being in plugins and some not creates issues with discovery, management and identification of responsible parties and _may_ imply a lack of "level playing field". But: * if other people don't have those concerns it's not worth pursuing * until we reach some kind of shared understanding and agreement about the concerns, speculating about solutions is premature Just compare the state of the documentation and tooling from 2 years ago (when tempest started adding the plugin interface) to today. Things have steadily improved over time and the situation now is much better. This will continue and in the future things will get even better. Yes, it's great. If you feel like I was suggesting otherwise, then my apologies for not being clear. As a general rule tempest and other QA tools have consistently done great work in terms of documentation and tooling. That there are plugins at all is fantastic; that we are having discussions about how to make the most effective and fair use of them is a sign that they work. -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote: > A lot of this results, in part, from there being no single guiding > pattern and principle for how (and where) the tests are to be > managed. It sounds like you want to write a general testing guide for openstack. Have you started this effort anywhere? I don't think anyone would be opposed to starting a document for that, it seems like a reasonable thing to have. But, I think you'll find there is not a one size fits all solution though, because every project has their own requirements and needs for testing. > When there's a choice between one, some and all, "some" is > almost always the wrong way to manage something. "some" is how we do > tempest (and fair few other OpenStack things). > > If it is the case that we want some projects to not put their tests > in the main tempest repo then the only conceivable pattern from a > memorability, discoverability, and equality standpoint is actually > for all the tests to be in plugins. > > If that isn't possible (and it is clear there are many reasons why > that may be the case) then we need to be extra sure that we explore > and uncover the issues that the "some" approach presents and provide > sufficient documentation, tooling, and guidance to help people get > around them. And that we recognize and acknowledge the impact it has. So have you read the documentation: https://docs.openstack.org/developer/tempest/ (or any of the other relevant documentation and filed bugs about where you think there are gaps? This is something that really bugs me sometimes (yes the pun is intended) just like anything else this is all about iterative improvements. These broad trends are things tempest and (every project hopefully) have been working on. But improvements don't just magically occur overnight it takes time to implement them. Just compare the state of the documentation and tooling from 2 years ago (when tempest started adding the plugin interface) to today. Things have steadily improved over time and the situation now is much better. This will continue and in the future things will get even better. The thing is this is open source collaborative development and there is an expectation that people who have issues with something in the project will report them or contribute a fix and communicate with the maintainers. The users of tempest's plugin interface tend to be other openstack projects (but not exclusively) and if there are something that's not clear we need to work together to fix them. Based on this paragraph I feel like you think the decision to add a tempest plugin interface and decrease it's scope was taken lightly without forethought or careful consideration. But, it's the exact opposite there was extensive debate and exploration of the problem space and took a long time to reach a consensus. > > If the answer to that is "who is going to do that?" or "who has the > time?" then I ask you to ask yourself why we think the "non-core" > projects have time to fiddle about with tempest plugins? I think this unfair simplification, no one is required to write a tempest plugin it's a choice the projects made. While I won't say the interface is perfect, things are always improving. If a project chooses to write a plugin, the expectation is that we'll all work together to help fix issues as they are encountered. No individual can do everything by themselves and it's a shared group effort. But, even so there is no shortage of work for anyone, it's all about prioritization of effort. > > And finally, I actually don't have too strong of a position in the > case of tempest and tempest plugins. What I take issue with is the > process whereby we discuss and decide these things and characterize > the various projects > > If I have any position on tempest at all it is that we should limit > it to gross cloud validation and maybe interop testing, and projects > should manage their own integration testing in tree using whatever > tooling they feel is most appropriate. If that turns out to be > tempest, cool. I fail to see how this is any different than how things work today. No one is required to use a tempest plugin and they can write tests however they want. Tempest itself has a well defined scope (which does evolve over time like any other project) and doesn't try to be all the testing everywhere. Almost every other project has it's own in tree testing outside of tempest or tempest plugins. Also, projects which have in-tree tempest tests also have tempest plugins to expand on that set of functionality. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
Excerpts from Chris Dent's message of 2017-06-01 11:09:56 +0100: > On Wed, 31 May 2017, Doug Hellmann wrote: > > Yeah, it sounds like the current organization of the repo is not > > ideal in terms of equal playing field for all of our project teams. > > I would be fine with all of the interop tests being in a plugin > > together, or of saying that the tempest repo should only contain > > those tests and that others should move to their own plugins. If we're > > going to reorganize all of that, we should decide what new structure we > > want and work it into the goal. > > I feel like the discussion about the interop tests has strayed this > conversation from the more general point about plugin "fairness" and > allowed the vagueness in plans for interop to control our thinking > and discussion about options in the bigger view. I should have prefaced my initial response with a statement like "For those of you who don't know or remember the history". It wasn't meant to imply we shouldn't be making any changes, just that we need to understand how we ended up where we are now so we don't make a change that then no longer meets old requirements. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Thu, Jun 1, 2017 at 3:39 PM, Chris Dent wrote: > On Wed, 31 May 2017, Doug Hellmann wrote: > >> Yeah, it sounds like the current organization of the repo is not >> ideal in terms of equal playing field for all of our project teams. >> I would be fine with all of the interop tests being in a plugin >> together, or of saying that the tempest repo should only contain >> those tests and that others should move to their own plugins. If we're >> going to reorganize all of that, we should decide what new structure we >> want and work it into the goal. >> > > I feel like the discussion about the interop tests has strayed this > conversation from the more general point about plugin "fairness" and > allowed the vagueness in plans for interop to control our thinking > and discussion about options in the bigger view. > > > This is pretty standard for an OpenStack conversation: > > * introduce a general idea or critique > * someone latches on to one small aspect of that idea that presents > some challenges, narrowing the context > * that latching and those challenges is used to kill the introspection > that the general idea was pursuing, effectively killing any > opportunities for learning and discovery that could lead to > improvement or innovation > > This _has_ to stop. We're at my three year anniversary in the > community and this has been and still is my main concern with the > how we collaborate. There is so much stop energy and chilling effect > in the way we discuss things in OpenStack. So much fatigue over > issues being brought up "over and over again" or things being > discussed without immediate solutions in mind. So what! Time moves > forward which means the context for issues is always changing. > Discussion is how we identify problems! Discussion is how we > get good solutions! > > It's clear from this thread and other conversations that the > management of tempest plugins is creating a multiplicity of issues > and confusions: > > * Some projects are required to use plugins and some are not. This > creates classes of projects. > > * Those second class projects now need to move their plugins to > other repos because rules. > > * Projects with plugins need to put their tests in their new repos, > except for some special tests which will be identified by a vague > process. > > * Review of changes is intermittent and hard to track because > stakeholders need to think about multiple locations, without > guidance. > > * People who want to do validation with tempest need to gather stuff > from a variety of locations. > > * Tempest is a hammer used for lots of different nails, but the > type of nail varies over time and with the whimsy of policy. > > * Discussion of using something other than tempest for interop is > curtailed by policy which appears to be based in "that's the way > it is done". > > A lot of this results, in part, from there being no single guiding > pattern and principle for how (and where) the tests are to be > managed. When there's a choice between one, some and all, "some" is > almost always the wrong way to manage something. "some" is how we do > tempest (and fair few other OpenStack things). > > If it is the case that we want some projects to not put their tests > in the main tempest repo then the only conceivable pattern from a > memorability, discoverability, and equality standpoint is actually > for all the tests to be in plugins. > > If that isn't possible (and it is clear there are many reasons why > that may be the case) then we need to be extra sure that we explore > and uncover the issues that the "some" approach presents and provide > sufficient documentation, tooling, and guidance to help people get > around them. And that we recognize and acknowledge the impact it has. > > If the answer to that is "who is going to do that?" or "who has the > time?" then I ask you to ask yourself why we think the "non-core" > projects have time to fiddle about with tempest plugins? > > +1 > And finally, I actually don't have too strong of a position in the > case of tempest and tempest plugins. What I take issue with is the > process whereby we discuss and decide these things and characterize > the various projects. > > If I have any position on tempest at all it is that we should limit > it to gross cloud validation and maybe interop testing, and projects > should manage their own integration testing in tree using whatever > tooling they feel is most appropriate. If that turns out to be > tempest, cool. > > I think it's a fair position and IMO should be the way forward. > > -- > Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ > freenode: cdent tw: @anticdent > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstac
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Wed, 31 May 2017, Doug Hellmann wrote: Yeah, it sounds like the current organization of the repo is not ideal in terms of equal playing field for all of our project teams. I would be fine with all of the interop tests being in a plugin together, or of saying that the tempest repo should only contain those tests and that others should move to their own plugins. If we're going to reorganize all of that, we should decide what new structure we want and work it into the goal. I feel like the discussion about the interop tests has strayed this conversation from the more general point about plugin "fairness" and allowed the vagueness in plans for interop to control our thinking and discussion about options in the bigger view. This is pretty standard for an OpenStack conversation: * introduce a general idea or critique * someone latches on to one small aspect of that idea that presents some challenges, narrowing the context * that latching and those challenges is used to kill the introspection that the general idea was pursuing, effectively killing any opportunities for learning and discovery that could lead to improvement or innovation This _has_ to stop. We're at my three year anniversary in the community and this has been and still is my main concern with the how we collaborate. There is so much stop energy and chilling effect in the way we discuss things in OpenStack. So much fatigue over issues being brought up "over and over again" or things being discussed without immediate solutions in mind. So what! Time moves forward which means the context for issues is always changing. Discussion is how we identify problems! Discussion is how we get good solutions! It's clear from this thread and other conversations that the management of tempest plugins is creating a multiplicity of issues and confusions: * Some projects are required to use plugins and some are not. This creates classes of projects. * Those second class projects now need to move their plugins to other repos because rules. * Projects with plugins need to put their tests in their new repos, except for some special tests which will be identified by a vague process. * Review of changes is intermittent and hard to track because stakeholders need to think about multiple locations, without guidance. * People who want to do validation with tempest need to gather stuff from a variety of locations. * Tempest is a hammer used for lots of different nails, but the type of nail varies over time and with the whimsy of policy. * Discussion of using something other than tempest for interop is curtailed by policy which appears to be based in "that's the way it is done". A lot of this results, in part, from there being no single guiding pattern and principle for how (and where) the tests are to be managed. When there's a choice between one, some and all, "some" is almost always the wrong way to manage something. "some" is how we do tempest (and fair few other OpenStack things). If it is the case that we want some projects to not put their tests in the main tempest repo then the only conceivable pattern from a memorability, discoverability, and equality standpoint is actually for all the tests to be in plugins. If that isn't possible (and it is clear there are many reasons why that may be the case) then we need to be extra sure that we explore and uncover the issues that the "some" approach presents and provide sufficient documentation, tooling, and guidance to help people get around them. And that we recognize and acknowledge the impact it has. If the answer to that is "who is going to do that?" or "who has the time?" then I ask you to ask yourself why we think the "non-core" projects have time to fiddle about with tempest plugins? And finally, I actually don't have too strong of a position in the case of tempest and tempest plugins. What I take issue with is the process whereby we discuss and decide these things and characterize the various projects. If I have any position on tempest at all it is that we should limit it to gross cloud validation and maybe interop testing, and projects should manage their own integration testing in tree using whatever tooling they feel is most appropriate. If that turns out to be tempest, cool. -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Wed, May 31, 2017 at 03:22:59PM +, Jeremy Stanley wrote: > On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote: > [...] > > it's news to me that they're considering reversing course. If the > > QA team isn't going to continue, we'll need to figure out what > > that means and potentially find another group to do it. > > I wasn't there for the discussion, but it sounds likely to be a > mischaracterization. > I'm going to assume it's not true (or much more > nuanced) at least until someone responds on behalf of the QA team. > This particular subthread is only going to go further into the weeds > until it is grounded in some authoritative details. +1 I'm very confused by this whole thread TBH. Was there a defcore test which was blocked from tempest? Quite frankly the amount of contribution to tempest specifically for defcore tests is very minimal. (at most 1 or 2 patches per cycle) It seems like this whole concern is based on a misunderstanding somewhere and just is going off in a weird direction. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote: [...] > it's news to me that they're considering reversing course. If the > QA team isn't going to continue, we'll need to figure out what > that means and potentially find another group to do it. I wasn't there for the discussion, but it sounds likely to be a mischaracterization. I'm going to assume it's not true (or much more nuanced) at least until someone responds on behalf of the QA team. This particular subthread is only going to go further into the weeds until it is grounded in some authoritative details. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On 31/05/17 09:43, Doug Hellmann wrote: Excerpts from Chris Dent's message of 2017-05-31 11:22:50 +0100: On Wed, 31 May 2017, Graham Hayes wrote: On 30/05/17 19:09, Doug Hellmann wrote: Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100: Note that this goal only applies to tempest _plugins_. Projects which have their tests in the core of tempest have nothing to do. I wonder if it wouldn't be more fair for all projects to use plugins for their tempest tests? All projects may have plugins, but all projects with tests used by the Interop WG (formerly DefCore) for trademark certification must place at least those tests in the tempest repo, to be managed by the QA team [1]. As new projects are added to those trademark programs, the tests are supposed to move to the central repo to ensure the additional review criteria are applied properly. Thanks for the clarification, Doug. I don't think it changes the main thrust of what I was trying to say (more below). [1] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html In the InterOp discussions in Boston, it was indicated that some people on the QA team were not comfortable with "non core" project (even in the InterOp program) having tests in core tempest. I do think that may be a bigger discussion though. I'm not suggesting we change everything (because that would take a lot of time and energy we probably don't have), but I had some thoughts in reaction to this and sharing is caring: The way in which the tempest _repo_ is a combination of smoke, integration, validation and trademark enforcement testing is very confusing to me. If we then lay on top of that the concept of "core" and "not core" with regard to who is supposed to put their tests in a plugin and who isn't (except when it is trademark related!) it all gets quite bewildering. The resolution above says: "the OpenStack community will benefit from having the interoperability tests used by DefCore in a central location". Findability is a good goal so this a reasonable assertion, but then the directive to lump those tests in with a bunch of other stuff seems off if the goal is to "easier to read and understand a set of tests". If, instead, Tempest is a framework and all tests are in plugins that each have their own repo then it is much easier to look for a repo (if there is a common pattern) and know "these are the interop tests for openstack" and "these are the integration tests for nova" and even "these are the integration tests for the thing we are currently describing as 'core'[1]". An area where this probably falls down is with validation. How do you know which plugins to assemble in order to validate this cloud you've just built? Except that we already have this problem now that we are requiring most projects to manage their tempest tests as plugins. Does it become worse by everything being a plugin? [1] We really need a better name for this. Yeah, it sounds like the current organization of the repo is not ideal in terms of equal playing field for all of our project teams. I would be fine with all of the interop tests being in a plugin together, or of saying that the tempest repo should only contain those tests and that others should move to their own plugins. If we're going to reorganize all of that, we should decide what new structure we want and work it into the goal. +1 - ZB The point of centralizing review of that specific set of tests was to make it easier for interop folks to help ensure the tests continue to follow the additionally stringent review criteria that comes with being used as part of the trademark program. The QA team agreed to do that, so it's news to me that they're considering reversing course. If the QA team isn't going to continue, we'll need to figure out what that means and potentially find another group to do it. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
Excerpts from Chris Dent's message of 2017-05-31 11:22:50 +0100: > On Wed, 31 May 2017, Graham Hayes wrote: > > On 30/05/17 19:09, Doug Hellmann wrote: > >> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100: > >>> Note that this goal only applies to tempest _plugins_. Projects > >>> which have their tests in the core of tempest have nothing to do. I > >>> wonder if it wouldn't be more fair for all projects to use plugins > >>> for their tempest tests? > >> > >> All projects may have plugins, but all projects with tests used by > >> the Interop WG (formerly DefCore) for trademark certification must > >> place at least those tests in the tempest repo, to be managed by > >> the QA team [1]. As new projects are added to those trademark > >> programs, the tests are supposed to move to the central repo to > >> ensure the additional review criteria are applied properly. > > Thanks for the clarification, Doug. I don't think it changes the > main thrust of what I was trying to say (more below). > > >> [1] > >> https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html > > > > In the InterOp discussions in Boston, it was indicated that some people > > on the QA team were not comfortable with "non core" project (even in > > the InterOp program) having tests in core tempest. > > > > I do think that may be a bigger discussion though. > > I'm not suggesting we change everything (because that would take a > lot of time and energy we probably don't have), but I had some > thoughts in reaction to this and sharing is caring: > > The way in which the tempest _repo_ is a combination of smoke, > integration, validation and trademark enforcement testing is very > confusing to me. If we then lay on top of that the concept of "core" > and "not core" with regard to who is supposed to put their tests in > a plugin and who isn't (except when it is trademark related!) it all > gets quite bewildering. > > The resolution above says: "the OpenStack community will benefit > from having the interoperability tests used by DefCore in a central > location". Findability is a good goal so this a reasonable > assertion, but then the directive to lump those tests in with a > bunch of other stuff seems off if the goal is to "easier to read and > understand a set of tests". > > If, instead, Tempest is a framework and all tests are in plugins > that each have their own repo then it is much easier to look for a > repo (if there is a common pattern) and know "these are the interop > tests for openstack" and "these are the integration tests for nova" > and even "these are the integration tests for the thing we are > currently describing as 'core'[1]". > > An area where this probably falls down is with validation. How do > you know which plugins to assemble in order to validate this cloud > you've just built? Except that we already have this problem now that > we are requiring most projects to manage their tempest tests as > plugins. Does it become worse by everything being a plugin? > > [1] We really need a better name for this. Yeah, it sounds like the current organization of the repo is not ideal in terms of equal playing field for all of our project teams. I would be fine with all of the interop tests being in a plugin together, or of saying that the tempest repo should only contain those tests and that others should move to their own plugins. If we're going to reorganize all of that, we should decide what new structure we want and work it into the goal. The point of centralizing review of that specific set of tests was to make it easier for interop folks to help ensure the tests continue to follow the additionally stringent review criteria that comes with being used as part of the trademark program. The QA team agreed to do that, so it's news to me that they're considering reversing course. If the QA team isn't going to continue, we'll need to figure out what that means and potentially find another group to do it. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On 31/05/17 11:22, Chris Dent wrote: > On Wed, 31 May 2017, Graham Hayes wrote: >> On 30/05/17 19:09, Doug Hellmann wrote: >>> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100: Note that this goal only applies to tempest _plugins_. Projects which have their tests in the core of tempest have nothing to do. I wonder if it wouldn't be more fair for all projects to use plugins for their tempest tests? >>> >>> All projects may have plugins, but all projects with tests used by >>> the Interop WG (formerly DefCore) for trademark certification must >>> place at least those tests in the tempest repo, to be managed by >>> the QA team [1]. As new projects are added to those trademark >>> programs, the tests are supposed to move to the central repo to >>> ensure the additional review criteria are applied properly. > > Thanks for the clarification, Doug. I don't think it changes the > main thrust of what I was trying to say (more below). > >>> [1] >>> https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html >>> >> >> In the InterOp discussions in Boston, it was indicated that some people >> on the QA team were not comfortable with "non core" project (even in >> the InterOp program) having tests in core tempest. >> >> I do think that may be a bigger discussion though. > > I'm not suggesting we change everything (because that would take a > lot of time and energy we probably don't have), but I had some > thoughts in reaction to this and sharing is caring: > > The way in which the tempest _repo_ is a combination of smoke, > integration, validation and trademark enforcement testing is very > confusing to me. If we then lay on top of that the concept of "core" > and "not core" with regard to who is supposed to put their tests in > a plugin and who isn't (except when it is trademark related!) it all > gets quite bewildering. > > The resolution above says: "the OpenStack community will benefit > from having the interoperability tests used by DefCore in a central > location". Findability is a good goal so this a reasonable > assertion, but then the directive to lump those tests in with a > bunch of other stuff seems off if the goal is to "easier to read and > understand a set of tests". > > If, instead, Tempest is a framework and all tests are in plugins > that each have their own repo then it is much easier to look for a > repo (if there is a common pattern) and know "these are the interop > tests for openstack" and "these are the integration tests for nova" > and even "these are the integration tests for the thing we are > currently describing as 'core'[1]". > > An area where this probably falls down is with validation. How do > you know which plugins to assemble in order to validate this cloud > you've just built? Except that we already have this problem now that > we are requiring most projects to manage their tempest tests as > plugins. Does it become worse by everything being a plugin? No - this was the gist of my point last year when I proposed the plugins first (or plugins for all as I called it at the time). It actually gets better - for a few reasons. 1. We can have a interop-tempest-plugin (under QA control) where all interop tests can go, and be tagged for each standard. This is good for many reasons, mainly that the tests are curated, and projects cannot change tests without someone from QA-core team checking it to make sure it does not break backwards compatibility. 2. With the some interop tests being out of the tempest tree, there is a very real possibility of a change in tempest blocking someone getting certification - tempest does not gate against plugins and has broken interfaces in the past. 3. With the new interop programs, the old definition of "core" is more murky - and it is looking more and more like the criteria for inclusion in openstack/tempest is "be there from the beginning". > [1] We really need a better name for this. 100% - but I have tried and failed to find a better descriptor, that everyone understands. - Graham > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] [tc] [all] more tempest plugins (was Re: [tc] [all] TC Report 22)
On Wed, 31 May 2017, Graham Hayes wrote: On 30/05/17 19:09, Doug Hellmann wrote: Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100: Note that this goal only applies to tempest _plugins_. Projects which have their tests in the core of tempest have nothing to do. I wonder if it wouldn't be more fair for all projects to use plugins for their tempest tests? All projects may have plugins, but all projects with tests used by the Interop WG (formerly DefCore) for trademark certification must place at least those tests in the tempest repo, to be managed by the QA team [1]. As new projects are added to those trademark programs, the tests are supposed to move to the central repo to ensure the additional review criteria are applied properly. Thanks for the clarification, Doug. I don't think it changes the main thrust of what I was trying to say (more below). [1] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html In the InterOp discussions in Boston, it was indicated that some people on the QA team were not comfortable with "non core" project (even in the InterOp program) having tests in core tempest. I do think that may be a bigger discussion though. I'm not suggesting we change everything (because that would take a lot of time and energy we probably don't have), but I had some thoughts in reaction to this and sharing is caring: The way in which the tempest _repo_ is a combination of smoke, integration, validation and trademark enforcement testing is very confusing to me. If we then lay on top of that the concept of "core" and "not core" with regard to who is supposed to put their tests in a plugin and who isn't (except when it is trademark related!) it all gets quite bewildering. The resolution above says: "the OpenStack community will benefit from having the interoperability tests used by DefCore in a central location". Findability is a good goal so this a reasonable assertion, but then the directive to lump those tests in with a bunch of other stuff seems off if the goal is to "easier to read and understand a set of tests". If, instead, Tempest is a framework and all tests are in plugins that each have their own repo then it is much easier to look for a repo (if there is a common pattern) and know "these are the interop tests for openstack" and "these are the integration tests for nova" and even "these are the integration tests for the thing we are currently describing as 'core'[1]". An area where this probably falls down is with validation. How do you know which plugins to assemble in order to validate this cloud you've just built? Except that we already have this problem now that we are requiring most projects to manage their tempest tests as plugins. Does it become worse by everything being a plugin? [1] We really need a better name for this. -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
