Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Fri, 29 Jun 2018 00:05:09 +0900 Dmitry Tantsur wrote > On 06/27/2018 03:17 AM, Ghanshyam Mann wrote: > > > > > > > > On Tue, 26 Jun 2018 23:12:30 +0900 Doug Hellmann > > wrote > > > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 > > +0100: > > > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > > wrote: > > > > > > > > > > > > > Dmitry Tantsur wrote: > > > > > > > > [...] > > > > > > > > My suggestion: tempest has to be compatible with all > > supported releases > > > > > > > > (of both services and plugins) OR be branched. > > > > > > > > [...] > > > > > > > I tend to agree with Dmitry... We have a model for things that > > need > > > > > > > release alignment, and that's the cycle-bound series. The > > reason tempest > > > > > > > is branchless was because there was no compatibility issue. If > > the split > > > > > > > of tempest plugins introduces a potential incompatibility, > > then I would > > > > > > > prefer aligning tempest to the existing model rather than > > introduce a > > > > > > > parallel tempest-specific cycle just so that tempest can stay > > > > > > > release-independent... > > > > > > > > > > > > > > I seem to remember there were drawbacks in branching tempest, > > though... > > > > > > > Can someone with functioning memory brain cells summarize them > > again ? > > > > > > > > > > > > > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > > > > > > > I'm sorry, but I'm having a hard time taking this statement > > seriously > > > > > when the current source of tension is that the Tempest API itself > > > > > is breaking for its plugins. > > > > > > > > > > Maybe rather than talking about how to release compatible things > > > > > together, we should go back and talk about why Tempest's API is > > changing > > > > > in a way that can't be made backwards-compatible. Can you give > > some more > > > > > detail about that? > > > > > > > > > > > > > Well it's not, if it did that would violate all the stability > > guarantees > > > > provided by Tempest's library and plugin interface. I've not ever > > heard of > > > > these kind of backwards incompatibilities in those interfaces and we > > go to > > > > all effort to make sure we don't break them. Where did the idea that > > > > backwards incompatible changes where being introduced come from? > > > > > > In his original post, gmann said, "There might be some changes in > > > Tempest which might not work with older version of Tempest Plugins." > > > I was surprised to hear that, but I'm not sure how else to interpret > > > that statement. > > > > I did not mean to say that Tempest will introduce the changes in backward > > incompatible way which can break plugins. That cannot happen as all > > plugins and tempest are branchless and they are being tested with master > > Tempest so if we change anything backward incompatible then it break the > > plugins gate. Even we have to remove any deprecated interfaces from > > Tempest, we fix all plugins first like - > > https://review.openstack.org/#/q/topic:remove-support-of-cinder-v1-api+(status:open+OR+status:merged) > > > > What I mean to say here is that adding new or removing deprecated > > interface in Tempest might not work with all released version or > > unreleased Plugins. That point is from point of view of using Tempest and > > Plugins in production cloud testing not gate(where we keep the > > compatibility). Production Cloud user use Tempest cycle based version. > > Pike based Cloud will be tested by Tempest 17.0.0 not latest version > > (though latest version might work). > > > > This thread is not just for gate testing point of view (which seems to be > > always interpreted), this is more for user using Tempest and Plugins for > > their cloud testing. I am looping operator mail list also which i forgot > > in initial post. > > > > We do not have any tag/release from plugins to know what version of plugin > > can work with what version of tempest. For Example If There is new > > interface introduced by Tempest 19.0.0 and pluginX start using it. Now it > > can create issues for pluginX in both release model 1. plugins with no > > release (I will call this PluginNR), 2. plugins with independent release > > (I will call it PluginIR). > > > > Users (Not Gate) will face below issues: > > - User cannot use PluginNR with Tempest <19.0.0 (where that new interface > > was not present). And there is no PluginNR release/tag as this is > > unreleased and not branched software. > > - User cannot find a PluginIR particular tag/r
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Excerpts from Dmitry Tantsur's message of 2018-06-28 17:05:09 +0200: > On 06/27/2018 03:17 AM, Ghanshyam Mann wrote: > > Users (Not Gate) will face below issues: > > - User cannot use PluginNR with Tempest <19.0.0 (where that new interface > > was not present). And there is no PluginNR release/tag as this is > > unreleased and not branched software. > > - User cannot find a PluginIR particular tag/release which can work with > > tempest <19.0.0 (where that new interface was not present). Only way for > > user to make it work is to manually find out the PluginIR tag/commit before > > PluginIR started consuming the new interface. > > In these discussions I always think: how is it solved outside of the > openstack > world. And the solutions seem to be: > 1. for PluginNR - do releases > 2. for PluginIR - declare their minimum version of tempest in requirements.txt > > Why isn't it sufficient for us? It is. We just haven't been doing it; in part I think because most developers interact with the plugins via the CI system and don't realize they are also "libraries" that need to be released so that refstack users can install them. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On 06/27/2018 03:17 AM, Ghanshyam Mann wrote: On Tue, 26 Jun 2018 23:12:30 +0900 Doug Hellmann wrote > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, wrote: > > > > > > > > > Dmitry Tantsur wrote: > > > > > > [...] > > > > > > My suggestion: tempest has to be compatible with all supported releases > > > > > > (of both services and plugins) OR be branched. > > > > > > [...] > > > > > I tend to agree with Dmitry... We have a model for things that need > > > > > release alignment, and that's the cycle-bound series. The reason tempest > > > > > is branchless was because there was no compatibility issue. If the split > > > > > of tempest plugins introduces a potential incompatibility, then I would > > > > > prefer aligning tempest to the existing model rather than introduce a > > > > > parallel tempest-specific cycle just so that tempest can stay > > > > > release-independent... > > > > > > > > > > I seem to remember there were drawbacks in branching tempest, though... > > > > > Can someone with functioning memory brain cells summarize them again ? > > > > > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > > > I'm sorry, but I'm having a hard time taking this statement seriously > > > when the current source of tension is that the Tempest API itself > > > is breaking for its plugins. > > > > > > Maybe rather than talking about how to release compatible things > > > together, we should go back and talk about why Tempest's API is changing > > > in a way that can't be made backwards-compatible. Can you give some more > > > detail about that? > > > > > > > Well it's not, if it did that would violate all the stability guarantees > > provided by Tempest's library and plugin interface. I've not ever heard of > > these kind of backwards incompatibilities in those interfaces and we go to > > all effort to make sure we don't break them. Where did the idea that > > backwards incompatible changes where being introduced come from? > > In his original post, gmann said, "There might be some changes in > Tempest which might not work with older version of Tempest Plugins." > I was surprised to hear that, but I'm not sure how else to interpret > that statement. I did not mean to say that Tempest will introduce the changes in backward incompatible way which can break plugins. That cannot happen as all plugins and tempest are branchless and they are being tested with master Tempest so if we change anything backward incompatible then it break the plugins gate. Even we have to remove any deprecated interfaces from Tempest, we fix all plugins first like - https://review.openstack.org/#/q/topic:remove-support-of-cinder-v1-api+(status:open+OR+status:merged) What I mean to say here is that adding new or removing deprecated interface in Tempest might not work with all released version or unreleased Plugins. That point is from point of view of using Tempest and Plugins in production cloud testing not gate(where we keep the compatibility). Production Cloud user use Tempest cycle based version. Pike based Cloud will be tested by Tempest 17.0.0 not latest version (though latest version might work). This thread is not just for gate testing point of view (which seems to be always interpreted), this is more for user using Tempest and Plugins for their cloud testing. I am looping operator mail list also which i forgot in initial post. We do not have any tag/release from plugins to know what version of plugin can work with what version of tempest. For Example If There is new interface introduced by Tempest 19.0.0 and pluginX start using it. Now it can create issues for pluginX in both release model 1. plugins with no release (I will call this PluginNR), 2. plugins with independent release (I will call it PluginIR). Users (Not Gate) will face below issues: - User cannot use PluginNR with Tempest <19.0.0 (where that new interface was not present). And there is no PluginNR release/tag as this is unreleased and not branched software. - User cannot find a PluginIR particular tag/release which can work with tempest <19.0.0 (where that new interface was not present). Only way for user to make it work is to manually find out the PluginIR tag/commit before PluginIR started consuming the new interface. In these discussions I always think: how is it solved outside of the openstack world. And the solutions seem to be: 1. for PluginNR - do releases 2. for PluginIR - declare their minimum version of tempest in requirements.txt Why isn't it sufficient for us? Dmitry Let me make it more clear via diagram:
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Thu, 28 Jun 2018 04:08:35 +0900 Sean McGinnis wrote > > > > There is no issue of backward incompatibility from Tempest and on Gate. > > GATE > > is always good as it is going with mater version or minimum supported > > version > > in plugins as you mentioned. We take care of all these things you mentioned > > which is our main goal also. > > > > But If we think from Cloud tester perspective where they use older version > > of > > tempest for particular OpenStack release but there is no corresponding > > tag/version from plugins to use them for that OpenStack release. > > > > Idea is here to have a tag from Plugins also like Tempest does currently > > for > > each OpenStack release so that user can pickup those tag and test their > > Complete Cloud. > > > > Thanks for the further explanation Ghanshyam. So it's not so much that newer > versions of tempest may break the current repo plugins, it's more to the fact > that any random plugin that gets pulled in has no way of knowing if it can > take > advantage of a potentially older version of tempest that had not yet > introduced > something the plugin is relying on. > > I think it makes sense for the tempest plugins to be following the > cycle-with-intermediary model. This would allow plugins to be released at any > point during a given cycle and would then have a way to match up a "release" > of > the plugin. > > Release repo deliverable placeholders are being proposed for all the tempest > plugin repos we could find. Thanks to Doug for pulling this all together: > > https://review.openstack.org/#/c/578141/ > > Please comment there if you see any issues. Thanks. That's correct understanding and goal of this thread which is from production cloud testing point of view not just *gate*. cycle-with-intermediary model fulfill the user's requirement which they asked in summit. Doug patch lgtm. -gmann > > Sean > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
> > There is no issue of backward incompatibility from Tempest and on Gate. GATE > is always good as it is going with mater version or minimum supported version > in plugins as you mentioned. We take care of all these things you mentioned > which is our main goal also. > > But If we think from Cloud tester perspective where they use older version of > tempest for particular OpenStack release but there is no corresponding > tag/version from plugins to use them for that OpenStack release. > > Idea is here to have a tag from Plugins also like Tempest does currently for > each OpenStack release so that user can pickup those tag and test their > Complete Cloud. > Thanks for the further explanation Ghanshyam. So it's not so much that newer versions of tempest may break the current repo plugins, it's more to the fact that any random plugin that gets pulled in has no way of knowing if it can take advantage of a potentially older version of tempest that had not yet introduced something the plugin is relying on. I think it makes sense for the tempest plugins to be following the cycle-with-intermediary model. This would allow plugins to be released at any point during a given cycle and would then have a way to match up a "release" of the plugin. Release repo deliverable placeholders are being proposed for all the tempest plugin repos we could find. Thanks to Doug for pulling this all together: https://review.openstack.org/#/c/578141/ Please comment there if you see any issues. Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018 19:32:59 +0900 Mehdi Abaakouk wrote > Hi, > > I have never understood the branchless tempest thing. Making Tempest > release is a great news for me. > > But about plugins... Tempest already provides a API for plugins. If you > are going to break this API, why not using stable branches and > deprecation process like any other software ? > > If you do that, plugin will be informed that Tempest will soon do a > breaking change. Their can update their plugin code and raise the > minimal tempest version required to work. > > Their can do that when they have times, and not because Tempest want to > release a version soon. > > Also the stable branch/deprecation process is well known by the > whole community. There is no issue of backward incompatibility from Tempest and on Gate. GATE is always good as it is going with mater version or minimum supported version in plugins as you mentioned. We take care of all these things you mentioned which is our main goal also. But If we think from Cloud tester perspective where they use older version of tempest for particular OpenStack release but there is no corresponding tag/version from plugins to use them for that OpenStack release. Idea is here to have a tag from Plugins also like Tempest does currently for each OpenStack release so that user can pickup those tag and test their Complete Cloud. -gmann > > And this will also allow them to release a version when their want. > > So I support making release of Tempest and Plugins, but do not support > a coordinated release. > > Regards, > > On Tue, Jun 26, 2018 at 06:18:52PM +0900, Ghanshyam Mann wrote: > >Hello Everyone, > > > >In Queens cycle, community goal to split the Tempest Plugin has been > >completed [1] and i think almost all the projects have separate repo for > >tempest plugin [2]. Which means each tempest plugins are being separated > >from their project release model. Few projects have started the > >independent release model for their plugins like kuryr-tempest-plugin, > >ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > >planning as chatted with amotoki. > > > >There might be some changes in Tempest which might not work with older > >version of Tempest Plugins. For example, If I am testing any production > >cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i > >will be using Tempest and Aodh's , Congress's Tempest plugins. With > >Independent release model of each Tempest Plugins, there might be chance > >that the Aodh's or Congress's Tempest plugin versions are not compatible > >with latest/known Tempest versions. It will become hard to find the > >compatible tag/release of Tempest and Tempest Plugins or in some cases i > >might need to patch up the things. > > > >During QA feedback sessions at Vancouver Summit, there was feedback to > >coordinating the release of all Tempest plugins and Tempest [4] (also > >amotoki talked to me on this as neutron-tempest-plugin is planning their > >first release). Idea is to release/tag all the Tempest plugins and Tempest > >together so that specific release/tag can be identified as compatible > >version of all the Plugins and Tempest for testing the complete stack. That > >way user can get to know what version of Tempest Plugins is compatible with > >what version of Tempest. > > > >For above use case, we need some coordinated release model among Tempest > >and all the Tempest Plugins. There should be single release of all Tempest > >Plugins with well defined tag whenever any Tempest release is happening. > >For Example, Tempest version 19.0.0 is to mark the "support of the Rocky > >release". When releasing the Tempest 19.0, we will release all the Tempest > >plugins also to tag the compatibility of plugins with Tempest for "support > >of the Rocky release". > > > >One way to make this coordinated release (just a initial thought): > >1. Release Each Tempest Plugins whenever there is any major version release > >of Tempest (like marking the support of OpenStack release in Tempest, EOL > >of OpenStack release in Tempest) > >1.1 Each plugin or Tempest can do their intermediate release of minor > > version change which are in backward compatible way. > >1.2 This coordinated Release can be started from latest Tempest Version > > for simple reading. Like if we start this coordinated release from > > Tempest version 19.0.0 then, > >each plugins will be released as 19.0.0 and so on. > > > >Giving the above background and use case of this coordinated release, > >A) I would like to ask each plugins owner if you are agree on this > >coordinated release? If no, please give more feedback or issue we can face > >due to this coordinated release. > > > > > >If we get the agreement from all Plugins t
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018 19:12:33 +0900 Dmitry Tantsur wrote > On 06/26/2018 11:57 AM, Ghanshyam Mann wrote: > > > > > > > > On Tue, 26 Jun 2018 18:37:42 +0900 Dmitry Tantsur > > wrote > > > On 06/26/2018 11:18 AM, Ghanshyam Mann wrote: > > > > Hello Everyone, > > > > > > > > In Queens cycle, community goal to split the Tempest Plugin has > > been completed [1] and i think almost all the projects have separate repo > > for tempest plugin [2]. Which means each tempest plugins are being > > separated from their project release model. Few projects have started the > > independent release model for their plugins like kuryr-tempest-plugin, > > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > > planning as chatted with amotoki. > > > > > > > > There might be some changes in Tempest which might not work with > > older version of Tempest Plugins. For example, If I am testing any > > production cloud which has Nova, Neutron, Cinder, Keystone , Aodh, > > Congress etc i will be using Tempest and Aodh's , Congress's Tempest > > plugins. With Independent release model of each Tempest Plugins, there > > might be chance that the Aodh's or Congress's Tempest plugin versions are > > not compatible with latest/known Tempest versions. It will become hard to > > find the compatible tag/release of Tempest and Tempest Plugins or in some > > cases i might need to patch up the things. > > > > > > FWIW this is solved by stable branches for all other projects. If we > > cannot keep > > > Tempest compatible with all supported branches, we should back off our > > decision > > > to make it branchless. The very nature of being branchless implies > > being > > > compatible with all supported releases. > > > > > > > > > > > During QA feedback sessions at Vancouver Summit, there was feedback > > to coordinating the release of all Tempest plugins and Tempest [4] (also > > amotoki talked to me on this as neutron-tempest-plugin is planning their > > first release). Idea is to release/tag all the Tempest plugins and Tempest > > together so that specific release/tag can be identified as compatible > > version of all the Plugins and Tempest for testing the complete stack. > > That way user can get to know what version of Tempest Plugins is > > compatible with what version of Tempest. > > > > > > > > For above use case, we need some coordinated release model among > > Tempest and all the Tempest Plugins. There should be single release of all > > Tempest Plugins with well defined tag whenever any Tempest release is > > happening. For Example, Tempest version 19.0.0 is to mark the "support of > > the Rocky release". When releasing the Tempest 19.0, we will release all > > the Tempest plugins also to tag the compatibility of plugins with Tempest > > for "support of the Rocky release". > > > > > > > > One way to make this coordinated release (just a initial thought): > > > > 1. Release Each Tempest Plugins whenever there is any major version > > release of Tempest (like marking the support of OpenStack release in > > Tempest, EOL of OpenStack release in Tempest) > > > > 1.1 Each plugin or Tempest can do their intermediate release of > > minor version change which are in backward compatible way. > > > > 1.2 This coordinated Release can be started from latest Tempest > > Version for simple reading. Like if we start this coordinated release > > from Tempest version 19.0.0 then, > > > > each plugins will be released as 19.0.0 and so on. > > > > > > > > Giving the above background and use case of this coordinated release, > > > > A) I would like to ask each plugins owner if you are agree on this > > coordinated release? If no, please give more feedback or issue we can > > face due to this coordinated release. > > > > > > Disclaimer: I'm not the PTL. > > > > > > Similarly to Luigi, I don't feel well about forcing a plugin release > > at the same > > > time as a tempest release, UNLESS tempest folks are going to > > coordinate their > > > releases with all how-many-do-we-have plugins. What I'd like to avoid > > is cutting > > > a release in the middle of a patch chain or some refactoring just > > because > > > tempest happened to have a release right now. > > > > I understand your point. But we can avoid that case if we only coordinate > > on major version bump only. as i mentioned in 1.2 point, Tempest and > > Tempest plugins can do their intermediate release anytime which are > > nothing but backward compatible release. In this proposed model, we can do > > a coordinated release for major version bump only which is happening only > > on OpenStack release and EOL of any stable branch. > > Even bigger concern: what if the plugin is actually not compatible yet? Say, > you're re
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
++ operator ML On Wed, 27 Jun 2018 10:17:33 +0900 Ghanshyam Mann wrote > > > > On Tue, 26 Jun 2018 23:12:30 +0900 Doug Hellmann > wrote > > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > wrote: > > > > > > > > > > > Dmitry Tantsur wrote: > > > > > > > [...] > > > > > > > My suggestion: tempest has to be compatible with all supported > releases > > > > > > > (of both services and plugins) OR be branched. > > > > > > > [...] > > > > > > I tend to agree with Dmitry... We have a model for things that > need > > > > > > release alignment, and that's the cycle-bound series. The reason > tempest > > > > > > is branchless was because there was no compatibility issue. If > the split > > > > > > of tempest plugins introduces a potential incompatibility, then I > would > > > > > > prefer aligning tempest to the existing model rather than > introduce a > > > > > > parallel tempest-specific cycle just so that tempest can stay > > > > > > release-independent... > > > > > > > > > > > > I seem to remember there were drawbacks in branching tempest, > though... > > > > > > Can someone with functioning memory brain cells summarize them > again ? > > > > > > > > > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > > > > > I'm sorry, but I'm having a hard time taking this statement seriously > > > > when the current source of tension is that the Tempest API itself > > > > is breaking for its plugins. > > > > > > > > Maybe rather than talking about how to release compatible things > > > > together, we should go back and talk about why Tempest's API is > changing > > > > in a way that can't be made backwards-compatible. Can you give some > more > > > > detail about that? > > > > > > > > > > Well it's not, if it did that would violate all the stability > guarantees > > > provided by Tempest's library and plugin interface. I've not ever heard > of > > > these kind of backwards incompatibilities in those interfaces and we go > to > > > all effort to make sure we don't break them. Where did the idea that > > > backwards incompatible changes where being introduced come from? > > > > In his original post, gmann said, "There might be some changes in > > Tempest which might not work with older version of Tempest Plugins." > > I was surprised to hear that, but I'm not sure how else to interpret > > that statement. > > I did not mean to say that Tempest will introduce the changes in backward > incompatible way which can break plugins. That cannot happen as all plugins > and tempest are branchless and they are being tested with master Tempest so > if we change anything backward incompatible then it break the plugins gate. > Even we have to remove any deprecated interfaces from Tempest, we fix all > plugins first like - > https://review.openstack.org/#/q/topic:remove-support-of-cinder-v1-api+(status:open+OR+status:merged) > > > What I mean to say here is that adding new or removing deprecated interface > in Tempest might not work with all released version or unreleased Plugins. > That point is from point of view of using Tempest and Plugins in production > cloud testing not gate(where we keep the compatibility). Production Cloud > user use Tempest cycle based version. Pike based Cloud will be tested by > Tempest 17.0.0 not latest version (though latest version might work). > > This thread is not just for gate testing point of view (which seems to be > always interpreted), this is more for user using Tempest and Plugins for > their cloud testing. I am looping operator mail list also which i forgot in > initial post. > > We do not have any tag/release from plugins to know what version of plugin > can work with what version of tempest. For Example If There is new interface > introduced by Tempest 19.0.0 and pluginX start using it. Now it can create > issues for pluginX in both release model 1. plugins with no release (I will > call this PluginNR), 2. plugins with independent release (I will call it > PluginIR). > > Users (Not Gate) will face below issues: > - User cannot use PluginNR with Tempest <19.0.0 (where that new interface > was not present). And there is no PluginNR release/tag as this is unreleased > and not branched software. > - User cannot find a PluginIR particular tag/release which can work with > tempest <19.0.0 (where that new interface was not present). Only way for > user to make it work is to manually find out the PluginIR tag/commit before > PluginIR started consuming t
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018 23:12:30 +0900 Doug Hellmann wrote > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > > > > wrote: > > > > > > > > > Dmitry Tantsur wrote: > > > > > > [...] > > > > > > My suggestion: tempest has to be compatible with all supported > > > > > > releases > > > > > > (of both services and plugins) OR be branched. > > > > > > [...] > > > > > I tend to agree with Dmitry... We have a model for things that need > > > > > release alignment, and that's the cycle-bound series. The reason > > > > > tempest > > > > > is branchless was because there was no compatibility issue. If the > > > > > split > > > > > of tempest plugins introduces a potential incompatibility, then I > > > > > would > > > > > prefer aligning tempest to the existing model rather than introduce a > > > > > parallel tempest-specific cycle just so that tempest can stay > > > > > release-independent... > > > > > > > > > > I seem to remember there were drawbacks in branching tempest, > > > > > though... > > > > > Can someone with functioning memory brain cells summarize them again > > > > > ? > > > > > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > > > I'm sorry, but I'm having a hard time taking this statement seriously > > > when the current source of tension is that the Tempest API itself > > > is breaking for its plugins. > > > > > > Maybe rather than talking about how to release compatible things > > > together, we should go back and talk about why Tempest's API is changing > > > in a way that can't be made backwards-compatible. Can you give some more > > > detail about that? > > > > > > > Well it's not, if it did that would violate all the stability guarantees > > provided by Tempest's library and plugin interface. I've not ever heard of > > these kind of backwards incompatibilities in those interfaces and we go to > > all effort to make sure we don't break them. Where did the idea that > > backwards incompatible changes where being introduced come from? > > In his original post, gmann said, "There might be some changes in > Tempest which might not work with older version of Tempest Plugins." > I was surprised to hear that, but I'm not sure how else to interpret > that statement. I did not mean to say that Tempest will introduce the changes in backward incompatible way which can break plugins. That cannot happen as all plugins and tempest are branchless and they are being tested with master Tempest so if we change anything backward incompatible then it break the plugins gate. Even we have to remove any deprecated interfaces from Tempest, we fix all plugins first like - https://review.openstack.org/#/q/topic:remove-support-of-cinder-v1-api+(status:open+OR+status:merged) What I mean to say here is that adding new or removing deprecated interface in Tempest might not work with all released version or unreleased Plugins. That point is from point of view of using Tempest and Plugins in production cloud testing not gate(where we keep the compatibility). Production Cloud user use Tempest cycle based version. Pike based Cloud will be tested by Tempest 17.0.0 not latest version (though latest version might work). This thread is not just for gate testing point of view (which seems to be always interpreted), this is more for user using Tempest and Plugins for their cloud testing. I am looping operator mail list also which i forgot in initial post. We do not have any tag/release from plugins to know what version of plugin can work with what version of tempest. For Example If There is new interface introduced by Tempest 19.0.0 and pluginX start using it. Now it can create issues for pluginX in both release model 1. plugins with no release (I will call this PluginNR), 2. plugins with independent release (I will call it PluginIR). Users (Not Gate) will face below issues: - User cannot use PluginNR with Tempest <19.0.0 (where that new interface was not present). And there is no PluginNR release/tag as this is unreleased and not branched software. - User cannot find a PluginIR particular tag/release which can work with tempest <19.0.0 (where that new interface was not present). Only way for user to make it work is to manually find out the PluginIR tag/commit before PluginIR started consuming the new interface. Let me make it more clear via diagram: PluginNR PluginIR Tempe
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Excerpts from Doug Hellmann's message of 2018-06-26 11:19:05 -0400: > 29 out of 40 repos that have "tempest" in the name have not been > tagged via the releases repo. Not all of those are plugins. Here's > a list: > > $ for repo in $(grep openstack/ reference/projects.yaml | grep tempest | > cut -f2- -d- | cut -f2 -d' ') ; do (echo -n $repo; cd ../releases/; if > grep -q $repo deliverables/*/*.yaml ; then echo ' FOUND'; else > echo ' NOT FOUND'; fi); done > > openstack/barbican-tempest-plugin NOT FOUND > openstack/blazar-tempest-plugin NOT FOUND > openstack/cinder-tempest-plugin NOT FOUND > openstack/cloudkitty-tempest-plugin NOT FOUND > openstack/congress-tempest-plugin NOT FOUND > openstack/designate-tempest-plugin FOUND > openstack/ec2api-tempest-plugin NOT FOUND > openstack/freezer-tempest-plugin NOT FOUND > openstack/heat-tempest-plugin NOT FOUND > openstack/tempest-horizon NOT FOUND > openstack/ironic-tempest-plugin FOUND > openstack/networking-generic-switch-tempest-plugin NOT FOUND > openstack/keystone-tempest-plugin NOT FOUND > openstack/kuryr-tempest-plugin FOUND > openstack/magnum-tempest-plugin NOT FOUND > openstack/manila-tempest-plugin NOT FOUND > openstack/mistral-tempest-plugin NOT FOUND > openstack/monasca-tempest-plugin FOUND > openstack/murano-tempest-plugin NOT FOUND > openstack/neutron-tempest-plugin NOT FOUND > openstack/octavia-tempest-plugin NOT FOUND > openstack/charm-tempest NOT FOUND > openstack/openstack-ansible-os_tempest FOUND > openstack/puppet-tempest FOUND > openstack/tempest FOUND > openstack/tempest-plugin-cookiecutter NOT FOUND > openstack/tempest-lib NOT FOUND > openstack/tempest-stress NOT FOUND > openstack/python-tempestconf FOUND > openstack/senlin-tempest-plugin NOT FOUND > openstack/solum-tempest-plugin NOT FOUND > openstack/telemetry-tempest-plugin NOT FOUND > openstack/tempest-tripleo-ui NOT FOUND > openstack/tripleo-common-tempest-plugin NOT FOUND > openstack/trove-tempest-plugin NOT FOUND > openstack/vitrage-tempest-plugin FOUND > openstack/watcher-tempest-plugin NOT FOUND > openstack/oswin-tempest-plugin FOUND > openstack/zaqar-tempest-plugin NOT FOUND > openstack/zun-tempest-plugin FOUND I have proposed https://review.openstack.org/578141 to update the deliverable files for rocky to include the plugins. I left out openstack/ec2api-tempest-plugin because it doesn't look like ec2-api has been tagged in quite a while. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Excerpts from Matthew Treinish's message of 2018-06-26 10:37:54 -0400: > On Tue, Jun 26, 2018 at 10:12:30AM -0400, Doug Hellmann wrote: > > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > > > > > wrote: > > > > > > > > > > > Dmitry Tantsur wrote: > > > As for this whole thread I don't understand any of the points being > > > brought up > > > in the original post or any of the follow ons, things seem to have been > > > confused > > > from the start. The ask from users at the summit was simple. When a new > > > OpenStack > > > release is pushed we push a tempest release to mark that (the next one > > > will be > > > 19.0.0 to mark Rocky). Users were complaining that many plugins don't > > > have a > > > corresponding version to mark support for a new release. So when trying > > > to run > > > against a rocky cloud you get tempest 19.0.0 and then a bunch of plugins > > > for > > > various services at different sha1s which have to be manually looked up > > > based > > > on dates. All users wanted at the summit was a tag for plugins like > > > tempest > > > does with the first number in: > > > > > > https://docs.openstack.org/tempest/latest/overview.html#release-versioning > > > > > > which didn't seem like a bad idea to me. I'm not sure the best mechanism > > > to > > > accomplish this, because I agree with much of what plugin maintainers were > > > saying on the thread about wanting to control their own releases. But the > > > desire to make sure users have a tag they can pull for the addition or > > > removal of a supported release makes sense as something a plugin should > > > do. > > > > We don't coordinate versions across projects anywhere else, for a > > bunch of reasons including the complexity of coordinating the details > > and the confusion it causes when the first version of something is > > 19.0.0. Instead, we list the compatible versions of everything > > together on a series-specific page on releases.o.o. That seems to > > be enough to help anyone wanting to know which versions of tools > > work together. The data is also available in YAML files, so it's easy > > enough to consume by automation. > > > > Would that work for tempest and it's plugins, too? > > That is exactly what I had in mind. I wasn't advocating all plugins use the > same > version number for releases, for the same reasons we don't do that for service > projects anymore. Just that there is a release for a plugin when they add > support for a new service release so that users can know which version to > install. > > > Is the problem that the versions are not the same, or that some of the > > plugins are not being tagged at all? > > > > While I don't pay attention to all the plugins, the impression I got was that > it was the latter and some plugins aren't pushing releases at all. Or if they > are there is no clear version to use for a specific openstack release. OK, that should be easy enough to work out a solution to. The release team can remind project teams to tag their tempest plugin(s) when they prepare their other releases at the end of the cycle, for example. 29 out of 40 repos that have "tempest" in the name have not been tagged via the releases repo. Not all of those are plugins. Here's a list: $ for repo in $(grep openstack/ reference/projects.yaml | grep tempest | cut -f2- -d- | cut -f2 -d' ') ; do (echo -n $repo; cd ../releases/; if grep -q $repo deliverables/*/*.yaml ; then echo ' FOUND'; else echo ' NOT FOUND'; fi); done openstack/barbican-tempest-plugin NOT FOUND openstack/blazar-tempest-plugin NOT FOUND openstack/cinder-tempest-plugin NOT FOUND openstack/cloudkitty-tempest-plugin NOT FOUND openstack/congress-tempest-plugin NOT FOUND openstack/designate-tempest-plugin FOUND openstack/ec2api-tempest-plugin NOT FOUND openstack/freezer-tempest-plugin NOT FOUND openstack/heat-tempest-plugin NOT FOUND openstack/tempest-horizon NOT FOUND openstack/ironic-tempest-plugin FOUND openstack/networking-generic-switch-tempest-plugin NOT FOUND openstack/keystone-tempest-plugin NOT FOUND openstack/kuryr-tempest-plugin FOUND openstack/magnum-tempest-plugin NOT FOUND openstack/manila-tempest-plugin NOT FOUND openstack/mistral-tempest-plugin NOT FOUND openstack/monasca-tempest-plugin FOUND openstack/murano-tempest-plugin NOT FOUND openstack/neutron-tempest-plugin NOT FOUND openstack/octavia-tempest-plugin NOT FOUND openstack/charm-tempest NOT FOUND openstack/openstack-ansible-os_tempest FOUND openstack/puppet-tempest FOUND openstack/tempest FOUND openstack/tempest-plugin-cookiecutter NOT FOUND openstack/tempest-lib NOT FOUND openstack/tempest-stress NOT FOUND openstack/python-tempestconf FOUND openstack/senlin-tempest-plugin NOT FOUND openstack/solum-tempest-plugin NOT FOUND openstack/
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, Jun 26, 2018 at 10:12:30AM -0400, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > > > > wrote: > > > > > > > > > Dmitry Tantsur wrote: > > > > > > [...] > > > > > > My suggestion: tempest has to be compatible with all supported > > > > > > releases > > > > > > (of both services and plugins) OR be branched. > > > > > > [...] > > > > > I tend to agree with Dmitry... We have a model for things that need > > > > > release alignment, and that's the cycle-bound series. The reason > > > > > tempest > > > > > is branchless was because there was no compatibility issue. If the > > > > > split > > > > > of tempest plugins introduces a potential incompatibility, then I > > > > > would > > > > > prefer aligning tempest to the existing model rather than introduce a > > > > > parallel tempest-specific cycle just so that tempest can stay > > > > > release-independent... > > > > > > > > > > I seem to remember there were drawbacks in branching tempest, > > > > > though... > > > > > Can someone with functioning memory brain cells summarize them again ? > > > > > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > > > I'm sorry, but I'm having a hard time taking this statement seriously > > > when the current source of tension is that the Tempest API itself > > > is breaking for its plugins. > > > > > > Maybe rather than talking about how to release compatible things > > > together, we should go back and talk about why Tempest's API is changing > > > in a way that can't be made backwards-compatible. Can you give some more > > > detail about that? > > > > > > > Well it's not, if it did that would violate all the stability guarantees > > provided by Tempest's library and plugin interface. I've not ever heard of > > these kind of backwards incompatibilities in those interfaces and we go to > > all effort to make sure we don't break them. Where did the idea that > > backwards incompatible changes where being introduced come from? > > In his original post, gmann said, "There might be some changes in > Tempest which might not work with older version of Tempest Plugins." > I was surprised to hear that, but I'm not sure how else to interpret > that statement. I have no idea what he means here either. If we went off and broke plugins using a defined stable interface with changes on master we would breaking all the stability guarantees Tempest provides on those interfaces. That's not something we do, and have review processes and testing to prevent. The only thing I can think of is removal of an interface, but that is pretty rare and when we do we go through the standard deprecation procedure when we do that. > > > As for this whole thread I don't understand any of the points being brought > > up > > in the original post or any of the follow ons, things seem to have been > > confused > > from the start. The ask from users at the summit was simple. When a new > > OpenStack > > release is pushed we push a tempest release to mark that (the next one will > > be > > 19.0.0 to mark Rocky). Users were complaining that many plugins don't have a > > corresponding version to mark support for a new release. So when trying to > > run > > against a rocky cloud you get tempest 19.0.0 and then a bunch of plugins for > > various services at different sha1s which have to be manually looked up > > based > > on dates. All users wanted at the summit was a tag for plugins like tempest > > does with the first number in: > > > > https://docs.openstack.org/tempest/latest/overview.html#release-versioning > > > > which didn't seem like a bad idea to me. I'm not sure the best mechanism to > > accomplish this, because I agree with much of what plugin maintainers were > > saying on the thread about wanting to control their own releases. But the > > desire to make sure users have a tag they can pull for the addition or > > removal of a supported release makes sense as something a plugin should do. > > We don't coordinate versions across projects anywhere else, for a > bunch of reasons including the complexity of coordinating the details > and the confusion it causes when the first version of something is > 19.0.0. Instead, we list the compatible versions of everything > together on a series-specific page on releases.o.o. That seems to > be enough to help anyone wanting to know which versions of tools > work together. The data is also available in YAML files, so it's easy > enough to consume by automation. > > Would that work for tempest and it's plugins, too? That is exactly what I had in mind. I wasn't advocating all plugins use the same version number for releases, for the same reasons we don't do that for service projects anymore. Just that there i
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Excerpts from Matthew Treinish's message of 2018-06-26 09:52:09 -0400: > On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, > > > wrote: > > > > > > > Dmitry Tantsur wrote: > > > > > [...] > > > > > My suggestion: tempest has to be compatible with all supported > > > > > releases > > > > > (of both services and plugins) OR be branched. > > > > > [...] > > > > I tend to agree with Dmitry... We have a model for things that need > > > > release alignment, and that's the cycle-bound series. The reason tempest > > > > is branchless was because there was no compatibility issue. If the split > > > > of tempest plugins introduces a potential incompatibility, then I would > > > > prefer aligning tempest to the existing model rather than introduce a > > > > parallel tempest-specific cycle just so that tempest can stay > > > > release-independent... > > > > > > > > I seem to remember there were drawbacks in branching tempest, though... > > > > Can someone with functioning memory brain cells summarize them again ? > > > > > > > > > > > > > Branchless Tempest enforces api stability across branches. > > > > I'm sorry, but I'm having a hard time taking this statement seriously > > when the current source of tension is that the Tempest API itself > > is breaking for its plugins. > > > > Maybe rather than talking about how to release compatible things > > together, we should go back and talk about why Tempest's API is changing > > in a way that can't be made backwards-compatible. Can you give some more > > detail about that? > > > > Well it's not, if it did that would violate all the stability guarantees > provided by Tempest's library and plugin interface. I've not ever heard of > these kind of backwards incompatibilities in those interfaces and we go to > all effort to make sure we don't break them. Where did the idea that > backwards incompatible changes where being introduced come from? In his original post, gmann said, "There might be some changes in Tempest which might not work with older version of Tempest Plugins." I was surprised to hear that, but I'm not sure how else to interpret that statement. > As for this whole thread I don't understand any of the points being brought up > in the original post or any of the follow ons, things seem to have been > confused > from the start. The ask from users at the summit was simple. When a new > OpenStack > release is pushed we push a tempest release to mark that (the next one will be > 19.0.0 to mark Rocky). Users were complaining that many plugins don't have a > corresponding version to mark support for a new release. So when trying to run > against a rocky cloud you get tempest 19.0.0 and then a bunch of plugins for > various services at different sha1s which have to be manually looked up based > on dates. All users wanted at the summit was a tag for plugins like tempest > does with the first number in: > > https://docs.openstack.org/tempest/latest/overview.html#release-versioning > > which didn't seem like a bad idea to me. I'm not sure the best mechanism to > accomplish this, because I agree with much of what plugin maintainers were > saying on the thread about wanting to control their own releases. But the > desire to make sure users have a tag they can pull for the addition or > removal of a supported release makes sense as something a plugin should do. We don't coordinate versions across projects anywhere else, for a bunch of reasons including the complexity of coordinating the details and the confusion it causes when the first version of something is 19.0.0. Instead, we list the compatible versions of everything together on a series-specific page on releases.o.o. That seems to be enough to help anyone wanting to know which versions of tools work together. The data is also available in YAML files, so it's easy enough to consume by automation. Would that work for tempest and it's plugins, too? Is the problem that the versions are not the same, or that some of the plugins are not being tagged at all? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, Jun 26, 2018 at 08:53:21AM -0400, Doug Hellmann wrote: > Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, wrote: > > > > > Dmitry Tantsur wrote: > > > > [...] > > > > My suggestion: tempest has to be compatible with all supported releases > > > > (of both services and plugins) OR be branched. > > > > [...] > > > I tend to agree with Dmitry... We have a model for things that need > > > release alignment, and that's the cycle-bound series. The reason tempest > > > is branchless was because there was no compatibility issue. If the split > > > of tempest plugins introduces a potential incompatibility, then I would > > > prefer aligning tempest to the existing model rather than introduce a > > > parallel tempest-specific cycle just so that tempest can stay > > > release-independent... > > > > > > I seem to remember there were drawbacks in branching tempest, though... > > > Can someone with functioning memory brain cells summarize them again ? > > > > > > > > > Branchless Tempest enforces api stability across branches. > > I'm sorry, but I'm having a hard time taking this statement seriously > when the current source of tension is that the Tempest API itself > is breaking for its plugins. > > Maybe rather than talking about how to release compatible things > together, we should go back and talk about why Tempest's API is changing > in a way that can't be made backwards-compatible. Can you give some more > detail about that? > Well it's not, if it did that would violate all the stability guarantees provided by Tempest's library and plugin interface. I've not ever heard of these kind of backwards incompatibilities in those interfaces and we go to all effort to make sure we don't break them. Where did the idea that backwards incompatible changes where being introduced come from? That being said things are definitely getting confused here, all andreaf was talking about the branchless nature is ensuring we run the same tests against service REST APIs, making sure we have API stability between releases in the services when we say we do. (to answer ttx's question) As for this whole thread I don't understand any of the points being brought up in the original post or any of the follow ons, things seem to have been confused from the start. The ask from users at the summit was simple. When a new OpenStack release is pushed we push a tempest release to mark that (the next one will be 19.0.0 to mark Rocky). Users were complaining that many plugins don't have a corresponding version to mark support for a new release. So when trying to run against a rocky cloud you get tempest 19.0.0 and then a bunch of plugins for various services at different sha1s which have to be manually looked up based on dates. All users wanted at the summit was a tag for plugins like tempest does with the first number in: https://docs.openstack.org/tempest/latest/overview.html#release-versioning which didn't seem like a bad idea to me. I'm not sure the best mechanism to accomplish this, because I agree with much of what plugin maintainers were saying on the thread about wanting to control their own releases. But the desire to make sure users have a tag they can pull for the addition or removal of a supported release makes sense as something a plugin should do. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Excerpts from Andrea Frittoli's message of 2018-06-26 13:35:11 +0100: > On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, wrote: > > > Dmitry Tantsur wrote: > > > [...] > > > My suggestion: tempest has to be compatible with all supported releases > > > (of both services and plugins) OR be branched. > > > [...] > > I tend to agree with Dmitry... We have a model for things that need > > release alignment, and that's the cycle-bound series. The reason tempest > > is branchless was because there was no compatibility issue. If the split > > of tempest plugins introduces a potential incompatibility, then I would > > prefer aligning tempest to the existing model rather than introduce a > > parallel tempest-specific cycle just so that tempest can stay > > release-independent... > > > > I seem to remember there were drawbacks in branching tempest, though... > > Can someone with functioning memory brain cells summarize them again ? > > > > > Branchless Tempest enforces api stability across branches. I'm sorry, but I'm having a hard time taking this statement seriously when the current source of tension is that the Tempest API itself is breaking for its plugins. Maybe rather than talking about how to release compatible things together, we should go back and talk about why Tempest's API is changing in a way that can't be made backwards-compatible. Can you give some more detail about that? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018, 1:08 pm Thierry Carrez, wrote: > Dmitry Tantsur wrote: > > [...] > > My suggestion: tempest has to be compatible with all supported releases > > (of both services and plugins) OR be branched. > > [...] > I tend to agree with Dmitry... We have a model for things that need > release alignment, and that's the cycle-bound series. The reason tempest > is branchless was because there was no compatibility issue. If the split > of tempest plugins introduces a potential incompatibility, then I would > prefer aligning tempest to the existing model rather than introduce a > parallel tempest-specific cycle just so that tempest can stay > release-independent... > > I seem to remember there were drawbacks in branching tempest, though... > Can someone with functioning memory brain cells summarize them again ? > Branchless Tempest enforces api stability across branches. For the same reason tempest plug ins should be branchless, which is one of the reasons that having them in the same repo as a service was an issue: services are branched but api/integration tests should be branchless. Andrea > -- > Thierry Carrez (ttx) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Dmitry Tantsur wrote: [...] My suggestion: tempest has to be compatible with all supported releases (of both services and plugins) OR be branched. [...] I tend to agree with Dmitry... We have a model for things that need release alignment, and that's the cycle-bound series. The reason tempest is branchless was because there was no compatibility issue. If the split of tempest plugins introduces a potential incompatibility, then I would prefer aligning tempest to the existing model rather than introduce a parallel tempest-specific cycle just so that tempest can stay release-independent... I seem to remember there were drawbacks in branching tempest, though... Can someone with functioning memory brain cells summarize them again ? -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Hi, I have never understood the branchless tempest thing. Making Tempest release is a great news for me. But about plugins... Tempest already provides a API for plugins. If you are going to break this API, why not using stable branches and deprecation process like any other software ? If you do that, plugin will be informed that Tempest will soon do a breaking change. Their can update their plugin code and raise the minimal tempest version required to work. Their can do that when they have times, and not because Tempest want to release a version soon. Also the stable branch/deprecation process is well known by the whole community. And this will also allow them to release a version when their want. So I support making release of Tempest and Plugins, but do not support a coordinated release. Regards, On Tue, Jun 26, 2018 at 06:18:52PM +0900, Ghanshyam Mann wrote: Hello Everyone, In Queens cycle, community goal to split the Tempest Plugin has been completed [1] and i think almost all the projects have separate repo for tempest plugin [2]. Which means each tempest plugins are being separated from their project release model. Few projects have started the independent release model for their plugins like kuryr-tempest-plugin, ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also planning as chatted with amotoki. There might be some changes in Tempest which might not work with older version of Tempest Plugins. For example, If I am testing any production cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i will be using Tempest and Aodh's , Congress's Tempest plugins. With Independent release model of each Tempest Plugins, there might be chance that the Aodh's or Congress's Tempest plugin versions are not compatible with latest/known Tempest versions. It will become hard to find the compatible tag/release of Tempest and Tempest Plugins or in some cases i might need to patch up the things. During QA feedback sessions at Vancouver Summit, there was feedback to coordinating the release of all Tempest plugins and Tempest [4] (also amotoki talked to me on this as neutron-tempest-plugin is planning their first release). Idea is to release/tag all the Tempest plugins and Tempest together so that specific release/tag can be identified as compatible version of all the Plugins and Tempest for testing the complete stack. That way user can get to know what version of Tempest Plugins is compatible with what version of Tempest. For above use case, we need some coordinated release model among Tempest and all the Tempest Plugins. There should be single release of all Tempest Plugins with well defined tag whenever any Tempest release is happening. For Example, Tempest version 19.0.0 is to mark the "support of the Rocky release". When releasing the Tempest 19.0, we will release all the Tempest plugins also to tag the compatibility of plugins with Tempest for "support of the Rocky release". One way to make this coordinated release (just a initial thought): 1. Release Each Tempest Plugins whenever there is any major version release of Tempest (like marking the support of OpenStack release in Tempest, EOL of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do their intermediate release of minor version change which are in backward compatible way. 1.2 This coordinated Release can be started from latest Tempest Version for simple reading. Like if we start this coordinated release from Tempest version 19.0.0 then, each plugins will be released as 19.0.0 and so on. Giving the above background and use case of this coordinated release, A) I would like to ask each plugins owner if you are agree on this coordinated release? If no, please give more feedback or issue we can face due to this coordinated release. If we get the agreement from all Plugins then, B) Release team or TC help to find the better model for this use case or improvement in above model. C) Once we define the release model, find out the team owning that release model (there are more than 40 Tempest plugins currently) . NOTE: Till we decide the best solution for this use case, each plugins can do/keep doing their plugin release as per independent release model. [1] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html [2] https://docs.openstack.org/tempest/latest/plugin-registry.html [3] https://github.com/openstack/kuryr-tempest-plugin/releases https://github.com/openstack/ironic-tempest-plugin/releases [4] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131011.html -gmann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht signa
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On 06/26/2018 11:57 AM, Ghanshyam Mann wrote: On Tue, 26 Jun 2018 18:37:42 +0900 Dmitry Tantsur wrote > On 06/26/2018 11:18 AM, Ghanshyam Mann wrote: > > Hello Everyone, > > > > In Queens cycle, community goal to split the Tempest Plugin has been completed [1] and i think almost all the projects have separate repo for tempest plugin [2]. Which means each tempest plugins are being separated from their project release model. Few projects have started the independent release model for their plugins like kuryr-tempest-plugin, ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also planning as chatted with amotoki. > > > > There might be some changes in Tempest which might not work with older version of Tempest Plugins. For example, If I am testing any production cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i will be using Tempest and Aodh's , Congress's Tempest plugins. With Independent release model of each Tempest Plugins, there might be chance that the Aodh's or Congress's Tempest plugin versions are not compatible with latest/known Tempest versions. It will become hard to find the compatible tag/release of Tempest and Tempest Plugins or in some cases i might need to patch up the things. > > FWIW this is solved by stable branches for all other projects. If we cannot keep > Tempest compatible with all supported branches, we should back off our decision > to make it branchless. The very nature of being branchless implies being > compatible with all supported releases. > > > > > During QA feedback sessions at Vancouver Summit, there was feedback to coordinating the release of all Tempest plugins and Tempest [4] (also amotoki talked to me on this as neutron-tempest-plugin is planning their first release). Idea is to release/tag all the Tempest plugins and Tempest together so that specific release/tag can be identified as compatible version of all the Plugins and Tempest for testing the complete stack. That way user can get to know what version of Tempest Plugins is compatible with what version of Tempest. > > > > For above use case, we need some coordinated release model among Tempest and all the Tempest Plugins. There should be single release of all Tempest Plugins with well defined tag whenever any Tempest release is happening. For Example, Tempest version 19.0.0 is to mark the "support of the Rocky release". When releasing the Tempest 19.0, we will release all the Tempest plugins also to tag the compatibility of plugins with Tempest for "support of the Rocky release". > > > > One way to make this coordinated release (just a initial thought): > > 1. Release Each Tempest Plugins whenever there is any major version release of Tempest (like marking the support of OpenStack release in Tempest, EOL of OpenStack release in Tempest) > > 1.1 Each plugin or Tempest can do their intermediate release of minor version change which are in backward compatible way. > > 1.2 This coordinated Release can be started from latest Tempest Version for simple reading. Like if we start this coordinated release from Tempest version 19.0.0 then, > > each plugins will be released as 19.0.0 and so on. > > > > Giving the above background and use case of this coordinated release, > > A) I would like to ask each plugins owner if you are agree on this coordinated release? If no, please give more feedback or issue we can face due to this coordinated release. > > Disclaimer: I'm not the PTL. > > Similarly to Luigi, I don't feel well about forcing a plugin release at the same > time as a tempest release, UNLESS tempest folks are going to coordinate their > releases with all how-many-do-we-have plugins. What I'd like to avoid is cutting > a release in the middle of a patch chain or some refactoring just because > tempest happened to have a release right now. I understand your point. But we can avoid that case if we only coordinate on major version bump only. as i mentioned in 1.2 point, Tempest and Tempest plugins can do their intermediate release anytime which are nothing but backward compatible release. In this proposed model, we can do a coordinated release for major version bump only which is happening only on OpenStack release and EOL of any stable branch. Even bigger concern: what if the plugin is actually not compatible yet? Say, you're releasing tempest 19.0. As the same point you're cutting ironic-tempest-plugin 19.0. Who guarantees that they're compatible? If we haven't had any patches for it in a month, it may well happen that it does not work. Or I am all open to have another release model which can be best suited for all plugins which can address the mentioned use case of coordinated release. My suggestion: tempest has to be compatible with all supported releases (of both services and plugins) OR be branched. -gmann > > >
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, Jun 26, 2018 at 2:48 PM, Ghanshyam Mann wrote: > Hello Everyone, > > In Queens cycle, community goal to split the Tempest Plugin has been > completed [1] and i think almost all the projects have separate repo for > tempest plugin [2]. Which means each tempest plugins are being separated > from their project release model. Few projects have started the > independent release model for their plugins like kuryr-tempest-plugin, > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > planning as chatted with amotoki. > > There might be some changes in Tempest which might not work with older > version of Tempest Plugins. I don't think that's a good premise. Isn't tempest branchless and by definition should be backward compatible with service releases? If there are changes in the plugin interface in tempest, I would also expect those to be backward compatible too. Likewise plugins should be backward compatible with their respective projects, so any kind of release model would work. Else, I think the whole branchless concept is of very little use. For example, If I am testing any production cloud which has Nova, Neutron, > Cinder, Keystone , Aodh, Congress etc i will be using Tempest and Aodh's , > Congress's Tempest plugins. With Independent release model of each Tempest > Plugins, there might be chance that the Aodh's or Congress's Tempest plugin > versions are not compatible with latest/known Tempest versions. It will > become hard to find the compatible tag/release of Tempest and Tempest > Plugins or in some cases i might need to patch up the things. > > During QA feedback sessions at Vancouver Summit, there was feedback to > coordinating the release of all Tempest plugins and Tempest [4] (also > amotoki talked to me on this as neutron-tempest-plugin is planning their > first release). Idea is to release/tag all the Tempest plugins and Tempest > together so that specific release/tag can be identified as compatible > version of all the Plugins and Tempest for testing the complete stack. That > way user can get to know what version of Tempest Plugins is compatible with > what version of Tempest. > > For above use case, we need some coordinated release model among Tempest > and all the Tempest Plugins. There should be single release of all Tempest > Plugins with well defined tag whenever any Tempest release is happening. > For Example, Tempest version 19.0.0 is to mark the "support of the Rocky > release". When releasing the Tempest 19.0, we will release all the Tempest > plugins also to tag the compatibility of plugins with Tempest for "support > of the Rocky release". > > One way to make this coordinated release (just a initial thought): > 1. Release Each Tempest Plugins whenever there is any major version > release of Tempest (like marking the support of OpenStack release in > Tempest, EOL of OpenStack release in Tempest) > 1.1 Each plugin or Tempest can do their intermediate release of minor > version change which are in backward compatible way. > 1.2 This coordinated Release can be started from latest Tempest > Version for simple reading. Like if we start this coordinated release from > Tempest version 19.0.0 then, > each plugins will be released as 19.0.0 and so on. > > Giving the above background and use case of this coordinated release, > A) I would like to ask each plugins owner if you are agree on this > coordinated release? If no, please give more feedback or issue we can face > due to this coordinated release. > > If we get the agreement from all Plugins then, > B) Release team or TC help to find the better model for this use case or > improvement in above model. > > C) Once we define the release model, find out the team owning that release > model (there are more than 40 Tempest plugins currently) . > > NOTE: Till we decide the best solution for this use case, each plugins can > do/keep doing their plugin release as per independent release model. > > [1] https://governance.openstack.org/tc/goals/queens/split-tempe > st-plugins.html > [2] https://docs.openstack.org/tempest/latest/plugin-registry.html > [3] https://github.com/openstack/kuryr-tempest-plugin/releases >https://github.com/openstack/ironic-tempest-plugin/releases > [4] http://lists.openstack.org/pipermail/openstack-dev/2018-June > /131011.html > > > -gmann > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Regards, Rabi Mishra __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tuesday, 26 June 2018 11:52:53 CEST Ghanshyam Mann wrote: > On Tue, 26 Jun 2018 18:28:03 +0900 Luigi Toscano > wrote > > On Tuesday, 26 June 2018 11:18:52 CEST Ghanshyam Mann wrote: > > > Hello Everyone, > > > > > > In Queens cycle, community goal to split the Tempest Plugin has been > > > completed [1] and i think almost all the projects have separate repo > > > for > > > tempest plugin [2]. Which means each tempest plugins are being > > > separated > > > from their project release model. Few projects have started the > > > independent release model for their plugins like kuryr-tempest-plugin, > > > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > > > planning as chatted with amotoki. > > > > > > There might be some changes in Tempest which might not work with older > > > version of Tempest Plugins. For example, If I am testing any > > > production > > > cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i > > > will be using Tempest and Aodh's , Congress's Tempest plugins. With > > > Independent release model of each Tempest Plugins, there might be > > > chance > > > that the Aodh's or Congress's Tempest plugin versions are not > > > compatible > > > with latest/known Tempest versions. It will become hard to find the > > > compatible tag/release of Tempest and Tempest Plugins or in some cases > > > i > > > might need to patch up the things. > > > > > > During QA feedback sessions at Vancouver Summit, there was feedback to > > > coordinating the release of all Tempest plugins and Tempest [4] (also > > > amotoki talked to me on this as neutron-tempest-plugin is planning > > > their > > > first release). Idea is to release/tag all the Tempest plugins and > > > Tempest > > > together so that specific release/tag can be identified as compatible > > > version of all the Plugins and Tempest for testing the complete stack. > > > That > > > way user can get to know what version of Tempest Plugins is compatible > > > with > > > what version of Tempest. > > > > > > For above use case, we need some coordinated release model among > > > Tempest and all the Tempest Plugins. There should be single release of > > > all Tempest Plugins with well defined tag whenever any Tempest release > > > is happening. For Example, Tempest version 19.0.0 is to mark the > > > "support of the Rocky release". When releasing the Tempest 19.0, we > > > will release all the Tempest plugins also to tag the compatibility of > > > plugins with Tempest for "support of the Rocky release". > > > > > > One way to make this coordinated release (just a initial thought): > > > 1. Release Each Tempest Plugins whenever there is any major version > > > release > > > of Tempest (like marking the support of OpenStack release in Tempest, > > > EOL > > > of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do > > > their > > > intermediate release of minor version change which are in backward > > > compatible way. 1.2 This coordinated Release can be started from latest > > > Tempest Version for simple reading. Like if we start this coordinated > > > release from Tempest version 19.0.0 then, each plugins will be released > > > as > > > 19.0.0 and so on. > > > > > > Giving the above background and use case of this coordinated release, > > > A) I would like to ask each plugins owner if you are agree on this > > > coordinated release? If no, please give more feedback or issue we can > > > face > > > due to this coordinated release. > > > > The Sahara PTL may disagree with me, but I disagree with forcing each > > team to release in a coordinate model. > > > > I already take care of releasing sahara-tests, which contains both the > > tempest plugin and the scenario tests, when a new major version of > > OpenStack is released, keeping the compatibility with the relevant > > versions of Tempest. > > > > tl;dr I agree with having Tempest plugins follow the same lifecycle of > > Tempest, but please allow me to do so manually. > > But with coordinated release, we can make sure we have particular tags > which can be used in OpenStack Complete testing. With independent release > model, there is no guarantee that all tempest plugins will be compatible > with Tempest versions. With the independent release model (in general, not just for Tempest plugins) it's up to the program maintainer to ensure that things are compatible. Let me make sure that: - I agree that Tempest plugins should follow the same lifecycle than Tempest. It's the policy that I applied for the Tempest plugin part of sahara-tests since Kilo. - most of plugins maintainer don't care, can't care or shouldn't care about the release process: then coordinated release is fine (as it happen also for the general usage of coordinated release vs independent relase in OpenStack) - I still prefer to manage this myself, because the repository does not co
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018 18:37:42 +0900 Dmitry Tantsur wrote > On 06/26/2018 11:18 AM, Ghanshyam Mann wrote: > > Hello Everyone, > > > > In Queens cycle, community goal to split the Tempest Plugin has been > > completed [1] and i think almost all the projects have separate repo for > > tempest plugin [2]. Which means each tempest plugins are being separated > > from their project release model. Few projects have started the > > independent release model for their plugins like kuryr-tempest-plugin, > > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > > planning as chatted with amotoki. > > > > There might be some changes in Tempest which might not work with older > > version of Tempest Plugins. For example, If I am testing any production > > cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i > > will be using Tempest and Aodh's , Congress's Tempest plugins. With > > Independent release model of each Tempest Plugins, there might be chance > > that the Aodh's or Congress's Tempest plugin versions are not compatible > > with latest/known Tempest versions. It will become hard to find the > > compatible tag/release of Tempest and Tempest Plugins or in some cases i > > might need to patch up the things. > > FWIW this is solved by stable branches for all other projects. If we cannot > keep > Tempest compatible with all supported branches, we should back off our > decision > to make it branchless. The very nature of being branchless implies being > compatible with all supported releases. > > > > > During QA feedback sessions at Vancouver Summit, there was feedback to > > coordinating the release of all Tempest plugins and Tempest [4] (also > > amotoki talked to me on this as neutron-tempest-plugin is planning their > > first release). Idea is to release/tag all the Tempest plugins and Tempest > > together so that specific release/tag can be identified as compatible > > version of all the Plugins and Tempest for testing the complete stack. > > That way user can get to know what version of Tempest Plugins is > > compatible with what version of Tempest. > > > > For above use case, we need some coordinated release model among Tempest > > and all the Tempest Plugins. There should be single release of all Tempest > > Plugins with well defined tag whenever any Tempest release is happening. > > For Example, Tempest version 19.0.0 is to mark the "support of the Rocky > > release". When releasing the Tempest 19.0, we will release all the Tempest > > plugins also to tag the compatibility of plugins with Tempest for "support > > of the Rocky release". > > > > One way to make this coordinated release (just a initial thought): > > 1. Release Each Tempest Plugins whenever there is any major version > > release of Tempest (like marking the support of OpenStack release in > > Tempest, EOL of OpenStack release in Tempest) > > 1.1 Each plugin or Tempest can do their intermediate release of minor > > version change which are in backward compatible way. > > 1.2 This coordinated Release can be started from latest Tempest > > Version for simple reading. Like if we start this coordinated release > > from Tempest version 19.0.0 then, > > each plugins will be released as 19.0.0 and so on. > > > > Giving the above background and use case of this coordinated release, > > A) I would like to ask each plugins owner if you are agree on this > > coordinated release? If no, please give more feedback or issue we can > > face due to this coordinated release. > > Disclaimer: I'm not the PTL. > > Similarly to Luigi, I don't feel well about forcing a plugin release at the > same > time as a tempest release, UNLESS tempest folks are going to coordinate > their > releases with all how-many-do-we-have plugins. What I'd like to avoid is > cutting > a release in the middle of a patch chain or some refactoring just because > tempest happened to have a release right now. I understand your point. But we can avoid that case if we only coordinate on major version bump only. as i mentioned in 1.2 point, Tempest and Tempest plugins can do their intermediate release anytime which are nothing but backward compatible release. In this proposed model, we can do a coordinated release for major version bump only which is happening only on OpenStack release and EOL of any stable branch. Or I am all open to have another release model which can be best suited for all plugins which can address the mentioned use case of coordinated release. -gmann > > > > > If we get the agreement from all Plugins then, > > B) Release team or TC help to find the better model for this use case or > > improvement in above model. > > > > C) Once we define the release model, find out the team owning that release > > model (there are more than 40 Tempest plugins curr
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tue, 26 Jun 2018 18:28:03 +0900 Luigi Toscano wrote > On Tuesday, 26 June 2018 11:18:52 CEST Ghanshyam Mann wrote: > > Hello Everyone, > > > > In Queens cycle, community goal to split the Tempest Plugin has been > > completed [1] and i think almost all the projects have separate repo for > > tempest plugin [2]. Which means each tempest plugins are being separated > > from their project release model. Few projects have started the > > independent release model for their plugins like kuryr-tempest-plugin, > > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > > planning as chatted with amotoki. > > > > There might be some changes in Tempest which might not work with older > > version of Tempest Plugins. For example, If I am testing any production > > cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i > > will be using Tempest and Aodh's , Congress's Tempest plugins. With > > Independent release model of each Tempest Plugins, there might be chance > > that the Aodh's or Congress's Tempest plugin versions are not compatible > > with latest/known Tempest versions. It will become hard to find the > > compatible tag/release of Tempest and Tempest Plugins or in some cases i > > might need to patch up the things. > > > > During QA feedback sessions at Vancouver Summit, there was feedback to > > coordinating the release of all Tempest plugins and Tempest [4] (also > > amotoki talked to me on this as neutron-tempest-plugin is planning their > > first release). Idea is to release/tag all the Tempest plugins and Tempest > > together so that specific release/tag can be identified as compatible > > version of all the Plugins and Tempest for testing the complete stack. > > That > > way user can get to know what version of Tempest Plugins is compatible > > with > > what version of Tempest. > > > > For above use case, we need some coordinated release model among Tempest > > and > > all the Tempest Plugins. There should be single release of all Tempest > > Plugins with well defined tag whenever any Tempest release is happening. > > For Example, Tempest version 19.0.0 is to mark the "support of the Rocky > > release". When releasing the Tempest 19.0, we will release all the Tempest > > plugins also to tag the compatibility of plugins with Tempest for "support > > of the Rocky release". > > > > One way to make this coordinated release (just a initial thought): > > 1. Release Each Tempest Plugins whenever there is any major version > > release > > of Tempest (like marking the support of OpenStack release in Tempest, EOL > > of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do their > > intermediate release of minor version change which are in backward > > compatible way. 1.2 This coordinated Release can be started from latest > > Tempest Version for simple reading. Like if we start this coordinated > > release from Tempest version 19.0.0 then, each plugins will be released as > > 19.0.0 and so on. > > > > Giving the above background and use case of this coordinated release, > > A) I would like to ask each plugins owner if you are agree on this > > coordinated release? If no, please give more feedback or issue we can > > face > > due to this coordinated release. > > > > The Sahara PTL may disagree with me, but I disagree with forcing each team > to > release in a coordinate model. > > I already take care of releasing sahara-tests, which contains both the > tempest > plugin and the scenario tests, when a new major version of OpenStack is > released, keeping the compatibility with the relevant versions of Tempest. > > tl;dr I agree with having Tempest plugins follow the same lifecycle of > Tempest, but please allow me to do so manually. But with coordinated release, we can make sure we have particular tags which can be used in OpenStack Complete testing. With independent release model, there is no guarantee that all tempest plugins will be compatible with Tempest versions. -gmann > > > -- > Luigi > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On 06/26/2018 11:18 AM, Ghanshyam Mann wrote: Hello Everyone, In Queens cycle, community goal to split the Tempest Plugin has been completed [1] and i think almost all the projects have separate repo for tempest plugin [2]. Which means each tempest plugins are being separated from their project release model. Few projects have started the independent release model for their plugins like kuryr-tempest-plugin, ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also planning as chatted with amotoki. There might be some changes in Tempest which might not work with older version of Tempest Plugins. For example, If I am testing any production cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i will be using Tempest and Aodh's , Congress's Tempest plugins. With Independent release model of each Tempest Plugins, there might be chance that the Aodh's or Congress's Tempest plugin versions are not compatible with latest/known Tempest versions. It will become hard to find the compatible tag/release of Tempest and Tempest Plugins or in some cases i might need to patch up the things. FWIW this is solved by stable branches for all other projects. If we cannot keep Tempest compatible with all supported branches, we should back off our decision to make it branchless. The very nature of being branchless implies being compatible with all supported releases. During QA feedback sessions at Vancouver Summit, there was feedback to coordinating the release of all Tempest plugins and Tempest [4] (also amotoki talked to me on this as neutron-tempest-plugin is planning their first release). Idea is to release/tag all the Tempest plugins and Tempest together so that specific release/tag can be identified as compatible version of all the Plugins and Tempest for testing the complete stack. That way user can get to know what version of Tempest Plugins is compatible with what version of Tempest. For above use case, we need some coordinated release model among Tempest and all the Tempest Plugins. There should be single release of all Tempest Plugins with well defined tag whenever any Tempest release is happening. For Example, Tempest version 19.0.0 is to mark the "support of the Rocky release". When releasing the Tempest 19.0, we will release all the Tempest plugins also to tag the compatibility of plugins with Tempest for "support of the Rocky release". One way to make this coordinated release (just a initial thought): 1. Release Each Tempest Plugins whenever there is any major version release of Tempest (like marking the support of OpenStack release in Tempest, EOL of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do their intermediate release of minor version change which are in backward compatible way. 1.2 This coordinated Release can be started from latest Tempest Version for simple reading. Like if we start this coordinated release from Tempest version 19.0.0 then, each plugins will be released as 19.0.0 and so on. Giving the above background and use case of this coordinated release, A) I would like to ask each plugins owner if you are agree on this coordinated release? If no, please give more feedback or issue we can face due to this coordinated release. Disclaimer: I'm not the PTL. Similarly to Luigi, I don't feel well about forcing a plugin release at the same time as a tempest release, UNLESS tempest folks are going to coordinate their releases with all how-many-do-we-have plugins. What I'd like to avoid is cutting a release in the middle of a patch chain or some refactoring just because tempest happened to have a release right now. If we get the agreement from all Plugins then, B) Release team or TC help to find the better model for this use case or improvement in above model. C) Once we define the release model, find out the team owning that release model (there are more than 40 Tempest plugins currently) . NOTE: Till we decide the best solution for this use case, each plugins can do/keep doing their plugin release as per independent release model. [1] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html [2] https://docs.openstack.org/tempest/latest/plugin-registry.html [3] https://github.com/openstack/kuryr-tempest-plugin/releases https://github.com/openstack/ironic-tempest-plugin/releases [4] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131011.html -gmann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe h
Re: [openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
On Tuesday, 26 June 2018 11:18:52 CEST Ghanshyam Mann wrote: > Hello Everyone, > > In Queens cycle, community goal to split the Tempest Plugin has been > completed [1] and i think almost all the projects have separate repo for > tempest plugin [2]. Which means each tempest plugins are being separated > from their project release model. Few projects have started the > independent release model for their plugins like kuryr-tempest-plugin, > ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also > planning as chatted with amotoki. > > There might be some changes in Tempest which might not work with older > version of Tempest Plugins. For example, If I am testing any production > cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i > will be using Tempest and Aodh's , Congress's Tempest plugins. With > Independent release model of each Tempest Plugins, there might be chance > that the Aodh's or Congress's Tempest plugin versions are not compatible > with latest/known Tempest versions. It will become hard to find the > compatible tag/release of Tempest and Tempest Plugins or in some cases i > might need to patch up the things. > > During QA feedback sessions at Vancouver Summit, there was feedback to > coordinating the release of all Tempest plugins and Tempest [4] (also > amotoki talked to me on this as neutron-tempest-plugin is planning their > first release). Idea is to release/tag all the Tempest plugins and Tempest > together so that specific release/tag can be identified as compatible > version of all the Plugins and Tempest for testing the complete stack. That > way user can get to know what version of Tempest Plugins is compatible with > what version of Tempest. > > For above use case, we need some coordinated release model among Tempest and > all the Tempest Plugins. There should be single release of all Tempest > Plugins with well defined tag whenever any Tempest release is happening. > For Example, Tempest version 19.0.0 is to mark the "support of the Rocky > release". When releasing the Tempest 19.0, we will release all the Tempest > plugins also to tag the compatibility of plugins with Tempest for "support > of the Rocky release". > > One way to make this coordinated release (just a initial thought): > 1. Release Each Tempest Plugins whenever there is any major version release > of Tempest (like marking the support of OpenStack release in Tempest, EOL > of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do their > intermediate release of minor version change which are in backward > compatible way. 1.2 This coordinated Release can be started from latest > Tempest Version for simple reading. Like if we start this coordinated > release from Tempest version 19.0.0 then, each plugins will be released as > 19.0.0 and so on. > > Giving the above background and use case of this coordinated release, > A) I would like to ask each plugins owner if you are agree on this > coordinated release? If no, please give more feedback or issue we can face > due to this coordinated release. > The Sahara PTL may disagree with me, but I disagree with forcing each team to release in a coordinate model. I already take care of releasing sahara-tests, which contains both the tempest plugin and the scenario tests, when a new major version of OpenStack is released, keeping the compatibility with the relevant versions of Tempest. tl;dr I agree with having Tempest plugins follow the same lifecycle of Tempest, but please allow me to do so manually. -- Luigi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][tempest-plugins][release][tc][ptl]: Coordinated Release Model proposal for Tempest & Tempest Plugins
Hello Everyone, In Queens cycle, community goal to split the Tempest Plugin has been completed [1] and i think almost all the projects have separate repo for tempest plugin [2]. Which means each tempest plugins are being separated from their project release model. Few projects have started the independent release model for their plugins like kuryr-tempest-plugin, ironic-tempest-plugin etc [3]. I think neutron-tempest-plugin also planning as chatted with amotoki. There might be some changes in Tempest which might not work with older version of Tempest Plugins. For example, If I am testing any production cloud which has Nova, Neutron, Cinder, Keystone , Aodh, Congress etc i will be using Tempest and Aodh's , Congress's Tempest plugins. With Independent release model of each Tempest Plugins, there might be chance that the Aodh's or Congress's Tempest plugin versions are not compatible with latest/known Tempest versions. It will become hard to find the compatible tag/release of Tempest and Tempest Plugins or in some cases i might need to patch up the things. During QA feedback sessions at Vancouver Summit, there was feedback to coordinating the release of all Tempest plugins and Tempest [4] (also amotoki talked to me on this as neutron-tempest-plugin is planning their first release). Idea is to release/tag all the Tempest plugins and Tempest together so that specific release/tag can be identified as compatible version of all the Plugins and Tempest for testing the complete stack. That way user can get to know what version of Tempest Plugins is compatible with what version of Tempest. For above use case, we need some coordinated release model among Tempest and all the Tempest Plugins. There should be single release of all Tempest Plugins with well defined tag whenever any Tempest release is happening. For Example, Tempest version 19.0.0 is to mark the "support of the Rocky release". When releasing the Tempest 19.0, we will release all the Tempest plugins also to tag the compatibility of plugins with Tempest for "support of the Rocky release". One way to make this coordinated release (just a initial thought): 1. Release Each Tempest Plugins whenever there is any major version release of Tempest (like marking the support of OpenStack release in Tempest, EOL of OpenStack release in Tempest) 1.1 Each plugin or Tempest can do their intermediate release of minor version change which are in backward compatible way. 1.2 This coordinated Release can be started from latest Tempest Version for simple reading. Like if we start this coordinated release from Tempest version 19.0.0 then, each plugins will be released as 19.0.0 and so on. Giving the above background and use case of this coordinated release, A) I would like to ask each plugins owner if you are agree on this coordinated release? If no, please give more feedback or issue we can face due to this coordinated release. If we get the agreement from all Plugins then, B) Release team or TC help to find the better model for this use case or improvement in above model. C) Once we define the release model, find out the team owning that release model (there are more than 40 Tempest plugins currently) . NOTE: Till we decide the best solution for this use case, each plugins can do/keep doing their plugin release as per independent release model. [1] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html [2] https://docs.openstack.org/tempest/latest/plugin-registry.html [3] https://github.com/openstack/kuryr-tempest-plugin/releases https://github.com/openstack/ironic-tempest-plugin/releases [4] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131011.html -gmann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
