On Fri, May 05, 2017 at 09:29:40AM +1200, Steve Baker wrote: > On Thu, May 4, 2017 at 3:56 PM, Matthew Treinish <[email protected]> > wrote: > > > On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote: > > > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish <[email protected]> > > > wrote: > > > > > > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote: > > > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <[email protected]> > > > > wrote: > > > > > > > > > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli < > > > > > >> [email protected]> wrote: > > > > > >> > > > > > >>> Dear stackers, > > > > > >>> > > > > > >>> starting in the Liberty cycle Tempest has defined a set of > > projects > > > > > >>> which are in scope for direct > > > > > >>> testing in Tempest [0]. The current list includes keystone, nova, > > > > > >>> glance, swift, cinder and neutron. > > > > > >>> All other projects can use the same Tempest testing > > infrastructure > > > > (or > > > > > >>> parts of it) by taking advantage > > > > > >>> the Tempest plugin and stable interfaces. > > > > > >>> > > > > > >>> Tempest currently hosts a set of API tests as well as a service > > > > client > > > > > >>> for the Heat project. > > > > > >>> The Heat service client is used by the tests in Tempest, which > > run in > > > > > >>> Heat gate as part of the grenade > > > > > >>> job, as well as in the Tempest gate (check pipeline) as part of > > the > > > > > >>> layer4 job. > > > > > >>> According to code search [3] the Heat service client is also > > used by > > > > > >>> Murano and Daisycore. > > > > > >>> > > > > > >> > > > > > >> For the heat grenade job, I've proposed two patches. > > > > > >> > > > > > >> 1. To run heat tree gabbi api tests as part of grenade > > 'post-upgrade' > > > > > >> phase > > > > > >> > > > > > >> https://review.openstack.org/#/c/460542/ > > > > > >> > > > > > >> 2. To remove tempest tests from the grenade job > > > > > >> > > > > > >> https://review.openstack.org/#/c/460810/ > > > > > >> > > > > > >> > > > > > >> > > > > > >>> I proposed a patch to Tempest to start the deprecation counter > > for > > > > Heat > > > > > >>> / orchestration related > > > > > >>> configuration items in Tempest [4], and I would like to make sure > > > > that > > > > > >>> all tests and the service client > > > > > >>> either find a new home outside of Tempest, or are removed, by > > the end > > > > > >>> the Pike cycle at the latest. > > > > > >>> > > > > > >>> Heat has in-tree integration tests and Gabbi based API tests, > > but I > > > > > >>> don't know if those provide > > > > > >>> enough coverage to replace the tests on Tempest side. > > > > > >>> > > > > > >>> > > > > > >> Yes, the heat gabbi api tests do not yet have the same coverage > > as the > > > > > >> tempest tree api tests (lacks tests using nova, neutron and swift > > > > > >> resources), but I think that should not stop us from *not* > > running > > > > the > > > > > >> tempest tests in the grenade job. > > > > > >> > > > > > >> I also don't know if the tempest tree heat tests are used by any > > other > > > > > >> upstream/downstream jobs. We could surely add more tests to bridge > > > > the gap. > > > > > >> > > > > > >> Also, It's possible to run the heat integration tests (we've > > enough > > > > > >> coverage there) with tempest plugin after doing some initial > > setup, > > > > as we > > > > > >> do in all our dsvm gate jobs. > > > > > >> > > > > > >> It would propose to move tests and client to a Tempest plugin > > owned / > > > > > >>> maintained by > > > > > >>> the Heat team, so that the Heat team can have full flexibility in > > > > > >>> consolidating their integration > > > > > >>> tests. For Murano and Daisycloud - and any other team that may > > want > > > > to > > > > > >>> use the Heat service > > > > > >>> client in their tests, even if the client is removed from > > Tempest, it > > > > > >>> would still be available via > > > > > >>> the Heat Tempest plugin. As long as the plugin implements the > > service > > > > > >>> client interface, > > > > > >>> the Heat service client will register automatically in the > > service > > > > > >>> client manager and be available > > > > > >>> for use as today. > > > > > >>> > > > > > >>> > > > > > >> if I understand correctly, you're proposing moving the existing > > > > tempest > > > > > >> tests and service clients to a separate repo managed by heat team. > > > > Though > > > > > >> that would be collective decision, I'm not sure that's something I > > > > would > > > > > >> like to do. To start with we may look at adding some of the > > missing > > > > pieces > > > > > >> in heat tree itself. > > > > > >> > > > > > > > > > > > > I'm proposing to move tests and the service client outside of > > tempest > > > > to a > > > > > > new home. > > > > > > > > > > > > I also suggested that the new home could be a dedicate repo, since > > that > > > > > > would allow you to maintain the > > > > > > current branchless nature of those tests. A more detailed > > discussion > > > > about > > > > > > the topic can be found > > > > > > in the corresponding proposed queens goal [5], > > > > > > > > > > > > Using a dedicated repo *is not* a precondition for moving tests and > > > > > > service client out of Tempest. > > > > > > > > > > > > > > > > > We probably are mixing two different things here. > > > > > > > > > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo > > > > > > > > > > Though we don't have any plans for it now, we may have to do it > > when/if > > > > > it's accepted as a community goal. > > > > > > > > > > 2. Moving tempest tree heat tests and heat service client to a new > > home > > > > > and owner. > > > > > > > > > > I don't think that's something heat team would like to do given that > > we > > > > > don't use these tests anywhere and would probably spend time > > improving > > > > the > > > > > coverage of the gabbi api tests we already have. > > > > > > > > > > > > > > > The test coverage provided by those tests is not available anywhere else > > at > > > the moment, however the tests are not really used, so I'm confused about > > > this. > > > > > > You could run the existing Tempest tests with very little extra effort > > > until the > > > corresponding coverage is available in Gabbi format. > > > > > > > > > > > > > > Ok, well if the heat team has no interest in maintaining these tests > > there > > > > is > > > > no point in keeping them around anymore. I've pushed up: > > > > > > > > https://review.openstack.org/461841 > > > > > > > > > Thanks for this. > > > We will need Heat PTL / QA Liaison +1 on the test removal before it goes > > > through. > > > > > > > > > > > > > > to remove the tests. As for the clients we can just move those to > > > > tempest.lib > > > > to not break the plugins that are using them. > > > > > > > > > > We could, but other projects maintain their own service clients, I'm not > > > convinced we should > > > make a special case for Heat. > > > > > > > I'm not viewing it as a special case for heat, more treating it like any > > other > > tempest interface that has a number of external consumers already but > > isn't in > > lib yet. I don't think it's at all unreasonable to maintain the 321 lines > > of > > heat client code in tempest.lib so that the plugins identified on this this > > thread (and any others out there) don't lose an interface they rely on. The > > maintenance burden for the clients is not high at all, especially since the > > Rest API is supposed to be stable interface nothing should really need to > > change in the client code often. We won't be adding new features just > > preserving the existing interfaces. We just will need to document that > > they're > > not self tested anymore because the tests no longer exist. > > > > Also, like it or not it this already is a special case. This is the first > > and > > only time a project team didn't take ownership of their project's tempest > > tests > > since we made the decision to migrate higher level services into plugins > > back > > at the Liberty summit. (it's also the last project left as part of that > > migration) > > > > The heat tree has 215 tests exposed as a tempest plugin, including one > which runs a gabbi suite. So I *really* struggle to respond to the > suggestion that we "didn't take ownership" without feelings of > defensiveness and anger. > > What we didn't do is use the tempest client, so moving existing tests over > isn't a simple copy. Also it took a while to get the gabbi stuff in place > to be able to rewrite the api coverage that the heat tempest tests have.
I didn't mean for it to come across as offensive, but the facts here are we're deleting these tests after waiting patiently ~2 years for heat team to migrate the tests out of tempest and there still really isn't equivalent coverage setup. And TBH I don't actually care, it's up to you what you end up using for your testing. Whether it's gabbi, tempest clients, etc or even whether you choose to expose any tests as a tempest plugin that's not what I was talking about. What I meant by take ownership is literal, which has been the thing we've been asking for the past ~2 years, and to just migrate the heat tests from the tempest tree and expose them into a plugin owned by heat. (to do whatever you wanted with, which could have been deprecate and delete) Every other project team for a higher level service did this. Heat is the only project that decided not to do this, which is fine. But, that's why this situation is unique. It's not a judgement on the heat team (although I do wish you had said something earlier about not wanting the tests so we could have cleaned things up in tempest a long time ago) I was just expressing this situation is unique because it has implications for other tempest users that are leveraging the heat clients. This wasn't a concern for any other projects because we just documented that the imports had to be updated after the migration. -Matt Treinish
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
