On Mon, Mar 13, 2017 at 7:37 PM, Andrea Frittoli <[email protected]> wrote:
> On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann <[email protected]> > wrote: > > On 3/10/2017 3:02 PM, Andrea Frittoli wrote: > > > > We had a couple of sessions related to this topic at the PTG [0][1]. > > > > We agreed that we want to still maintain integration tests only in > > Tempest, which means that API micro versions that have no integration > > impact can be tested via functional tests. > > To be clear, "integration" here means tests that span operations across > multiple services, correct? Like a compute test that first creates a > port in the networking service and a volume in the block storage service > and then uses those to create a server and maybe take a snapshot of it > which is then verified was uploaded to the image service. > > Yes, indeed. > > > > The non-integration things are self-contained in a single service, like > if all you need to do is create an aggregate, show it's details and > validate the response, at a particular microversion, we can just do that > in nova functional tests, and it's not necessary in Tempest. > > It might be worth having a definition of this policy in the Tempest docs > so when people ask this question again you can just point at the docs. > > > I agree, we need to go a better job at documenting this. > I just want to avoid having a black and white rule that would not work. > > To me tests that should be in Tempest are tests that make sense to run > against any change in any repo, and the reasons for that can be many: > - the test involves multiple services (more that "it uses a token though") > - the test covers a feature which is key for interoperability > - the test must be reliable and not take too long > > +1. I added those in https://review.openstack.org/#/c/444727/ > Based on this I don't it's possible to completely avoid the discussion > about > which test should in Tempest and which not, it's something to be considered > on a test by test basis, based on a set of guidelines. > > We have an initial definition of scope in [0], but it's probably worth to > elaborate > more on it. I'll put up a patch so that the discussion on it can continue > in > gerrit. > > [0] https://docs.openstack.org/developer/tempest/test- > removal.html#tempest-scope > > > > > > In terms of which versions we test in the gate, for nova we always run > > with min_microversion = None and max_microversion = latest, which means > > that all tests will be executed. > > Since micro versions are incremental, and each micro version usually > > involves no or one test on Tempest side, I think it will be a while > > before this becomes an issue for the common gate. > > We test max_microversion=latest only on master. On the devstack stable > branch we cap the max_microversion, e.g.: > > Thanks, good point. > > > > https://github.com/openstack-dev/devstack/blob/stable/ > newton/lib/tempest#L339 > > -- > > Thanks, > > Matt > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
