On Mon, Mar 13, 2017 at 6:52 AM Ghanshyam Mann <[email protected]> wrote:
> On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann <[email protected]> > wrote: > > On 3/10/2017 3:02 PM, Andrea Frittoli wrote: > > > We had a couple of sessions related to this topic at the PTG [0][1]. > > We agreed that we want to still maintain integration tests only in > Tempest, which means that API micro versions that have no integration > impact can be tested via functional tests. > > > To be clear, "integration" here means tests that span operations across > multiple services, correct? Like a compute test that first creates a port > in the networking service and a volume in the block storage service and > then uses those to create a server and maybe take a snapshot of it which is > then verified was uploaded to the image service. > > The non-integration things are self-contained in a single service, like if > all you need to do is create an aggregate, show it's details and validate > the response, at a particular microversion, we can just do that in nova > functional tests, and it's not necessary in Tempest. > > It might be worth having a definition of this policy in the Tempest docs > so when people ask this question again you can just point at the docs. > > > Yes, that will be helpful to have those agreement in doc. I am adding > those in existing microversion testing doc [0] and later will be > refactoring to give them better and more visible shape. > > Tempest maintain the info about what microversion tests are is implemented > on Tempest side which will be helpful for projects to check the coverage > [1]. Cinder one was missed which I added in [2]. > > As summary: > Tempest Scope: > - Only Integration tests for microversion is allowed in Tempest > If a microversion is important for interoperability it may end up in Tempest even if doesn't involve too much integration > - Exception for non integration tests to fill the schema gap if exists. > Yeah when we fill schema gaps we need at least one test for the top version to verify that the latest implemented schema is valid > > Project Scope: > - Remaining tests coverage of microversion should be on Projects side. > - Project can cover those as functional tests etc. > - Nova currently have at least one functional tests per each > microversion and plan to extend coverage like [3] > - IMO, only running tests with 'latest' microversion is not enough, > each microversion should be tested with positive and negative testing > (w.r.t at least immediate previous microversion). > > > > > In terms of which versions we test in the gate, for nova we always run > with min_microversion = None and max_microversion = latest, which means > that all tests will be executed. > Since micro versions are incremental, and each micro version usually > involves no or one test on Tempest side, I think it will be a while > before this becomes an issue for the common gate. > > > We test max_microversion=latest only on master. On the devstack stable > branch we cap the max_microversion, e.g.: > > > https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339 > > -- > > Thanks, > > Matt > > > > > ..0 https://review.openstack.org/#/c/444727/1 > ..1 > https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest > > ..2 https://review.openstack.org/#/c/444711/ > ..3 > https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests > > > Thanks > gmann > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
