On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann <mriede...@gmail.com> wrote:
On 3/10/2017 3:02 PM, Andrea Frittoli wrote: > > We had a couple of sessions related to this topic at the PTG [0][1]. > > We agreed that we want to still maintain integration tests only in > Tempest, which means that API micro versions that have no integration > impact can be tested via functional tests. To be clear, "integration" here means tests that span operations across multiple services, correct? Like a compute test that first creates a port in the networking service and a volume in the block storage service and then uses those to create a server and maybe take a snapshot of it which is then verified was uploaded to the image service. Yes, indeed. The non-integration things are self-contained in a single service, like if all you need to do is create an aggregate, show it's details and validate the response, at a particular microversion, we can just do that in nova functional tests, and it's not necessary in Tempest. It might be worth having a definition of this policy in the Tempest docs so when people ask this question again you can just point at the docs. I agree, we need to go a better job at documenting this. I just want to avoid having a black and white rule that would not work. To me tests that should be in Tempest are tests that make sense to run against any change in any repo, and the reasons for that can be many: - the test involves multiple services (more that "it uses a token though") - the test covers a feature which is key for interoperability - the test must be reliable and not take too long Based on this I don't it's possible to completely avoid the discussion about which test should in Tempest and which not, it's something to be considered on a test by test basis, based on a set of guidelines. We have an initial definition of scope in [0], but it's probably worth to elaborate more on it. I'll put up a patch so that the discussion on it can continue in gerrit. [0] https://docs.openstack.org/developer/tempest/test-removal.html#tempest-scope > > In terms of which versions we test in the gate, for nova we always run > with min_microversion = None and max_microversion = latest, which means > that all tests will be executed. > Since micro versions are incremental, and each micro version usually > involves no or one test on Tempest side, I think it will be a while > before this becomes an issue for the common gate. We test max_microversion=latest only on master. On the devstack stable branch we cap the max_microversion, e.g.: Thanks, good point. https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339 -- Thanks, Matt __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev