On Wed, Jul 09, 2014 at 05:41:10AM -0400, Eoghan Glynn wrote: > > TL;DR: branchless Tempest shouldn't impact on backporting policy, yet > makes it difficult to test new features not discoverable via APIs > > Folks, > > At the project/release status meeting yesterday[1], I raised the issue > that featureful backports to stable are beginning to show up[2], purely > to facilitate branchless Tempest. We had a useful exchange of views on > IRC but ran out of time, so this thread is intended to capture and > complete the discussion.
So, [2] is definitely not something that should be backported. But, doesn't it mean that cinder snapshot notifications don't work at all in icehouse? Is this reflected in the release notes or docs somewhere, because it seems like something that would be expected to work, which, I think, is actually a bigger bug being exposed by branchless tempest. As a user how do I know that with the cloud I'm using whether cinder snapshot notifications are supported? > > The issues, as I see it, are: > > * Tempest is expected to do double-duty as both the integration testing > harness for upstream CI and as a tool for externally probing capabilities > in public clouds > > * Tempest has an implicit bent towards pure API tests, yet not all > interactions between OpenStack services that we want to test are > mediated by APIs I think this is the bigger issue. If there is cross-service communication it should have an API contract. (and probably be directly tested too) It doesn't necessarily have to be a REST API, although in most cases that's easier. This is probably something for the TC to discuss/mandate, though. > > * We don't have another integration test harness other than Tempest > that we could use to host tests that don't just make assertions > about the correctness/presence of versioned APIs > > * We want to be able to add new features to Juno, or fix bugs of > omission, in ways that aren't necessarily discoverable in the API; > without backporting these patches to stable if we wouldn't have > done so under the normal stable-maint policy[3] > > * Integrated projects are required[4] to provide Tempest coverage, > so the rate of addition of tests to Tempest is unlikely to slow > down anytime soon > > So the specific type of test that I have in mind would be common > for Ceilometer, but also possibly for Ironic and others: > > 1. an end-user initiates some action via an API > (e.g. calls the cinder snapshot API) > > 2. this initiates some actions behind the scenes > (e.g. a volume is snapshot'd and a notification emitted) > > 3. the test reasons over some expected side-effect > (e.g. some metering data shows up in ceilometer) > > The branchless Tempest spec envisages new features will be added and > need to be skipped when testing stable/previous, but IIUC requires > that the presence of new behaviors is externally discoverable[5]. I think the test case you proposed is fine. I know some people will argue that it is expanding the scope of tempest to include more whitebox like testing, because the notification are an internal side-effect of the api call, but I don't see it that way. It feels more like exactly what tempest is there to enable testing, a cross-project interaction using the api. I'm pretty sure that most of the concerns around tests like this were from the gate maintenance and debug side of things. In other words when things go wrong how impossible will it be to debug that a notification wasn't generated or not counted? Right now I think it would be pretty difficult to debug a notification test failure, which is where the problem is. While I think testing like this is definitely valid, that doesn't mean we should rush in a bunch of sloppy tests that are impossible to debug, because that'll just make everyone sad panda. But, they're is also a slight misunderstanding here. Having a feature be externally discoverable isn't a hard requirement for a config option in tempest, it's just *strongly* recommended. Mostly, because if there isn't a way to discover it how are end users expected to know what will work. For this specific case I think it's definitely fair to have an option for which notifications services are expected to be generated. That's something that is definitely a configurable option when setting up a deployment, and is something that feels like a valid tempest config option, so we know which tests will work. We already have similar feature flags for config time options in the services, and having options like that would also get you out of that backport mess you have right now. However, it does raise the question of being an end user how am I expected to know which notifications get counted? Which is why having the feature discoverability is generally a really good idea. > > One approach mooted for allowing these kind of scenarios to be tested > was to split off the pure-API aspects of Tempest so that it can be used > for probing public-cloud-capabilities as well as upstream CI, and then > build project-specific mini-Tempests to test integration with other > projects. > > Personally, I'm not a fan of that approach as it would require a lot > of QA expertise in each project, lead to inefficient use of CI > nodepool resources to run all the mini-Tempests, and probably lead to > a divergent hotchpotch of per-project approaches. I think the proposal here was for people interested in doing whitebox testing, where there is a desire to test an internal project mechanism. I could see the argument for testing notifications this way, but that would have to be for every project individually. There are already several projects that have functional testing like this in tree and run them as a gating job. There are definitely certain classes of testing where doing this makes sense. > > Another idea would be to keep all tests in Tempest, while also > micro-versioning the services such that tests can be skipped on the > basis of whether a particular feature-adding commit is present. > > When this micro-versioning can't be discovered by the test (as in the > public cloud capabilities probing case), those tests would be skipped > anyway. Yeah, I'm not a fan of this approach at all. It is just a bad way of reimplementing a temporally-aware tempest. But, instead of using branches we have arbitrary service versions. It sacrifices all the real advantages of having branchless tempest, but adds more complexity around the version discovery to all the projects and tempest. If we want to revert back to a temporally aware tempest, which I don't think we should, then going back the branched model is what we should do. > > The final, less palatable, approach that occurs to me would be to > revert to branchful Tempest. So I don't think we're anywhere near this. I think what we're hitting here is more a matter of projects trying to map out exactly how to test things for real in the gate with tempest. While at the same time coming to understand that things don't quite work as well as we expected. I think that we have to remember that this is the first cycle with branchless tempest it's still new for everyone and what we're hitting here are just some of the growing pains around it. Having discussions like this and mapping out the requirements more completely is th I recognize that for projects that didn't have any real testing before we started branchless tempest it's harder to get things going with it. Especially because in my experience the adage "if it isn't tested it's broken" tends to hold true. So I expect there will be a lot of non-backportable fixes just to enable testing. What this friction with branchless tempest is showing us is that these fixes, besides fixing the bug, will also have implications for people using OpenStack clouds. Which I feel is invaluable information to collect, and definitely something we should gate on. The open question is how do we make it easier the to enable testing for new things. > > Any other ideas, or preferences among the options laid out above? > > Cheers, > Eoghan > > [1] > http://eavesdrop.openstack.org/meetings/project/2014/project.2014-07-08-21.03.html > [2] https://review.openstack.org/104863 > [3] https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes > [4] > https://github.com/openstack/governance/blob/master/reference/incubation-integration-requirements.rst#qa-1 > [5] > https://github.com/openstack/qa-specs/blob/master/specs/implemented/branchless-tempest.rst#scenario-1-new-tests-for-new-features > -Matt Treinish
pgpdvT5oYpl7m.pgp
Description: PGP signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
