On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote: > A lot of this results, in part, from there being no single guiding > pattern and principle for how (and where) the tests are to be > managed.
It sounds like you want to write a general testing guide for openstack. Have you started this effort anywhere? I don't think anyone would be opposed to starting a document for that, it seems like a reasonable thing to have. But, I think you'll find there is not a one size fits all solution though, because every project has their own requirements and needs for testing. > When there's a choice between one, some and all, "some" is > almost always the wrong way to manage something. "some" is how we do > tempest (and fair few other OpenStack things). > > If it is the case that we want some projects to not put their tests > in the main tempest repo then the only conceivable pattern from a > memorability, discoverability, and equality standpoint is actually > for all the tests to be in plugins. > > If that isn't possible (and it is clear there are many reasons why > that may be the case) then we need to be extra sure that we explore > and uncover the issues that the "some" approach presents and provide > sufficient documentation, tooling, and guidance to help people get > around them. And that we recognize and acknowledge the impact it has. So have you read the documentation: https://docs.openstack.org/developer/tempest/ (or any of the other relevant documentation and filed bugs about where you think there are gaps? This is something that really bugs me sometimes (yes the pun is intended) just like anything else this is all about iterative improvements. These broad trends are things tempest and (every project hopefully) have been working on. But improvements don't just magically occur overnight it takes time to implement them. Just compare the state of the documentation and tooling from 2 years ago (when tempest started adding the plugin interface) to today. Things have steadily improved over time and the situation now is much better. This will continue and in the future things will get even better. The thing is this is open source collaborative development and there is an expectation that people who have issues with something in the project will report them or contribute a fix and communicate with the maintainers. The users of tempest's plugin interface tend to be other openstack projects (but not exclusively) and if there are something that's not clear we need to work together to fix them. Based on this paragraph I feel like you think the decision to add a tempest plugin interface and decrease it's scope was taken lightly without forethought or careful consideration. But, it's the exact opposite there was extensive debate and exploration of the problem space and took a long time to reach a consensus. > > If the answer to that is "who is going to do that?" or "who has the > time?" then I ask you to ask yourself why we think the "non-core" > projects have time to fiddle about with tempest plugins? I think this unfair simplification, no one is required to write a tempest plugin it's a choice the projects made. While I won't say the interface is perfect, things are always improving. If a project chooses to write a plugin, the expectation is that we'll all work together to help fix issues as they are encountered. No individual can do everything by themselves and it's a shared group effort. But, even so there is no shortage of work for anyone, it's all about prioritization of effort. > > And finally, I actually don't have too strong of a position in the > case of tempest and tempest plugins. What I take issue with is the > process whereby we discuss and decide these things and characterize > the various projects > > If I have any position on tempest at all it is that we should limit > it to gross cloud validation and maybe interop testing, and projects > should manage their own integration testing in tree using whatever > tooling they feel is most appropriate. If that turns out to be > tempest, cool. I fail to see how this is any different than how things work today. No one is required to use a tempest plugin and they can write tests however they want. Tempest itself has a well defined scope (which does evolve over time like any other project) and doesn't try to be all the testing everywhere. Almost every other project has it's own in tree testing outside of tempest or tempest plugins. Also, projects which have in-tree tempest tests also have tempest plugins to expand on that set of functionality. -Matt Treinish
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev