On Thursday, 4 May 2017 15:41:04 CEST Dan Prince wrote: > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > > ----- Original Message ----- > >
> > > > > > Running a subset of Tempest tests isn't the same thing as designing > > > (and owning) your own test suite that targets the things that mean > > > the > > > most to our community (namely speed and coverage). Even giving up > > > 5-10 > > > minutes of runtime...just to be able to run Tempest isn't something > > > that some of us would be willing to do. > > > > As I mentioned, you can do it with Tempest (the library). You can > > have your own test suite that does exactly what you are asking > > (namely, a set of scenario tests based on Heat which targets the > > TripleO use case) in a Tempest plugin and there is no absolute reason > > that those tests should add 5-10 minutes of runtime compared to > > pingtest. > > > > It/they would be exactly pingtest, only implemented using a different > > library and running with a different runner, with the *exact* same > > run time. > > > > Obvious advantages: only one technology used to run tests, so if > > anyone else want to run additional tests, there is no need to > > maintain two code paths; reuse on a big and proven library of test > > and test runner tools. > > I like the idea of getting pingtest out of tripleo.sh as more of a > stand alone tool. I would support an effort that re-implemented it... > and using tempest-lib would be totally fine. And as you point out one > could even combine these tests with a more common "Tempest" run that > incorporates the scenarios, etc. That's the idea, yes: anyone would be able to consume it easily with the other tests; just a regexp away. > To me the message is clear that we DO NOT want to consume the normal > Tempest scenarios in TripleO upstream CI at this point. Sure there is > overlap there, but the focus of those tests is just plain different... > speed isn't a primary concern there as it is for us so I don't think we > should do it now. And probably not ever unless the CI job time is less > than an hour. Like even if we were able to tune a set of stock Tempest > smoke tests today to our liking unless TripleO proper gates on the > runtime of those not increasing we'd be at risk of breaking our CI > queues as the wall time would potentially get too long. In this regard > this entire thread is poorly named I think in that we are no longer > talking about 'pingtest vs. tempest' but rather the implementation > details of how we reimplement our existing pingtest to better suite the > community. > > So ++ for the idea of experimenting with the use of tempest.lib. But > stay away from the idea of using Tempest smoke tests and the like for > TripleO I think ATM. That would be good! > > Its also worth noting there is some risk when maintaining your own in- > tree Tempest tests [1]. If I understood that thread correctly that > breakage wouldn't have occurred if the stable branch tests were gating > Tempest proper... which is a very hard thing to do if we have our own > in-tree stuff. So there is a cost to doing what you suggest here, but > probably one that we'd be willing to accept. About this, the idea is not to put the Tempest plugin in-tree, but keep it in a separate repository (and keep it branchless like Tempest; as you test the API. We did this for Sahara tests since liberty with good results. Moreover, there is a proposed global goal for Queen to decouple the Tempest plugin in separate repositories: https://review.openstack.org/#/c/369749/ -- Luigi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev