On Thu, May 4, 2017 at 7:13 PM Emilien Macchi <emil...@redhat.com> wrote:
> On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dpri...@redhat.com> wrote: > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > >> ----- Original Message ----- > >> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > >> > > (cross-posting) > >> > > >> > > Instead of running the Pingtest, we would execute a Tempest > >> > > Scenario > >> > > that boot an instance from volume (like Pingstest is already > >> > > doing) > >> > > and see how it goes (in term of coverage and runtime). > >> > > I volunteer to kick-off the work with someone more expert than I > >> > > am > >> > > with quickstart (Arx maybe?). > >> > > > >> > > Another iteration could be to start building an easy interface to > >> > > select which Tempest tests we want a TripleO CI job to run and > >> > > plug > >> > > it > >> > > to our CI tooling (tripleo-quickstart I presume). > >> > > >> > Running a subset of Tempest tests isn't the same thing as designing > >> > (and owning) your own test suite that targets the things that mean > >> > the > >> > most to our community (namely speed and coverage). Even giving up > >> > 5-10 > >> > minutes of runtime...just to be able to run Tempest isn't something > >> > that some of us would be willing to do. > >> > >> As I mentioned, you can do it with Tempest (the library). You can > >> have your own test suite that does exactly what you are asking > >> (namely, a set of scenario tests based on Heat which targets the > >> TripleO use case) in a Tempest plugin and there is no absolute reason > >> that those tests should add 5-10 minutes of runtime compared to > >> pingtest. > >> > >> It/they would be exactly pingtest, only implemented using a different > >> library and running with a different runner, with the *exact* same > >> run time. > >> > >> Obvious advantages: only one technology used to run tests, so if > >> anyone else want to run additional tests, there is no need to > >> maintain two code paths; reuse on a big and proven library of test > >> and test runner tools. > > > > I like the idea of getting pingtest out of tripleo.sh as more of a > > stand alone tool. I would support an effort that re-implemented it... > > and using tempest-lib would be totally fine. And as you point out one > > could even combine these tests with a more common "Tempest" run that > > incorporates the scenarios, etc. > > I don't understand why we would re-implement the pingtest in a tempest > plugin. > Could you please tell us what is the technical difference between what > does this scenario : > > https://github.com/openstack/tempest/blob/master/tempest/scenario/test_volume_boot_pattern.py > > And this pingtest: > > https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pingtests/tenantvm_floatingip.yaml > > They both create a volume Cinder, snapshot it in Glance and and spawn > a Nova server from the volume. > > What one does that the other one doesn't? > > > To me the message is clear that we DO NOT want to consume the normal > > Tempest scenarios in TripleO upstream CI at this point. Sure there is > > overlap there, but the focus of those tests is just plain different... > > I haven't seen strong pushback in this thread except you. > I'm against overlap in general and this one is pretty obvious. Why > would we maintain a tripleo-specific Tempest scenario while existing > ones would work for us. Please give me a technical reason in what is > not good enough in the existing scenarios. > > > speed isn't a primary concern there as it is for us so I don't think we > > should do it now. And probably not ever unless the CI job time is less > > than an hour. Like even if we were able to tune a set of stock Tempest > > smoke tests today to our liking unless TripleO proper gates on the > > runtime of those not increasing we'd be at risk of breaking our CI > > queues as the wall time would potentially get too long. In this regard > > this entire thread is poorly named I think in that we are no longer > > talking about 'pingtest vs. tempest' but rather the implementation > > details of how we reimplement our existing pingtest to better suite the > > community. > > What I would like to see if we're going to use Tempest in our gate, is > to run at least one TripleO jobs as voting in Tempest. > > Tempest folks: I need your support here. We have been running Puppet > jobs as non-voting and we have seen a quite number of patches that > broke us because folks were ignore the jobs. If we switch TripleO to > use more Tempest, being in your gate might be required. We'll run the > fastest and more stable job that we have to make sure the impact for > you is minimal. > > For TripleO I guess the risks of failures might be about deploying tempest tests and their dependencies more than else. We're increasing our stable API surface constantly, and I believe the number of breakages have decreased over time - I don't have data to support this though. >From a test pov, any cloud deployed by TripleO should be able to run a few basic scenarios and I would say the likelihood of Tempest breaking that is rather low. We could start with a non-voting job like for puppet and take it from there. > So ++ for the idea of experimenting with the use of tempest.lib. But > > stay away from the idea of using Tempest smoke tests and the like for > > TripleO I think ATM. > > > > Its also worth noting there is some risk when maintaining your own in- > > tree Tempest tests [1]. If I understood that thread correctly that > > breakage wouldn't have occurred if the stable branch tests were gating > > Tempest proper... which is a very hard thing to do if we have our own > > in-tree stuff. So there is a cost to doing what you suggest here, but > > probably one that we'd be willing to accept. > > I'm not sure we have the resources to write and maintain our own > in-tree tempest plugin, tbh. > > > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116172. > > html > > > > Dan > > > > It sounds like we haven't reached full consensus yet but at least we > agreed on experimenting with Tempest. > > While we make progress on the discussion, I'll start working with some > folks as experimental jobs, so we can make progress in the meantime. > > Thanks, > -- > Emilien Macchi > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev