On Thu, May 4, 2017 at 11:11 PM Dan Prince <dpri...@redhat.com> wrote:
> On Thu, 2017-05-04 at 14:11 -0400, Emilien Macchi wrote: > > On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dpri...@redhat.com> > > wrote: > > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > > > > ----- Original Message ----- > > > > > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > > > > > > (cross-posting) > > > > > > Instead of running the Pingtest, we would execute a Tempest > > > > > > Scenario > > > > > > that boot an instance from volume (like Pingstest is already > > > > > > doing) > > > > > > and see how it goes (in term of coverage and runtime). > > > > > > I volunteer to kick-off the work with someone more expert > > > > > > than I > > > > > > am > > > > > > with quickstart (Arx maybe?). > > > > > > > > > > > > Another iteration could be to start building an easy > > > > > > interface to > > > > > > select which Tempest tests we want a TripleO CI job to run > > > > > > and > > > > > > plug > > > > > > it > > > > > > to our CI tooling (tripleo-quickstart I presume). > > > > > > > > > > Running a subset of Tempest tests isn't the same thing as > > > > > designing > > > > > (and owning) your own test suite that targets the things that > > > > > mean > > > > > the > > > > > most to our community (namely speed and coverage). Even giving > > > > > up > > > > > 5-10 > > > > > minutes of runtime...just to be able to run Tempest isn't > > > > > something > > > > > that some of us would be willing to do. > > > > > > > > As I mentioned, you can do it with Tempest (the library). You can > > > > have your own test suite that does exactly what you are asking > > > > (namely, a set of scenario tests based on Heat which targets the > > > > TripleO use case) in a Tempest plugin and there is no absolute > > > > reason > > > > that those tests should add 5-10 minutes of runtime compared to > > > > pingtest. > > > > > > > > It/they would be exactly pingtest, only implemented using a > > > > different > > > > library and running with a different runner, with the *exact* > > > > same > > > > run time. > > > > > > > > Obvious advantages: only one technology used to run tests, so if > > > > anyone else want to run additional tests, there is no need to > > > > maintain two code paths; reuse on a big and proven library of > > > > test > > > > and test runner tools. > > > > > > I like the idea of getting pingtest out of tripleo.sh as more of a > > > stand alone tool. I would support an effort that re-implemented > > > it... > > > and using tempest-lib would be totally fine. And as you point out > > > one > > > could even combine these tests with a more common "Tempest" run > > > that > > > incorporates the scenarios, etc. > > > > I don't understand why we would re-implement the pingtest in a > > tempest plugin. > > Could you please tell us what is the technical difference between > > what > > does this scenario : > > https://github.com/openstack/tempest/blob/master/tempest/scenario/tes > > t_volume_boot_pattern.py > > > > And this pingtest: > > https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pi > > ngtests/tenantvm_floatingip.yaml > > > > They both create a volume Cinder, snapshot it in Glance and and spawn > > a Nova server from the volume. > > > > What one does that the other one doesn't? > > I don't think these are the same things. Does the Tempest test even > create a floating IP? And in the case of pingtest we also cover Heat > API in the overcloud (also valuable coverage). And even if they could > be made to match today is there any guarantee that they would diverge > in the future or maintain the same speed goals as that test lives in > Tempest (and most TripleO cores don't review there). > > The main difference that I care about is it is easier for us to > maintain and fix the pingtest varient at this point. We care a lot > about our CI, and like I said before increasing the runtime isn't > something we could easily tolerate. I'm willing to entertain reuse so > long as it also allows us the speed and control we desire. > > > > > > To me the message is clear that we DO NOT want to consume the > > > normal > > > Tempest scenarios in TripleO upstream CI at this point. Sure there > > > is > > > overlap there, but the focus of those tests is just plain > > > different... > > > > I haven't seen strong pushback in this thread except you. > > Perhaps most cores haven't weighed in on this issue because moving to > Tempest(-lib) isn't the most pressing issue ATM. We have a lot of > architectural changes happening at the moment for example and that is > why I only replied to this thread this week. > > > I'm against overlap in general and this one is pretty obvious. Why > > would we maintain a tripleo-specific Tempest scenario while existing > > ones would work for us. Please give me a technical reason in what is > > not good enough in the existing scenarios. > > We maintain it because we care about speed I think. We (QA) care about speed as well :) Even when deploying with devstack, we are rather resource and time contained. Granted, not to the level of TripleO perhaps. We use a "slow" tag to mark tests that take longer to run, which could be useful for you as well. > Also, the re-use > you talk about here has a fairly large cost to us in that we'd be > introducing new dependencies and code reviews for TripleO. All of this > has a cost too... when it really comes down to it I think the simpler > implementation of ping test suites what we do and need in TripleO quite > fine already. > > My assumption was that some of the people chiming into this thread > would do the heavy lifting to give us a tempest.lib parity test that > covers the same things we do today (in a similar runtime). Call it an > experiment I guess... just to see what it would look like if we went > down this path. If it works out... and everyone likes it great. If not > then perhaps we still keep some of the simplicity we have today with > pingtest. But I'm still not convinced that this test lives in Tempest > proper because then we'd loose our guarantees of speed (runtime) and > coverage as it is out of our control. > > Dan > > > > > > > speed isn't a primary concern there as it is for us so I don't > > > think we > > > should do it now. And probably not ever unless the CI job time is > > > less > > > than an hour. Like even if we were able to tune a set of stock > > > Tempest > > > smoke tests today to our liking unless TripleO proper gates on the > > > runtime of those not increasing we'd be at risk of breaking our CI > > > queues as the wall time would potentially get too long. In this > > > regard > > > this entire thread is poorly named I think in that we are no longer > > > talking about 'pingtest vs. tempest' but rather the implementation > > > details of how we reimplement our existing pingtest to better suite > > > the > > > community. > > > > What I would like to see if we're going to use Tempest in our gate, > > is > > to run at least one TripleO jobs as voting in Tempest. > > > > Tempest folks: I need your support here. We have been running Puppet > > jobs as non-voting and we have seen a quite number of patches that > > broke us because folks were ignore the jobs. If we switch TripleO to > > use more Tempest, being in your gate might be required. We'll run the > > fastest and more stable job that we have to make sure the impact for > > you is minimal. > > > > > So ++ for the idea of experimenting with the use of tempest.lib. > > > But > > > stay away from the idea of using Tempest smoke tests and the like > > > for > > > TripleO I think ATM. > > > > > > Its also worth noting there is some risk when maintaining your own > > > in- > > > tree Tempest tests [1]. If I understood that thread correctly that > > > breakage wouldn't have occurred if the stable branch tests were > > > gating > > > Tempest proper... which is a very hard thing to do if we have our > > > own > > > in-tree stuff. So there is a cost to doing what you suggest here, > > > but > > > probably one that we'd be willing to accept. > > > > I'm not sure we have the resources to write and maintain our own > > in-tree tempest plugin, tbh. > > > > > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116 > > > 172. > > > html > > > > > > Dan > > > > > > > It sounds like we haven't reached full consensus yet but at least we > > agreed on experimenting with Tempest. > > > > While we make progress on the discussion, I'll start working with > > some > > folks as experimental jobs, so we can make progress in the meantime. > > > > Thanks, > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev