Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-09-06 Thread Steven Hardy
On Thu, Apr 6, 2017 at 12:29 PM, Justin Kilpatrick wrote: > Maybe I'm getting a little off topic with this question, but why was > Tempest removed last time? > > I'm not well versed in the history of this discussion, but from what I > understand Tempest in the gate has > been an off and on again t

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-09-05 Thread Emilien Macchi
On Tue, Sep 5, 2017 at 11:35 PM, Giulio Fidente wrote: [...] > slightly OT but notable mention, scenario001/container is also the one > gating Ceph via ceph-ansible currently :D > > ... soon to convert scenario004/container as well Really good work Giulio and John to make that happen, thanks agai

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-09-05 Thread Giulio Fidente
On 09/05/2017 09:26 PM, Emilien Macchi wrote: > On Wed, Apr 5, 2017 at 1:49 PM, Emilien Macchi wrote: > [...] > >> == Solutions >> >> 1) Switch pingtest to Tempest run on some specific tests, with feature >> parity of what we had with pingtest. >> For example, we could imagine to run the scenario

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-09-05 Thread Emilien Macchi
On Wed, Apr 5, 2017 at 1:49 PM, Emilien Macchi wrote: [...] > == Solutions > > 1) Switch pingtest to Tempest run on some specific tests, with feature > parity of what we had with pingtest. > For example, we could imagine to run the scenarios that deploys VM and > boot from volume. It would test t

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-05 Thread Luigi Toscano
On Thursday, 4 May 2017 20:11:14 CEST Emilien Macchi wrote: > On Thu, May 4, 2017 at 9:41 AM, Dan Prince wrote: > > I like the idea of getting pingtest out of tripleo.sh as more of a > > stand alone tool. I would support an effort that re-implemented it... > > and using tempest-lib would be total

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Andrea Frittoli
On Thu, May 4, 2017 at 7:13 PM Emilien Macchi wrote: > On Thu, May 4, 2017 at 9:41 AM, Dan Prince wrote: > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > >> - Original Message - > >> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > >> > > (cross-posting) > >> > >

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Andrea Frittoli
On Thu, May 4, 2017 at 11:11 PM Dan Prince wrote: > On Thu, 2017-05-04 at 14:11 -0400, Emilien Macchi wrote: > > On Thu, May 4, 2017 at 9:41 AM, Dan Prince > > wrote: > > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > > > > - Original Message - > > > > > On Wed, 2017-05-03 a

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Dan Prince
On Thu, 2017-05-04 at 14:11 -0400, Emilien Macchi wrote: > On Thu, May 4, 2017 at 9:41 AM, Dan Prince > wrote: > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > > > - Original Message - > > > > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > > > > > (cross-posting) >

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Emilien Macchi
On Thu, May 4, 2017 at 9:41 AM, Dan Prince wrote: > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: >> - Original Message - >> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: >> > > (cross-posting) >> > >> > > Instead of running the Pingtest, we would execute a Tempest >

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Luigi Toscano
On Thursday, 4 May 2017 15:41:04 CEST Dan Prince wrote: > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > > - Original Message - > > > > > > > > Running a subset of Tempest tests isn't the same thing as designing > > > (and owning) your own test suite that targets the things th

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Dan Prince
On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote: > - Original Message - > > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > > > (cross-posting) > > > > > Instead of running the Pingtest, we would execute a Tempest > > > Scenario > > > that boot an instance from volume (li

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Chandan kumar
On Thu, May 4, 2017 at 5:34 PM, Arx Cruz wrote: > > > On Wed, May 3, 2017 at 11:53 PM, Emilien Macchi wrote: >> >> (cross-posting) >> >> I've seen a bunch of interesting thoughts here. >> The most relevant feedback I've seen so far: >> >> - TripleO folks want to keep testing fast and efficient. >

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Arx Cruz
On Wed, May 3, 2017 at 11:53 PM, Emilien Macchi wrote: > (cross-posting) > > I've seen a bunch of interesting thoughts here. > The most relevant feedback I've seen so far: > > - TripleO folks want to keep testing fast and efficient. > - Tempest folks understand this problematic and is willing to

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-04 Thread Luigi Toscano
- Original Message - > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > > (cross-posting) > > > Instead of running the Pingtest, we would execute a Tempest Scenario > > that boot an instance from volume (like Pingstest is already doing) > > and see how it goes (in term of covera

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-03 Thread Dan Prince
On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote: > (cross-posting) > > I've seen a bunch of interesting thoughts here. > The most relevant feedback I've seen so far: > > - TripleO folks want to keep testing fast and efficient. > - Tempest folks understand this problematic and is willing t

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-05-03 Thread Emilien Macchi
(cross-posting) I've seen a bunch of interesting thoughts here. The most relevant feedback I've seen so far: - TripleO folks want to keep testing fast and efficient. - Tempest folks understand this problematic and is willing to collaborate. I propose that we move forward and experiment the usage

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-18 Thread Attila Fazekas
On Tue, Apr 18, 2017 at 11:04 AM, Arx Cruz wrote: > > > On Tue, Apr 18, 2017 at 10:42 AM, Steven Hardy wrote: > >> On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote: >> > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec >> wrote: >> > > Tempest isn't really either of those things. A

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-18 Thread Luigi Toscano
On Monday, 17 April 2017 18:28:24 CEST Ben Nemec wrote: > On 04/17/2017 10:51 AM, Emilien Macchi wrote: > > We haven't got much feedback from TripleO core reviewers, who are > > usually more involved on this topic. I'll give a chance to let them > > talk because we take some actions based on the fe

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-18 Thread Andrea Frittoli
On Tue, Apr 18, 2017 at 10:07 AM Arx Cruz wrote: > On Tue, Apr 18, 2017 at 10:42 AM, Steven Hardy wrote: > >> On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote: >> > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec >> wrote: >> > > Tempest isn't really either of those things. Accord

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-18 Thread Arx Cruz
On Tue, Apr 18, 2017 at 10:42 AM, Steven Hardy wrote: > On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote: > > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec > wrote: > > > Tempest isn't really either of those things. According to another > message > > > in this thread it takes aro

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-18 Thread Steven Hardy
On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote: > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec wrote: > > Tempest isn't really either of those things. According to another message > > in this thread it takes around 15 minutes to run just the smoke tests. > > That's unacceptable

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-17 Thread Justin Kilpatrick
ts >>> we are trying to bring up again, if it is done today wil ping you and >>> schedule again >>> >>> Otherwise i will schedule for Monday >>> >>> Thanks, >>> Ravi >>> - Original Message - >>> From:

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-17 Thread Ben Nemec
ou and schedule again Otherwise i will schedule for Monday Thanks, Ravi - Original Message - From: jtale...@redhat.com To: openstack-dev@lists.openstack.org, jkilp...@redhat.com Sent: Thursday, April 6, 2017 4:15:58 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: Re: [openstac

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-17 Thread Emilien Macchi
sage - > From: jtale...@redhat.com > To: openstack-dev@lists.openstack.org, jkilp...@redhat.com > Sent: Thursday, April 6, 2017 4:15:58 PM GMT +05:30 Chennai, Kolkata, Mumbai, > New Delhi > Subject: Re: [openstack-dev] [tripleo] pingtest vs tempest > > On Thu, Apr 6, 2017 at 5:32 A

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-07 Thread Ravi Sekhar Reddy Konda
@lists.openstack.org, jkilp...@redhat.com Sent: Thursday, April 6, 2017 4:15:58 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: Re: [openstack-dev] [tripleo] pingtest vs tempest On Thu, Apr 6, 2017 at 5:32 AM, Sagi Shnaidman wrote: > HI, > > I think Rally or Browbeat and other pe

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Flavio Percoco
On 06/04/17 12:32 +0300, Sagi Shnaidman wrote: HI, I think Rally or Browbeat and other performance oriented solutions won't serve our needs, because we run TripleO CI on virtualized environment with very limited resources. Actually we are pretty close to full utilizing these resources when deplo

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Assaf Muller
On Wed, Apr 5, 2017 at 4:49 PM, Emilien Macchi wrote: > Greetings dear owls, > > I would like to bring back an old topic: running tempest in the gate. > > == Context > > Right now, TripleO gate is running something called pingtest to > validate that the OpenStack cloud is working. It's an Heat st

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Arx Cruz
I would say smoke tests, and at least the minimum scenario tests. Smoke tests takes 14 minutes (113 tests) to run, and I can check how long it takes the minimum scenario tests later. So it won't take a long time running. Kind regards, Arx Cruz On Thu, Apr 6, 2017 at 2:44 PM, Andrea Frittoli wrot

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Andrea Frittoli
I don't really have much context in what the decision is going to be based on here, so I'll just add some random comments here and there. On Thu, Apr 6, 2017 at 12:48 PM Arx Cruz wrote: > Having tempest running will allow these jobs to appear in openstack-health > system as well. > I agree that

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Luigi Toscano
On Thursday, 6 April 2017 13:29:32 CEST Justin Kilpatrick wrote: > Maybe I'm getting a little off topic with this question, but why was > Tempest removed last time? > > I'm not well versed in the history of this discussion, but from what I > understand Tempest in the gate has > been an off and on

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Arx Cruz
Having tempest running will allow these jobs to appear in openstack-health system as well. On Thu, Apr 6, 2017 at 1:29 PM, Justin Kilpatrick wrote: > Maybe I'm getting a little off topic with this question, but why was > Tempest removed last time? > > I'm not well versed in the history of this d

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Justin Kilpatrick
Maybe I'm getting a little off topic with this question, but why was Tempest removed last time? I'm not well versed in the history of this discussion, but from what I understand Tempest in the gate has been an off and on again thing for a while but I've never heard the story of why it got removed.

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Chris Dent
On Thu, 6 Apr 2017, Sagi Shnaidman wrote: It may be useful to run a "limited edition" of API tests that maximize coverage and don't duplicate, for example just to check service working basically, without covering all its functionality. It will take very little time (i.e. 5 tests for each service

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Joe Talerico
On Thu, Apr 6, 2017 at 5:32 AM, Sagi Shnaidman wrote: > HI, > > I think Rally or Browbeat and other performance oriented solutions won't > serve our needs, because we run TripleO CI on virtualized environment with > very limited resources. Actually we are pretty close to full utilizing these > res

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Boris Pavlovic
Sagi, I think Rally or Browbeat and other performance oriented solutions won't > serve our needs, because we run TripleO CI on virtualized environment with > very limited resources. Actually we are pretty close to full utilizing > these resources when deploying openstack, so very little is availab

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Dmitry Tantsur
On 04/05/2017 10:49 PM, Emilien Macchi wrote: Greetings dear owls, I would like to bring back an old topic: running tempest in the gate. == Context Right now, TripleO gate is running something called pingtest to validate that the OpenStack cloud is working. It's an Heat stack, that deploys a N

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-06 Thread Sagi Shnaidman
HI, I think Rally or Browbeat and other performance oriented solutions won't serve our needs, because we run TripleO CI on virtualized environment with very limited resources. Actually we are pretty close to full utilizing these resources when deploying openstack, so very little is available for t

Re: [openstack-dev] [tripleo] pingtest vs tempest

2017-04-05 Thread Joe Talerico
On Wed, Apr 5, 2017 at 4:49 PM, Emilien Macchi wrote: > Greetings dear owls, > > I would like to bring back an old topic: running tempest in the gate. > > == Context > > Right now, TripleO gate is running something called pingtest to > validate that the OpenStack cloud is working. It's an Heat sta