Re: [openstack-dev] Which program for Rally

2014-08-18 Thread Matthew Treinish
On Fri, Aug 15, 2014 at 01:57:29AM +0400, Boris Pavlovic wrote: > Matt, > >> One thing did just occur to me while writing this though it's probably worth > > investigating splitting out the stress test framework as an external > > tool/project after we start work on the tempest library. [3] > >

Re: [openstack-dev] Which program for Rally

2014-08-15 Thread Jeremy Stanley
On 2014-08-13 19:28:48 -0700 (-0700), Joe Gordon wrote: > We actually run out of nodes almost every day now (except > weekends), we have about 800 nodes, and hit that quota most days [...] It's worth noting that a lot of the recent exhaustion has been due to OpenStack bugs impacting the providers

Re: [openstack-dev] Which program for Rally

2014-08-14 Thread Boris Pavlovic
Matt, One thing did just occur to me while writing this though it's probably worth > investigating splitting out the stress test framework as an external > tool/project after we start work on the tempest library. [3] I fully agree with the fact that stress testing doesn't belong to Tempest. Th

Re: [openstack-dev] Which program for Rally

2014-08-14 Thread Matthew Treinish
On Wed, Aug 13, 2014 at 03:48:59PM -0600, Duncan Thomas wrote: > On 13 August 2014 13:57, Matthew Treinish wrote: > > On Tue, Aug 12, 2014 at 01:45:17AM +0400, Boris Pavlovic wrote: > >> Keystone, Glance, Cinder, Neutron and Heat are running rally performance > >> jobs, that can be used for perfor

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Tom Fifield
On 13/08/14 19:55, Boris Pavlovic wrote: > Matt, > > > On Mon, Aug 11, 2014 at 07:06:11PM -0400, Zane Bitter wrote: > > On 11/08/14 16:21, Matthew Treinish wrote: > > >I'm sorry, but the fact that the > > >docs in the rally tree has a section for user testimonials [4] I feel > s

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Joe Gordon
On Wed, Aug 13, 2014 at 2:48 PM, Duncan Thomas wrote: > On 13 August 2014 13:57, Matthew Treinish wrote: > > On Tue, Aug 12, 2014 at 01:45:17AM +0400, Boris Pavlovic wrote: > >> Keystone, Glance, Cinder, Neutron and Heat are running rally performance > >> jobs, that can be used for performance t

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Robert Collins
On 14 August 2014 09:48, Duncan Thomas wrote: > On 13 August 2014 13:57, Matthew Treinish wrote: >> So this is actually the communication problem I mentioned before. Singling >> out >> individual projects and getting them to add a rally job is not "cross >> project" >> communication. (this is

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Angus Salkeld
On Wed, 2014-08-13 at 15:48 -0600, Duncan Thomas wrote: > On 13 August 2014 13:57, Matthew Treinish wrote: > > On Tue, Aug 12, 2014 at 01:45:17AM +0400, Boris Pavlovic wrote: > >> Keystone, Glance, Cinder, Neutron and Heat are running rally performance > >> jobs, that can be used for performance t

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Duncan Thomas
On 13 August 2014 13:57, Matthew Treinish wrote: > On Tue, Aug 12, 2014 at 01:45:17AM +0400, Boris Pavlovic wrote: >> Keystone, Glance, Cinder, Neutron and Heat are running rally performance >> jobs, that can be used for performance testing, benchmarking, regression >> testing (already now). These

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Matthew Treinish
On Tue, Aug 12, 2014 at 01:45:17AM +0400, Boris Pavlovic wrote: > Hi stackers, > > > I would like to put some more details on current situation. > > > > > The issue is with what Rally is in it's > > current form. It's scope is too large and monolithic, and it duplicates > > much of > > the funct

Re: [openstack-dev] Which program for Rally

2014-08-13 Thread Boris Pavlovic
Matt, On Mon, Aug 11, 2014 at 07:06:11PM -0400, Zane Bitter wrote: > > On 11/08/14 16:21, Matthew Treinish wrote: > > >I'm sorry, but the fact that the > > >docs in the rally tree has a section for user testimonials [4] I feel > speaks a > > >lot about the intent of the project. > Yes, you are a

Re: [openstack-dev] Which program for Rally

2014-08-12 Thread Matthew Treinish
On Mon, Aug 11, 2014 at 07:06:11PM -0400, Zane Bitter wrote: > On 11/08/14 16:21, Matthew Treinish wrote: > >I'm sorry, but the fact that the > >docs in the rally tree has a section for user testimonials [4] I feel speaks > >a > >lot about the intent of the project. > > What... does that even mea

Re: [openstack-dev] Which program for Rally

2014-08-12 Thread Doug Hellmann
On Aug 11, 2014, at 12:00 PM, David Kranz wrote: > On 08/06/2014 05:48 PM, John Griffith wrote: >> I have to agree with Duncan here. I also don't know if I fully understand >> the limit in options. Stress test seems like it could/should be different >> (again overlap isn't a horrible thing)

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Zane Bitter
On 11/08/14 16:21, Matthew Treinish wrote: I'm sorry, but the fact that the docs in the rally tree has a section for user testimonials [4] I feel speaks a lot about the intent of the project. What... does that even mean? "They seem like just the type of guys that would help Keystone with perf

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Boris Pavlovic
Hi stackers, I would like to put some more details on current situation. > > The issue is with what Rally is in it's > current form. It's scope is too large and monolithic, and it duplicates > much of > the functionality we either already have or need in current QA or Infra > projects. But, noth

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread David Kranz
On 08/11/2014 04:21 PM, Matthew Treinish wrote: I apologize for the delay in my response to this thread, between travelling and having a stuck 'a' key on my laptop this is the earliest I could respond. I opted for a separate branch on this thread to summarize my views and I'll respond inline

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Matthew Treinish
I apologize for the delay in my response to this thread, between travelling and having a stuck 'a' key on my laptop this is the earliest I could respond. I opted for a separate branch on this thread to summarize my views and I'll respond inline later on some of the previous discussion. On Wed, Aug

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Zane Bitter
On 08/08/14 10:41, Anne Gentle wrote: - Would have to ensure Rally is what we want "first" as getting to be PTL since you are first to propose seems to be the model. I know that at one time it was popular in the trade/gutter press to cast aspersions on new projects by saying that someone getti

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread David Kranz
On 08/06/2014 05:48 PM, John Griffith wrote: I have to agree with Duncan here. I also don't know if I fully understand the limit in options. Stress test seems like it could/should be different (again overlap isn't a horrible thing) and I don't see it as siphoning off resources so not sure of

Re: [openstack-dev] Which program for Rally

2014-08-08 Thread Anne Gentle
On Wed, Aug 6, 2014 at 5:30 AM, Thierry Carrez wrote: > Hi everyone, > > At the TC meeting yesterday we discussed Rally program request and > incubation request. We quickly dismissed the incubation request, as > Rally appears to be able to live happily on top of OpenStack and would > benefit from

Re: [openstack-dev] Which program for Rally

2014-08-08 Thread Neependra Kumar Khare
- Original Message - From: "Thierry Carrez" To: "OpenStack Development Mailing List" Sent: Wednesday, August 6, 2014 4:00:35 PM Subject: [openstack-dev] Which program for Rally 1. Rally as an essential QA tool Performance testing (and especially performance regre

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread John Griffith
On Thu, Aug 7, 2014 at 9:02 AM, John Griffith wrote: > > > > On Thu, Aug 7, 2014 at 6:20 AM, Sean Dague wrote: > >> On 08/07/2014 07:58 AM, Angus Salkeld wrote: >> > On Wed, 2014-08-06 at 15:48 -0600, John Griffith wrote: >> >> I have to agree with Duncan here. I also don't know if I fully >> >

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread John Griffith
On Thu, Aug 7, 2014 at 6:20 AM, Sean Dague wrote: > On 08/07/2014 07:58 AM, Angus Salkeld wrote: > > On Wed, 2014-08-06 at 15:48 -0600, John Griffith wrote: > >> I have to agree with Duncan here. I also don't know if I fully > >> understand the limit in options. Stress test seems like it > >> c

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread Sean Dague
On 08/07/2014 07:31 AM, Rohan Kanade wrote: > Date: Wed, 06 Aug 2014 09:44:12 -0400 > From: Sean Dague mailto:s...@dague.net>> > To: openstack-dev@lists.openstack.org > <mailto:openstack-dev@lists.openstack.org> > Subject: Re: [openstack-d

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread Sean Dague
On 08/07/2014 07:58 AM, Angus Salkeld wrote: > On Wed, 2014-08-06 at 15:48 -0600, John Griffith wrote: >> I have to agree with Duncan here. I also don't know if I fully >> understand the limit in options. Stress test seems like it >> could/should be different (again overlap isn't a horrible thing

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread Sean Dague
On 08/06/2014 05:48 PM, John Griffith wrote: > I have to agree with Duncan here. I also don't know if I fully > understand the limit in options. Stress test seems like it could/should > be different (again overlap isn't a horrible thing) and I don't see it > as siphoning off resources so not sure

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread Angus Salkeld
On Wed, 2014-08-06 at 15:48 -0600, John Griffith wrote: > I have to agree with Duncan here. I also don't know if I fully > understand the limit in options. Stress test seems like it > could/should be different (again overlap isn't a horrible thing) and I > don't see it as siphoning off resources

[openstack-dev] Which program for Rally

2014-08-07 Thread Rohan Kanade
> > Date: Wed, 06 Aug 2014 09:44:12 -0400 > From: Sean Dague > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] Which program for Rally > Message-ID: <53e2312c.8000...@dague.net> > Content-Type: text/plain; charset=utf-8 > > Like the fact

Re: [openstack-dev] Which program for Rally

2014-08-07 Thread marc
Hi John, see below. Zitat von John Griffith : I have to agree with Duncan here. I also don't know if I fully understand the limit in options. Stress test seems like it could/should be different This is correct, Rally and Tempest Stress test have a different focus. The stress test framework

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Swapnil Kulkarni
I agree with Duncan and John here, I am not as old contributor in OpneStack as most of the people commenting here are, but I see we have done this right throughout the OpenStack lifecycle, at the start we only had Nova and we could have always said "hey lets have everything in Nova" but we went ahe

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Yingjun Li
From a user’s aspect i do think Rally is more suitable for a product-ready cloud, and seems like it is where it focused on. It’s very easy to evaluate that if the performance of the cloud is better after we adjust some configs or some other tuning. It also provides SLA which maybe not so powerf

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread John Griffith
I have to agree with Duncan here. I also don't know if I fully understand the limit in options. Stress test seems like it could/should be different (again overlap isn't a horrible thing) and I don't see it as siphoning off resources so not sure of the issue. We've become quite wrapped up in proj

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Duncan Thomas
I'm not following here - you complain about rally being monolithic, then suggest that parts of it should be baked into tempest - a tool that is already huge and difficult to get into. I'd rather see tools that do one thing well and some overlap than one tool to rule them all. On 6 August 2014 14:4

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Russell Bryant
On 08/06/2014 09:44 AM, Sean Dague wrote: > Something that we need to figure out is given where we are in the > release cycle do we want to ask the QA team to go off and do Rally deep > dive now to try to pull it apart into the parts that make sense for > other programs to take in. There are always

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Sean Dague
On 08/06/2014 09:11 AM, Russell Bryant wrote: > On 08/06/2014 06:30 AM, Thierry Carrez wrote: >> Hi everyone, >> >> At the TC meeting yesterday we discussed Rally program request and >> incubation request. We quickly dismissed the incubation request, as >> Rally appears to be able to live happily o

Re: [openstack-dev] Which program for Rally

2014-08-06 Thread Russell Bryant
On 08/06/2014 06:30 AM, Thierry Carrez wrote: > Hi everyone, > > At the TC meeting yesterday we discussed Rally program request and > incubation request. We quickly dismissed the incubation request, as > Rally appears to be able to live happily on top of OpenStack and would > benefit from having a

[openstack-dev] Which program for Rally

2014-08-06 Thread Thierry Carrez
Hi everyone, At the TC meeting yesterday we discussed Rally program request and incubation request. We quickly dismissed the incubation request, as Rally appears to be able to live happily on top of OpenStack and would benefit from having a release cycle decoupled from the OpenStack "integrated re