Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-24 Thread Renat Akhmerov 
Thanks Boris!

Renat Akhmerov
@ Mirantis Inc.


> On 24 Dec 2014, at 23:54, Boris Pavlovic  wrote:
> 
> Guys, 
> 
> I added patch to infra:
> https://review.openstack.org/#/c/143879/ 
> 
> 
> That allows to run Rally against Mistral in gates.
> 
> Best regards,
> Boris Pavlovic 
> 
> On Mon, Dec 22, 2014 at 4:25 PM, Anastasia Kuznetsova 
> mailto:akuznets...@mirantis.com>> wrote:
> Dmitry,
> 
> Now I see that my comments are not so informative, I will try to describe 
> environment and scenarios in more details.
> 
> 1) 1 api 1 engine 1 executor  it means that there were 3 Mistral processes 
> running on the same box
> 2) list-workbooks scenario was run when there were no workflow executions at 
> the same time, I will notice this your comment and I will measure time in 
> such situation, but I guess that it will take more time, the question is as 
> far as.
> 3) 60 % of success means that only 60 % of number of times execution of 
> scenario 'list-workbooks' were successful, at the moment I have observed only 
> one type of error: 
> error connection to Rabbit : Error ConnectionError: ('Connection aborted.', 
> error(104, 'Connection reset by peer'))
> 4) we don't know the durability criteria of Mistral and under what load 
> Mistral will 'die', we want to define the threshold.
> 
> P.S. Dmitry, if you have any ideas/scenarios which you want to test, please 
> share them.
> 
> On Sat, Dec 20, 2014 at 9:35 AM, Dmitri Zimine  > wrote:
> Anastasia, any start is a good start. 
> 
> > 1 api 1 engine 1 executor, list-workbooks
> 
> what exactly doest it mean: 1) is mistral deployed on 3 boxes with component 
> per box, or all three are processes on the same box? 2) is list-workbooks 
> test running while workflow executions going on? How many? what’s the 
> character of the load 3) when it says 60% success what exactly does it mean, 
> what kind of failures? 4) what is the durability criteria, how long do we 
> expect Mistral to withstand the load.  
> 
> Let’s discuss this in details on the next IRC meeting? 
> 
> Thanks again for getting this started. 
> 
> DZ.
> 
> 
> On Dec 19, 2014, at 7:44 AM, Anastasia Kuznetsova  > wrote:
> 
>> Boris,
>> 
>> Thanks for feedback! 
>> 
>> > But I belive that you should put bigger load here: 
>> > https://etherpad.openstack.org/p/mistral-rally-testing-results 
>> > 
>> 
>> As I said it is only beginning and  I will increase the load and change its 
>> type. 
>> 
>> >As well concurrency should be at least 2-3 times bigger than times 
>> >otherwise it won't generate proper load and you won't collect >enough data 
>> >for statistical analyze.  
>> >
>> >As well use  "rps" runner that generates more real life load.
>> >Plus it will be nice to share as well output of "rally task report" command.
>> 
>> Thanks for the advice, I will consider it in further testing and reporting.
>> 
>> Answering to your question about using Rally for integration testing, as I 
>> mentioned in our load testing plan published on wiki page,  one of our final 
>> goals is to have a Rally gate in one of Mistral repositories, so we are 
>> interested in it and I already prepare first commits to Rally.
>> 
>> Thanks,
>> Anastasia Kuznetsova
>> 
>> On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic > > wrote:
>> Anastasia, 
>> 
>> Nice work on this. But I belive that you should put bigger load here: 
>> https://etherpad.openstack.org/p/mistral-rally-testing-results 
>> 
>> 
>> As well concurrency should be at least 2-3 times bigger than times otherwise 
>> it won't generate proper load and you won't collect enough data for 
>> statistical analyze.  
>> 
>> As well use  "rps" runner that generates more real life load.
>> Plus it will be nice to share as well output of "rally task report" command.
>> 
>> 
>> By the way what do you think about using Rally scenarios (that you already 
>> wrote) for integration testing as well? 
>> 
>> 
>> Best regards,
>> Boris Pavlovic 
>> 
>> On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova 
>> mailto:akuznets...@mirantis.com>> wrote:
>> Hello everyone,
>> 
>> I want to announce that Mistral team has started work on load and 
>> performance testing in this release cycle.
>> 
>> Brief information about scope of our work can be found here: 
>> https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing 
>> 
>> 
>> First results are published here:
>> https://etherpad.openstack.org/p/mistral-rally-testing-results 
>> 
>> 
>> Thanks,
>> Anastasia Kuznetsova
>> @ Mirantis Inc.
>> 
>> ___
>> OpenStack-dev mailing list

Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-24 Thread Boris Pavlovic 
Guys,

I added patch to infra:
https://review.openstack.org/#/c/143879/

That allows to run Rally against Mistral in gates.

Best regards,
Boris Pavlovic

On Mon, Dec 22, 2014 at 4:25 PM, Anastasia Kuznetsova <
akuznets...@mirantis.com> wrote:

> Dmitry,
>
> Now I see that my comments are not so informative, I will try to describe
> environment and scenarios in more details.
>
> 1) *1 api 1 engine 1 executor  *it means that there were 3 Mistral
> processes running on the same box
> 2) list-workbooks scenario was run when there were no workflow executions
> at the same time, I will notice this your comment and I will measure time
> in such situation, but I guess that it will take more time, the question is
> as far as.
> 3) 60 % of success means that only 60 % of number of times execution of
> scenario 'list-workbooks' were successful, at the moment I have observed
> only one type of error:
> error connection to Rabbit : Error ConnectionError: ('Connection
> aborted.', error(104, 'Connection reset by peer'))
> 4) we don't know the durability criteria of Mistral and under what load
> Mistral will 'die', we want to define the threshold.
>
> P.S. Dmitry, if you have any ideas/scenarios which you want to test,
> please share them.
>
> On Sat, Dec 20, 2014 at 9:35 AM, Dmitri Zimine 
> wrote:
>
>> Anastasia, any start is a good start.
>>
>> *> 1 api 1 engine 1 executor, list-workbooks*
>>
>> what exactly doest it mean: 1) is mistral deployed on 3 boxes with
>> component per box, or all three are processes on the same box? 2) is
>> list-workbooks test running while workflow executions going on? How many?
>> what’s the character of the load 3) when it says 60% success what exactly
>> does it mean, what kind of failures? 4) what is the durability criteria,
>> how long do we expect Mistral to withstand the load.
>>
>> Let’s discuss this in details on the next IRC meeting?
>>
>> Thanks again for getting this started.
>>
>> DZ.
>>
>>
>> On Dec 19, 2014, at 7:44 AM, Anastasia Kuznetsova <
>> akuznets...@mirantis.com> wrote:
>>
>> Boris,
>>
>> Thanks for feedback!
>>
>> > But I belive that you should put bigger load here:
>> https://etherpad.openstack.org/p/mistral-rally-testing-results
>>
>> As I said it is only beginning and  I will increase the load and change
>> its type.
>>
>> >As well concurrency should be at least 2-3 times bigger than times
>> otherwise it won't generate proper load and you won't collect >enough data
>> for statistical analyze.
>> >
>> >As well use  "rps" runner that generates more real life load.
>> >Plus it will be nice to share as well output of "rally task report"
>> command.
>>
>> Thanks for the advice, I will consider it in further testing and
>> reporting.
>>
>> Answering to your question about using Rally for integration testing, as
>> I mentioned in our load testing plan published on wiki page,  one of our
>> final goals is to have a Rally gate in one of Mistral repositories, so we
>> are interested in it and I already prepare first commits to Rally.
>>
>> Thanks,
>> Anastasia Kuznetsova
>>
>> On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic 
>> wrote:
>>>
>>> Anastasia,
>>>
>>> Nice work on this. But I belive that you should put bigger load here:
>>> https://etherpad.openstack.org/p/mistral-rally-testing-results
>>>
>>> As well concurrency should be at least 2-3 times bigger than times
>>> otherwise it won't generate proper load and you won't collect enough data
>>> for statistical analyze.
>>>
>>> As well use  "rps" runner that generates more real life load.
>>> Plus it will be nice to share as well output of "rally task report"
>>> command.
>>>
>>>
>>> By the way what do you think about using Rally scenarios (that you
>>> already wrote) for integration testing as well?
>>>
>>>
>>> Best regards,
>>> Boris Pavlovic
>>>
>>> On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova <
>>> akuznets...@mirantis.com> wrote:

 Hello everyone,

 I want to announce that Mistral team has started work on load and
 performance testing in this release cycle.

 Brief information about scope of our work can be found here:

 https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing

 First results are published here:
 https://etherpad.openstack.org/p/mistral-rally-testing-results

 Thanks,
 Anastasia Kuznetsova
 @ Mirantis Inc.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>

Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-22 Thread Anastasia Kuznetsova 
Dmitry,

Now I see that my comments are not so informative, I will try to describe
environment and scenarios in more details.

1) *1 api 1 engine 1 executor  *it means that there were 3 Mistral
processes running on the same box
2) list-workbooks scenario was run when there were no workflow executions
at the same time, I will notice this your comment and I will measure time
in such situation, but I guess that it will take more time, the question is
as far as.
3) 60 % of success means that only 60 % of number of times execution of
scenario 'list-workbooks' were successful, at the moment I have observed
only one type of error:
error connection to Rabbit : Error ConnectionError: ('Connection aborted.',
error(104, 'Connection reset by peer'))
4) we don't know the durability criteria of Mistral and under what load
Mistral will 'die', we want to define the threshold.

P.S. Dmitry, if you have any ideas/scenarios which you want to test, please
share them.

On Sat, Dec 20, 2014 at 9:35 AM, Dmitri Zimine 
wrote:

> Anastasia, any start is a good start.
>
> *> 1 api 1 engine 1 executor, list-workbooks*
>
> what exactly doest it mean: 1) is mistral deployed on 3 boxes with
> component per box, or all three are processes on the same box? 2) is
> list-workbooks test running while workflow executions going on? How many?
> what’s the character of the load 3) when it says 60% success what exactly
> does it mean, what kind of failures? 4) what is the durability criteria,
> how long do we expect Mistral to withstand the load.
>
> Let’s discuss this in details on the next IRC meeting?
>
> Thanks again for getting this started.
>
> DZ.
>
>
> On Dec 19, 2014, at 7:44 AM, Anastasia Kuznetsova <
> akuznets...@mirantis.com> wrote:
>
> Boris,
>
> Thanks for feedback!
>
> > But I belive that you should put bigger load here:
> https://etherpad.openstack.org/p/mistral-rally-testing-results
>
> As I said it is only beginning and  I will increase the load and change
> its type.
>
> >As well concurrency should be at least 2-3 times bigger than times
> otherwise it won't generate proper load and you won't collect >enough data
> for statistical analyze.
> >
> >As well use  "rps" runner that generates more real life load.
> >Plus it will be nice to share as well output of "rally task report"
> command.
>
> Thanks for the advice, I will consider it in further testing and reporting.
>
> Answering to your question about using Rally for integration testing, as I
> mentioned in our load testing plan published on wiki page,  one of our
> final goals is to have a Rally gate in one of Mistral repositories, so we
> are interested in it and I already prepare first commits to Rally.
>
> Thanks,
> Anastasia Kuznetsova
>
> On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic 
> wrote:
>>
>> Anastasia,
>>
>> Nice work on this. But I belive that you should put bigger load here:
>> https://etherpad.openstack.org/p/mistral-rally-testing-results
>>
>> As well concurrency should be at least 2-3 times bigger than times
>> otherwise it won't generate proper load and you won't collect enough data
>> for statistical analyze.
>>
>> As well use  "rps" runner that generates more real life load.
>> Plus it will be nice to share as well output of "rally task report"
>> command.
>>
>>
>> By the way what do you think about using Rally scenarios (that you
>> already wrote) for integration testing as well?
>>
>>
>> Best regards,
>> Boris Pavlovic
>>
>> On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova <
>> akuznets...@mirantis.com> wrote:
>>>
>>> Hello everyone,
>>>
>>> I want to announce that Mistral team has started work on load and
>>> performance testing in this release cycle.
>>>
>>> Brief information about scope of our work can be found here:
>>>
>>> https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing
>>>
>>> First results are published here:
>>> https://etherpad.openstack.org/p/mistral-rally-testing-results
>>>
>>> Thanks,
>>> Anastasia Kuznetsova
>>> @ Mirantis Inc.
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-19 Thread Dmitri Zimine 
Anastasia, any start is a good start. 

> 1 api 1 engine 1 executor, list-workbooks

what exactly doest it mean: 1) is mistral deployed on 3 boxes with component 
per box, or all three are processes on the same box? 2) is list-workbooks test 
running while workflow executions going on? How many? what’s the character of 
the load 3) when it says 60% success what exactly does it mean, what kind of 
failures? 4) what is the durability criteria, how long do we expect Mistral to 
withstand the load.  

Let’s discuss this in details on the next IRC meeting? 

Thanks again for getting this started. 

DZ.


On Dec 19, 2014, at 7:44 AM, Anastasia Kuznetsova  
wrote:

> Boris,
> 
> Thanks for feedback! 
> 
> > But I belive that you should put bigger load here: 
> > https://etherpad.openstack.org/p/mistral-rally-testing-results
> 
> As I said it is only beginning and  I will increase the load and change its 
> type. 
> 
> >As well concurrency should be at least 2-3 times bigger than times otherwise 
> >it won't generate proper load and you won't collect >enough data for 
> >statistical analyze.  
> >
> >As well use  "rps" runner that generates more real life load.
> >Plus it will be nice to share as well output of "rally task report" command.
> 
> Thanks for the advice, I will consider it in further testing and reporting.
> 
> Answering to your question about using Rally for integration testing, as I 
> mentioned in our load testing plan published on wiki page,  one of our final 
> goals is to have a Rally gate in one of Mistral repositories, so we are 
> interested in it and I already prepare first commits to Rally.
> 
> Thanks,
> Anastasia Kuznetsova
> 
> On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic  
> wrote:
> Anastasia, 
> 
> Nice work on this. But I belive that you should put bigger load here: 
> https://etherpad.openstack.org/p/mistral-rally-testing-results
> 
> As well concurrency should be at least 2-3 times bigger than times otherwise 
> it won't generate proper load and you won't collect enough data for 
> statistical analyze.  
> 
> As well use  "rps" runner that generates more real life load.
> Plus it will be nice to share as well output of "rally task report" command.
> 
> 
> By the way what do you think about using Rally scenarios (that you already 
> wrote) for integration testing as well? 
> 
> 
> Best regards,
> Boris Pavlovic 
> 
> On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova 
>  wrote:
> Hello everyone,
> 
> I want to announce that Mistral team has started work on load and performance 
> testing in this release cycle.
> 
> Brief information about scope of our work can be found here: 
> https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing
> 
> First results are published here:
> https://etherpad.openstack.org/p/mistral-rally-testing-results
> 
> Thanks,
> Anastasia Kuznetsova
> @ Mirantis Inc.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-19 Thread Anastasia Kuznetsova 
Boris,

Thanks for feedback!

> But I belive that you should put bigger load here:
https://etherpad.openstack.org/p/mistral-rally-testing-results

As I said it is only beginning and  I will increase the load and change its
type.

>As well concurrency should be at least 2-3 times bigger than times
otherwise it won't generate proper load and you won't collect >enough data
for statistical analyze.
>
>As well use  "rps" runner that generates more real life load.
>Plus it will be nice to share as well output of "rally task report"
command.

Thanks for the advice, I will consider it in further testing and reporting.

Answering to your question about using Rally for integration testing, as I
mentioned in our load testing plan published on wiki page,  one of our
final goals is to have a Rally gate in one of Mistral repositories, so we
are interested in it and I already prepare first commits to Rally.

Thanks,
Anastasia Kuznetsova

On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic 
wrote:
>
> Anastasia,
>
> Nice work on this. But I belive that you should put bigger load here:
> https://etherpad.openstack.org/p/mistral-rally-testing-results
>
> As well concurrency should be at least 2-3 times bigger than times
> otherwise it won't generate proper load and you won't collect enough data
> for statistical analyze.
>
> As well use  "rps" runner that generates more real life load.
> Plus it will be nice to share as well output of "rally task report"
> command.
>
>
> By the way what do you think about using Rally scenarios (that you already
> wrote) for integration testing as well?
>
>
> Best regards,
> Boris Pavlovic
>
> On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova <
> akuznets...@mirantis.com> wrote:
>>
>> Hello everyone,
>>
>> I want to announce that Mistral team has started work on load and
>> performance testing in this release cycle.
>>
>> Brief information about scope of our work can be found here:
>>
>> https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing
>>
>> First results are published here:
>> https://etherpad.openstack.org/p/mistral-rally-testing-results
>>
>> Thanks,
>> Anastasia Kuznetsova
>> @ Mirantis Inc.
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Plans to load and performance testing

2014-12-19 Thread Boris Pavlovic 
Anastasia,

Nice work on this. But I belive that you should put bigger load here:
https://etherpad.openstack.org/p/mistral-rally-testing-results

As well concurrency should be at least 2-3 times bigger than times
otherwise it won't generate proper load and you won't collect enough data
for statistical analyze.

As well use  "rps" runner that generates more real life load.
Plus it will be nice to share as well output of "rally task report" command.


By the way what do you think about using Rally scenarios (that you already
wrote) for integration testing as well?


Best regards,
Boris Pavlovic

On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova <
akuznets...@mirantis.com> wrote:
>
> Hello everyone,
>
> I want to announce that Mistral team has started work on load and
> performance testing in this release cycle.
>
> Brief information about scope of our work can be found here:
>
> https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing
>
> First results are published here:
> https://etherpad.openstack.org/p/mistral-rally-testing-results
>
> Thanks,
> Anastasia Kuznetsova
> @ Mirantis Inc.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Plans to load and performance testing

2014-12-19 Thread Anastasia Kuznetsova 
Hello everyone,

I want to announce that Mistral team has started work on load and
performance testing in this release cycle.

Brief information about scope of our work can be found here:
https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing

First results are published here:
https://etherpad.openstack.org/p/mistral-rally-testing-results

Thanks,
Anastasia Kuznetsova
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev