Re: [openstack-dev] [rally][users]: Synchronizing between multiple scenario instances.

2014-10-24 Thread Boris Pavlovic
Behazd,

Ok, for now we can do it in such way.

What I am thinking is that this *logic* should be implemented on
benchmark.runner level.
Cause different load generators are using different approaches to generate
load.
So in case of serial runner it's even impossible to make locking.

So what about adding in
https://github.com/stackforge/rally/blob/master/rally/benchmark/runners/base.py#L140
2 abstract methods (one for incrementing, another for waiting)

And implement them for different runners?


Best regards,
Boris Pavlovic


On Thu, Oct 23, 2014 at 12:05 PM, Behzad Dastur (bdastur) bdas...@cisco.com
 wrote:

  Hi Boris,

 I am still getting my feet wet with rally so some concepts are new, and
 did not quite get your statement regarding the different load generators. I
 am presuming you are referring to the Scenario runner and the different
 “types” of runs.



 What I was looking at is the runner, where we specify the type, times and
 concurrency.  We could have an additional field(s) which would specify the
 synchronization property.



 Essentially, what I have found most useful in the cases where we run
 scenarios/tests in parallel;  is some sort of “barrier”, where at a certain
 point in the run you want all the parallel tasks to reach a specific point
 before continuing.



 Also, I am also considering cases where synchronization is needed within a
 single benchmark case, where the same benchmark scenario:

 creates some vms, performs some tasks, deletes the vm



 Just for simplicity as a POC, I tried something with shared memory
 (multiprocessing.Value), which looks something like this:



 class Barrier(object):

 __init__(self, concurrency)

 self.shmem = multiprocessing.Value(‘I’, concurrency)

 self.lock = multiprocessing.Lock()



 def wait_at_barrier ():

while self.shmem.value  0:

time.sleep(1)

return



 def decrement_shm_concurrency_cnt ():

  with self.lock:

  self.shmem.value -=  1



 And from the scenario, it can be called as:



 scenario:

  -- do some action –

   barrobj.decrement_shm_concurrency_cnt()

  sync_helper.wait_at_barrier()

 -- do some action –   ß all processes will do this action at almost the
 same time.



 I would be happy to discuss more to get a good common solution.



 regards,

 Behzad











 *From:* bo...@pavlovic.ru [mailto:bo...@pavlovic.ru] *On Behalf Of *Boris
 Pavlovic
 *Sent:* Tuesday, October 21, 2014 3:23 PM
 *To:* Behzad Dastur (bdastur)
 *Cc:* OpenStack Development Mailing List (not for usage questions);
 Pradeep Chandrasekar (pradeech); John Wei-I Wu (weiwu)
 *Subject:* Re: [openstack-dev] [rally][users]: Synchronizing between
 multiple scenario instances.



 Behzad,



 Unfortunately at this point there is no support of locking between
 scenarios.





 It will be quite tricky for implementation, because we have different load
 generators, and we will need to find

 common solutions for all of them.



 If you have any ideas about how to implement it in such way, I will be
 more than happy to get this in upstream.





 One of the way that I see is to having some kind of chain-of-benchmarks:



 1) Like first benchmark is running N VMs

 2) Second benchmarking is doing something with all those benchmarks

 3) Third benchmark is deleting all these VMs



 (where chain elements are atomic actions)



 Probably this will be better long term solution.

 Only thing that we should understand is how to store those results and how
 to display them.





 If you would like to help with this let's start discussing it, in some
 kind of google docs.



 Thoughts?





 Best regards,

 Boris Pavlovic





 On Wed, Oct 22, 2014 at 2:13 AM, Behzad Dastur (bdastur) 
 bdas...@cisco.com wrote:

 Does rally provide any synchronization mechanism to synchronize between
 multiple scenario, when running in parallel? Rally spawns multiple
 processes, with each process running the scenario.  We need a way to
 synchronize between these to start a perf test operation at the same time.





 regards,

 Behzad





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]: Synchronizing between multiple scenario instances.

2014-10-23 Thread Behzad Dastur (bdastur)
Hi Boris,
I am still getting my feet wet with rally so some concepts are new, and did not 
quite get your statement regarding the different load generators. I am 
presuming you are referring to the Scenario runner and the different “types” of 
runs.

What I was looking at is the runner, where we specify the type, times and 
concurrency.  We could have an additional field(s) which would specify the 
synchronization property.

Essentially, what I have found most useful in the cases where we run 
scenarios/tests in parallel;  is some sort of “barrier”, where at a certain 
point in the run you want all the parallel tasks to reach a specific point 
before continuing.

Also, I am also considering cases where synchronization is needed within a 
single benchmark case, where the same benchmark scenario:
creates some vms, performs some tasks, deletes the vm

Just for simplicity as a POC, I tried something with shared memory 
(multiprocessing.Value), which looks something like this:

class Barrier(object):
__init__(self, concurrency)
self.shmem = multiprocessing.Value(‘I’, concurrency)
self.lock = multiprocessing.Lock()

def wait_at_barrier ():
   while self.shmem.value  0:
   time.sleep(1)
   return

def decrement_shm_concurrency_cnt ():
 with self.lock:
 self.shmem.value -=  1


And from the scenario, it can be called as:

scenario:
 -- do some action –
  barrobj.decrement_shm_concurrency_cnt()
 sync_helper.wait_at_barrier()
-- do some action –   -- all processes will do this action at almost the same 
time.

I would be happy to discuss more to get a good common solution.

regards,
Behzad





From: bo...@pavlovic.ru [mailto:bo...@pavlovic.ru] On Behalf Of Boris Pavlovic
Sent: Tuesday, October 21, 2014 3:23 PM
To: Behzad Dastur (bdastur)
Cc: OpenStack Development Mailing List (not for usage questions); Pradeep 
Chandrasekar (pradeech); John Wei-I Wu (weiwu)
Subject: Re: [openstack-dev] [rally][users]: Synchronizing between multiple 
scenario instances.

Behzad,

Unfortunately at this point there is no support of locking between scenarios.


It will be quite tricky for implementation, because we have different load 
generators, and we will need to find
common solutions for all of them.

If you have any ideas about how to implement it in such way, I will be more 
than happy to get this in upstream.


One of the way that I see is to having some kind of chain-of-benchmarks:

1) Like first benchmark is running N VMs
2) Second benchmarking is doing something with all those benchmarks
3) Third benchmark is deleting all these VMs

(where chain elements are atomic actions)

Probably this will be better long term solution.
Only thing that we should understand is how to store those results and how to 
display them.


If you would like to help with this let's start discussing it, in some kind of 
google docs.

Thoughts?


Best regards,
Boris Pavlovic


On Wed, Oct 22, 2014 at 2:13 AM, Behzad Dastur (bdastur) 
bdas...@cisco.commailto:bdas...@cisco.com wrote:
Does rally provide any synchronization mechanism to synchronize between 
multiple scenario, when running in parallel? Rally spawns multiple processes, 
with each process running the scenario.  We need a way to synchronize between 
these to start a perf test operation at the same time.


regards,
Behzad


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [rally][users]: Synchronizing between multiple scenario instances.

2014-10-21 Thread Behzad Dastur (bdastur)
Does rally provide any synchronization mechanism to synchronize between 
multiple scenario, when running in parallel? Rally spawns multiple processes, 
with each process running the scenario.  We need a way to synchronize between 
these to start a perf test operation at the same time.


regards,
Behzad

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]: Synchronizing between multiple scenario instances.

2014-10-21 Thread Boris Pavlovic
Behzad,

Unfortunately at this point there is no support of locking between
scenarios.


It will be quite tricky for implementation, because we have different load
generators, and we will need to find
common solutions for all of them.

If you have any ideas about how to implement it in such way, I will be more
than happy to get this in upstream.


One of the way that I see is to having some kind of chain-of-benchmarks:

1) Like first benchmark is running N VMs
2) Second benchmarking is doing something with all those benchmarks
3) Third benchmark is deleting all these VMs

(where chain elements are atomic actions)

Probably this will be better long term solution.
Only thing that we should understand is how to store those results and how
to display them.


If you would like to help with this let's start discussing it, in some kind
of google docs.

Thoughts?


Best regards,
Boris Pavlovic


On Wed, Oct 22, 2014 at 2:13 AM, Behzad Dastur (bdastur) bdas...@cisco.com
wrote:

  Does rally provide any synchronization mechanism to synchronize between
 multiple scenario, when running in parallel? Rally spawns multiple
 processes, with each process running the scenario.  We need a way to
 synchronize between these to start a perf test operation at the same time.





 regards,

 Behzad



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]

2014-10-20 Thread Behzad Dastur (bdastur)
Hi Boris,
Does rally provide any synchronization mechanism to synchronize between 
multiple scenario, when running in parallel? Rally spawns multiple processes, 
with each process running the scenario.  We need a way to synchronize between 
these to start a perf test operation at the same time.


regards,
Behzad



From: Boris Pavlovic [mailto:bpavlo...@mirantis.com]
Sent: Wednesday, September 24, 2014 11:24 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [rally][users]

Ajay,

Ya adding support of  benchmarking OpenStack clouds using ordinary user 
accounts that already exist is one of our majors goals for already more then 
half of year. As I said in my previous message, we will support it soon finally.



Btw we have feature request page:
https://github.com/stackforge/rally/tree/master/doc/feature_request
With the list of features that we are working now.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:30 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi Boris
Existing users is one thing but according to Rally page it says admin account 
benchmarking is already supported

Rally is on its way to support of benchmarking OpenStack clouds using ordinary 
user accounts that already exist. Rally lacked such functionality (it only 
supported benchmarking either from an admin account or from a bunch of 
temporarily created users), which posed a problem since some deployments don't 
allow temporary users creation. There have been 
twohttps://review.openstack.org/#/c/116766/ 
patcheshttps://review.openstack.org/#/c/119344/ that prepare the code for 
this new functionality. It is going to come very soon - stay tuned.


Ajay

From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, September 24, 2014 at 6:13 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally][users]

Ajay,

I am working that feature. It's almost ready.
I'll let you know when I finish.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:02 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
Our default mode of execution of rally is allowing Rally to create a new user 
and tenant. Is there a way to have rally use the existing admin tenant and user.
I need to use Rally for some tests which would need a admin access so I would 
like Rally to use existing admin tenant and admin user for tests
Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]

2014-09-25 Thread Boris Pavlovic
Ajay,

Ya adding support of  *benchmarking OpenStack clouds using ordinary user
accounts that already exist *is one of our majors goals for already more
then half of year. As I said in my previous message, we will support it
soon finally.

Btw we have feature request page:
https://github.com/stackforge/rally/tree/master/doc/feature_request
With the list of features that we are working now.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:30 AM, Ajay Kalambur (akalambu) 
akala...@cisco.com wrote:

  Hi Boris
 Existing users is one thing but according to Rally page it says admin
 account benchmarking is already supported

  Rally is on its way to support of *benchmarking OpenStack clouds using
 ordinary user accounts that already exist*. Rally lacked such
 functionality (it only supported benchmarking either from an admin account
 or from a bunch of temporarily created users), which posed a problem since
 some deployments don't allow temporary users creation. There have been two
 https://review.openstack.org/#/c/116766/ patches
 https://review.openstack.org/#/c/119344/ that prepare the code for this
 new functionality. It is going to come very soon - stay tuned.


  Ajay

   From: Boris Pavlovic bpavlo...@mirantis.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Wednesday, September 24, 2014 at 6:13 PM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [rally][users]

   Ajay,

  I am working that feature. It's almost ready.
 I'll let you know when I finish.


  Best regards,
 Boris Pavlovic

 On Thu, Sep 25, 2014 at 5:02 AM, Ajay Kalambur (akalambu) 
 akala...@cisco.com wrote:

  Hi
 Our default mode of execution of rally is allowing Rally to create a new
 user and tenant. Is there a way to have rally use the existing admin tenant
 and user.
 I need to use Rally for some tests which would need a admin access so I
 would like Rally to use existing admin tenant and admin user for tests
  Ajay


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [rally][users]

2014-09-24 Thread Ajay Kalambur (akalambu)
Hi
Our default mode of execution of rally is allowing Rally to create a new user 
and tenant. Is there a way to have rally use the existing admin tenant and user.
I need to use Rally for some tests which would need a admin access so I would 
like Rally to use existing admin tenant and admin user for tests
Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]

2014-09-24 Thread Boris Pavlovic
Ajay,

I am working that feature. It's almost ready.
I'll let you know when I finish.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:02 AM, Ajay Kalambur (akalambu) 
akala...@cisco.com wrote:

  Hi
 Our default mode of execution of rally is allowing Rally to create a new
 user and tenant. Is there a way to have rally use the existing admin tenant
 and user.
 I need to use Rally for some tests which would need a admin access so I
 would like Rally to use existing admin tenant and admin user for tests
 Ajay


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally][users]

2014-09-24 Thread Ajay Kalambur (akalambu)
Hi Boris
Existing users is one thing but according to Rally page it says admin account 
benchmarking is already supported

Rally is on its way to support of benchmarking OpenStack clouds using ordinary 
user accounts that already exist. Rally lacked such functionality (it only 
supported benchmarking either from an admin account or from a bunch of 
temporarily created users), which posed a problem since some deployments don't 
allow temporary users creation. There have been 
twohttps://review.openstack.org/#/c/116766/ 
patcheshttps://review.openstack.org/#/c/119344/ that prepare the code for 
this new functionality. It is going to come very soon - stay tuned.


Ajay

From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, September 24, 2014 at 6:13 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally][users]

Ajay,

I am working that feature. It's almost ready.
I'll let you know when I finish.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:02 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
Our default mode of execution of rally is allowing Rally to create a new user 
and tenant. Is there a way to have rally use the existing admin tenant and user.
I need to use Rally for some tests which would need a admin access so I would 
like Rally to use existing admin tenant and admin user for tests
Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev