Re: [Openstack] Queue Service Implementation Thoughts
Yup, those are options we can try when we're ready to optimize. -Eric On Tue, Mar 08, 2011 at 10:55:34PM +, Sandy Walsh wrote: > I'm sure you've seen this: > http://nichol.as/benchmark-of-python-web-servers?source=g > > -S > > From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net > [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf > of Eric Day [e...@oddments.org] > Sent: Tuesday, March 08, 2011 5:58 PM > To: Todd Willey > Cc: openstack@lists.launchpad.net > Subject: Re: [Openstack] Queue Service Implementation Thoughts > > On Tue, Mar 08, 2011 at 04:47:38PM -0500, Todd Willey wrote: > > With this switch to python, does it make sense to revisit the concept > > of openstack-common for things like logging, flag parsing, etc? What > > components would you like to just be able to drop in from nova, > > glance, or swift? > > Yes, I'm planning on putting as much as possible into openstack-common, > picking the best from nova/swift/glance as we move along. Nova, swift, > and glance can start using those modules as they see fit. > > -Eric > > > > > -todd[1] > > > > On Tue, Mar 8, 2011 at 4:05 PM, Eric Day wrote: > > > Hi everyone, > > > > > > I added a sqlite backend to the prototype and ran some tests. Initially > > > things were very slow, but after some further testing I was able > > > to figure out where the time was being spent. In order to do this I > > > added a very simple binary protocol interface to insert only. These > > > tests are with a single server process and multiple client processes > > > (so don't compare to previous email numbers that were two process). The > > > numbers given are requests/second. > > > > > > echo (no parsing) - 17k > > > > > > binary - 13k > > > binary+insert msg into dict - 11k > > > binary+insert msg into sqlite - 8.7k > > > > > > wsgi - 4.9k > > > wsgi+webob - 3.5k > > > wsgi+webob+insert msg into dict - 3.4k > > > wsgi+webob+insert msg into sqlite - 2.8k > > > > > > wsgi+webob+routes - 1.9k > > > wsgi+webob+routes+insert msg into dict - 1.8k > > > wsgi+webob+routes+insert msg into sqlite - 1.5k > > > > > > This shows that without wsgi/webob/routes, the performance is pretty > > > decent). This would be the case when using an efficient binary protocol > > > or perhaps a more efficient HTTP parser. > > > > > > Next, it shows WSGI adds significant overhead. The webob and routes > > > modules also add a fair amount. > > > > > > I'm going to rework the current code in the prototype into a proper > > > project in the burrow trunk with modular front-ends and back-ends so > > > we can easily test new options. I'll stick with the current wsgi code > > > for now just to get things functioning and we can look at optimizing > > > later. For the proxy-server communication, we'll definitely need to > > > use something more efficient than stock wsgi modules in the long run. > > > > > > No red flags yet with Python, and we're in the ballpark for efficiency > > > with a binary protocol. A quick test with other servers showed > > > rabbitmq at about 9k messages/sec (binary protocol, Erlang server) > > > and Gearman at about 13k messages/sec (binary protocol, C server). > > > > > > -Eric > > > > > > On Mon, Mar 07, 2011 at 01:32:55PM -0800, Eric Day wrote: > > >> I ran the tests again to verify: > > >> > > >> 500k requests - 10 processes each running 50k requests. > > >> > > >> time req/s cs us sy id > > >> 2 thread/proc > > >> echo c++ 7.19 69541 142182 23 77 0 > > >> echo erlang 9.53 52465 105871 39 61 0 > > >> echo python 9.58 52192 108420 42 58 0 > > >> 2 thread/proc > > >> wsgi python 58.74 8512 18132 86 14 0 > > >> webob python 78.75 6349 13678 89 11 0 > > >> webmachine* 63.50 7874 11834 89 11 0 > > >> openstack20.48 24414 49897 68 32 0 > > >> > > >> cs/us/sys/id are from vmstat while running the tests. > > >> > > >> * webmachine degrades over time with long-lived, multi-request > > >> connections. This number was estimated with 1000 requests per > > >> connection. At 50k requests per connection, the rate dropped to > > >> 2582 req/s. > > >> > > >> As you can see I was able to reproduce the same numbers. If > > >> anyone would like to do the same, you can grab lp:burrow for the > > >> "openstack" Erlang application (compile and run ./start), webmachine > > >> is at https://github.com/basho/webmachine (you need to create a demo > > >> app and make sure you set nodelay for the socket options), and I've > > >> attached the python server and client (start 10 client processes when > > >> testing). Find me on irc (eday in #openstack) if you have questions. > > >> > > >> If we hit performance issues with this type of application, we'll > > >> probably hit them around the same time with both Erlang and Python > > >> (then we'll look to C/C++). Since most OpenStack developers are a lot > > >> more comfortable with Python, I sugg
Re: [Openstack] Queue Service Implementation Thoughts
I'm sure you've seen this: http://nichol.as/benchmark-of-python-web-servers?source=g -S From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Eric Day [e...@oddments.org] Sent: Tuesday, March 08, 2011 5:58 PM To: Todd Willey Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Queue Service Implementation Thoughts On Tue, Mar 08, 2011 at 04:47:38PM -0500, Todd Willey wrote: > With this switch to python, does it make sense to revisit the concept > of openstack-common for things like logging, flag parsing, etc? What > components would you like to just be able to drop in from nova, > glance, or swift? Yes, I'm planning on putting as much as possible into openstack-common, picking the best from nova/swift/glance as we move along. Nova, swift, and glance can start using those modules as they see fit. -Eric > > -todd[1] > > On Tue, Mar 8, 2011 at 4:05 PM, Eric Day wrote: > > Hi everyone, > > > > I added a sqlite backend to the prototype and ran some tests. Initially > > things were very slow, but after some further testing I was able > > to figure out where the time was being spent. In order to do this I > > added a very simple binary protocol interface to insert only. These > > tests are with a single server process and multiple client processes > > (so don't compare to previous email numbers that were two process). The > > numbers given are requests/second. > > > > echo (no parsing) - 17k > > > > binary - 13k > > binary+insert msg into dict - 11k > > binary+insert msg into sqlite - 8.7k > > > > wsgi - 4.9k > > wsgi+webob - 3.5k > > wsgi+webob+insert msg into dict - 3.4k > > wsgi+webob+insert msg into sqlite - 2.8k > > > > wsgi+webob+routes - 1.9k > > wsgi+webob+routes+insert msg into dict - 1.8k > > wsgi+webob+routes+insert msg into sqlite - 1.5k > > > > This shows that without wsgi/webob/routes, the performance is pretty > > decent). This would be the case when using an efficient binary protocol > > or perhaps a more efficient HTTP parser. > > > > Next, it shows WSGI adds significant overhead. The webob and routes > > modules also add a fair amount. > > > > I'm going to rework the current code in the prototype into a proper > > project in the burrow trunk with modular front-ends and back-ends so > > we can easily test new options. I'll stick with the current wsgi code > > for now just to get things functioning and we can look at optimizing > > later. For the proxy-server communication, we'll definitely need to > > use something more efficient than stock wsgi modules in the long run. > > > > No red flags yet with Python, and we're in the ballpark for efficiency > > with a binary protocol. A quick test with other servers showed > > rabbitmq at about 9k messages/sec (binary protocol, Erlang server) > > and Gearman at about 13k messages/sec (binary protocol, C server). > > > > -Eric > > > > On Mon, Mar 07, 2011 at 01:32:55PM -0800, Eric Day wrote: > >> I ran the tests again to verify: > >> > >> 500k requests - 10 processes each running 50k requests. > >> > >> time req/s cs us sy id > >> 2 thread/proc > >> echo c++ 7.19 69541 142182 23 77 0 > >> echo erlang 9.53 52465 105871 39 61 0 > >> echo python 9.58 52192 108420 42 58 0 > >> 2 thread/proc > >> wsgi python 58.74 8512 18132 86 14 0 > >> webob python 78.75 6349 13678 89 11 0 > >> webmachine* 63.50 7874 11834 89 11 0 > >> openstack20.48 24414 49897 68 32 0 > >> > >> cs/us/sys/id are from vmstat while running the tests. > >> > >> * webmachine degrades over time with long-lived, multi-request > >> connections. This number was estimated with 1000 requests per > >> connection. At 50k requests per connection, the rate dropped to > >> 2582 req/s. > >> > >> As you can see I was able to reproduce the same numbers. If > >> anyone would like to do the same, you can grab lp:burrow for the > >> "openstack" Erlang application (compile and run ./start), webmachine > >> is at https://github.com/basho/webmachine (you need to create a demo > >> app and make sure you set nodelay for the socket options), and I've > >> attached the python server and client (start 10 client processes when > >> testing). Find me on irc (eday in #openstack) if you have questions. > >> > >> If we hit performance issues with this type of application, we'll > >> probably hit them around the same time with both Erlang and Python > >> (then we'll look to C/C++). Since most OpenStack developers are a lot > >> more comfortable with Python, I suggest we make the switch. Please > >> response with thoughts on this. I'll add a sqlite backend to the > >> Python prototype and run some performance tests against that to see > >> if any red flags come up. > >> > >> -Eric > >> > >> On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote: > >> >Eric, > >> > Thanks for your exper
Re: [Openstack] Queue Service Implementation Thoughts
On Tue, Mar 08, 2011 at 04:47:38PM -0500, Todd Willey wrote: > With this switch to python, does it make sense to revisit the concept > of openstack-common for things like logging, flag parsing, etc? What > components would you like to just be able to drop in from nova, > glance, or swift? Yes, I'm planning on putting as much as possible into openstack-common, picking the best from nova/swift/glance as we move along. Nova, swift, and glance can start using those modules as they see fit. -Eric > > -todd[1] > > On Tue, Mar 8, 2011 at 4:05 PM, Eric Day wrote: > > Hi everyone, > > > > I added a sqlite backend to the prototype and ran some tests. Initially > > things were very slow, but after some further testing I was able > > to figure out where the time was being spent. In order to do this I > > added a very simple binary protocol interface to insert only. These > > tests are with a single server process and multiple client processes > > (so don't compare to previous email numbers that were two process). The > > numbers given are requests/second. > > > > echo (no parsing) - 17k > > > > binary - 13k > > binary+insert msg into dict - 11k > > binary+insert msg into sqlite - 8.7k > > > > wsgi - 4.9k > > wsgi+webob - 3.5k > > wsgi+webob+insert msg into dict - 3.4k > > wsgi+webob+insert msg into sqlite - 2.8k > > > > wsgi+webob+routes - 1.9k > > wsgi+webob+routes+insert msg into dict - 1.8k > > wsgi+webob+routes+insert msg into sqlite - 1.5k > > > > This shows that without wsgi/webob/routes, the performance is pretty > > decent). This would be the case when using an efficient binary protocol > > or perhaps a more efficient HTTP parser. > > > > Next, it shows WSGI adds significant overhead. The webob and routes > > modules also add a fair amount. > > > > I'm going to rework the current code in the prototype into a proper > > project in the burrow trunk with modular front-ends and back-ends so > > we can easily test new options. I'll stick with the current wsgi code > > for now just to get things functioning and we can look at optimizing > > later. For the proxy-server communication, we'll definitely need to > > use something more efficient than stock wsgi modules in the long run. > > > > No red flags yet with Python, and we're in the ballpark for efficiency > > with a binary protocol. A quick test with other servers showed > > rabbitmq at about 9k messages/sec (binary protocol, Erlang server) > > and Gearman at about 13k messages/sec (binary protocol, C server). > > > > -Eric > > > > On Mon, Mar 07, 2011 at 01:32:55PM -0800, Eric Day wrote: > >> I ran the tests again to verify: > >> > >> 500k requests - 10 processes each running 50k requests. > >> > >> time req/s cs us sy id > >> 2 thread/proc > >> echo c++ 7.19 69541 142182 23 77 0 > >> echo erlang 9.53 52465 105871 39 61 0 > >> echo python 9.58 52192 108420 42 58 0 > >> 2 thread/proc > >> wsgi python 58.74 8512 18132 86 14 0 > >> webob python 78.75 6349 13678 89 11 0 > >> webmachine* 63.50 7874 11834 89 11 0 > >> openstack 20.48 24414 49897 68 32 0 > >> > >> cs/us/sys/id are from vmstat while running the tests. > >> > >> * webmachine degrades over time with long-lived, multi-request > >> connections. This number was estimated with 1000 requests per > >> connection. At 50k requests per connection, the rate dropped to > >> 2582 req/s. > >> > >> As you can see I was able to reproduce the same numbers. If > >> anyone would like to do the same, you can grab lp:burrow for the > >> "openstack" Erlang application (compile and run ./start), webmachine > >> is at https://github.com/basho/webmachine (you need to create a demo > >> app and make sure you set nodelay for the socket options), and I've > >> attached the python server and client (start 10 client processes when > >> testing). Find me on irc (eday in #openstack) if you have questions. > >> > >> If we hit performance issues with this type of application, we'll > >> probably hit them around the same time with both Erlang and Python > >> (then we'll look to C/C++). Since most OpenStack developers are a lot > >> more comfortable with Python, I suggest we make the switch. Please > >> response with thoughts on this. I'll add a sqlite backend to the > >> Python prototype and run some performance tests against that to see > >> if any red flags come up. > >> > >> -Eric > >> > >> On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote: > >> > Eric, > >> > Thanks for your experimentation and analysis. Somewhat illustrates > >> > the > >> > point about premature optimization. First cut, have to agree with you > >> > and > >> > conclude that python implementation is effective, overall. As you > >> > said,if > >> > we find performance bottlenecks, especially as the payload size > >> > increases > >> > (as well as if we require any complex processing at the http server > >> > layer) > >> > we can optimize specific
Re: [Openstack] Queue Service Implementation Thoughts
With this switch to python, does it make sense to revisit the concept of openstack-common for things like logging, flag parsing, etc? What components would you like to just be able to drop in from nova, glance, or swift? -todd[1] On Tue, Mar 8, 2011 at 4:05 PM, Eric Day wrote: > Hi everyone, > > I added a sqlite backend to the prototype and ran some tests. Initially > things were very slow, but after some further testing I was able > to figure out where the time was being spent. In order to do this I > added a very simple binary protocol interface to insert only. These > tests are with a single server process and multiple client processes > (so don't compare to previous email numbers that were two process). The > numbers given are requests/second. > > echo (no parsing) - 17k > > binary - 13k > binary+insert msg into dict - 11k > binary+insert msg into sqlite - 8.7k > > wsgi - 4.9k > wsgi+webob - 3.5k > wsgi+webob+insert msg into dict - 3.4k > wsgi+webob+insert msg into sqlite - 2.8k > > wsgi+webob+routes - 1.9k > wsgi+webob+routes+insert msg into dict - 1.8k > wsgi+webob+routes+insert msg into sqlite - 1.5k > > This shows that without wsgi/webob/routes, the performance is pretty > decent). This would be the case when using an efficient binary protocol > or perhaps a more efficient HTTP parser. > > Next, it shows WSGI adds significant overhead. The webob and routes > modules also add a fair amount. > > I'm going to rework the current code in the prototype into a proper > project in the burrow trunk with modular front-ends and back-ends so > we can easily test new options. I'll stick with the current wsgi code > for now just to get things functioning and we can look at optimizing > later. For the proxy-server communication, we'll definitely need to > use something more efficient than stock wsgi modules in the long run. > > No red flags yet with Python, and we're in the ballpark for efficiency > with a binary protocol. A quick test with other servers showed > rabbitmq at about 9k messages/sec (binary protocol, Erlang server) > and Gearman at about 13k messages/sec (binary protocol, C server). > > -Eric > > On Mon, Mar 07, 2011 at 01:32:55PM -0800, Eric Day wrote: >> I ran the tests again to verify: >> >> 500k requests - 10 processes each running 50k requests. >> >> time req/s cs us sy id >> 2 thread/proc >> echo c++ 7.19 69541 142182 23 77 0 >> echo erlang 9.53 52465 105871 39 61 0 >> echo python 9.58 52192 108420 42 58 0 >> 2 thread/proc >> wsgi python 58.74 8512 18132 86 14 0 >> webob python 78.75 6349 13678 89 11 0 >> webmachine* 63.50 7874 11834 89 11 0 >> openstack 20.48 24414 49897 68 32 0 >> >> cs/us/sys/id are from vmstat while running the tests. >> >> * webmachine degrades over time with long-lived, multi-request >> connections. This number was estimated with 1000 requests per >> connection. At 50k requests per connection, the rate dropped to >> 2582 req/s. >> >> As you can see I was able to reproduce the same numbers. If >> anyone would like to do the same, you can grab lp:burrow for the >> "openstack" Erlang application (compile and run ./start), webmachine >> is at https://github.com/basho/webmachine (you need to create a demo >> app and make sure you set nodelay for the socket options), and I've >> attached the python server and client (start 10 client processes when >> testing). Find me on irc (eday in #openstack) if you have questions. >> >> If we hit performance issues with this type of application, we'll >> probably hit them around the same time with both Erlang and Python >> (then we'll look to C/C++). Since most OpenStack developers are a lot >> more comfortable with Python, I suggest we make the switch. Please >> response with thoughts on this. I'll add a sqlite backend to the >> Python prototype and run some performance tests against that to see >> if any red flags come up. >> >> -Eric >> >> On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote: >> > Eric, >> > Thanks for your experimentation and analysis. Somewhat illustrates >> > the >> > point about premature optimization. First cut, have to agree with you >> > and >> > conclude that python implementation is effective, overall. As you >> > said,if >> > we find performance bottlenecks, especially as the payload size >> > increases >> > (as well as if we require any complex processing at the http server >> > layer) >> > we can optimize specific areas. >> > Just for sure, might be better for someone else to recheck. That way >> > we have done our due diligence. >> > Cheers >> > >> > >> > Original Message >> > Subject: [Openstack] Queue Service Implementation Thoughts >> > From: Eric Day >> > Date: Sat, March 05, 2011 4:07 pm >> > To: openstack@lists.launchpad.net >> > >> > Hi everyone, >> > >> > When deciding to move forward with Erlang, I first tried o
Re: [Openstack] Queue Service Implementation Thoughts
Hi everyone, I added a sqlite backend to the prototype and ran some tests. Initially things were very slow, but after some further testing I was able to figure out where the time was being spent. In order to do this I added a very simple binary protocol interface to insert only. These tests are with a single server process and multiple client processes (so don't compare to previous email numbers that were two process). The numbers given are requests/second. echo (no parsing) - 17k binary - 13k binary+insert msg into dict - 11k binary+insert msg into sqlite - 8.7k wsgi - 4.9k wsgi+webob - 3.5k wsgi+webob+insert msg into dict - 3.4k wsgi+webob+insert msg into sqlite - 2.8k wsgi+webob+routes - 1.9k wsgi+webob+routes+insert msg into dict - 1.8k wsgi+webob+routes+insert msg into sqlite - 1.5k This shows that without wsgi/webob/routes, the performance is pretty decent). This would be the case when using an efficient binary protocol or perhaps a more efficient HTTP parser. Next, it shows WSGI adds significant overhead. The webob and routes modules also add a fair amount. I'm going to rework the current code in the prototype into a proper project in the burrow trunk with modular front-ends and back-ends so we can easily test new options. I'll stick with the current wsgi code for now just to get things functioning and we can look at optimizing later. For the proxy-server communication, we'll definitely need to use something more efficient than stock wsgi modules in the long run. No red flags yet with Python, and we're in the ballpark for efficiency with a binary protocol. A quick test with other servers showed rabbitmq at about 9k messages/sec (binary protocol, Erlang server) and Gearman at about 13k messages/sec (binary protocol, C server). -Eric On Mon, Mar 07, 2011 at 01:32:55PM -0800, Eric Day wrote: > I ran the tests again to verify: > > 500k requests - 10 processes each running 50k requests. > > time req/s cs us sy id > 2 thread/proc > echo c++ 7.19 69541 142182 23 77 0 > echo erlang 9.53 52465 105871 39 61 0 > echo python 9.58 52192 108420 42 58 0 > 2 thread/proc > wsgi python 58.74 8512 18132 86 14 0 > webob python 78.75 6349 13678 89 11 0 > webmachine* 63.50 7874 11834 89 11 0 > openstack20.48 24414 49897 68 32 0 > > cs/us/sys/id are from vmstat while running the tests. > > * webmachine degrades over time with long-lived, multi-request > connections. This number was estimated with 1000 requests per > connection. At 50k requests per connection, the rate dropped to > 2582 req/s. > > As you can see I was able to reproduce the same numbers. If > anyone would like to do the same, you can grab lp:burrow for the > "openstack" Erlang application (compile and run ./start), webmachine > is at https://github.com/basho/webmachine (you need to create a demo > app and make sure you set nodelay for the socket options), and I've > attached the python server and client (start 10 client processes when > testing). Find me on irc (eday in #openstack) if you have questions. > > If we hit performance issues with this type of application, we'll > probably hit them around the same time with both Erlang and Python > (then we'll look to C/C++). Since most OpenStack developers are a lot > more comfortable with Python, I suggest we make the switch. Please > response with thoughts on this. I'll add a sqlite backend to the > Python prototype and run some performance tests against that to see > if any red flags come up. > > -Eric > > On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote: > >Eric, > > Thanks for your experimentation and analysis. Somewhat illustrates the > >point about premature optimization. First cut, have to agree with you and > >conclude that python implementation is effective, overall. As you said,if > >we find performance bottlenecks, especially as the payload size increases > >(as well as if we require any complex processing at the http server > > layer) > >we can optimize specific areas. > >Just for sure, might be better for someone else to recheck. That way > >we have done our due diligence. > >Cheers > > > > > > Original Message > > Subject: [Openstack] Queue Service Implementation Thoughts > > From: Eric Day > > Date: Sat, March 05, 2011 4:07 pm > > To: openstack@lists.launchpad.net > > > > Hi everyone, > > > > When deciding to move forward with Erlang, I first tried out the Erlang > > REST framework webmachine (it is built on top of mochiweb and used > > by projects like Riak). After some performance testing, I decided to > > write a simple wrapper over the HTTP packet parsing built into Erlang > > (also used by mochiweb/webmachine) to see if I could make things a > > bit more efficient. Here are the results: > > > > Erlang (2 threads) > > echo - 58823 reqs/sec > >
Re: [Openstack] [Merge] lp:~mdragon/nova/multi-tenant-accounting into lp:nova
On 3/4/11 7:33 PM, Eric Day wrote:
On Fri, Mar 04, 2011 at 01:35:48PM -0600, Monsyne Dragon wrote:
I think, really, we are getting off on a tangent here. The purpose
of multitenant is to have a label ('account' or 'project' or
whatever ) that we tag resources (instances, etc) in nova with
so that we can group together usage reports, etc, that go to some
system outside of openstack for reporting/billing/etc purpose.
The whole thing is pretty tangential to auth. for multitenant, we
really don't care how the user logs in, or where the account label
comes from. Just that it's there, so when someone takes a billable
action, we can record it under the right label for billing, and if
an entity, like an instance, exists we can count it under such a
label for the same.
It's actually not, since the concept of 'project' (which you're mapping
account on top of) is going away. Resources will have an owner, and
these can be acted on by the owner or other accounts (or tenants)
the owner gives permission to. So the account you're talking about
is not just a label, it's going to be another tenant within the
system. A deployment can treat tenants differently of course (ie,
users, projects, billable accounts, ...).
I've read through termie & vish's authn/authz branch, and it appears
that it does the same thing as this patch doing in this respect, namely
it re-uses 'project' as the account/owner. Again, all we really need,
as far as the function of this blueprint is concerned, is a label we can
slap on system usages. The owner might eventually be an account with
all sorts of acl's and delegations, etc, but for this functionality, we
just need an identifier that will go on usage reports, i.e.
'owner.name', so some external system can group them all together and do
billing off them.
I have removed the piece of this branch that added the account to the
service url. I agree that we can wait on that till auth is sorted out.
(really, just removing the hardcoding of the account/project to
"openstack" does the most in enabling us to start working on the
system-usages blueprints.)
I think the outstanding authn/authz branch needs to land (with some
further work) before going to much further.
What is the timeline on that authn/authz branch looking like? I'm
assuming it won't be landing til' diablo.
-Eric
--
--
-Monsyne Dragon
work: 210-312-4190
mobile210-441-0965
google voice: 210-338-0336
Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is
prohibited.
If you receive this transmission in error, please notify us immediately by
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service Implementation Thoughts
nice.
On Mar 8, 2011, at 1:14 PM, Eric Day wrote:
> Hi Curtis,
>
> On Tue, Mar 08, 2011 at 04:31:59PM +, Curtis Carter wrote:
>> Couple of questions.
>> Where did you come up with 300 bytes per message for ets?
>
> Raw message = key(16) + ttl(8) + hide(8) + body(100) = 132 bytes
>
> I inserted the raw message data as the tuple {key/binary, ttl/integer,
> hide/integer, body/binary} in ets (5 million) and inspected the
> memory usage. It lines up with what should be used with the given
> type overheads in:
>
> http://www.erlang.org/doc/efficiency_guide/advanced.html#id66654
>
> It was very fast, but it did come with that overhead.
>
> As far as queue storage, we won't be using a traditional queue
> structure alone. We need a few other indexes into the data store for
> efficient operations, not just inspecting beginning of the queue. For
> example, TTL and hide properties require timestamp-based indexes into
> the queue as well, which are probably best done with a secondary heap
> index or btree. We'll also need insert order and message ID indexed.
>
>> When you say 2 threads for Erlang what do you mean? Do you mean 2
>> schedulers and/or 2 run queues, Erlang processes, or Erlang vms.
>
> Sorry, two scheduler threads in one vm (os kernel threads). For C++
> this was also 2 threads with PTHREAD_SCOPE_SYSTEM. The main point
> is that all connections (erlang processes, eventlet green threads,
> state structures in C++ event loop) were running on two cores. For
> Python this was just a os.fork() after creating the listening socket.
>
>> I wrote a basic queue using the queue module(double ended FIFO) that was
>> handling around 526k a second with 100 clients on an old core2 duo laptop.
>> 20 8 byte messages per client (I do not know how the queue module is
>> implemented)
>> 100 clients/queues
>> Was very unscientific in the front seat of my car while waiting on someone.
>> NO HTTP
>
> The main thing I was trying to test was language overhead, especially
> with HTTP parsing. For the raw echo test, erlang and python are pretty
> even. Python WSGI does have some overhead (just as Erlang modules
> like webmachine did), but that may be solved with an implementation
> optimization (like I did by not using webmachine).
>
>> I know very apple/oranges here. From what I've seen in the past Erlang does
>> not pass other http servers until you hit a large amount of concurrent
>> requests.
> See: http://oddments.org/?p=494
>
> Eventlet came in a bit behind C++ for the larger payload (512k data
> for each of the 32k connections), but it held up quite well. Eventlet
> was only running with a single process too, so I imagine with two
> processes the time would improve. In a later test I used Erlang and
> it came in between the C++ 2-thread test and Eventlet single thread
> (as you would expect from my other tests).
>
> -Eric
>
>> On Mar 7, 2011, at 3:32 PM, Eric Day wrote:
>>
>>> I ran the tests again to verify:
>>>
>>> 500k requests - 10 processes each running 50k requests.
>>>
>>> time req/s cs us sy id
>>> 2 thread/proc
>>> echo c++ 7.19 69541 142182 23 77 0
>>> echo erlang 9.53 52465 105871 39 61 0
>>> echo python 9.58 52192 108420 42 58 0
>>> 2 thread/proc
>>> wsgi python 58.74 8512 18132 86 14 0
>>> webob python 78.75 6349 13678 89 11 0
>>> webmachine* 63.50 7874 11834 89 11 0
>>> openstack20.48 24414 49897 68 32 0
>>>
>>> cs/us/sys/id are from vmstat while running the tests.
>>>
>>> * webmachine degrades over time with long-lived, multi-request
>>> connections. This number was estimated with 1000 requests per
>>> connection. At 50k requests per connection, the rate dropped to
>>> 2582 req/s.
>>>
>>> As you can see I was able to reproduce the same numbers. If
>>> anyone would like to do the same, you can grab lp:burrow for the
>>> "openstack" Erlang application (compile and run ./start), webmachine
>>> is at https://github.com/basho/webmachine (you need to create a demo
>>> app and make sure you set nodelay for the socket options), and I've
>>> attached the python server and client (start 10 client processes when
>>> testing). Find me on irc (eday in #openstack) if you have questions.
>>>
>>> If we hit performance issues with this type of application, we'll
>>> probably hit them around the same time with both Erlang and Python
>>> (then we'll look to C/C++). Since most OpenStack developers are a lot
>>> more comfortable with Python, I suggest we make the switch. Please
>>> response with thoughts on this. I'll add a sqlite backend to the
>>> Python prototype and run some performance tests against that to see
>>> if any red flags come up.
>>>
>>> -Eric
>>>
>>> On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote:
Eric,
Thanks for your experimentation and analysis. Somewhat illustrates the
point about premature optimization. First cut, have to agree with you and
conclude that python implem
Re: [Openstack] Queue Service Implementation Thoughts
Hi Curtis,
On Tue, Mar 08, 2011 at 04:31:59PM +, Curtis Carter wrote:
> Couple of questions.
> Where did you come up with 300 bytes per message for ets?
Raw message = key(16) + ttl(8) + hide(8) + body(100) = 132 bytes
I inserted the raw message data as the tuple {key/binary, ttl/integer,
hide/integer, body/binary} in ets (5 million) and inspected the
memory usage. It lines up with what should be used with the given
type overheads in:
http://www.erlang.org/doc/efficiency_guide/advanced.html#id66654
It was very fast, but it did come with that overhead.
As far as queue storage, we won't be using a traditional queue
structure alone. We need a few other indexes into the data store for
efficient operations, not just inspecting beginning of the queue. For
example, TTL and hide properties require timestamp-based indexes into
the queue as well, which are probably best done with a secondary heap
index or btree. We'll also need insert order and message ID indexed.
> When you say 2 threads for Erlang what do you mean? Do you mean 2 schedulers
> and/or 2 run queues, Erlang processes, or Erlang vms.
Sorry, two scheduler threads in one vm (os kernel threads). For C++
this was also 2 threads with PTHREAD_SCOPE_SYSTEM. The main point
is that all connections (erlang processes, eventlet green threads,
state structures in C++ event loop) were running on two cores. For
Python this was just a os.fork() after creating the listening socket.
> I wrote a basic queue using the queue module(double ended FIFO) that was
> handling around 526k a second with 100 clients on an old core2 duo laptop.
> 20 8 byte messages per client (I do not know how the queue module is
> implemented)
> 100 clients/queues
> Was very unscientific in the front seat of my car while waiting on someone.
> NO HTTP
The main thing I was trying to test was language overhead, especially
with HTTP parsing. For the raw echo test, erlang and python are pretty
even. Python WSGI does have some overhead (just as Erlang modules
like webmachine did), but that may be solved with an implementation
optimization (like I did by not using webmachine).
> I know very apple/oranges here. From what I've seen in the past Erlang does
> not pass other http servers until you hit a large amount of concurrent
> requests.
See: http://oddments.org/?p=494
Eventlet came in a bit behind C++ for the larger payload (512k data
for each of the 32k connections), but it held up quite well. Eventlet
was only running with a single process too, so I imagine with two
processes the time would improve. In a later test I used Erlang and
it came in between the C++ 2-thread test and Eventlet single thread
(as you would expect from my other tests).
-Eric
> On Mar 7, 2011, at 3:32 PM, Eric Day wrote:
>
> > I ran the tests again to verify:
> >
> > 500k requests - 10 processes each running 50k requests.
> >
> >time req/s cs us sy id
> > 2 thread/proc
> > echo c++ 7.19 69541 142182 23 77 0
> > echo erlang 9.53 52465 105871 39 61 0
> > echo python 9.58 52192 108420 42 58 0
> > 2 thread/proc
> > wsgi python 58.74 8512 18132 86 14 0
> > webob python 78.75 6349 13678 89 11 0
> > webmachine* 63.50 7874 11834 89 11 0
> > openstack20.48 24414 49897 68 32 0
> >
> > cs/us/sys/id are from vmstat while running the tests.
> >
> > * webmachine degrades over time with long-lived, multi-request
> > connections. This number was estimated with 1000 requests per
> > connection. At 50k requests per connection, the rate dropped to
> > 2582 req/s.
> >
> > As you can see I was able to reproduce the same numbers. If
> > anyone would like to do the same, you can grab lp:burrow for the
> > "openstack" Erlang application (compile and run ./start), webmachine
> > is at https://github.com/basho/webmachine (you need to create a demo
> > app and make sure you set nodelay for the socket options), and I've
> > attached the python server and client (start 10 client processes when
> > testing). Find me on irc (eday in #openstack) if you have questions.
> >
> > If we hit performance issues with this type of application, we'll
> > probably hit them around the same time with both Erlang and Python
> > (then we'll look to C/C++). Since most OpenStack developers are a lot
> > more comfortable with Python, I suggest we make the switch. Please
> > response with thoughts on this. I'll add a sqlite backend to the
> > Python prototype and run some performance tests against that to see
> > if any red flags come up.
> >
> > -Eric
> >
> > On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote:
> >> Eric,
> >> Thanks for your experimentation and analysis. Somewhat illustrates the
> >> point about premature optimization. First cut, have to agree with you and
> >> conclude that python implementation is effective, overall. As you said,if
> >> we find performance bottlenecks, especially as the payload size increases
> >> (as well as
Re: [Openstack] Queue Service Implementation Thoughts
Couple of questions. Where did you come up with 300 bytes per message for ets? When you say 2 threads for Erlang what do you mean? Do you mean 2 schedulers and/or 2 run queues, Erlang processes, or Erlang vms. I wrote a basic queue using the queue module(double ended FIFO) that was handling around 526k a second with 100 clients on an old core2 duo laptop. 20 8 byte messages per client (I do not know how the queue module is implemented) 100 clients/queues Was very unscientific in the front seat of my car while waiting on someone. NO HTTP I know very apple/oranges here. From what I've seen in the past Erlang does not pass other http servers until you hit a large amount of concurrent requests. On Mar 7, 2011, at 3:32 PM, Eric Day wrote: > I ran the tests again to verify: > > 500k requests - 10 processes each running 50k requests. > >time req/s cs us sy id > 2 thread/proc > echo c++ 7.19 69541 142182 23 77 0 > echo erlang 9.53 52465 105871 39 61 0 > echo python 9.58 52192 108420 42 58 0 > 2 thread/proc > wsgi python 58.74 8512 18132 86 14 0 > webob python 78.75 6349 13678 89 11 0 > webmachine* 63.50 7874 11834 89 11 0 > openstack20.48 24414 49897 68 32 0 > > cs/us/sys/id are from vmstat while running the tests. > > * webmachine degrades over time with long-lived, multi-request > connections. This number was estimated with 1000 requests per > connection. At 50k requests per connection, the rate dropped to > 2582 req/s. > > As you can see I was able to reproduce the same numbers. If > anyone would like to do the same, you can grab lp:burrow for the > "openstack" Erlang application (compile and run ./start), webmachine > is at https://github.com/basho/webmachine (you need to create a demo > app and make sure you set nodelay for the socket options), and I've > attached the python server and client (start 10 client processes when > testing). Find me on irc (eday in #openstack) if you have questions. > > If we hit performance issues with this type of application, we'll > probably hit them around the same time with both Erlang and Python > (then we'll look to C/C++). Since most OpenStack developers are a lot > more comfortable with Python, I suggest we make the switch. Please > response with thoughts on this. I'll add a sqlite backend to the > Python prototype and run some performance tests against that to see > if any red flags come up. > > -Eric > > On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote: >> Eric, >> Thanks for your experimentation and analysis. Somewhat illustrates the >> point about premature optimization. First cut, have to agree with you and >> conclude that python implementation is effective, overall. As you said,if >> we find performance bottlenecks, especially as the payload size increases >> (as well as if we require any complex processing at the http server layer) >> we can optimize specific areas. >> Just for sure, might be better for someone else to recheck. That way >> we have done our due diligence. >> Cheers >> >> >> Original Message >> Subject: [Openstack] Queue Service Implementation Thoughts >> From: Eric Day >> Date: Sat, March 05, 2011 4:07 pm >> To: openstack@lists.launchpad.net >> >> Hi everyone, >> >> When deciding to move forward with Erlang, I first tried out the Erlang >> REST framework webmachine (it is built on top of mochiweb and used >> by projects like Riak). After some performance testing, I decided to >> write a simple wrapper over the HTTP packet parsing built into Erlang >> (also used by mochiweb/webmachine) to see if I could make things a >> bit more efficient. Here are the results: >> >> Erlang (2 threads) >> echo - 58823 reqs/sec >> webmachine - 7782 reqs/sec >> openstack - 24154 reqs/sec >> >> The test consists of four concurrent connections focused on packet >> parsing speed and framework overhead. A simple echo test was also >> done for a baseline (no parsing, just a simple recv/send loop). As >> you can see, the simple request/response wrapper I wrote did get some >> gains, although it's a little more hands-on to use (looks more like >> wsgi+webob in python). >> >> I decided to run the same tests against Python just for comparison. I >> ran echo, wsgi, and wsgi+webob decorators all using eventlet. I ran >> both single process and two process in order to compare with Erlang >> which was running with two threads. >> >> Python (eventlet) >> echo (1 proc) - 17857 reqs/sec >> echo (2 proc) - 52631 reqs/sec >> wsgi (1 proc) - 4859 reqs/sec >> wsgi (2 proc) - 8695 reqs/sec >> wsgi webob (1 proc) - 3430 reqs/sec >> wsgi webob (2 proc) - 6142 reqs/sec >> >> As you can see, the two process Python echo server was not too far >> behind the two thread Erlang echo server. The
Re: [Openstack] Queue Service Implementation Thoughts
Agree with Eric and Thierry on this. John On Mar 8, 2011, at 1:09 AM, Thierry Carrez wrote: > Eric Day wrote: >> If we hit performance issues with this type of application, we'll >> probably hit them around the same time with both Erlang and Python >> (then we'll look to C/C++). Since most OpenStack developers are a lot >> more comfortable with Python, I suggest we make the switch. Please >> response with thoughts on this. I'll add a sqlite backend to the >> Python prototype and run some performance tests against that to see >> if any red flags come up. > > Looking at it (again) from a developer community perspective: with > Python we already know we have the community to support it, while with > Erlang and C++ it's an unknown. +1 for consistency, +1 for the switch. > > -- > Thierry Carrez (ttx) > Release Manager, OpenStack > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Reminder: OpenStack team meeting - 21:00 UTC
Hello everyone, As a reminder, our weekly team meeting will take place at 21:00 UTC this Tuesday in #openstack-meeting on IRC. Check out how that time translates for *your* timezone: http://timeanddate.com/s/1zc8 See the meeting agenda, edit the wiki to add new topics for discussion: http://wiki.openstack.org/Meetings Cheers, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service Implementation Thoughts
Eric Day wrote: > If we hit performance issues with this type of application, we'll > probably hit them around the same time with both Erlang and Python > (then we'll look to C/C++). Since most OpenStack developers are a lot > more comfortable with Python, I suggest we make the switch. Please > response with thoughts on this. I'll add a sqlite backend to the > Python prototype and run some performance tests against that to see > if any red flags come up. Looking at it (again) from a developer community perspective: with Python we already know we have the community to support it, while with Erlang and C++ it's an unknown. +1 for consistency, +1 for the switch. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OS API server password generation
Jay Pipes wrote:
> On Mon, Mar 7, 2011 at 10:43 AM, Dan Prince wrote:
>> Bugs vs. Blueprints is sort of a gray area. When a larger blue print is
>> accepted (like the Openstack 1.1 API) should we file component features as
>> bugs or blueprints?
>>
>> In my case I could have easily considered this a bug as well since we should
>> already support the v1.0 API right?
>
> Ya, probably should have been a bug, but it's OK. I agree there is
> some confusion over bugs vs. blueprints...
Right, there is definitely some overlap between feature (blueprint) and
bug in a corner case: incomplete features. When you develop feature A
and land parts A1 A2 and A3, then realize that A4 is missing, it's a bug
("A is missing support for A4") and a feature ("Add A4 support") at the
same time. Those are usually good targets for exceptions, if they are
relatively self-contained.
My rule of thumb would be to consider them "features" until you release
a version where they are missing, at which point it becomes a "bug"
(i.e. acknowledge that it's an incomplete feature at release time).
Cheers,
--
Thierry Carrez (ttx)
Release Manager, OpenStack
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
