Re: [Openstack] Scaling

[Steven Hardy]

On Mon, Jul 01, 2013 at 08:48:24AM +, John Ashford wrote:
> I am a newbie and trying to understand how to scale. If i have a web server 
> on a VM within Openstack is it done by simply assigning more cores to the web 
> server VM? And is there a way to do so automatically within Openstack? And 
> when you reach the maximum number of cores on one physical machine, can you 
> spread load across multiple physical machines without a load balancer in 
> front? 

There are several ways you could approach this problem, but I'll mention
the AutoScaling capability provided by heat, as it does exactly what you
want AFAICT.

We provide orchestration capabilities which allow you to spin up groups of
instances, e.g running web server applications, and balance requests via
another instance running a simple haproxy based loadbalancer (so you don't
need an actual physical loadbalancer appliance).

You can use the AutoScaling functionality without any loadbalancer resource
if you wish, such as in the OpenShift demo linked below (where the broker
node is doing the load balancing)

The trigger for scaling out can either be manual (using the InstanceGroup
resource combined with stack updates), or automatic (using the
AutoScalingGroup resource combined with alarms.

There are example templates in the heat-templates repository:

https://github.com/openstack/heat-templates/blob/master/cfn/AutoScalingMultiAZSample.yaml

Here are some demo videos which describe/demonstrate this feature in more 
detail:

http://www.youtube.com/watch?v=9lLSZi6njq4

http://www.youtube.com/watch?v=nEzK-2lsEVg&feature=youtu.be&t=34m35s

HTH,

Steve

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[William DeCoste]

Hi all,

Apologies for coming in so late to this conversation. Thought I'd 
mention the current capability that Red Hat's OpenShift PaaS provides 
for application auto-scaling. A user can choose to create a scaled 
application which will:


1) Create a primary gear 
(https://openshift.redhat.com/community/faq#t6n11272) which will run an 
instance of the application stack and an HAProxy load-balancer.

2) Create a secondary gear running another instance of the application stack
3) HAProxy will load-balance between the primary and secondary gear(s)
4) As load increases/descreases through HAProxy, OpenShift will 
automatically spin up/down additional secondary gears.


OpenShift does not require the HEAT API, but this could be used to spin 
up the various machines needed to make a HA and stable OpenShift 
environment. Also, when nodes run out of resources to host new 
applications, we could use the HEAT API to spin up new nodes.


Thanks -Bill

 * **To**: openstack@xxx 
 * **From**: Frans Thamura mailto:frans@DOMAIN.HIDDEN>>
 * **Date**: Fri, 26 Oct 2012 12:46:49 +0700

Hi All

Anyone can give me reference, related to scaling PaaS system in OpenStack?

how (more basic better) scalable is implementing PaaS in OpenStack?

right now, we create virtual machine and install ubuntu inside, and
run CloudFoundry or OpenShift to make it PaaS enable.

my target for PaaS is to run our Java apps inside cloud environment.

in another world, we have Liquid VM, but it is not opensource yet,
part of Java VE Virtual Edition. The JVM can boot direct from the
hypervisor.

I still researching the theory behind scalability of cloud esp in
openstack + cloudfoundry.

F

--
Bill DeCoste
Principal Software Engineer, Red Hat
978-204-0920
wdeco...@redhat.com

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[Frans Thamura]

Your url help one glue. DEA need to be configure

Will work on this.

Nb: finding how dea work now... heheh

Frans Thamura
Meruvian
On Oct 27, 2012 3:55 AM, "Diane Mueller"  wrote:

> Stackato (ActiveState's PaaS) auto-scales on OpenStack nicely. I just
> finished deploying it on Folsom
>
> You can find the Stackato docs on auto-scaling are here, and do include an
> OpenStack section for each step:
>
> http://docs.stackato.com/cluster/autoscaling.html?highlight=scaling
>
> Please feel free to go on our #stackato irc channel and ask any further
> questions you have or drop me a note privately if you prefer.
>
> Kind Regards,
>
> Diane Mueller
> ActiveState Stackato
> twitter/irc: pythondj
> http://activestate.com/stackato
>
>
>
>
> On Fri, Oct 26, 2012 at 4:02 AM, Angus Salkeld wrote:
>
>> On 26/10/12 13:07 +0700, Frans Thamura wrote:
>>
>>> Yes, we use it here, but still finding to configure with OpenStack, to
>>> bring scale in this case communicate with openstack nova controller,
>>> we just use it now here..
>>>
>>>
>> You could use the heat project to provide autoscaling.
>> The way this would work is you:
>> 1 create an CloudFormations style template with your application
>> (OpenShift/CloudFoundry)
>> 2 you setup an autoscale group and alarm resource in the template
>> 3 you post the metric of interest in your application to our Cloudwatch
>>   (see the calls to cfn-push-stats)
>> as an example look at:
>> https://github.com/heat-api/**heat/blob/master/templates/**
>> AutoScalingMultiAZSample.**template
>>
>> What happens is you setup a threshold that triggers a scale up and scale
>> down action.
>>
>> also see:
>> https://github.com/heat-api/**heat/blob/master/templates/**
>> OpenShift.template
>> https://github.com/heat-api/**heat/wiki
>>
>>
>> -Angus
>>
>>>
>>>
>>> On Fri, Oct 26, 2012 at 1:00 PM, Ray Sun 
>>> wrote:
>>>
 Have you hearad BOSH, a deployment tool for CloudFoundry on
 cloud(including

 AWS and openstack)?
 https://github.com/**cloudfoundry/bosh

 - Ray
 Yours faithfully, Kind regards.

 CIeNET Technologies (Beijing) Co., Ltd
 Email: qsun01...@cienet.com.cn
 Office Phone: +86-01081470088-7079
 Mobile Phone: +86-13581988291



 On Fri, Oct 26, 2012 at 1:46 PM, Frans Thamura 
 wrote:

>
> Hi All
>
> Anyone can give me reference, related to scaling PaaS system in
> OpenStack?
>
> how (more basic better) scalable is implementing PaaS in OpenStack?
>
> right now, we create virtual machine and install ubuntu inside, and
> run CloudFoundry or OpenShift to make it PaaS enable.
>
> my target for PaaS is to run our Java apps inside cloud environment.
>
> in another world, we have Liquid VM, but it is not opensource yet,
> part of Java VE Virtual Edition. The JVM can boot direct from the
> hypervisor.
>
> I still researching the theory behind scalability of cloud esp in
> openstack + cloudfoundry.
>
> F
>
> __**_
> Mailing list: 
> https://launchpad.net/~**openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : 
> https://launchpad.net/~**openstack
> More help   : 
> https://help.launchpad.net/**ListHelp
>



>>> __**_
>>> Mailing list: 
>>> https://launchpad.net/~**openstack
>>> Post to : openstack@lists.launchpad.net
>>> Unsubscribe : 
>>> https://launchpad.net/~**openstack
>>> More help   : 
>>> https://help.launchpad.net/**ListHelp
>>>
>>
>> __**_
>> Mailing list: 
>> https://launchpad.net/~**openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : 
>> https://launchpad.net/~**openstack
>> More help   : 
>> https://help.launchpad.net/**ListHelp
>>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[Diane Mueller]

Stackato (ActiveState's PaaS) auto-scales on OpenStack nicely. I just
finished deploying it on Folsom

You can find the Stackato docs on auto-scaling are here, and do include an
OpenStack section for each step:

http://docs.stackato.com/cluster/autoscaling.html?highlight=scaling

Please feel free to go on our #stackato irc channel and ask any further
questions you have or drop me a note privately if you prefer.

Kind Regards,

Diane Mueller
ActiveState Stackato
twitter/irc: pythondj
http://activestate.com/stackato




On Fri, Oct 26, 2012 at 4:02 AM, Angus Salkeld  wrote:

> On 26/10/12 13:07 +0700, Frans Thamura wrote:
>
>> Yes, we use it here, but still finding to configure with OpenStack, to
>> bring scale in this case communicate with openstack nova controller,
>> we just use it now here..
>>
>>
> You could use the heat project to provide autoscaling.
> The way this would work is you:
> 1 create an CloudFormations style template with your application
> (OpenShift/CloudFoundry)
> 2 you setup an autoscale group and alarm resource in the template
> 3 you post the metric of interest in your application to our Cloudwatch
>   (see the calls to cfn-push-stats)
> as an example look at:
> https://github.com/heat-api/**heat/blob/master/templates/**
> AutoScalingMultiAZSample.**template
>
> What happens is you setup a threshold that triggers a scale up and scale
> down action.
>
> also see:
> https://github.com/heat-api/**heat/blob/master/templates/**
> OpenShift.template
> https://github.com/heat-api/**heat/wiki
>
>
> -Angus
>
>>
>>
>> On Fri, Oct 26, 2012 at 1:00 PM, Ray Sun  wrote:
>>
>>> Have you hearad BOSH, a deployment tool for CloudFoundry on
>>> cloud(including
>>>
>>> AWS and openstack)?
>>> https://github.com/**cloudfoundry/bosh
>>>
>>> - Ray
>>> Yours faithfully, Kind regards.
>>>
>>> CIeNET Technologies (Beijing) Co., Ltd
>>> Email: qsun01...@cienet.com.cn
>>> Office Phone: +86-01081470088-7079
>>> Mobile Phone: +86-13581988291
>>>
>>>
>>>
>>> On Fri, Oct 26, 2012 at 1:46 PM, Frans Thamura 
>>> wrote:
>>>

 Hi All

 Anyone can give me reference, related to scaling PaaS system in
 OpenStack?

 how (more basic better) scalable is implementing PaaS in OpenStack?

 right now, we create virtual machine and install ubuntu inside, and
 run CloudFoundry or OpenShift to make it PaaS enable.

 my target for PaaS is to run our Java apps inside cloud environment.

 in another world, we have Liquid VM, but it is not opensource yet,
 part of Java VE Virtual Edition. The JVM can boot direct from the
 hypervisor.

 I still researching the theory behind scalability of cloud esp in
 openstack + cloudfoundry.

 F

 __**_
 Mailing list: 
 https://launchpad.net/~**openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : 
 https://launchpad.net/~**openstack
 More help   : 
 https://help.launchpad.net/**ListHelp

>>>
>>>
>>>
>> __**_
>> Mailing list: 
>> https://launchpad.net/~**openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : 
>> https://launchpad.net/~**openstack
>> More help   : 
>> https://help.launchpad.net/**ListHelp
>>
>
> __**_
> Mailing list: 
> https://launchpad.net/~**openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : 
> https://launchpad.net/~**openstack
> More help   : 
> https://help.launchpad.net/**ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[Angus Salkeld]

On 26/10/12 13:07 +0700, Frans Thamura wrote:

Yes, we use it here, but still finding to configure with OpenStack, to
bring scale in this case communicate with openstack nova controller,
we just use it now here..



You could use the heat project to provide autoscaling.
The way this would work is you:
1 create an CloudFormations style template with your application 
(OpenShift/CloudFoundry)
2 you setup an autoscale group and alarm resource in the template
3 you post the metric of interest in your application to our Cloudwatch
  (see the calls to cfn-push-stats)

as an example look at:

https://github.com/heat-api/heat/blob/master/templates/AutoScalingMultiAZSample.template

What happens is you setup a threshold that triggers a scale up and scale down 
action.

also see:
https://github.com/heat-api/heat/blob/master/templates/OpenShift.template
https://github.com/heat-api/heat/wiki


-Angus



On Fri, Oct 26, 2012 at 1:00 PM, Ray Sun  wrote:

Have you hearad BOSH, a deployment tool for CloudFoundry on cloud(including
AWS and openstack)?
https://github.com/cloudfoundry/bosh

- Ray
Yours faithfully, Kind regards.

CIeNET Technologies (Beijing) Co., Ltd
Email: qsun01...@cienet.com.cn
Office Phone: +86-01081470088-7079
Mobile Phone: +86-13581988291



On Fri, Oct 26, 2012 at 1:46 PM, Frans Thamura  wrote:


Hi All

Anyone can give me reference, related to scaling PaaS system in OpenStack?

how (more basic better) scalable is implementing PaaS in OpenStack?

right now, we create virtual machine and install ubuntu inside, and
run CloudFoundry or OpenShift to make it PaaS enable.

my target for PaaS is to run our Java apps inside cloud environment.

in another world, we have Liquid VM, but it is not opensource yet,
part of Java VE Virtual Edition. The JVM can boot direct from the
hypervisor.

I still researching the theory behind scalability of cloud esp in
openstack + cloudfoundry.

F

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp





___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[Frans Thamura]

Yes, we use it here, but still finding to configure with OpenStack, to
bring scale in this case communicate with openstack nova controller,
we just use it now here..



On Fri, Oct 26, 2012 at 1:00 PM, Ray Sun  wrote:
> Have you heard BOSH, a deployment tool for CloudFoundry on cloud(including
> AWS and openstack)?
> https://github.com/cloudfoundry/bosh
>
> - Ray
> Yours faithfully, Kind regards.
>
> CIeNET Technologies (Beijing) Co., Ltd
> Email: qsun01...@cienet.com.cn
> Office Phone: +86-01081470088-7079
> Mobile Phone: +86-13581988291
>
>
>
> On Fri, Oct 26, 2012 at 1:46 PM, Frans Thamura  wrote:
>>
>> Hi All
>>
>> Anyone can give me reference, related to scaling PaaS system in OpenStack?
>>
>> how (more basic better) scalable is implementing PaaS in OpenStack?
>>
>> right now, we create virtual machine and install ubuntu inside, and
>> run CloudFoundry or OpenShift to make it PaaS enable.
>>
>> my target for PaaS is to run our Java apps inside cloud environment.
>>
>> in another world, we have Liquid VM, but it is not opensource yet,
>> part of Java VE Virtual Edition. The JVM can boot direct from the
>> hypervisor.
>>
>> I still researching the theory behind scalability of cloud esp in
>> openstack + cloudfoundry.
>>
>> F
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Scaling PaaS in OpenStack

[Ray Sun]

Have you heard BOSH, a deployment tool for CloudFoundry on cloud(including
AWS and openstack)?
https://github.com/cloudfoundry/bosh

- Ray
Yours faithfully, Kind regards.

CIeNET Technologies (Beijing) Co., Ltd
Email: qsun01...@cienet.com.cn
Office Phone: +86-01081470088-7079
Mobile Phone: +86-13581988291



On Fri, Oct 26, 2012 at 1:46 PM, Frans Thamura  wrote:

> Hi All
>
> Anyone can give me reference, related to scaling PaaS system in OpenStack?
>
> how (more basic better) scalable is implementing PaaS in OpenStack?
>
> right now, we create virtual machine and install ubuntu inside, and
> run CloudFoundry or OpenShift to make it PaaS enable.
>
> my target for PaaS is to run our Java apps inside cloud environment.
>
> in another world, we have Liquid VM, but it is not opensource yet,
> part of Java VE Virtual Edition. The JVM can boot direct from the
> hypervisor.
>
> I still researching the theory behind scalability of cloud esp in
> openstack + cloudfoundry.
>
> F
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Thierry Carrez]

Alejandro Comisario wrote:
> PS: Is the plan to commit the new Zones code into Milestone 3 ? that
> would be fantastic news !

Essex-3 being out, that sounds unlikely. And Essex being feature-frozen,
it also sounds unlikely that such a far-reaching change would land in Essex.

New features and architectural improvements are exciting, but we also
need to be careful to produce something usable in production at the end:
constantly adding new code (and therefore new bugs) is not the way to
win that fight.

Note that, in retrospect, 6-month release schedules with monthly
milestones might not be the best way to get fast availability of new
features together with a bugfixing period. We end up getting features
out fast, but with too many bugs for anyone to consider using them
before final release, which happens too rarely.

We will certainly revisit that for the Folsom cycle -- expect a session
at the Design Summit to discuss that.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Blake Yeager]

Sandy,

This is exactly what I was looking for, thanks for sending it along.  I am
glad to see that you all are thinking about this and have some targets in
mind.

This weekend I will pull your numbers into a wiki page so we can begin to
expand and elaborate on other scale metrics around Nova.  In general I
think they look like reasonable numbers although the number of hosts per
zone looks a little low to me and the response time of 3 seconds seems a
little generous ;-)

Thanks again!

-Blake


On Thu, Jan 26, 2012 at 10:40 AM, Sandy Walsh wrote:

>  Thanks Blake ... all very valid points.
>
> Based on our discussions yesterday (the ink is still wet on the
> whiteboard) we've been kicking around numbers in the following ranges:
>
> 500-1000 hosts per zone (zone = single nova deployment. 1 db, 1 rabbit)
> 25-100 instances per host (minimum flavor)
> 3s api response time fully loaded (over that would be considered a
> failure). 'nova list' being the command that can bring down the house. But
> also 'nova boot' is another concern. We're always trying to get more async
> operations in there.
>
> Hosts per zone is a tricky one because we run into so many issues around
> network architecture, so your mileage may vary. Network is the limiting
> factor in this regard.
>
> All of our design decisions are being made with these metrics in mind.
>
> That said, we'd love to get more feedback on realistic metric expectations
> to ensure we're in the right church.
>
> Hope this is what you're looking for?
>
> -S
>
>
>  --
> *From:* Blake Yeager [blake.yea...@gmail.com]
> *Sent:* Thursday, January 26, 2012 12:13 PM
> *To:* Sandy Walsh
> *Cc:* openstack@lists.launchpad.net
> *Subject:* Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS:
> [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova
> list" (KeyError: 'uuid') + doubts
>
>  Sandy,
>
>  I am excited to hear about the work that is going on around
> communication between trusted zones and look forward to seeing what you
> have created.
>
>  In general, the scalability of Nova is an area where I think we need to
> put additional emphasis.  Rackspace has done a lot of work on zones, but
> they don't seem to be receiving a lot of support from the rest of the
> community.
>
>  The OpenStack mission statement indicates the mission of the project is*:
> * "To produce the ubiquitous Open Source cloud computing platform that
> will meet the needs of public and private cloud providers regardless of
> size, by being simple to implement and massively scalable."
>
>  I would challenge the community to ensure that scale is being given the
> appropriate focus in upcoming releases, especially Nova.  Perhaps we need
> to start by setting very specific scale targets for a single Nova zone in
> terms of nodes, instances, volumes, etc.  I did a quick search of the wiki
> but I didn't find anything about scale targets.  Does anyone know if
> something exists and I am just missing it?  Obviously scale will depend a
> lot on your specific hardware and configuration but we could start by
> saying with this minimum hardware spec and this configuration we want to be
> able to hit this scale.  Likewise it would be nice to publish some
> statistics about the scale that we believe a given release can operate at
> safely.  This would tie into some of the QA/Testing work that Jay & team
> are working on.
>
>  Does anyone have other thoughts about how we ensure we are all working
> toward building a massively scalable system?
>
>  -Blake
>
>  On Thu, Jan 26, 2012 at 9:20 AM, Sandy Walsh 
> wrote:
>
>> Zones is going through some radical changes currently.
>>
>> Specifically, we're planning to use direct Rabbit-to-Rabbit communication
>> between trusted Zones to avoid the complication of changes to OS API,
>> Keystone and novaclient.
>>
>> To the user deploying Nova not much will change, there may be a new
>> service to deploy (a Zones service), but that would be all. To a developer,
>> the code in OS API will greatly simplify and the Distributed Scheduler will
>> be able to focus on single zone scheduling (vs doing both zone and host
>> scheduling as it does today).
>>
>> We'll have more details soon, but we aren't planning on introducing the
>> new stuff until we have a working replacement in place. The default Essex
>> Scheduler now will largely be the same and the filters/weight functions
>> will still carry forward, so any investments there won't be lost.
>>
>> Stay tuned, we're hoping to get all this in a new

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Caitlin Bestler]

The approach here looks solid, but I'm not sure if it goes far enough.

One issue that Keystone has to resolve eventually is how to authenticate 
request for tenant-specific file system users.
Basically the core authentication system allows satellite authentication 
systems to authenticate users within defined scopes.
That is a Tenant X authentication server authenticates file system users for 
Tenant X. TenantX:Jsmith is a different user than
TenantY:Jsmith.

What you probably want to avoid for that sort of system is *mapping* all of 
users from each of the Tenants to the central
authentication server. Adding and deleting file system users from *all* tenants 
could end up being a bit too many transactions
and ultimately requires excessive and error-prone replication of data.

What we need is for TenantX's server to provide the information about who 
"Jsmith" is, and what jsmith is allowed to do,
But in a way where it cannot reference any of TenantY's resources.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Alejandro Comisario]

This is getting really interesting !!
I really hope to see the new Zones code merged into Essex, since we are 
really planning a production implementation on Essex, as soon as it is 
marked as a release ( nova, keystone, glance & swift, wich also we got 
it working on a big lab environment with Milestone 2 )
As the expectation, because we mainly use Cactus into production and 
because of the network layer we inherit, today we are using 1 zone per 
VLAN (thats about 16 hosts of 96GB of RAM each, enough to fill the VLAN 
with the flavors we use), so yes, the limitation here is the networking.


Thats why we are testing Essex with Quantum, cause we really want to 
increase the capacity of a zone ( +50 hosts ) by assigning a/several 
network/s to a project on any zone, and the new MultiZone code, to be 
able to spread the instances across datacenters (and inside the 
datacenter also at the same time), we are thinking also (maybe this out 
of the scope of the subject) that the parent zone might be an instance 
with no compute nodes but with all the zones loaded into the db, and 
many nova-api spawning on different ports, being load balanced at the 
same time just to handle the request for the cloud management.


Just to add a little of our metrics if it helps.

PS: Is the plan to commit the new Zones code into Milestone 3 ? that 
would be fantastic news !


Cheers !

On 01/26/2012 01:40 PM, Sandy Walsh wrote:

Thanks Blake ... all very valid points.

Based on our discussions yesterday (the ink is still wet on the 
whiteboard) we've been kicking around numbers in the following ranges:


500-1000 hosts per zone (zone = single nova deployment. 1 db, 1 rabbit)
25-100 instances per host (minimum flavor)
3s api response time fully loaded (over that would be considered a 
failure). 'nova list' being the command that can bring down the house. 
But also 'nova boot' is another concern. We're always trying to get 
more async operations in there.


Hosts per zone is a tricky one because we run into so many issues 
around network architecture, so your mileage may vary. Network is the 
limiting factor in this regard.


All of our design decisions are being made with these metrics in mind.

That said, we'd love to get more feedback on realistic metric 
expectations to ensure we're in the right church.


Hope this is what you're looking for?

-S



*From:* Blake Yeager [blake.yea...@gmail.com]
*Sent:* Thursday, January 26, 2012 12:13 PM
*To:* Sandy Walsh
*Cc:* openstack@lists.launchpad.net
*Subject:* Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: 
[Question #185840]: Multi-Zone finally working on ESSEX but cant "nova 
list" (KeyError: 'uuid') + doubts


Sandy,

I am excited to hear about the work that is going on around 
communication between trusted zones and look forward to seeing what 
you have created.


In general, the scalability of Nova is an area where I think we need 
to put additional emphasis.  Rackspace has done a lot of work on 
zones, but they don't seem to be receiving a lot of support from the 
rest of the community.


The OpenStack mission statement indicates the mission of the project 
is*:* "To produce the ubiquitous Open Source cloud computing platform 
that will meet the needs of public and private cloud providers 
regardless of size, by being simple to implement and massively scalable."


I would challenge the community to ensure that scale is being given 
the appropriate focus in upcoming releases, especially Nova.  Perhaps 
we need to start by setting very specific scale targets for a single 
Nova zone in terms of nodes, instances, volumes, etc.  I did a quick 
search of the wiki but I didn't find anything about scale targets. 
 Does anyone know if something exists and I am just missing it? 
 Obviously scale will depend a lot on your specific hardware and 
configuration but we could start by saying with this minimum hardware 
spec and this configuration we want to be able to hit this scale. 
 Likewise it would be nice to publish some statistics about the scale 
that we believe a given release can operate at safely.  This would tie 
into some of the QA/Testing work that Jay & team are working on.


Does anyone have other thoughts about how we ensure we are all working 
toward building a massively scalable system?


-Blake

On Thu, Jan 26, 2012 at 9:20 AM, Sandy Walsh 
mailto:sandy.wa...@rackspace.com>> wrote:


Zones is going through some radical changes currently.

Specifically, we're planning to use direct Rabbit-to-Rabbit
communication between trusted Zones to avoid the complication of
changes to OS API, Keystone and novaclient.

To the user deploying Nova not much will change, there may be a
new service to deploy (a Zones service), but that would be all. To
a developer, the cod

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Kevin L. Mitchell]

On Thu, 2012-01-26 at 10:13 -0600, Blake Yeager wrote:
> Does anyone have other thoughts about how we ensure we are all working
> toward building a massively scalable system?

I recently discussed with both Sandy and Ziad a "multi-realm" extension
to Keystone.  I've documented my thoughts on it as the following
blueprint:

https://blueprints.launchpad.net/keystone/+spec/multi-realm

The spec is at:

http://wiki.openstack.org/MultiRealmKeystone

And your email provides a perfect starting point for kicking off a
discussion on the concept :)
-- 
Kevin L. Mitchell 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Sandy Walsh]

Thanks Blake ... all very valid points.

Based on our discussions yesterday (the ink is still wet on the whiteboard) 
we've been kicking around numbers in the following ranges:

500-1000 hosts per zone (zone = single nova deployment. 1 db, 1 rabbit)
25-100 instances per host (minimum flavor)
3s api response time fully loaded (over that would be considered a failure). 
'nova list' being the command that can bring down the house. But also 'nova 
boot' is another concern. We're always trying to get more async operations in 
there.

Hosts per zone is a tricky one because we run into so many issues around 
network architecture, so your mileage may vary. Network is the limiting factor 
in this regard.

All of our design decisions are being made with these metrics in mind.

That said, we'd love to get more feedback on realistic metric expectations to 
ensure we're in the right church.

Hope this is what you're looking for?

-S



From: Blake Yeager [blake.yea...@gmail.com]
Sent: Thursday, January 26, 2012 12:13 PM
To: Sandy Walsh
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question 
#185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 
'uuid') + doubts

Sandy,

I am excited to hear about the work that is going on around communication 
between trusted zones and look forward to seeing what you have created.

In general, the scalability of Nova is an area where I think we need to put 
additional emphasis.  Rackspace has done a lot of work on zones, but they don't 
seem to be receiving a lot of support from the rest of the community.

The OpenStack mission statement indicates the mission of the project is: "To 
produce the ubiquitous Open Source cloud computing platform that will meet the 
needs of public and private cloud providers regardless of size, by being simple 
to implement and massively scalable."

I would challenge the community to ensure that scale is being given the 
appropriate focus in upcoming releases, especially Nova.  Perhaps we need to 
start by setting very specific scale targets for a single Nova zone in terms of 
nodes, instances, volumes, etc.  I did a quick search of the wiki but I didn't 
find anything about scale targets.  Does anyone know if something exists and I 
am just missing it?  Obviously scale will depend a lot on your specific 
hardware and configuration but we could start by saying with this minimum 
hardware spec and this configuration we want to be able to hit this scale.  
Likewise it would be nice to publish some statistics about the scale that we 
believe a given release can operate at safely.  This would tie into some of the 
QA/Testing work that Jay & team are working on.

Does anyone have other thoughts about how we ensure we are all working toward 
building a massively scalable system?

-Blake

On Thu, Jan 26, 2012 at 9:20 AM, Sandy Walsh 
mailto:sandy.wa...@rackspace.com>> wrote:
Zones is going through some radical changes currently.

Specifically, we're planning to use direct Rabbit-to-Rabbit communication 
between trusted Zones to avoid the complication of changes to OS API, Keystone 
and novaclient.

To the user deploying Nova not much will change, there may be a new service to 
deploy (a Zones service), but that would be all. To a developer, the code in OS 
API will greatly simplify and the Distributed Scheduler will be able to focus 
on single zone scheduling (vs doing both zone and host scheduling as it does 
today).

We'll have more details soon, but we aren't planning on introducing the new 
stuff until we have a working replacement in place. The default Essex Scheduler 
now will largely be the same and the filters/weight functions will still carry 
forward, so any investments there won't be lost.

Stay tuned, we're hoping to get all this in a new blueprint soon.

Hope it helps,
Sandy


From: boun...@canonical.com<mailto:boun...@canonical.com> 
[boun...@canonical.com<mailto:boun...@canonical.com>] on behalf of Alejandro 
Comisario 
[question185...@answers.launchpad.net<mailto:question185...@answers.launchpad.net>]
Sent: Thursday, January 26, 2012 8:50 AM
To: Sandy Walsh
Subject: Re: [Question #185840]: Multi-Zone finally working on ESSEX but cant   
"nova list" (KeyError: 'uuid') + doubts

Question #185840 on OpenStack Compute (nova) changed:
https://answers.launchpad.net/nova/+question/185840

   Status: Answered => Open

Alejandro Comisario is still having a problem:
Sandy, Vish !

Thanks for the replies ! let me get to the relevant points.

#1 I totally agree with you guys, the policy for spawning instances
maybe very special of each company strategy, but, as you can pass from
"Fill First" to "Spread First" just adding a &qu

Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

[Blake Yeager]

Sandy,

I am excited to hear about the work that is going on around communication
between trusted zones and look forward to seeing what you have created.

In general, the scalability of Nova is an area where I think we need to put
additional emphasis.  Rackspace has done a lot of work on zones, but they
don't seem to be receiving a lot of support from the rest of the community.

The OpenStack mission statement indicates the mission of the project is*:* "To
produce the ubiquitous Open Source cloud computing platform that will meet
the needs of public and private cloud providers regardless of size, by
being simple to implement and massively scalable."

I would challenge the community to ensure that scale is being given the
appropriate focus in upcoming releases, especially Nova.  Perhaps we need
to start by setting very specific scale targets for a single Nova zone in
terms of nodes, instances, volumes, etc.  I did a quick search of the wiki
but I didn't find anything about scale targets.  Does anyone know if
something exists and I am just missing it?  Obviously scale will depend a
lot on your specific hardware and configuration but we could start by
saying with this minimum hardware spec and this configuration we want to be
able to hit this scale.  Likewise it would be nice to publish some
statistics about the scale that we believe a given release can operate at
safely.  This would tie into some of the QA/Testing work that Jay & team
are working on.

Does anyone have other thoughts about how we ensure we are all working
toward building a massively scalable system?

-Blake

On Thu, Jan 26, 2012 at 9:20 AM, Sandy Walsh wrote:

> Zones is going through some radical changes currently.
>
> Specifically, we're planning to use direct Rabbit-to-Rabbit communication
> between trusted Zones to avoid the complication of changes to OS API,
> Keystone and novaclient.
>
> To the user deploying Nova not much will change, there may be a new
> service to deploy (a Zones service), but that would be all. To a developer,
> the code in OS API will greatly simplify and the Distributed Scheduler will
> be able to focus on single zone scheduling (vs doing both zone and host
> scheduling as it does today).
>
> We'll have more details soon, but we aren't planning on introducing the
> new stuff until we have a working replacement in place. The default Essex
> Scheduler now will largely be the same and the filters/weight functions
> will still carry forward, so any investments there won't be lost.
>
> Stay tuned, we're hoping to get all this in a new blueprint soon.
>
> Hope it helps,
> Sandy
>
> 
> From: boun...@canonical.com [boun...@canonical.com] on behalf of
> Alejandro Comisario [question185...@answers.launchpad.net]
> Sent: Thursday, January 26, 2012 8:50 AM
> To: Sandy Walsh
> Subject: Re: [Question #185840]: Multi-Zone finally working on ESSEX but
> cant   "nova list" (KeyError: 'uuid') + doubts
>
> Question #185840 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/185840
>
>Status: Answered => Open
>
> Alejandro Comisario is still having a problem:
> Sandy, Vish !
>
> Thanks for the replies ! let me get to the relevant points.
>
> #1 I totally agree with you guys, the policy for spawning instances
> maybe very special of each company strategy, but, as you can pass from
> "Fill First" to "Spread First" just adding a "reverse=True" on
> nova.scheduler.least_cost.weighted_sum" and
> "nova.scheduler.distributed_scheduler._schedule" maybe its a harmless
> addition to manipulate (since we are going to have a lot of zones across
> datacenters, and many different departments are going to create many
> instances to load-balance their applications, we really preffer
> SpreadFirst to make sure hight availability of the pools )
>
> #2 As we are going to test essex-3, i would like if you can tell me if
> the zones code from Chris Behrens is going to be added on Final Essex /
> Milestone 4, so we can keep testing other features, or you preffer us to
> load this as a bug to be fixed since maybe the code that broke is not
> going to have major changes.
>
> Kindest regards !
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp