[Openstack] deal with booting lots of instance simultaneously

2013-02-19 Thread gtt116
Hi all, When create lots of instance simultaneously, there will be lots of instance in ERROR state. And most of them are caused by network rpc request timeout. This result is not so graceful. I think it will be better if scheduler keep a queue of creating request. when he find all the hosts are

Re: [Openstack] deal with booting lots of instance simultaneously

2013-02-19 Thread Diego Parrilla SantamarĂ­a
Hi gtt, what does it mean for you 'lots of instance simultaneously'? 100, 1000, 1, more? We have launched 100 (but less than 1000) simultaneously without any issue. Rabbit running in a multicore with several gigs of RAM with out of the box configuration. Cheers Diego -- Diego Parrilla

Re: [Openstack] deal with booting lots of instance simultaneously

2013-02-19 Thread gtt116
Hi Diego Thanks for you reply. How many hosts do you have? I have 4 hosts. And in this bug, https://bugs.launchpad.net/nova/+bug/1094226, The N is 20. In my environment N is about 16. I found that nova-network is too busy to deal with so many rpc request at the same time. The Rabbitmq is strong

Re: [Openstack] deal with booting lots of instance simultaneously

2013-02-19 Thread Diego Parrilla SantamarĂ­a
Increasing the RPC timeout should help. I have seen this problem in nova-network in the past. Vish suggestion sounds good. Recently we launched by mistake 128 VMs in a production environment of a customer: 0 errors. They are using 12 cores and several gigs for the nova-network servers with dual

Re: [Openstack] deal with booting lots of instance simultaneously

2013-02-19 Thread Jay Pipes
Are you using multi_host setup? If not, as Vish suggested, that will alleviate much of the problem. Best, -jay On 02/19/2013 04:09 AM, gtt116 wrote: Hi Diego Thanks for you reply. How many hosts do you have? I have 4 hosts. And in this bug, https://bugs.launchpad.net/nova/+bug/1094226, The