Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Steve Heistand
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ok a followup to this. its not any sort of disk access issue. I created a new disk to hold the glance directory and running atop while launching shows all the activity on the original root drive nothing using glance. Normally at idle the root drive

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Eric_E_Smith
12:43 PM To: openstack@lists.launchpad.net Subject: Re: [Openstack] launching multiple VMs takes very long time -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ok a followup to this. its not any sort of disk access issue. I created a new disk to hold the glance directory and running atop while

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Rick Jones
On 05/01/2013 10:43 AM, Steve Heistand wrote: there may be some network issues going on here, trying to shove some amount of data bigger then a few Gig seems to start slowing things down. If you think there is an actual network problem, you could I suppose try exploring that with netperf or

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Steve Heistand
so I dont think there is a network or disk bottle neck now. in watching instances launch I see that the disk image itself gets copied out to the compute node in 1 minute, the xml file for libvirt gets created. then a pause of ~ 20 minutes then the kvm process starts up and off the VM goes just

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Mark Lehrer
where should I start looking to find out why things are so slow? Check the glance node. The first prototype I set up had an incredibly slow glance node, which led to this kind of behavior. Are the compute nodes downloading at the same time, or sequentially? Mark

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Rick Jones
On 04/30/2013 11:42 AM, Steve Heistand wrote: if I launch one vm at a time its doesnt very long to start up the instance. maybe a minute. if I launch 4 instances (of the same snapshot as before) it takes 30 minutes. they are all launching to different compute nodes, the controllers are all

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Eric_E_Smith
Perhaps network throughput where the snapshot is hosted (The controller node)? -Original Message- From: Openstack [mailto:openstack-bounces+eric_e_smith=dell@lists.launchpad.net] On Behalf Of Steve Heistand Sent: Tuesday, April 30, 2013 1:43 PM To: openstack@lists.launchpad.net

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Steve Heistand
They are all booting the same image. there still may be a random read issues if they arent mostly synced in copying the file to the compute node. But there isnt much of a load on the disk while the instances are being launched. Its just a sata drive at this point so the performance isnt great but

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Melanie Witt
This presentation from the summit might be of interest to you: http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-the-boot-barrier-identifying-and-eliminating-contention-in-openstack I couldn't find just the slide deck anywhere so far. Your issue is different

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Steve Heistand
this certainly looks promising.. thanks s On 04/30/2013 01:36 PM, Melanie Witt wrote: This presentation from the summit might be of interest to you:

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Rick Jones
On 04/30/2013 01:36 PM, Melanie Witt wrote: This presentation from the summit might be of interest to you: http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-the-boot-barrier-identifying-and-eliminating-contention-in-openstack A nice presentation. Based on his