Re: Blades running out of space
Thanks Alex. On Apr 19, 2011, at 12:15 PM, Alexander Patterson wrote: > Hello, > > We have this same issue at CSUEB. We have X amount of resources and X > amount of limited local storage space on our current Blade hardware. > > For example, we have X core blades at X mhz per core with X gb of ram > and X GB HD of local storage that is not upgradable on our current > hardware. I have then calculated X number of VM's can run with X > amount of performance to failure ratio. > > We are running 20 to 22 VM"s per blade that is about 1536mb of ram > each and we give each VM tmp directory about 1 to 3 gb of space per > VM. This makes each VM able to have 1 core at 2400mhz per VM, with our > current hardware. We have our Vmware tmp space set to /dev/shm and > have giving ourself 87 gb. We had it set to 40 GB but were also > running out of space. We have the main images and VMguest load on to > our netappfiler02:/vol/sata1/virtualmachines . > > Here is the our solution for changing the local VMware TMP directory > on our blades > > -Add to /etc/fstab: > none /dev/shm tmpfs size=87G 0 0 (You can set yours to whatever) > -Run 'mount -a' > -Add or edit the line in /etc/vmware/config: > tmpDirectory="/dev/shm" > -Restart VMware: 'service vmware restart' > -Restart the VMs on the host > -Verify that the memory-mapped files are being stored in /dev/shm: > 'lsof | grep vmware-root' > -You should see the space used on /dev/shm increase. Verify this by > running 'df /dev/shm'. > > We are running a NFS share with our VCL, I also want to point that > out. We have tried to use the nFS share as the tmpDirectory for > VMware, this lead to failures and very slow speeds, the Vmware > tmpDirectory should be hosted local if you have the space. > > I have noticed in testing as more software or data is loaded the VM's > increases in size on both the copy image (Vmguest-XX) this is storage > on our local NFS mount and the local blade VM directory. In campus > testing we have found out the hard way with this method due to > failures poor performance when reloading mutipal VM's on our small > amount of blades. We had originaly set our system to run 30 VM"s per > blades but ran out of resources and local storage space. When the > system reloads 30 to 50 + VM's it's going to pushing X amount of > network space/local storage ect. If you run out of tmp space on your > local blade you get all failures on all reservations once your hit the > limit. Also when you reload 10 or 15 VM's at one time on a blade, It > can get very slow. We don't have 100's of blades like NC state, to > prevision and ease the load on our hardware, we had to cut the number > of VM's per blades to increase performance. To help with consistence > reloading, we have also changed some of the Virtual Environments with > 4 or 5 softwares into one image; so multiple classes can use the same > image and not have to reload a image for each class. This has helped > tremendously here on our campus. > > Here is an example of our local storage setup on one of our Blades > here. We are also using VCL 2.1 and Vmware 1.X with our older setup. > > [root@vclesx04 ~]# df -h > FilesystemSize Used Avail Use% Mounted on > /dev/sda2 7.8G 3.6G 3.8G 49% / > /dev/sda6 3.8G 571M 3.1G 16% /var > /dev/sda5 3.8G 188M 3.5G 6% /home > /dev/sda1 97M 18M 75M 19% /boot > tmpfs 87G 36G 52G 42% /dev/shm > netappfiler02:/vol/sata1/virtualmachines > 5.3T 2.2T 3.1T 42% /virtualmachines > > -Alex > > > On Tue, Apr 19, 2011 at 9:26 AM, IBRAHIM MOMOH wrote: >> hi Andy, >> >> Thanks for your quick response. Here are the requested details: >> >> vm guest files are taking up all the space. i think they should be deleted >> after a session ends. >> Provisioning module: Vmware server >> VCL Version: VCL2.1 >> VMware version: 1.0.10 >> >> >> On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote: >> >>> Hello, >>> More information is necessary. Which files are taking up the space? Which >>> provisioning module are you using? Which version of VCL and VMware are you >>> running? >>> >>> -Andy >>> >>> On 4/19/2011 8:15 AM, momoh Ibrahim wrote: Hi Everyone, My university have VCL (using vmware) running on 7 blades, it is difficult to make reservations because the blades are out of space. I have a feeling something is not being deleted after reservations are expired/completed. Any suggestions on what to do? Thank you. Ibrahim Momoh >> >> > > > > -- > Thanks, > Alex Patterson > User Support Services > Operating System Analyst > California State University, East Bay
Re: Blades running out of space
Hello, We have this same issue at CSUEB. We have X amount of resources and X amount of limited local storage space on our current Blade hardware. For example, we have X core blades at X mhz per core with X gb of ram and X GB HD of local storage that is not upgradable on our current hardware. I have then calculated X number of VM's can run with X amount of performance to failure ratio. We are running 20 to 22 VM"s per blade that is about 1536mb of ram each and we give each VM tmp directory about 1 to 3 gb of space per VM. This makes each VM able to have 1 core at 2400mhz per VM, with our current hardware. We have our Vmware tmp space set to /dev/shm and have giving ourself 87 gb. We had it set to 40 GB but were also running out of space. We have the main images and VMguest load on to our netappfiler02:/vol/sata1/virtualmachines . Here is the our solution for changing the local VMware TMP directory on our blades -Add to /etc/fstab: none /dev/shm tmpfs size=87G 0 0 (You can set yours to whatever) -Run 'mount -a' -Add or edit the line in /etc/vmware/config: tmpDirectory="/dev/shm" -Restart VMware: 'service vmware restart' -Restart the VMs on the host -Verify that the memory-mapped files are being stored in /dev/shm: 'lsof | grep vmware-root' -You should see the space used on /dev/shm increase. Verify this by running 'df /dev/shm'. We are running a NFS share with our VCL, I also want to point that out. We have tried to use the nFS share as the tmpDirectory for VMware, this lead to failures and very slow speeds, the Vmware tmpDirectory should be hosted local if you have the space. I have noticed in testing as more software or data is loaded the VM's increases in size on both the copy image (Vmguest-XX) this is storage on our local NFS mount and the local blade VM directory. In campus testing we have found out the hard way with this method due to failures poor performance when reloading mutipal VM's on our small amount of blades. We had originaly set our system to run 30 VM"s per blades but ran out of resources and local storage space. When the system reloads 30 to 50 + VM's it's going to pushing X amount of network space/local storage ect. If you run out of tmp space on your local blade you get all failures on all reservations once your hit the limit. Also when you reload 10 or 15 VM's at one time on a blade, It can get very slow. We don't have 100's of blades like NC state, to prevision and ease the load on our hardware, we had to cut the number of VM's per blades to increase performance. To help with consistence reloading, we have also changed some of the Virtual Environments with 4 or 5 softwares into one image; so multiple classes can use the same image and not have to reload a image for each class. This has helped tremendously here on our campus. Here is an example of our local storage setup on one of our Blades here. We are also using VCL 2.1 and Vmware 1.X with our older setup. [root@vclesx04 ~]# df -h FilesystemSize Used Avail Use% Mounted on /dev/sda2 7.8G 3.6G 3.8G 49% / /dev/sda6 3.8G 571M 3.1G 16% /var /dev/sda5 3.8G 188M 3.5G 6% /home /dev/sda1 97M 18M 75M 19% /boot tmpfs 87G 36G 52G 42% /dev/shm netappfiler02:/vol/sata1/virtualmachines 5.3T 2.2T 3.1T 42% /virtualmachines -Alex On Tue, Apr 19, 2011 at 9:26 AM, IBRAHIM MOMOH wrote: > hi Andy, > > Thanks for your quick response. Here are the requested details: > > vm guest files are taking up all the space. i think they should be deleted > after a session ends. > Provisioning module: Vmware server > VCL Version: VCL2.1 > VMware version: 1.0.10 > > > On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote: > >> Hello, >> More information is necessary. Which files are taking up the space? Which >> provisioning module are you using? Which version of VCL and VMware are you >> running? >> >> -Andy >> >> On 4/19/2011 8:15 AM, momoh Ibrahim wrote: >>> Hi Everyone, >>> >>> My university have VCL (using vmware) running on 7 blades, it is >>> difficult to make reservations because the blades are out of space. I >>> have a feeling something is not being deleted after reservations are >>> expired/completed. Any suggestions on what to do? >>> >>> Thank you. >>> >>> >>> Ibrahim Momoh >>> > > -- Thanks, Alex Patterson User Support Services Operating System Analyst California State University, East Bay
Re: Blades running out of space
hi Andy, Thanks for your quick response. Here are the requested details: vm guest files are taking up all the space. i think they should be deleted after a session ends. Provisioning module: Vmware server VCL Version: VCL2.1 VMware version: 1.0.10 On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote: > Hello, > More information is necessary. Which files are taking up the space? Which > provisioning module are you using? Which version of VCL and VMware are you > running? > > -Andy > > On 4/19/2011 8:15 AM, momoh Ibrahim wrote: >> Hi Everyone, >> >> My university have VCL (using vmware) running on 7 blades, it is >> difficult to make reservations because the blades are out of space. I >> have a feeling something is not being deleted after reservations are >> expired/completed. Any suggestions on what to do? >> >> Thank you. >> >> >> Ibrahim Momoh >>
Re: Blades running out of space
Hello, More information is necessary. Which files are taking up the space? Which provisioning module are you using? Which version of VCL and VMware are you running? -Andy On 4/19/2011 8:15 AM, momoh Ibrahim wrote: Hi Everyone, My university have VCL (using vmware) running on 7 blades, it is difficult to make reservations because the blades are out of space. I have a feeling something is not being deleted after reservations are expired/completed. Any suggestions on what to do? Thank you. Ibrahim Momoh
Blades running out of space
Hi Everyone, My university have VCL (using vmware) running on 7 blades, it is difficult to make reservations because the blades are out of space. I have a feeling something is not being deleted after reservations are expired/completed. Any suggestions on what to do? Thank you. Ibrahim Momoh