Hi Zak, I've raised the max VM limit to 8191 in r36988). We can provide a test build if you are unable to build from SVN, just let us know.
Kind Regards, Knut On May 4, 2011, at 12:41 AM, Zak Perschon wrote: > I had to restart the machine running all of the VMs, so I decided to increase > the amount of files per process like you suggested, relaunched, and we are > now hitting the projected 1023 virtual machine cap! Here is the error that > VboxManage pushes out if anyone is interested (A vbox.log file was also > created for the VM): > > VBoxManage: error: VM creation failed (GVMM) (VERR_GVM_TOO_MANY_VMS) > VBoxManage: error: Unknown error creating VM (VERR_GVM_TOO_MANY_VMS) > VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component > Console, interface IConsole, callee > > Now that we made it this far, I guess the next step would be attempting to > remove the 1023 machine limit that virtualbox is imposing on us? Thanks again > for the help, its been very uplifting to have something actually show > progress (bashing our heads against a slew of different virtualization > solutions has been exhausting). > > -Zak > > > Date: Tue, 3 May 2011 20:39:14 +0200 > > From: [email protected] > > To: [email protected] > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > On 03.05.2011 17:39, Zak Perschon wrote: > > > After letting my mass-deploy script complete, we seem to have hit a cap > > > at 1010 running VMs (A heck of a lot better than 128!). There doesn't > > > seem to be a log created for the 1010th VM that failed, but here is the > > > error message that I get when attempting to start it: > > > > > > VBoxManage: error: The Virtual Machine 'TinyCore-1011' has terminated > > > unexpectidly during startup with exit code 1 > > > VboxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), > > > component Machine, interface IMachine, callee > > > > > > Any ideas as to where to go from here? > > > > Of course :) It looks like you're running into the maximum number of > > open files _per process_ now, which defaults to 1024 on most Linux > > distros. The critical component in this respect is the VBoxXPCOMIPCD > > process (automatically started). This one is the central message broker > > for API requests. Since it also has a couple of files open the limit of > > API clients is a bit lower than 1024. > > > > Not sure if it's worth the effort to get 13 VMs more, but if you want to > > get to the current hardcoded maximum you have to stop all VMs again. > > Please double check with the "ps" command that VBoxXPCOMIPCD has > > terminated (should happen approx. 5 seconds after the last API client is > > gone). > > > > Then set a higher limit either straight with ulimit -n 2048, or > > following the descriptions (the comments are interesting as well) in > > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ > > > > > The current system resource usage is as follows: > > > > > > CPU Usage: 10~% average idling, 20~% when we are utilizing all of the > > > 1010 machines. > > > > This sounds very good. Must be a really stripped down distribution which > > doesn't do much user friendly (i.e. CPU wasting) background activities. > > Another reason for the low utilization might actually be lock > > contention, as all those VMs need to coordinate their activities to some > > degree, and that could cause delays. We'd have some ideas what to change > > if this really happens. > > > > > Memory: 146.8GB of 2019.9GB (which seems low, we have 1010 machines > > > using 150MB of ram and 8 MB of vram, but we are getting replies from all > > > of the machines) > > > Disk Space: 48.6GB of 1.3TB (The machine is supposed to have an attached > > > SAS drive, but its still on order) > > > > The memory usage is suspiciously low. There could be a reason though - > > VirtualBox allocates VM memory lazily. If a VM doesn't touch all its > > memory then it will be below the expected value. > > > > > The original goal of 1800 was our "conservative" estimate, we are > > > actually aiming to get around 4,000 if possible. So far, Virtualbox has > > > been the only VM solution we've tried that has gotten us this close > > > (most either have a cap at around 300~ VMs that can't increase, or they > > > don't support our host machine's hardware and crash). If you'd like, > > > once we are finished with this I can provide a bit more detail on the VM > > > solutions we've tried and the limitations we've ran in to. > > > > For over 1023 it needs some code tweaking... we'll help you with that of > > course. Don't know yet when our expert can spare a few minutes for > > looking into this. > > > > Klaus > > > > > > > > -Zak > > > > > > > Date: Tue, 3 May 2011 17:00:55 +0200 > > > > From: [email protected] > > > > To: [email protected] > > > > CC: [email protected] > > > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > > > Hi Zak, > > > > > > > > On 03.05.2011 00:29, Zak Perschon wrote: > > > > > Hello, > > > > > > > > > > Here is a quick update for you (thanks a ton for the help by the > > > > > way!): > > > > > > > > > > I've been able to breach the 128 cap we had before w/ the fix you > > > > > suggested, but have ran into some "user error issues" that have caused > > > > > me to have to re-deploy/restart my VMs. The highest I've gotten so far > > > > > is 675 running VMs, but I'm sure I'll be able to hit the 1023 mark. It > > > > > takes a uprising amount of time to deploy/start 1000 VMS.... > > > > > > > > Keeping fingers crossed... with 675 VMs, 150MB RAM each (how much VRAM, > > > > I guessed 16MB?), you're around 280GB of RAM used by all of them, using > > > > the rough rule of thumb that RAM usage of a VM is approx. > > > > 1.02*(RAM+VRAM)+250MB. Varies greatly with the VM config details of > > > course. > > > > > > > > With your monster gear having 1.8TB of free RAM the theoretical maximum > > > > would be around 4300 of the tiny VMs you outlined (ignoring the disk and > > > > CPU capacity). > > > > > > > > I'm working on updating the manual. The information for Solaris is > > > > already there (in chapter 9), and the Linux information should also go > > > > there. > > > > > > > > Just out of curiosity - are other virtualizers significantly faster with > > > > such insanely high VM counts? It's really a very exotic use case. > > > > > > > > BTW, we'd appreciate keeping such threads on the mailing list - they > > > > show that VirtualBox is a serious virtualization solution, and also help > > > > others if they run into such extreme situations. > > > > > > > > Klaus > > > > > > > > > > > > > > -Zak > > > > > > > > > > > Date: Mon, 2 May 2011 19:01:21 +0200 > > > > > > From: [email protected] > > > > > > To: [email protected] > > > > > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > > > > > > > On 01.05.2011 17:56, Knut St. Osmundsen wrote: > > > > > > > Hi Zak! > > > > > > > > > > > > > > The VMM imposes a limit of 1023 VMs on 64-bit hosts and 127 VMs on > > > > > > > 32-bit hosts. See: > > > > > > > > > > > > > > > http://www.virtualbox.org/browser/trunk/src/VBox/VMM/VMMR0/GVMMR0.cpp#L121 > > > > > > > > > > > > That means your plan to have 1800 VMs will not work without > > > bumping the > > > > > > hardcoded limits, and in this area it needs much more thinking > > > than just > > > > > > changing a #define. > > > > > > > > > > > > Anyway, let's get you to that limit first... > > > > > > > > > > > > If you run "ipcs -ls" you'll get a "max number of arrays = 128" > > > > > > line, > > > > > > which is what you're most likely bumping into. > > > > > > > > > > > > You can read the kernel defaults with "sysctl kernel.sem" as > > > well, and > > > > > > the last number needs to be increased to a bit more than 1023, > > > just to > > > > > > be on the safe side if some other app also needs a few semaphores. > > > > > > > > > > > > "sysctl -w kernel.sem=250 32000 32 1200" should eliminate this > > > > > > limit. > > > > > > You can also put this in /etc/sysctl.conf, so that it is set > > > straight on > > > > > > system boot, as otherwise these settings are lost on a reboot. > > > > > > > > > > > > So... how many VMs can you start now? > > > > > > > > > > > > Klaus > > > > > > > > > > > > > > > > > > > > Seeing you have 2TB of memory and 128 cores, I would assume you're > > > > > > > running 64-bit Ubuntu. So, it is probably some other limit that > > > you're > > > > > > > hitting. It could be SysV semaphores, it could be memory available > > > > > below > > > > > > > 4GB, ... Could you please provide some more details regarding what > > > > > > > happens when you are unable to launch more VMs? For instance, does > > > > > a new > > > > > > > VBox.log file get created for the VM? If it does, please zip it > > > up and > > > > > > > send it to me (or Klaus) per private mail and we'll work it out. > > > > > > > > > > > > I think the 64bit host OS assumption is safe... I would seriously > > > > > > ask > > > > > > people to have their brain inspected if they'd use a 32bit OS for > > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Apr 29, 2011, at 7:43 PM, Zak Perschon wrote: > > > > > > > > > > > > > >> Ah, sorry forgot to include the original support thread. The > > > limit I > > > > > > >> hit was 128 active VMs at once. Once I hit that number, I was > > > unable > > > > > > >> to launch any more unless I stopped one that was already > > > running. My > > > > > > >> original thread can be found here (though there is little > > > information > > > > > > >> contained in > > > > > > >> it):http://forums.virtualbox.org/viewtopic.php?f=7&t=40955 > > > > > > >> <http://forums.virtualbox.org/viewtopic.php?f=7&t=40955> > > > > > > >> > > > > > > >> -Zak > > > > > > >> > > > > > > >> > Date: Fri, 29 Apr 2011 16:25:44 +0200 > > > > > > >> > From:[email protected] > > > <mailto:[email protected]> > > > > > > >> > To:[email protected] <mailto:[email protected]> > > > > > > >> > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > >> > > > > > > > >> > On 25.04.2011 18:00, Zak Perschon wrote: > > > > > > >> > > > > > > > > >> > > I've come across what is seemingly a hard cap for the > > > amount of > > > > > > >> > > concurrent VMs I can run on a Linux based machine (after > > > posting > > > > > > >> on the > > > > > > >> > > support forums, I was directed here). Apparently this same > > > cap was > > > > > > >> > > encountered on a solaris host and a fix was deployed to > > > > > increase the > > > > > > >> > > amount of VMs that were allowed to run at the same time. Was > > > > > this fix > > > > > > >> > > supposed to be included in other OS releases? Any information > > > > > on this > > > > > > >> > > would be greatly appreciated. > > > > > > >> > > > > > > > >> > That's a rather vague statement, and if you could tell what > > > > > number of > > > > > > >> > concurrently running VMs you can achieve I might be able to > > > > > infer what > > > > > > >> > limit is causing trouble for you. > > > > > > >> > > > > > > > >> > It could be the SysV semaphore count, the number of open > > > files per > > > > > > >> > process, and so on. All these defaults depend on the kernel > > > > > > >> > configuration and/or distribution, so it's hard to generalize. > > > > > > >> > > > > > > > >> > Solaris is easy compared to that, as it is very uniform > > > compared to > > > > > > >> Linux. > > > > > > >> > > > > > > > >> > Klaus > > > > > > >> > > > > > > > >> > > > > > > > > >> > > Here is the information on the hardware/VM profiles I am > > > > > > >> > > using > > > > > > >> > > > > > > > > >> > > Host Machine: > > > > > > >> > > 128 Processor Cores (64 dual core Xenon processors > > > > > [email protected] GHz) > > > > > > >> > > 2.0 TB Memory (1.8~ Usable) > > > > > > >> > > 1.8 TB RAID 0 (6 x 300GB HDD running at 10k RPM) > > > > > > >> > > 16TB SAS > > > > > > >> > > > > > > > > >> > > VM Profile > > > > > > >> > > Compiled TinyCore Linux 2.6 > > > > > > >> > > Required Memory: 150~MB > > > > > > >> > > Required HDD Space: 50MB > > > > > > >> > > > > > > > > >> > > -Zak > > > > > > >> > > > > > > > >> > _______________________________________________ > > > > > > >> > vbox-dev mailing list > > > > > > >> >[email protected] <mailto:[email protected]> > > > > > > >> >http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > >> _______________________________________________ > > > > > > >> vbox-dev mailing list > > > > > > >> [email protected] <mailto:[email protected]> > > > > > > >> http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Kind regards / Mit freundlichen Gruessen / Vennlig hilsen, > > > > > > > bird > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG Knut St. Osmundsen > > > > > > > Werkstrasse 24 Senior Staff Engineer, VirtualBox > > > > > > > 71384 Weinstadt, Germany mailto:[email protected] > > > > > > > > > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 Muenchen > > > > > > > Registergericht: Amtsgericht Muenchen, HRA 95603 > > > > > > > > > > > > > > Komplementaerin: ORACLE Deutschland Verwaltung B.V. > > > > > > > Rijnzathe 6, 3454PV De Meern, Niederlande > > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > > > > > > > Geschaeftsfuehrer: J. Kunz, M. van de Molen, A. van der Ven > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > vbox-dev mailing list > > > > > > > [email protected] > > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > > > > > > > > > -- > > > > > > Oracle <http://www.oracle.com> > > > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop > > > > > > Virtualization > > > > > > Phone: +49 7151 60405 205 <tel:+49715160405205> > > > > > > Oracle VM VirtualBox > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 München > > > > > > Registergericht: Amtsgericht München, HRA 95603 > > > > > > > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > > > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > > > > > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van > > > der Ven > > > > > > > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is > > > committed to > > > > > > developing practices and products that help protect the environment > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > vbox-dev mailing list > > > > > > [email protected] > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > > > -- > > > > Oracle <http://www.oracle.com> > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop > > > > Virtualization > > > > Phone: +49 7151 60405 205 <tel:+49715160405205> > > > > Oracle VM VirtualBox > > > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt > > > > > > > > ORACLE Deutschland B.V. & Co. KG > > > > Hauptverwaltung: Riesstr. 25, D-80992 München > > > > Registergericht: Amtsgericht München, HRA 95603 > > > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > > > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven > > > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is committed to > > > > developing practices and products that help protect the environment > > > > > > > > > > > > > > > > _______________________________________________ > > > vbox-dev mailing list > > > [email protected] > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > -- > > Oracle <http://www.oracle.com> > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop > > Virtualization > > Phone: +49 7151 60405 205 <tel:+49715160405205> > > Oracle VM VirtualBox > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt > > > > ORACLE Deutschland B.V. & Co. KG > > Hauptverwaltung: Riesstr. 25, D-80992 München > > Registergericht: Amtsgericht München, HRA 95603 > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is committed to > > developing practices and products that help protect the environment > > > > > > _______________________________________________ > > vbox-dev mailing list > > [email protected] > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > _______________________________________________ > vbox-dev mailing list > [email protected] > http://vbox.innotek.de/mailman/listinfo/vbox-dev
_______________________________________________ vbox-dev mailing list [email protected] http://vbox.innotek.de/mailman/listinfo/vbox-dev
