Re: Practical limit to number of jails on a given host?
Thanks everyone for the very helpful responses. :) -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Practical limit to number of jails on a given host?
On 6. Feb 2012, at 20:29 , Doug Barton wrote: > Howdy, > > Thinking about implementing a poor-man's virtualization solution with > lots'o'jails, and wondering what people think about the practical limits > of such a system. I realize that part of the answer is going to depend > on CPU and RAM, so let's assume for the sake of argument that the answer > to that bit is, "Lots of both." > > So first question is, is there some sort of hard-coded limit somewhere? > If not, what is the largest number of jails that you've created > successfully/reliably on a system, and what are the specs for that system? Yes, jails provide you 6 9s ... though that's not 99.% but 99 is the maximum number of jails. And yes, I have started this many before -- without processes or anything. It took a couple of days, due to some list handling, which could be optimized. You will find that once you get there, you'll have a syscall which never returns... You notice once the start loop slows down if you print a . every 100 or 1000. The machine was a 4 or 8 core amd64 with 8G of memory. I think I had a slide in there: http://www.bsdcan.org/2010/schedule/attachments/130_2010-bz-the-new-vvorld.pdf I know if using vnets; you can get the 4k (or more) but I also have reports that it may not scale. The other limit you'll run into is the number of PIDs. And eventually scheduling depending on what you want to do. > And finally, has anyone run into trouble with a large number of IP > addresses for the jails? ISTR that way back when, the IP addresses > associated with a particular interface were stored in a linked list, so > as you added more you would start seeing O(N) slowdown on a lot of > network stuff in the kernel. Yeah, we still do list walks here and there. /bz -- Bjoern A. Zeeb You have to have visions! It does not matter how good you are. It matters what good you do! ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Practical limit to number of jails on a given host?
On Feb 6, 2012, at 21:29, Doug Barton wrote: > Howdy, > > Thinking about implementing a poor-man's virtualization solution with > lots'o'jails, and wondering what people think about the practical limits > of such a system. I realize that part of the answer is going to depend > on CPU and RAM, so let's assume for the sake of argument that the answer > to that bit is, "Lots of both." Worry more about disk I/O. ZFS with fast spindles in raid-Z combined with SSD L2ARC and ZIL got me much, much further than only spindles, but in the end I caved and did SSD across the board on the most busy jail hosts. They have anywhere between 40 and 70 jails running, many of them very busy, all of them different. The process count seen from the host is in the low four digits. > So first question is, is there some sort of hard-coded limit somewhere? > If not, what is the largest number of jails that you've created > successfully/reliably on a system, and what are the specs for that system? I've - for the sake of testing - had about 350 jails on one system, each with a mysql, a java/tomcat, and an nginx. They all worked and responded fine to queries. I have no reason to think it would be a problem to add more. The system in question was a 12-core (2 CPU), 48GB system. > On a related note, what are the limits in terms of mount points on the > system and/or jails? I'm thinking of a fairly typical "nullfs mount the > system, devfs, and 2 or 3 NFS mount points" per jail type of situation. I have no idea about NFS in such a setting; I use nullfs (ro) for all the system stuff (6 per jail iirc), and use zfs datasets for /, /tmp, /var, /etc and /usr/local inside the jails. Devfs of course. I implement filesystem quotas and the likes using zfs, along with compression for datasets that generally benefit from that. Make sure you allow for enough open files. Also make sure any postgreses you allow are on different UIDs (unless 9.x has a new way of "fixing" that sysv limitation). If you use ZFS, it might be an idea to limit the ARC size (loader.conf) to avoid ZFS gobbling up all the free memory after booting but before processes in the jails have ballooned). And make sure you have plenty of swap. You don't want to swap, but if things get hot it's better to have a slowdown from swapping than having random processes being killed off ;) > And finally, has anyone run into trouble with a large number of IP > addresses for the jails? ISTR that way back when, the IP addresses > associated with a particular interface were stored in a linked list, so > as you added more you would start seeing O(N) slowdown on a lot of > network stuff in the kernel. I remember DES complained about his 1-something ghz athlon getting slow with 1500 jails due to this. That was back around ..5-BETA? I remember laughing long and hard at the insanity of 1500 jails on one box, and even more at him being surprised that "something" would barf .. But I am pretty sure it was fixed soon after. > Any thoughts or advice along these lines will be greatly appreciated. :) > > > Doug > > -- > > It's always a long day; 86400 doesn't fit into a short. > > Breadth of IT experience, and depth of knowledge in the DNS. > Yours for the right price. :) http://SupersetSolutions.com/ > > ___ > freebsd-jail@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org" > ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Practical limit to number of jails on a given host?
On 2012-02-06, at 3:29 PM, Doug Barton wrote: > > So first question is, is there some sort of hard-coded limit somewhere? > If not, what is the largest number of jails that you've created > successfully/reliably on a system, and what are the specs for that system? ~150 full jails (postfix+cyrus-imapd+apache) on a Quad Core Xeon with 32G of RAM ... they weren't high load sites, obviously, but loading was generally <1, and the machine was perfectly responsive …___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Practical limit to number of jails on a given host?
In response to Doug Barton : > > Any thoughts or advice along these lines will be greatly appreciated. :) I don't know of any hardcoded limits, but I can give you some specifics of what I know works. We have production servers running 40 jails. They perform a variety of functions from database jails that are constantly busy to batch job jails that sit idle 99% of the time. These systems have 16G of RAM, 16 CPUs, and about 200 nullfs mounts. They run like production-quality stuff should, long uptimes (only rebooted for OS upgrades) and very little maintenance or babysitting required. HTH -- http://www.intermedix.com Bill Moran Senior Vice President of Databases Phone: 412-422-3463x4023 The information contained in this message is confidential and may be privileged and/or protected under law. If you received this message in error, please notify us immediately by forwarding a copy to karen.coll...@intermedix.com and then deleting the original message and any attachments. ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Practical limit to number of jails on a given host?
- Original Message - From: "Doug Barton" So first question is, is there some sort of hard-coded limit somewhere? If not, what is the largest number of jails that you've created successfully/reliably on a system, and what are the specs for that system? We happilly run up ~80 single process jails on 24 core machines without issue. One thing to be aware of is the issue with prison0->uref becoming 0 and panicing the machine if a fix for this hasnt been commited yet. If you need a working patch for this I can provide one. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Practical limit to number of jails on a given host?
Howdy, Thinking about implementing a poor-man's virtualization solution with lots'o'jails, and wondering what people think about the practical limits of such a system. I realize that part of the answer is going to depend on CPU and RAM, so let's assume for the sake of argument that the answer to that bit is, "Lots of both." So first question is, is there some sort of hard-coded limit somewhere? If not, what is the largest number of jails that you've created successfully/reliably on a system, and what are the specs for that system? On a related note, what are the limits in terms of mount points on the system and/or jails? I'm thinking of a fairly typical "nullfs mount the system, devfs, and 2 or 3 NFS mount points" per jail type of situation. And finally, has anyone run into trouble with a large number of IP addresses for the jails? ISTR that way back when, the IP addresses associated with a particular interface were stored in a linked list, so as you added more you would start seeing O(N) slowdown on a lot of network stuff in the kernel. Any thoughts or advice along these lines will be greatly appreciated. :) Doug -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
