Re: Experience with Open Source load balancers?
Mattew, We run high volume SSL but not nearly the 12Gbps you are talking about so that hasn't been an issue for us. Thanks for the information. Looks like the Citrix ANG rep owes me another lunch to explain himself. :) I'm gonna do some research on NGINX... -Hammer- I was a normal American nerd. -Jack Herer On Wed, May 18, 2011 at 2:23 PM, Andreas Echavez andr...@livejournalinc.com wrote: We're using both an F5 BigIP as well as Nginx (open source software) in a production environment. They both have their merits, but when we recently came under some advanced DDoSes (slowloris, slow POST, and more), we couldn't process certain types of layer 7 insepction/modification because it was too heavy for the F5 to handle. Nginx was more cost effective because we could scale laterally with cheap commodity hardware. This isn't a knock on the BigIP though; it's a much better piece of equipment, has commercial support, and a fantastic web interface. With Nginx you might find yourself compiling modules in by hand and writing config files. Ultimately, the open source solution is going to stand the test of time better. It all depends on who's paying the bills, and what your time is worth. Nginx was specifically worth the effort for us because we had unique traffic demands that change too quickly for a commercial solution. Thanks, Andreas On Mon, May 16, 2011 at 4:15 PM, Welch, Bryan bryan.we...@arrisi.com wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? TIA, replies off list are welcomed. Regards, Bryan
Re: Experience with Open Source load balancers?
I've worked with everything over the years. BigIP, CSS, CSM, ACE (blows), NetScaler, say when. I've been thru a few RFPs and bake offs and also evaluated open source options. 1. If you are looking for simple round robin load balancing with decent load capabilities then there are several open source options in this thread that may work. As long as you understand that you are going to be expected to support them. 2. If you are pushing features. SSL termination. Header rewrites. Payload inspection (NetScaler does application firewalling on the same appliance). Or other complexities and you are having to deal with enterprise traffic volume you might be better off with one of the big vendors. Applications these days are more and more complicated and a high end load balancer with a stable feature set can often rescue your AppDev team and make you a hero. Recommend: F5 and Citrix Netscaler. If you are looking to combine your L7 FW into your LB then you might lean towards NetScaler. If you are looking at seperating those duties you can look at F5. IRules (F5) are the bomb. -Hammer- I was a normal American nerd. -Jack Herer On Wed, May 18, 2011 at 12:31 AM, matthew zeier m...@velvet.org wrote: I'll pile on here too - there's very little of Mozilla's web infrastructure that isn't behind Zeus. +1 for Zeus. Use it in our production network with great success. Magnitudes cheaper than a solution from F5, and doesn't hide the inner workings of the product if you want to do some things outside the scope of support.
Re: Experience with Open Source load balancers?
Recommend: F5 and Citrix Netscaler. If you are looking to combine your L7 FW into your LB then you might lean towards NetScaler. If you are looking at seperating those duties you can look at F5. IRules (F5) are the bomb. Except that under (Mozilla) load, Netscaler fell apart. F5, at the time, could not handle the logging rate I required. Mozilla load is typically defined as high connection rate, low traffic per connection and mostly all SSL. During the Firefox 4 release, we peaked globally at 12Gbps, a significant portion of which was pushed out of three Zeus clusters with L7 rules and some non-trivial traffic script rules and a heck of a lot of content caching. Of all the systems seeing increased usage during the Fx4 release, this wasn't where my worries were :) A slightly older post, http://blog.mozilla.com/mrz/2008/12/04/load-balancer-performance-issues-fxfeedsmozillaorg-versioncheck/
Re: Experience with Open Source load balancers?
We're using both an F5 BigIP as well as Nginx (open source software) in a production environment. They both have their merits, but when we recently came under some advanced DDoSes (slowloris, slow POST, and more), we couldn't process certain types of layer 7 insepction/modification because it was too heavy for the F5 to handle. Nginx was more cost effective because we could scale laterally with cheap commodity hardware. This isn't a knock on the BigIP though; it's a much better piece of equipment, has commercial support, and a fantastic web interface. With Nginx you might find yourself compiling modules in by hand and writing config files. Ultimately, the open source solution is going to stand the test of time better. It all depends on who's paying the bills, and what your time is worth. Nginx was specifically worth the effort for us because we had unique traffic demands that change too quickly for a commercial solution. Thanks, Andreas On Mon, May 16, 2011 at 4:15 PM, Welch, Bryan bryan.we...@arrisi.comwrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? TIA, replies off list are welcomed. Regards, Bryan
Re: Experience with Open Source load balancers?
On Mon, May 16, 2011 at 5:15 PM, Welch, Bryan bryan.we...@arrisi.com wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? Honestly I think to get *all* those features you're much better off with commercial solutions like the ones you're already using from F5, or something from Cisco, Coyote Point, Brocade, or others. You can absolutely put together a solution based on any number of open source products, but you won't get the single integrated front end for management and configuration that any of the commercial options will provide, you may be missing features, and ultimately, you're on the hook for making it work. In particular the stateful failover has been problematic in open source solutions in my experience. They've come a VERY long way, but it is a hard problem to tackle. I've worked with open source and commercial solutions, and while the open source systems were almost always far more flexible, and cheaper up front, they certainly required more work to get going.. Once setup and running though both types of solutions had pretty equal amounts of maintenance, with the commercial solutions requiring somewhat less time/babysitting for upgrades and to enable or use new features or functionality.
Re: Experience with Open Source load balancers?
We've use Linux LVS for many many years with success. http://www.linuxvirtualserver.org/ On Mon, May 16, 2011 at 7:15 PM, Welch, Bryan bryan.we...@arrisi.comwrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? TIA, replies off list are welcomed. Regards, Bryan -- ~Jeff It is not the critic who counts, nor the man who points how the strong man stumbled or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena; whose face is marred by dust and sweat and blood; who strives valiantly...who knows the great enthusiasms, the great devotions, and spends himself in a worthy cause; who, at best, knows the triumph of high achievement; and who, at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who know neither victory nor defeat. Theodore Roosevelt (1858 - 1919), Man in the Arena Speech given April 23, 1910
Re: Experience with Open Source load balancers?
On Tue, 2011-05-17 at 11:03 -0600, Michael Loftis wrote: On Mon, May 16, 2011 at 5:15 PM, Welch, Bryan bryan.we...@arrisi.com wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? Honestly I think to get *all* those features you're much better off with commercial solutions like the ones you're already using from F5, or something from Cisco, Coyote Point, Brocade, or others. You can absolutely put together a solution based on any number of open source products, but you won't get the single integrated front end for management and configuration that any of the commercial options will provide, you may be missing features, and ultimately, you're on the hook for making it work. In particular the stateful failover has been problematic in open source solutions in my experience. They've come a VERY long way, but it is a hard problem to tackle. +1. I think the list of features covers more than just one FOSS project. Whilst I've had no end of good experiences using LVS (as some others have mentioned), I wouldn't expect it to do all that is requested in the original post. At least, not by itself. I've worked with open source and commercial solutions, and while the open source systems were almost always far more flexible, and cheaper up front, they certainly required more work to get going.. Once setup and running though both types of solutions had pretty equal amounts of maintenance, with the commercial solutions requiring somewhat less time/babysitting for upgrades and to enable or use new features or functionality. I worry far more about upgrades to proprietary appliances (where it's often the whole system image), than I do about a few package updates on a Linux machine (followed by a service restart, or two). But still, pretty well worded. :) Tom
Re: Experience with Open Source load balancers?
On 05/17/2011 08:23 AM, Tom Hill wrote: I've worked with open source and commercial solutions, and while the open source systems were almost always far more flexible, and cheaper up front, they certainly required more work to get going.. Once setup and running though both types of solutions had pretty equal amounts of maintenance, with the commercial solutions requiring somewhat less time/babysitting for upgrades and to enable or use new features or functionality. I worry far more about upgrades to proprietary appliances (where it's often the whole system image), than I do about a few package updates on a Linux machine (followed by a service restart, or two). But still, pretty well worded. :) Tom Can't speak for other brands these days but F5s have two hard disks in them. You can upgrade the software on the hot-spare, boot off that and confirm everything is working. If it isn't you can just switch back. Paul
Re: Experience with Open Source load balancers?
On Mon, May 16, 2011 at 04:15:45PM -0700, Welch, Bryan wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? I've used LVS and other Open Source solutions in the past. As others have alluded to, these require knowledge and experience with the underlying OS and network stack that's often lacking in many organizations. A good hybrid solution which implements all (I think) of your requirements is Zeus (http://www.zeus.com/) It's a software solution which you can deploy on your own hardware. It's been very solid in my experience. You can deploy the software in a clustered configuration if needed, though I've only used it in an HA pair. LaDerrick TIA, replies off list are welcomed. Regards, Bryan
Re: Experience with Open Source load balancers?
In message BANLkTimxkNx5=__jxd9056fao19v1zo...@mail.gmail.com, Michael Loftis writes: On Mon, May 16, 2011 at 5:15 PM, Welch, Bryan bryan.we...@arrisi.com wrot= e: Greetings all. I've been tasked with comparing the use of open source load balancing sof= tware against commercially available off the shelf hardware such as F5, whi= ch is what we currently use. =A0We use the load balancers for traditional l= oad balancing, full proxy for http/ssl traffic, ssl termination and certifi= cate management, ssl and http header manipulation, nat, high availability o= f the physical hardware and stateful failover of the tcp sessions. =A0These= units will be placed at the customer prem supporting our applications and = services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. =A0It= is the heart and soul of our data center network after all. =A0However, on= ce I started to think about it I realized that I hadn't had any real experi= ence with this solution beyond tinkering with it at home and reading about = it in years past. Can anyone offer any operational insight and real world experiences with = these solutions? Honestly I think to get *all* those features you're much better off with commercial solutions like the ones you're already using from F5, or something from Cisco, Coyote Point, Brocade, or others. You can absolutely put together a solution based on any number of open source products, but you won't get the single integrated front end for management and configuration that any of the commercial options will provide, you may be missing features, and ultimately, you're on the hook for making it work. In particular the stateful failover has been problematic in open source solutions in my experience. They've come a VERY long way, but it is a hard problem to tackle. I've worked with open source and commercial solutions, and while the open source systems were almost always far more flexible, and cheaper up front, they certainly required more work to get going.. Once setup and running though both types of solutions had pretty equal amounts of maintenance, with the commercial solutions requiring somewhat less time/babysitting for upgrades and to enable or use new features or functionality. Just make sure the DNS components return valid responses to queries as well as valid responses to A queries. Many load balancers get this wrong. They return NXDOMAIN instead of NOERROR, they drop queries, they don't return CNAMEs when the A response returns a CNAME, they return the wrong SOA record (doesn't match the zone delegated to the box). Better still would be for them to return records but until one is ready to do that the negative responses need to be correct. If they are returning queries check NS, SOA, TXT and MX responses for similar errors. is just more visible as browsers make queries and the others are done in the background. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Experience with Open Source load balancers?
On Tue, May 17, 2011 at 6:23 PM, Mark Andrews ma...@isc.org wrote: [snip] Better still would be for them to return records but until one is ready to do that the negative responses need to be correct. Hm... better would be for load balancers operate transparently at Layer 3 and not tamper with the contents of answers from proper DNS servers. Eating traffic based on application content, or turning NOERROR, 0 matches into NXDOMAIN is seriously f***'ed up. I look forward to more domains having DS records published by TLDs w/ signed zones... and possibly browsers displaying warnings trying to visit HTTPS domains without a signed zone. perhaps load balancers/middle box manufacturers will start to become a little bit more honest in what they do with DNS traffic :) -- -JH
Re: Experience with Open Source load balancers?
On Tue, May 17, 2011 at 11:57 AM, LaDerrick H. na...@lacutt.com wrote: On Mon, May 16, 2011 at 04:15:45PM -0700, Welch, Bryan wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? I've used LVS and other Open Source solutions in the past. As others have alluded to, these require knowledge and experience with the underlying OS and network stack that's often lacking in many organizations. A good hybrid solution which implements all (I think) of your requirements is Zeus (http://www.zeus.com/) It's a software solution which you can deploy on your own hardware. It's been very solid in my experience. You can deploy the software in a clustered configuration if needed, though I've only used it in an HA pair. LaDerrick TIA, replies off list are welcomed. Regards, Bryan +1 for Zeus. Use it in our production network with great success. Magnitudes cheaper than a solution from F5, and doesn't hide the inner workings of the product if you want to do some things outside the scope of support. Zeus also does licensing just based on throughput, not arbitrary transactions per second like F5 does/did. If you're hardware can push the traffic, theres no limitations on the number of transactions or sessions. -- Brent Jones br...@servuhome.net
Re: Experience with Open Source load balancers?
In response to your query on dnssec in the browser, I use this. https://addons.mozilla.org/en-us/firefox/addon/dnssec-validator/ --Original Message-- From: Jimmy Hess To: Mark Andrews Cc: Welch, Bryan Cc: nanog@nanog.org Subject: Re: Experience with Open Source load balancers? Sent: May 17, 2011 7:07 PM On Tue, May 17, 2011 at 6:23 PM, Mark Andrews ma...@isc.org wrote: [snip] Better still would be for them to return records but until one is ready to do that the negative responses need to be correct. Hm... better would be for load balancers operate transparently at Layer 3 and not tamper with the contents of answers from proper DNS servers. Eating traffic based on application content, or turning NOERROR, 0 matches into NXDOMAIN is seriously f***'ed up. I look forward to more domains having DS records published by TLDs w/ signed zones... and possibly browsers displaying warnings trying to visit HTTPS domains without a signed zone. perhaps load balancers/middle box manufacturers will start to become a little bit more honest in what they do with DNS traffic :) -- -JH Sent via BlackBerry from T-Mobile
Re: Experience with Open Source load balancers?
I'll pile on here too - there's very little of Mozilla's web infrastructure that isn't behind Zeus. +1 for Zeus. Use it in our production network with great success. Magnitudes cheaper than a solution from F5, and doesn't hide the inner workings of the product if you want to do some things outside the scope of support.
Re: Experience with Open Source load balancers?
S/W vs H/W is really a question rooted in performance and feature needs vs cost... weigh your options carefully. On Mon, May 16, 2011 at 7:15 PM, Welch, Bryan bryan.we...@arrisi.com wrote: Greetings all. I've been tasked with comparing the use of open source load balancing software against commercially available off the shelf hardware such as F5, which is what we currently use. We use the load balancers for traditional load balancing, full proxy for http/ssl traffic, ssl termination and certificate management, ssl and http header manipulation, nat, high availability of the physical hardware and stateful failover of the tcp sessions. These units will be placed at the customer prem supporting our applications and services and we'll need to support them accordingly. Now my knee jerk reaction to this is that it's a really bad idea. It is the heart and soul of our data center network after all. However, once I started to think about it I realized that I hadn't had any real experience with this solution beyond tinkering with it at home and reading about it in years past. Can anyone offer any operational insight and real world experiences with these solutions? TIA, replies off list are welcomed. Regards, Bryan
Re: Experience with Open Source load balancers?
We used Pound (http://www.apsis.ch/pound) on a couple of FreeBSD servers some years ago. Configuration is simple and the software has lots of good and interesting features. The only problem was that always our traffic had a spike, serving pages through it became a nightmare. Eventually we ended up buying a couple of Foundry/Brocade load balancers (Server Iron). I don't know what is software's current development state but if they managed to solve those performance issues it would be an interesting choice, if you really want to go that way. HTH
