________________________________________ From: Jose Hernandez [mailto:[email protected]] Sent: June 12, 2009 5:29 AM To: [email protected] Subject: [pfSense Support] Inbound load balancer performance under heavy load.
Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time... We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before... We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds... So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine... so the only other thing that could be causing the problem was pfSense... however I couldn't find any indication of anything going wrong but the traffic graph... memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc... After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend... and that fixed the problem of requests not getting through and the traffic graph was again stable... I wonder if it is there any known issue with the inbound load balancer... I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~210000, as when load testing we tested from a bunch of ~10 IPs... The problem is that we do need load balancing, mainly for redundancy of our systems at the back end... The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don't know if pfSense is actually able to take the load... Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE(tm) What about cpu load at the time the problem was happening? Do you use smp kernel? http://forum.pfsense.org/index.php/topic,6805.0.html Eugene --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] Commercial support available - https://portal.pfsense.org
