Re: Need help with HAProxy
Could this be related to the number of concurrent connections being run? maxconn 1024 under global seems quite low, and if your servers are holding connections open due to responding slowly could this be the cause? Chris On 13/01/2012 03:37, John Lauro wrote: If the refused connection concerns you, and you can't avoid starting haproxy, one option is to put up a firewall rule to block syn packets while haproxy reloads, and then unblock. That way clients will retry the connection in about 3 seconds instead of being refused.
Re: Virtual Hosting and logs
Hey Willy, LOL Then I was confused by other comments I got back when I posted about analyzing the logs the other day. :-) Your right about syslog-ng, I would definitely recommend it to anyone also. Joe On Fri, Jan 13, 2012 at 1:54 AM, Willy Tarreau w...@1wt.eu wrote: Hi Joe, On Thu, Jan 12, 2012 at 08:40:01PM -0500, Joseph Hardeman wrote: Hey Chris, What flavor of linux will you be putting syslog-ng on? Be sure the syslog-ng you install can handle multi-threading of its processes, so version 3.0 or newer I believe, otherwise it will eat up all of 1 CPU and could most certainly lose logs then if you have a lot of traffic going through haproxy. My experience with syslog-ng has already been extremely good since version 1.4 around 10 years ago. I remember reaching 2 logs per second with zero losses on a pentium-3 933 MHz. You need to tune it to use large buffers to cover disk latency, and that's all. Syslog-ng is an excellent piece of software, which is why I always recommend it to everyone who needs high logging rates. We have it setup for one of our customers now, actually I just finished setting it up four days ago and I have syslog-ng splitting out logs per hour. I don't really see much in the way of missing logs, if anything they now have more information than they were getting for the visits to their site from Google Analytics. But just as an idea, using option httplog clf in the listen section for mode http, yesterday I receiving around 12G of logs from a single haproxy box while today they are at 4.9G and the day isn't over yet. So today may end up around 10G as the west coast is now getting off of work. And the clf option sends through less data than the normal option httplog so the amount of data is a bit lower than if you log normal logs from haproxy. This point surprizes me a little bit because CLF logs contain the same info with more delimiters. Maybe they compress better but I'm surprized you find them smaller. For instance : normal: Jan 13 07:52:50 pcw haproxy[839]: 127.0.0.1:56837[13/Jan/2012:07:52:46.258] echo echo/NOSRV 0/0/0/3325/3789 200 14 - - 0/0/0/0/0 0/0 GET / HTTP/1.1 clf: Jan 13 07:52:34 pcw haproxy[834]: 127.0.0.1 - - [13/Jan/2012:06:52:31 +] GET / HTTP/1.1 200 14 - - 56835 759 echo echo NOSRV 0 0 0 2285 2845 0 0 0 0 0 0 0 - - Regards, Willy
VM vs bare metal and threading
All, I'm not sure what the issue is here, but I wanted to know if there was an easy explanation for this. We've been doing some load testing of HAProxy and have found the following: HAProxy (both 1.4.15 and 1.4.19 builds) running under Gentoo in a 2 vCPU VM (Vsphere 4.x) running on a box with a Xeon x5675 (3.06 GHz current gen Westmere) maxes out (starts throwing 50x errors) at around a session rate of 3500. However, copies of the same binaries pointed at the same backend servers on a Gentoo box (bare metal) with 2x E5405 (2.00GHz - Q4,2007 launch) top out at a session rate of around 8000 - at which point the back end servers start to fall over. And that HAProxy machine is doing LOTS of other things at the same time. Here's the reason for the query: We're not sure why, but the bare metal box seems to be balancing the load better across cpu's. (We're using the same config file, so nbproc is set to 1 for both setups). Most of our HAProxy setups aren't really getting hit hard enough to tell if multiple CPU's are being used or not as their session rates typically stay around 300-400. We know it's not virtualization in general because we have a virtual machine in the production version of this system that achieves higher numbers on lesser hardware. Just wondering if there is somewhere we should start looking. TIA. matt
RE: VM vs bare metal and threading
There are all sorts of kernel tuning parameters under /proc that can make a big difference, not to mention what type of virtual NIC you have in the VM. Are they running the same kernel version and Gentoo release? Have you compared sysctl.conf (or whatever gento uses to customize settings in /proc)? Generally I prefer to run haproxy (and only haproxy) in 1 CPU vms (less CPUs, lower latency from the vm scheduler), with haproxy, my only exception is if I also want ssl and load is higher. When dealing with high rates and larger number of connections make sure you don't go low on RAM. Haproxy goes exponentially worse as it starts to swap, in fact running swapoff -a isn't a bad idea, especially for bench testing... and it takes a lot more ram to support 8000 connections/sec than 300. In summary, check RAM, and /proc tuning. -Original Message- From: Matt Banks [mailto:mattba...@gmail.com] Sent: Friday, January 13, 2012 2:40 PM To: haproxy@formilux.org Subject: VM vs bare metal and threading All, I'm not sure what the issue is here, but I wanted to know if there was an easy explanation for this. We've been doing some load testing of HAProxy and have found the following: HAProxy (both 1.4.15 and 1.4.19 builds) running under Gentoo in a 2 vCPU VM (Vsphere 4.x) running on a box with a Xeon x5675 (3.06 GHz current gen Westmere) maxes out (starts throwing 50x errors) at around a session rate of 3500. However, copies of the same binaries pointed at the same backend servers on a Gentoo box (bare metal) with 2x E5405 (2.00GHz - Q4,2007 launch) top out at a session rate of around 8000 - at which point the back end servers start to fall over. And that HAProxy machine is doing LOTS of other things at the same time. Here's the reason for the query: We're not sure why, but the bare metal box seems to be balancing the load better across cpu's. (We're using the same config file, so nbproc is set to 1 for both setups). Most of our HAProxy setups aren't really getting hit hard enough to tell if multiple CPU's are being used or not as their session rates typically stay around 300-400. We know it's not virtualization in general because we have a virtual machine in the production version of this system that achieves higher numbers on lesser hardware. Just wondering if there is somewhere we should start looking. TIA. matt
