I need your assistance to configure load balancer servers.
Hello Support, As I am a newbie on HA Proxy, can anybody please provide me the sample configuration file for the following scenario? I need your assistance to configure load balancer servers in such a way, 1] I have hosted one IDENTICAL application on my two following servers, srv1.artificialmachines.com: 10.1.138.10 (FYI..Public IP:x.x.x.x) srv2.artificialmachines.com: 10.1.138.11 (FYI..Public IP:x.x.x.x) Now, I want the hosted application should be load-balanced by our high-availability load balancers in an active/passive configuration. In addition to that I need a virtual IP address: x.x.x.x with two ports. It will be assigned to the hosted application by the load balancer so that hosted application have a single IP address to access it. The required ports on wich the hosted application will accept the connections are as, -- virtual IP: x.x.x.x:37425 and 41836. 2] Communication should be haapen between servers and client in following manner, i] On port virtual IP: x.x.x.x:41836 the requests or connections should only acceptable from theses two IP addresses: 174.x.x.x and 174.x.x.x.(public IP addresses of my another servers). ii] All other requests or connections from internet are allowable or acceptable on the port virtual IP: x.x.x.x:37425. So, please confirm me if it is possible to implement the same. Please help me. Also please let me know about, 1] Can I change or modify the existing configured of HA? 2] If yes then, How do I modify(add/remove)existing entries from the configuration file?(If I want to modify any parameter or section in future) 3] Can I change the existing entries from default section? or how to reset the existing parameters from default section? 4] How to make new sever entries in configuration file? 4] How to restart the haproxy after some modification done in configuration file? Please help me. -- Manoj M. Burande, Artificial Machines Pvt Ltd, System Administrator.
RE: Nbproc question
Here's the output of top on the system: top - 09:50:36 up 4 days, 18:50, 1 user, load average: 1.31, 1.59, 1.55 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 2.5%us, 9.9%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.5%hi, 12.1%si, 0.0%st Mem: 8179536k total, 997748k used, 7181788k free, 139236k buffers Swap: 9976356k total,0k used, 9976356k free, 460396k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 752741 daemon20 0 34760 24m 860 R 100 0.3 871:15.76 haproxy It's a quad core system, but haproxy is taking 100% of one core. We're doing less than 5k req/sec and the box has two 2.6ghz Opterons in it. Do you know how much health checks affect cpu utilization of an haproxy process? We have about 100 backend servers and we're running inter 500 rise 2 fall 1 I haven't tried adjusting that, although when it was set to the default our error rates were much higher. Thanks, Jonah -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Monday, September 28, 2009 9:50 PM To: Jonah Horowitz Cc: haproxy@formilux.org Subject: Re: Nbproc question On Mon, Sep 28, 2009 at 06:43:58PM -0700, Jonah Horowitz wrote: In the documentation it seems to discourage using the nbproc directive. What¹s the situation with this? I¹m running a server with 8 cores, so I¹m tempted to up the nbproc. Is the process normally multithreaded? no the process is not multithreaded. Is nbproc something I can use for performance tuning, or is it just for file handles? It can bring you small performance gains at the expense of a more complex monitoring, since the stats will still only reflect the process which receives the stats request. Also, health-checks will be performed by each process, causing an increased load on your servers. And the connection limitation will not work anymore, as any process won't know that there are other processes already using a server. It was initially designed to workaround per-process file handle limitations on some systems, but it is true that it brings a minor performance advantage. However, considering that you can reach 4 connections per second with a single process on a cheap core2duo 2.66 GHz, and that forwarding data at 10 Gbps on this machine consumes only 20% of a core, you can certainly understand why I don't see the situations where it would make sense to use nbproc. Regards, Willy
Re: ssl health check question
Hi Willy, Thanks. I'll correct the config. Basically, I split the proxy into frontend and backend. I followed example given in the haproxy manual to achieve what I wanted to do. Thanks for your help, Ninad Examples : # Relay HTTPS traffic to Apache instance and check service availability # using HTTP request OPTIONS * HTTP/1.1 on port 80. backend https_relay mode tcp option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www server apache1 192.168.1.1:443 check port 80 From: Willy Tarreau w...@1wt.eu To: Ninad Raje nina...@yahoo.com Cc: haproxy@formilux.org Sent: Monday, September 28, 2009 9:54:35 PM Subject: Re: ssl health check question Hi Ninad, On Mon, Sep 28, 2009 at 09:57:11AM -0700, Ninad Raje wrote: Hi Willy, I want to use httpchk in addition to ssk-hello-chk to check the health of for our backend SSL servers. An example below: listen ssl bind xx.xx.xx.xxx:443 mode tcp balance roundrobin optionhttpclose log global optionhttplog optiondontlognull optionforwardfor except 127.0.0.1/8 optionssl-hello-chk optionhttpchk GET /hplbhealth maxconn 200 servervmvpe xx.xx.xx.xxx:443 check inter 2000 weight 1 So you mean you'd like to use two distinct checks in parallel ? If so, it's not yet possible, this requires the support of scripted health checks. In your case, I see that the config is wrong anyway, as you have enabled some HTTP options on this TCP proxy which appears to work in SSL only. So as you can guess, haproxy will not be able to modify the contents nor log them since it would require it to decipher the contents. Willy
Re: Nbproc question
On Tue, Sep 29, 2009 at 10:30 AM, Willy Tarreau w...@1wt.eu wrote: On Tue, Sep 29, 2009 at 09:56:51AM -0700, Jonah Horowitz wrote: Here's the output of top on the system: top - 09:50:36 up 4 days, 18:50, 1 user, load average: 1.31, 1.59, 1.55 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 2.5%us, 9.9%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.5%hi, 12.1%si, 0.0%st Mem: 8179536k total, 997748k used, 7181788k free, 139236k buffers Swap: 9976356k total, 0k used, 9976356k free, 460396k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 752741 daemon 20 0 34760 24m 860 R 100 0.3 871:15.76 haproxy It's a quad core system, but haproxy is taking 100% of one core. We're doing less than 5k req/sec and the box has two 2.6ghz Opterons in it. huh! surely there's something unusually wrong here. Do you know how much health checks affect cpu utilization of an haproxy process? We have about 100 backend servers and we're running inter 500 rise 2 fall 1 It means only 200 checks per second, that's not much at all. I've ran tests up to 4 checks per second, so you should not even notice it. I haven't tried adjusting that, although when it was set to the default our error rates were much higher. Could you send me your conf in private ? Also, what's your data rate ? I'm seeing the following CPU usage : Cpu(s): 2.5%us, 9.9%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.5%hi, 12.1%si, 0.0%st Which translates into that for one CPU : 10% user 40% system 50% soft-irq This means that 90% of the time is spent in the kernel (network stack+drivers). Do you have a high bit rate (multi-gigabit) ? Are you sure you aren't running with any ip_conntrack/nf_conntrack module loaded ? Can you show the output of do you recommend against these modules? we have a stock fedora 10 kernel that have nf_conntrack compiled in statically. i've increased: /proc/sys/net/netfilter/nf_conntrack_max but is it correct to expect connection tracking to add kernel networking cpu overhead due to netfilter? i've speculated that it might, but fruitless searches for discussions that would suggest so have restrained me from bothering to re-compile a custom kernel for our haproxy machines. haproxy -vv ? I'd like to see if epoll support is correctly enabled. Also, please send the output of uname -a. Ah, please also check the clock sources : # cat /sys/devices/system/clocksource/clocksource0/current_clocksource # cat /sys/devices/system/clocksource/clocksource0/available_clocksource Many dual-core opterons had no synchronization for their internal timestamp counters, so those were often disabled and replaced with slow external clock sources, resulting in poor network performance. Something nice on an opteron however is that you can nearly double the performance by binding the network interrupts to one core and the process on the other one of the same socket. this is intriguing. can this be done with other multi-core cpus? do you have any documentation that i could read to learn more about this? Regards, Willy
Re: Nbproc question
On Tue, Sep 29, 2009 at 10:41:28AM -0700, David Birdsong wrote: (...) Which translates into that for one CPU : 10% user 40% system 50% soft-irq This means that 90% of the time is spent in the kernel (network stack+drivers). Do you have a high bit rate (multi-gigabit) ? Are you sure you aren't running with any ip_conntrack/nf_conntrack module loaded ? Can you show the output of do you recommend against these modules? we have a stock fedora 10 kernel that have nf_conntrack compiled in statically. By default I recommend against it because it's never tuned for server usage, and if people don't know if they are using it, then they might be using it with inadequate desktop tuning. i've increased: /proc/sys/net/netfilter/nf_conntrack_max but is it correct to expect connection tracking to add kernel networking cpu overhead due to netfilter? i've speculated that it might, but fruitless searches for discussions that would suggest so have restrained me from bothering to re-compile a custom kernel for our haproxy machines. Yes, from my experience, using conntrack on a machine (with large enough hash buckets) still results in 1/3 of the CPU being usable for haproxy+system and 2/3 being consumed by conntrack. You must understand that when running conntrack on a proxy, it has to setup and tear down two connections per proxy connection, explaining why it ends up with that amount of CPU used. Often if you absolutely need conntrack to NAT packets, the solution consist in setting it on one front machine and having the proxies on a second level machine (run both in series). It will *triple* the performance because the number of conntrack entries will be halved and it will have more CPU to run. Something nice on an opteron however is that you can nearly double the performance by binding the network interrupts to one core and the process on the other one of the same socket. this is intriguing. can this be done with other multi-core cpus? yes it can but it's only possible/efficient when the L2 cache is shared, which is the case on opterons. With an L3 cache, it will not be as efficient but will still be. But when your caches are completely independant, having packets being parsed by one core and pass to the other core through slow memory is horribly inefficient as the data pass twice on the memory bus for nothing ! do you have any documentation that i could read to learn more about this? Not that much. I remember there are some useful tuning tricks on the Myricom site and/or in some of their drivers' READMEs. That's where I discovered the DCA mechanism that I was not aware of. Willy
[PATCH] [MEDIUM] Collect provide separate statistics for sockets
From 9f3e91786298783690dd8b0a113b8a5c7f1ed137 Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Tue, 29 Sep 2009 20:54:26 +0200
Subject: [MEDIUM] Collect provide separate statistics for sockets
This patch allows to collect provide separate statistics for each socket.
It can be very useful if you would like to distinguish between traffic
generate by local and remote users or between different types of remote
clients (peerings, domestic, foreign).
Currently only Sessions, Bytes, Denied and Req Errors stats are supported,
but adding Session rate or even Session Limit should be possible if
we found it useful.
---
doc/configuration.txt | 19 +++
include/proto/dumpstats.h |1 +
include/proto/proxy.h |8 ++-
include/types/proto_http.h |1 +
include/types/protocols.h |4 ++
include/types/proxy.h | 12 ++---
include/types/session.h|1 +
src/cfgparse.c | 96 +-
src/client.c | 19 +--
src/dumpstats.c| 112 +++-
src/haproxy.c |6 +-
src/log.c |2 +-
src/proto_http.c | 76 --
src/proto_tcp.c|6 ++-
src/proxy.c|4 +-
src/session.c | 29 ---
16 files changed, 327 insertions(+), 69 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 1416386..c9c5caf 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -744,6 +744,7 @@ option httplog X X X X
option originalto X X X X
[no] option persist X - X X
[no] option redispatch X - X X
+[no] option socket-statsX X X -
option smtpchk X - X X
[no] option splice-auto X X X X
[no] option splice-request X X X X
@@ -1074,6 +1075,8 @@ bind [address]:port [, ...]
bind [address]:port [, ...] interface interface
bind [address]:port [, ...] mss maxseg
bind [address]:port [, ...] transparent
+bind [address]:port [, ...] id id
+bind [address]:port [, ...] name name
Define one or several listening addresses and/or ports in a frontend.
May be used in sections : defaults | frontend | listen | backend
no |yes | yes | no
@@ -1107,6 +1110,13 @@ bind [address]:port [, ...] transparent
work on other operating systems. The commonly advertised
value on Ethernet networks is 1460 = 1500(MTU) - 40(IP+TCP).
+id is a persistent value for socket ID. Must be unique and
+ larger than 1000, as smaller values are reserved for
+ auto-assigned ids. Can only be used when defining only
+ a single socket.
+
+nameis an optional name provided for stats
+
transparent is an optional keyword which is supported only on certain
Linux kernels. It indicates that the addresses will be bound
even if they do not belong to the local machine. Any packet
@@ -2665,6 +2675,15 @@ no option redispatch
See also : redispatch, retries
+option socket-stats
+no option socket-stats
+
+ Enable or disable collecting providing separate statistics for each socket.
+ May be used in sections:defaults | frontend | listen
+
+ Arguments : none
+
+
option smtpchk
option smtpchk hello domain
Use SMTP health checks for server testing
diff --git a/include/proto/dumpstats.h b/include/proto/dumpstats.h
index a0167a7..81aae1a 100644
--- a/include/proto/dumpstats.h
+++ b/include/proto/dumpstats.h
@@ -38,6 +38,7 @@
#define STATS_TYPE_FE 0
#define STATS_TYPE_BE 1
#define STATS_TYPE_SV 2
+#define STATS_TYPE_SO 3
#define STATS_ST_INIT 0
#define STATS_ST_REQ 1
diff --git a/include/proto/proxy.h b/include/proto/proxy.h
index 0ad1e7f..ce62b3f 100644
--- a/include/proto/proxy.h
+++ b/include/proto/proxy.h
@@ -66,9 +66,13 @@ static inline void proxy_reset_timeouts(struct proxy *proxy)
}
/* increase the number of cumulated connections on the designated frontend */
-static void inline proxy_inc_fe_ctr(struct proxy *fe)
+static void inline proxy_inc_fe_ctr(struct listener *l, struct proxy *fe)
{
- fe-cum_feconn++;
+ fe-counters.cum_feconn++;
+
+ if (l-counters)
+ l-counters-cum_feconn++;
+
update_freq_ctr(fe-fe_sess_per_sec, 1);
if (fe-fe_sess_per_sec.curr_ctr fe-fe_sps_max)
fe-fe_sps_max = fe-fe_sess_per_sec.curr_ctr;
diff --git a/include/types/proto_http.h b/include/types/proto_http.h
index c0350d2..b91228e 100644
--- a/include/types/proto_http.h
+++ b/include/types/proto_http.h
@@ -159,6 +159,7 @@ enum {
