Re: weights
Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? On Tue, Mar 4, 2014 at 12:15 PM, Willy Tarreau w...@1wt.eu wrote: On Sat, Mar 01, 2014 at 11:06:32PM +0530, vijeesh vijayan wrote: Thanks. will share screenshot shortly. roundrobin recommented for mysql also? What Baptiste is explaining is that leastconn focuses on balancing the number of established connections and not the cumulated number of connections. If one server responds slowly and the other responds fast, the slow one will always have a certain number of open connections while the fast one will have very few. Thus it is normal that haproxy will pick the fast one more often than the slow one. And this is precisely the purpose of leastconn. Some people use leastconn to avoid servers which are suffering from some local system perturbations (eg: backups). And in general, what you're observing means exactly that one server is working much better than another one. So round robin will equally distribute the number of requests to your servers, but will degrade the quality of service since the slow one will get more requests than right now, and the fast one will remain mostly idle waiting for the slow one to get its share. Willy -- = Vijeesh K The temptation to quit will be greatest just before you are about to succeed
Re: [PATCH] MINOR: set IP_FREEBIND on IPv6 sockets in transparent mode
On 03.03.2014 21:31, Willy Tarreau wrote: On Mon, Mar 03, 2014 at 09:10:51PM +0100, Lukas Tribus wrote: Lets set IP_FREEBIND on IPv6 sockets as well, this works since Linux 3.3 and doesn't require CAP_NET_ADMIN privileges (IPV6_TRANSPARENT does). This allows unprivileged users to bind to non-local IPv6 addresses, which can be useful when setting up the listening sockets or when connecting to backend servers with a specific, non-local source IPv6 address (at that point we usually dropped root privileges already). Patch applied, thank you Lukas! I will test the patch. Stupid question, but is it really supported from 3.3 and higher? A quick test with dev22 yesterday seemed to be working but I didn't put any traffic through it. It was late so I didn't give it enough attention ;-) Sander
Re: weights
On Tue, Mar 04, 2014 at 02:20:32PM +0530, vijeesh vijayan wrote: Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? Yes it does affect it. In leastconn, the weight affects the number of concurrent connections. So if a server has a weight of 125 and another one of 2, then haproxy will ensure that the first one has 125/127 connections while the second has 2/127 connections. But again, these are *concurrent* connections, not cumulated connections. Willy
Re: [PATCH] MINOR: set IP_FREEBIND on IPv6 sockets in transparent mode
Hi Sander, Patch applied, thank you Lukas! I will test the patch. Stupid question, but is it really supported from 3.3 and higher? A quick test with dev22 yesterday seemed to be working but I didn't put any traffic through it. It was late so I didn't give it enough attention ;-) Just tested it with plain dev22 and 3.2 and IPv6 seems to work nicely. What does the patch do? Because I don't quite understand. Sorry for the confusion. In your case, as long as you start haproxy initially as root, you don't need this patch and you don't need linux 3.3. All you need is start haproxy as root, it will set IPV6_TRANSPARENT on the socket and it will work in all kernels starting with 2.6.37. The problem with the behavior before this patch was that IPV6_TRANSPARENT requires superuser privileges (or more specifically the CAP_NET_ADMIN capability). There are 2 use cases where we may not have this capability: - when HAProxy is not started as root initially (and listens only to ports 1024) - I suspect this is not very common - when HAProxy drops root privileges after the initial setup, and the socket option is needed on backend connections (which are not setup at HAproxy start of course, but when the connection is actually needed, but at that point we don't have the capabilities anymore) IP_FREEBIND doesn't need special privileges, so this fixes those 2 cases. In your case however you don't need IP_FREEBIND, because: - you only need the socket options on frontend connections (specified on the bind line) - which are setup before dropping to a normal user - you start haproxy with root privileges (most likely you bind to port 80 or 443, so you need to start privileged anyway) So the patch is not necessary for you. IP_FREEBIND on IPv6 sockets requires Linux 3.3, but IPV6_TRANSPARENT only requires 2.6.37. Regards, Lukas
Re: weights
This distribution happens only when server x and y has same number of open connections? On Tue, Mar 4, 2014 at 3:10 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Mar 04, 2014 at 02:20:32PM +0530, vijeesh vijayan wrote: Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? Yes it does affect it. In leastconn, the weight affects the number of concurrent connections. So if a server has a weight of 125 and another one of 2, then haproxy will ensure that the first one has 125/127 connections while the second has 2/127 connections. But again, these are *concurrent* connections, not cumulated connections. Willy -- = Vijeesh K The temptation to quit will be greatest just before you are about to succeed
Re: weights
On Tue, Mar 04, 2014 at 07:50:04PM +0530, vijeesh vijayan wrote: This distribution happens only when server x and y has same number of open connections? no, the distribution happens all the time. To make it simpler to understand, imagine that you have weight=1 for all servers. Haproxy will then try to balance the established connections so that all servers have the same number. Now if one server has weight 2, haproxy will try to load it with twice the number of connections as the first one. And so on... That's why leastconn is normally used with long-lived connections (eg: RDP, LDAP, SQL, ...). Willy
Re: weights
Thanks. please check my last reply Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? in this case , server x should be getting 5 folds of connections of y ideally. but something is preventing this . Am i right? in our case x is getting only 50 percent of y ( we are calculating the number of connections/sec) . how do we know how many connections haproxy keep it open for a particular server? On Tue, Mar 4, 2014 at 7:52 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Mar 04, 2014 at 07:50:04PM +0530, vijeesh vijayan wrote: This distribution happens only when server x and y has same number of open connections? no, the distribution happens all the time. To make it simpler to understand, imagine that you have weight=1 for all servers. Haproxy will then try to balance the established connections so that all servers have the same number. Now if one server has weight 2, haproxy will try to load it with twice the number of connections as the first one. And so on... That's why leastconn is normally used with long-lived connections (eg: RDP, LDAP, SQL, ...). Willy -- = Vijeesh K The temptation to quit will be greatest just before you are about to succeed
Re: weights
On Tue, Mar 04, 2014 at 08:27:03PM +0530, vijeesh vijayan wrote: Thanks. please check my last reply Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? in this case , server x should be getting 5 folds of connections of y ideally. but something is preventing this . Am i right? in our case x is getting only 50 percent of y ( we are calculating the number of connections/sec) . how do we know how many connections haproxy keep it open for a particular server? No, unfortunately you definitely don't understand the difference between *concurrent* connections and *cumulated* connections. You're measuring the number of connections distributed over time. I'm talking about concurrent connections, which is what leastconn is about. Willy
haproxy loosing connections
Hello, I'm maintain postgresql cluster with streaming replication for php-based webapp. And for a few days I'm trying to get rid of errors in my setup: Application serverDB server | PHP - pgbouncer - haproxy | - | postgresql | pgbouncer pools connections from php (session-based) and haproxy load-balance and failovering 3 backend postgresql servers. Every ~10 min haproxy drops connection and pgbouncer reports: application logs: failed to execute the SQL statement: SQLSTATE[08P01]: Unknown error: 7 ERROR: server conn crashed? syslog: Mar 4 22:16:12 app1 pgbouncer[15572]: C-0x1d0c130: mydb/pgsql@unix:5432 Pooler Error: server conn crashed? When I remove haproxy from this setup or change balancer to any other tcp-balancer: balancer-ng for example everything works fine! I tried almost everything I can imagine: - changing connection between php, pgbouncer and haproxy to tcp/ip or unix-socket - changing timeouts, conn lifetimes, keepalive, addition tcp options, pool modes - downgrading to older versions of pgbouncer and haproxy - reduced number of TW-sockets by changing connectivity of other components to unix-socket where possible Any ideas what to look for? software versions: php5 5.4.23 pgbouncer 1.5.4 haproxy 1.5dev22 pgbouncer config: == [databases] * = host=127.0.0.1 port=6432 [pgbouncer] syslog = 1 pidfile = /var/run/postgresql/pgbouncer.pid listen_addr = 127.0.0.1 listen_port = 5432 unix_socket_dir = /var/run/postgresql listen_backlog = -1 auth_type = trust auth_file = /etc/pgbouncer/userlist.txt admin_users = stats_users = pgsql pool_mode = session server_reset_query = DISCARD ALL; ignore_startup_parameters = application_name server_check_query = select 1 server_check_delay = 10 max_client_conn = 5120 default_pool_size = 16 reserve_pool_size = 0 reserve_pool_timeout = 0 log_connections = 0 log_disconnections = 0 log_pooler_errors = 1 server_lifetime = 1200 server_idle_timeout = 60 query_timeout = 0 client_login_timeout = 60 == haproxy config: == defaults option splice-auto option tcpka timeout connect 5s timeout client 2s timeout server 10s listen stats :18080 mode http stats enable stats uri / listen pgsql 127.0.0.1:6432 maxconn 3000 mode tcp balance roundrobin option tcp-smart-accept option tcp-smart-connect option pgsql-check user postgres server slave1 10.0.0.1:5432 server slave2 10.0.0.2:5432 server slave3 10.0.0.3:5432 ==
HAProxy 1.5 possible bug
Hello,
Is this a known bug in HAProxy 1.5?
When I use 0.0.0.0 or * as server address for a certain host, HAProxy
crashes with a oom_killer log. This is what is in the man page:
server
...
Address “0.0.0.0″ or “*” has a special meaning.
It indicates that the connection will be forwarded to the same IP
address as the one from the client connection. This is useful in
transparent proxy architectures where the client’s connection is
intercepted and haproxy must forward to the original destination
address.
eg in the backend:
use-server www.speedtest.net if { hdr_sub(host) speedtest.net }
server www.speedtest.net *
so what should happen is that the alias 'www.speedtest.net' should be
equal to the same IP address as was transmitted.
Or am I doing something wrong?
Thanks
Fred
Re: inspecting incoming tcp content
On Tue, 4 Mar 2014 07:40:48 +0100 Willy Tarreau w...@1wt.eu wrote: Hi, On Mon, Mar 03, 2014 at 09:12:27PM +0100, PiBa-NL wrote: Hi, Im not sure if this is the exact issue that Anup was having, and maybe i'm hijacking his thread, if so i'm sorry for that, but when try to check how it works i also having difficulties getting it to work as i expected it to. I'm using HAProxy v1.5dev21 on FreeBSD 8.3. Ive written in a frontend the following which checks for a GET web request to determine which backend to use, this works..: mode tcp tcp-request inspect-delay 5s acl PAYLOADcheck req.payload(0,3) -m bin 474554 use_backend web_80_tcp if PAYLOADcheck tcp-request content accept if PAYLOADcheck However when changing the match line to the following it fails: acl PAYLOADcheck req.payload(0,3) -m str GET or acl PAYLOADcheck req.payload(0,3) -m sub GET or acl PAYLOADcheck req.payload(0,3) -m reg -i GET The req.payload returns a piece of 'binary' data, but the 'compatibility matrix' seems to say that converting for use with sub/reg/others should not be an issue. Then the next step is of course to not match only the first 3 characters but some content further in the 'middle' of the data stream.. Am i missing something ? Or might there be an issue with the implementation? What you've done is absolutely correct. It is possible that there's a bug somewhere in the cast. I'm CCing Thierry who has a pending patch set of about 50 patches to rework ACLs (merge ACL+map and allow to update them on-the-fly) to ensure he checks this case. The match bin get the configuration string 474554 and convert it as the binary sequence GET. The match str get the configuration string GET and use it as is. The fetch req.payload() returns a binary content. When you try to match with str method, the binary content is converted as string. The converter produce string representing hexadecimal content: 474554. If you write acl PAYLOADcheck req.payload(0,3) -m str 474554 The system works perfectly. This behavior is not intuitive. Maybe it can be change later. Thierry
Re: inspecting incoming tcp content
On Tue, Mar 04, 2014 at 04:51:56PM +0100, Thierry FOURNIER wrote: The match bin get the configuration string 474554 and convert it as the binary sequence GET. The match str get the configuration string GET and use it as is. The fetch req.payload() returns a binary content. When you try to match with str method, the binary content is converted as string. The converter produce string representing hexadecimal content: 474554. If you write acl PAYLOADcheck req.payload(0,3) -m str 474554 The system works perfectly. This behavior is not intuitive. Maybe it can be change later. Indeed, thank you for diagnosing this. Originally we chose to cast bin to str as hex dump because it was only used in stick tables. But now that we support other storage and usages, it becomes less and less natural. I think we'll change this before the final release so that bin automatically casts to str as-is and we'll add a tohex converter for people who want to explicitly convert a bin to an hex string. Willy
RE: haproxy loosing connections
Hi, Hello, I'm maintain postgresql cluster with streaming replication for php-based webapp. And for a few days I'm trying to get rid of errors in my setup: Application serverDB server | PHP - pgbouncer - haproxy | - | postgresql | pgbouncer pools connections from php (session-based) and haproxy load-balance and failovering 3 backend postgresql servers. Every ~10 min haproxy drops connection and pgbouncer reports: application logs: failed to execute the SQL statement: SQLSTATE[08P01]: Unknown error: 7 ERROR: server conn crashed? syslog: Mar 4 22:16:12 app1 pgbouncer[15572]: C-0x1d0c130: mydb/pgsql@unix:5432 Pooler Error: server conn crashed? When I remove haproxy from this setup or change balancer to any other tcp-balancer: balancer-ng for example everything works fine! I tried almost everything I can imagine: - changing connection between php, pgbouncer and haproxy to tcp/ip or unix-socket - changing timeouts, conn lifetimes, keepalive, addition tcp options, pool modes - downgrading to older versions of pgbouncer and haproxy - reduced number of TW-sockets by changing connectivity of other components to unix-socket where possible Any ideas what to look for? software versions: php5 5.4.23 pgbouncer 1.5.4 haproxy 1.5dev22 Whats the kernel release? option splice-auto Try without splice. Enable logging and catch a failed request. Regards, Lukas
RE: HAProxy 1.5 possible bug
Hi Fred,
Is this a known bug in HAProxy 1.5?
When I use 0.0.0.0 or * as server address for a certain host, HAProxy
crashes with a oom_killer log.
Thats certainly not expected. Does the OOM conditional really come
from HAProxy?
server
...
Address “0.0.0.0″ or “*” has a special meaning.
It indicates that the connection will be forwarded to the same IP
address as the one from the client connection. This is useful in
transparent proxy architectures where the client’s connection is
intercepted and haproxy must forward to the original destination
address.
eg in the backend:
use-server www.speedtest.net if { hdr_sub(host) speedtest.net }
server www.speedtest.net *
so what should happen is that the alias 'www.speedtest.net' should be
equal to the same IP address as was transmitted.
Or am I doing something wrong?
Please post the full configuration and explain what you are trying to
do. Also, post the output of ./haproxy -vv.
Regards,
Lukas
Re: weights
Willy, Exactly right, but it is a common misunderstanding. Out of interest, How hard would it be to get a least connection scheduler to take account of cumulated connections? It would/might make it far more useful for HTTP.. Off the top of my head I think least conns in LVS is based on cummulative for 60 seconds (which again causes a lot of confusion) Just had a quick look here: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.ipvsadm.html and to calculate active conns for LC: active connections = ActConn * K + InActConn Where K is between 32 and 50? So probably way more confusing and yet most of our customers prefer the LeastConnection handling for HTTP in LVS rather than HAProxy I also slightly think that they just instinctively like the bigger numbers for connection count ;-). http://blog.loadbalancer.org/look-why-cant-you-just-tell-me-how-many-people-are-connected-to-the-load-balancer/ Just thinking the new keepalive functionality will probably effect this as well? On 4 March 2014 14:59, Willy Tarreau w...@1wt.eu wrote: On Tue, Mar 04, 2014 at 08:27:03PM +0530, vijeesh vijayan wrote: Thanks. please check my last reply Thanks. Am talking about the weights , if one server (x) assigned with weight 125 and other server (y) with weight 12 ( added twice in the file) , we see x is getting half of the traffic compared to y. that means weigt has no affects here? in this case , server x should be getting 5 folds of connections of y ideally. but something is preventing this . Am i right? in our case x is getting only 50 percent of y ( we are calculating the number of connections/sec) . how do we know how many connections haproxy keep it open for a particular server? No, unfortunately you definitely don't understand the difference between *concurrent* connections and *cumulated* connections. You're measuring the number of connections distributed over time. I'm talking about concurrent connections, which is what leastconn is about. Willy -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: inspecting incoming tcp content
Ok seems to work now knowing this. Though it hase some side affects. i could now match param=TEST using the following acl: acl PAYLOADcheck req.payload(0,0) -m reg -i 706172616d3D54455354 Case insensitive matching works 'perfectly', but for the hex code (see the D and d above), but doesnt match different cases of letters which one would probably expect. So even though i use -i, if i use the word TEST in lower case it doesn't match anymore. There might be a workaround for that with the ,lower option (i didnt confirm if that is applied before the hex conversion.) Also the current documentation gives several examples which indicate a different working: On systems where the regex library is much slower when using -i, it is possible to convert the sample to lowercase before matching, like this : acl script_tag payload(0,500),lower -m reg script This doesn't work for detecting the text script as its hex equivalent should be there, also if less than 500 bytes are send in the initial request it doesn't match at all. So seems like this part of the manual could use a little more clarification. (Praise though for the overall completeness/clarity of the manual!) Though if implementation now changes to match the manual, and possibly a additional tohex option that would be great. As its used on mode tcp certainly the option should exist to match binary/hex values that cannot be easily expressed with normal text. So the original design implementation does make sense, just not for 'textual' protocols. Thanks for investigating. PiBa-NL Willy Tarreau schreef op 4-3-2014 17:28: On Tue, Mar 04, 2014 at 04:51:56PM +0100, Thierry FOURNIER wrote: The match bin get the configuration string 474554 and convert it as the binary sequence GET. The match str get the configuration string GET and use it as is. The fetch req.payload() returns a binary content. When you try to match with str method, the binary content is converted as string. The converter produce string representing hexadecimal content: 474554. If you write acl PAYLOADcheck req.payload(0,3) -m str 474554 The system works perfectly. This behavior is not intuitive. Maybe it can be change later. Indeed, thank you for diagnosing this. Originally we chose to cast bin to str as hex dump because it was only used in stick tables. But now that we support other storage and usages, it becomes less and less natural. I think we'll change this before the final release so that bin automatically casts to str as-is and we'll add a tohex converter for people who want to explicitly convert a bin to an hex string. Willy
Re: rewrite URI help
The haproxy log contains the original request, not the rewritten one. If
you want to see the rewritten URL you need to look at the backend server
which is receiving the request.
-Patrick
*From: *Steve Phillips stw...@gmail.com
*Sent: * 2014-03-04 19:54:44 E
*To: *HAProxy haproxy@formilux.org
*Subject: *rewrite URI help
Trying to reverse proxy all requests to
/slideshare
to
www.slideshare.net/api/2/get_slideshow
http://www.slideshare.net/api/2/get_slideshow
my front-end config:
acl url_slideshare path_dir slideshare
use_backend slideshare if url_slideshare
and back-end:
backend slideshare
option http-server-close
option httpclose
reqrep ^([^\ ]*)\ /slideshare(.*) \1\ /api/2/get_slideshow\2
server slideshare www.slideshare.net:443
http://www.slideshare.net:443 ssl verify none
requests to /slideshow however, are not being rewritten:
173.11.67.214:60821 http://173.11.67.214:60821
[04/Mar/2014:19:49:03.257] main slideshare/slideshare
6142/0/289/121/6552 404 9299 - - 0/0/0/0/0 0/0 {} GET
/slideshare?slideshow_url=http%3A%2F%2Fwww.slideshare.net
http://2Fwww.slideshare.net%2FAaronKlein1%2Foptimizing-aws-economicsdetailed=1api_key=msCpLON8hash=a7fe5fd52cc86e4a4a3d1022cb7c63476b79e044ts=1393980574
HTTP/1.1
Is my regex incorrect? Am I missing something else?
Thanks.
Steve
Re: inspecting incoming tcp content
On Wed, Mar 05, 2014 at 12:55:47AM +0100, PiBa-NL wrote: Ok seems to work now knowing this. Though it hase some side affects. i could now match param=TEST using the following acl: acl PAYLOADcheck req.payload(0,0) -m reg -i 706172616d3D54455354 Case insensitive matching works 'perfectly', but for the hex code (see the D and d above), but doesnt match different cases of letters which one would probably expect. So even though i use -i, if i use the word TEST in lower case it doesn't match anymore. Indeed, you'd have to match it this way in order to match the input bytes, not the hex string : acl PAYLOADcheck req.payload(0,0) -m reg -i [57]0[46]1[57]2[46]1[46]d3D[57]4[46]5[57]3[57]4 There might be a workaround for that with the ,lower option (i didnt confirm if that is applied before the hex conversion.) Yes it would be much easier. The way the match is done is : 1) sample fetch function. Here, it is req.payload(). 2) converters. Here none, unless you add ,lower 3) cast to the input type of the ACL match (here, reg takes a string so it remains the same) 4) execution of the match function (here reg) for all patterns. Also the current documentation gives several examples which indicate a different working: On systems where the regex library is much slower when using -i, it is possible to convert the sample to lowercase before matching, like this : acl script_tag payload(0,500),lower -m reg script This doesn't work for detecting the text script as its hex equivalent should be there, also if less than 500 bytes are send in the initial request it doesn't match at all. You're absolutely right. We really need to change this confusing behaviour before the release. I'm sure we'll break one or two setups, but it we're still in the development phase until we release, and the fix will be trivial. So seems like this part of the manual could use a little more clarification. (Praise though for the overall completeness/clarity of the manual!) I tend to consider that the doc is the reference which people use to write their confs. So when something has never been working properly, I prefer to make the code work as documented than fix the doc. Though if implementation now changes to match the manual, and possibly a additional tohex option that would be great. Yes it will be necessary so that the very few users (if any) who rely on the current behaviour can fix their configs. As its used on mode tcp certainly the option should exist to match binary/hex values that cannot be easily expressed with normal text. So the original design implementation does make sense, just not for 'textual' protocols. I agree. Thanks, Willy
