haproxy-systemd-wrapper exit code problem

[Gabriele Cerami]

Hi,

We're having a problem with version 1.5.14 of haproxy, packaged for
CentOS 7, but it seems even the code in master is affected.

In situations where bind is not possible (in our case, the address was
already in use) tcp_connect_server returns with a status of 256
(ERR_ALERT). This value is then passed down as exit code for
haproxy-systemd-wrapper.

The problem is that exit value is truncated to the least significant 8
bits, so even if haproxy fails, systemd gets an exit code of 0 and
thinks the service start succeded.

Can you confirm this analysis ? (credits should go to Sofer Athlan-Guyot )
Any workarounds ?
Thanks.

Re: Getting 502 Gateway Timeout for BOSH over HAPRoxy

[Willy Tarreau]

Hi Vijaya,

On Mon, Oct 17, 2016 at 05:33:24AM +, Vijayalakshmi Devi A M wrote:
> Hi,
> 
> Our web application uses XMPP for chat. We are using ejabberd xmpp server .
> We have configured xmpp BOSH( http://xmpp.org/extensions/xep-0206.html ) via
> HAProxy. But sometimes haproxy is throwing error with Status Code:504 Gateway
> Time-out. Our application shows error as No 'Access-Control-Allow-Origin'
> header is present on the requested resource. Origin '. But the same request
> initially it works without any error. Can you please guide us on how to
> resolve this issue

It's not very clear, but at least from what I remember, BOSH uses long
connection polling, so if your haproxy timeout is smaller than the server's,
it's normal that unused connections ultimately time out and are logged as
504, and it should be irrelevant to the headers you're using.

Regards,
Willy

Re: HAProxy reloads lets old and outdated processes

[Willy Tarreau]

Hi Pierre,

On Fri, Oct 14, 2016 at 10:54:43AM +, Pierre Cheynier wrote:
> Hi Lukas,
> 
> > I did not meant no-reuseport to workaround or "solve" the problem 
> definitely, but rather to see if the problems can still be triggered, 
> since you can reproduce the problem easily.
> 
> This still happens using snapshot 20161005 with no-reuseport set, a bit less 
> probably because reload is faster.
> 
> Here is what I observe after reloading 50 times, waiting 0.1 sec between 
> each: 
> 
> $ ps fauxww | tail -9
> root 50253  0.1  0.0  46340  1820 ?Ss   10:43   0:00 
> /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p 
> /run/haproxy.pid
> haproxy  51003  0.0  0.0  78256  9144 ?S10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51000
> haproxy  51025  0.3  0.0  78256  9208 ?Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51000
> haproxy  51777  0.0  0.0  78256  9144 ?S10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51771
> haproxy  51834  0.3  0.0  78256  9208 ?Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51771
> haproxy  51800  0.0  0.0  78256  9140 ?S10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51785
> haproxy  51819  0.3  0.0  78256  9204 ?Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51785
> haproxy  52083  0.0  0.0  78256  9144 ?S10:47   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 52076
> haproxy  52084  0.3  0.0  78256  3308 ?Ss   10:47   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 52076
> 
> $ sudo ss -tanp |grep -i listen | grep 80
> LISTEN 0  128  *:80   *:* 
>   users:(("haproxy",pid=52084,fd=8))
> LISTEN 0  128  *:8080 *:* 
>   users:(("haproxy",pid=52084,fd=6))
> LISTEN 0  12810.5.6.7:8000 *:*
>users:(("haproxy",pid=52084,fd=7))
> 
> $ head -12 /etc/haproxy/haproxy.cfg
> global
>  log 127.0.0.1 local0 warning
>  log 127.0.0.1 local1 notice
>  maxconn 262144
>  user haproxy
>  group haproxy
>  nbproc 1
>  chroot /var/lib/haproxy
>  pidfile /var/run/haproxy.pid
>  stats socket /var/lib/haproxy/stats
>  noreuseport
> 
> Definitely, some instances seems to be "lost" (not referenced by another) and
> will never be stopped.

Is this with live traffic or on a test machine ? Could you please check
whether these instances have one connection attached ? I don't see any
valid reason for a dying process not to leave once it doesn't have any
more connection. And during my last attempts at fixing such issues by
carefully reviewing the code and hammering the systemd-wrapper like mad,
I couldn't get this behaviour to happen a single time. Thus it would be
nice to know what these processes are doing there and why they don't
stop.

Regards,
Willy

Queuing connections for a backend service?

[VINCE KAY]

Hi All,
I asked this question couple of weeks back didn't get any response.
Re-sending to bubble up this request to the top of the mailing list.
Does anyone out knows if this feature (see below original post on my
requirement) is available in committed open source release. Appreciate
your help.

I want to hold/queue incoming client connections to a database backend that
sits behind HAProxy dynamically during maintenance tasks and allow
connections back in once done. I searched through the mailing list archive
and found that a patch was provided for similar request in the past where
setting number of  connections for frontend/backend to dynamically
enable/disable connection queuing. Is this feature already committed to
latest official version of HAproxy?

If not can you please provide the patch for this support? Appreciate your
help.


Thanks,
Vincent

Re: HaProxy for SFTP load balancing

[Andrew Smalley]

Hello Malreddy,

Below is a working VIP I have created on our loadbalancer.org appliance
which will do what you want without the ACL and

With regard the ACL, You will not be able to do some of this in TCP Mode

https://www.haproxy.com/doc/aloha/7.0/haproxy/acls.html

listen stfp
bind 192.168.100.100:8022 transparent
mode tcp
balance leastconn
stick on src
stick-table type ip size 10240k expire 30m peers loadbalancer_replication
server backup 127.0.0.1:9081 backup  non-stick
option redispatch
option abortonclose
maxconn 4
server sftp-1 192.168.100.101:22  weight 100  check port 22 inter
4000  rise 2  fall 2  minconn 0  maxconn 0  on-marked-down
shutdown-sessions



Regards

Andrew Smalley

Loadbalancer.org Ltd.



On 18 October 2016 at 15:16,  wrote:

> Hi Andrew,
>
>
>
> We need High availability for SFTP
>
>
>
>
>
> Haproxy installed in server with ip (1.2.3.4)
>
>
>
> for this IP several clients are mapped as mentioned below
>
>
>
> client1.hh.com mapped to IP 1.2.3.4
>
> client2.hh.com mapped to IP 1.2.3.4
>
> client3.hh.com mapped to IP 1.2.3.4
>
>
>
>
>
> *For client1 the associated sftp servers are sftp1,sftp2,sftp3.*
>
>
>
>
>
> When request comes from *client1.hh.com *, then it
> should be serviced by any of the sftp servers associated to this client i.e
> sftp1 or sftp2 or sftp3.
>
>
>
>
>
> To achieve this, below is the haproxy.cfg
>
>
>
>
>
> listen sftp-server
> bind :2121
> mode tcp
> maxconn 2000
> option redis-check
> retries 3
> option redispatch
> #checking if the request is coming from client1
> *acl devclient1 ssl_fc_sni_reg -i devclient1.healthhub.net.in
> *
> #req.ssl_sni ssl_fc_sni_reg
> balance roundrobin
>
> *use_backend srvs_devclient1 if devclient1*
>
>
>
>
>
> *backend srvs_devclient1*
>
>
> * balance roundrobin*
> * server ftp01 172.31.10.247:22  check weight 2*
> * server ftp02 172.31.10.156:22  check weight 2*
>
>
>
> But when I try to transfer file i'm getting exception as connection closed
> by foreign client.
>
> I'm able to transfer file to sftp server directly(sftp1) which is up and
> running, but through haproxy it's not working.
>
>
>
> Kindly suggest how to fetch the servername from which the request is
> coming so that I can map that particular client to the associated sftp
> servers.
>
>
>
> More over if any of sftp server is down, haproxy should route the request
> to any of the associated sftp servers which are up.
>
>
>
> Eg: sftp1 is down, haproxy proxy should able to route the request to sftp2
> or sftp3 which are up and running.
>
>
>
> Requesting you to assist us to resolve the issue.
>
>
>
>
>
> Thanks in advance.
>
>
>
>
>
>
>
> -Original Message-
> From: "Andrew Smalley" 
> Sent: Thursday, October 6, 2016 7:43pm
> To: "vi...@abinnovative.com" 
> Cc: "Lukas Tribus" , "HAProxy" ,
> malreddy.t...@abinnovative.com
> Subject: Re: HaProxy for SFTP load balancing
>
> If you want a connect to port check you can use the below example
>
> listen sftp bind 192.168.100.100:8022 transparent mode http balance
> leastconn option forwardfor if-none stick on hdr(X-Forwarded-For,-1)
> stick on src stick-table type string len 64 size 10240k expire 30m peers
> loadbalancer_replication server backup 127.0.0.1:9081 backup non-stick
> option http-keep-alive option redispatch option abortonclose maxconn 4
> server RIP_ 192.168.100.0:80 weight RIP_Name check port 8022 inter 4000
> rise 2 fall 2 minconn 100 maxconn 0 on-marked-down shutdown-sessions
>
> Of if you wish to define an external file to use then something like below
> will work.
>
> listen sftp bind 192.168.100.100:8022 transparent mode http balance
> leastconn option forwardfor if-none stick on hdr(X-Forwarded-For,-1)
> stick on src stick-table type string len 64 size 10240k expire 30m peers
> loadbalancer_replication server backup 127.0.0.1:9081 backup non-stick
> option external-check external-check command /var/lib/loadbalancer.org/
> check/sftp_check.sh option http-keep-alive option redispatch option
> abortonclose maxconn 4 server RIP_ 192.168.100.0:80 weight RIP_Name
> check inter 4000 rise 2 fall 2 minconn 100 maxconn 0 on-marked-down
> shutdown-sessions
>
>
> I hope this helps?
>
>
> Regards
>
> Andrew Smalley
>
> Loadbalancer.org Ltd.
>
>
> On 6 October 2016 at 15:07, vi...@abinnovative.com  > wrote:
>
>> We gave, that doesn't works.
>>
>>
>> Vijay
>>
>>
>> On 06-10-2016 07:34 PM, Lukas Tribus wrote:
>>
>> Hi Vijay,
>>
>>
>> enable health-checks, by adding the "check" keyword to both your server
>> configuration lines.
>>
>>
>> Lukas
>>
>>
>>
>>
>>
>> --
>> *Thanks*
>> *Vijay*
>>
>

HaProxy for SFTP load balancing

[malreddy . tugu]

Hi Andrew,
 
We need High availability for SFTP
 
 
Haproxy installed in server with ip (1.2.3.4)
 
for this IP several clients are mapped as mentioned below
 
client1.hh.com mapped to IP 1.2.3.4
client2.hh.com mapped to IP 1.2.3.4
client3.hh.com mapped to IP 1.2.3.4
 
 
For client1 the associated sftp servers are sftp1,sftp2,sftp3.
 
 
When request comes from client1.hh.com, then it should be serviced by any of 
the sftp servers associated to this client i.e sftp1 or sftp2 or sftp3.
 
 
To achieve this, below is the haproxy.cfg
 
 
listen sftp-server
 bind :2121
 mode tcp
 maxconn 2000
 option redis-check
 retries 3
 option redispatch
 #checking if the request is coming from client1
 acl devclient1 ssl_fc_sni_reg -i devclient1.healthhub.net.in
 #req.ssl_sni ssl_fc_sni_reg
 balance roundrobin
 
 use_backend srvs_devclient1 if devclient1
 
 
backend srvs_devclient1

 balance roundrobin
 server ftp01 172.31.10.247:22 check weight 2
 server ftp02 172.31.10.156:22 check weight 2
 
But when I try to transfer file i'm getting exception as connection closed by 
foreign client.
I'm able to transfer file to sftp server directly(sftp1) which is up and 
running, but through haproxy it's not working.
 
Kindly suggest how to fetch the servername from which the request is coming so 
that I can map that particular client to the associated sftp servers.
 
More over if any of sftp server is down, haproxy should route the request to 
any of the associated sftp servers which are up.
 
Eg: sftp1 is down, haproxy proxy should able to route the request to sftp2 or 
sftp3 which are up and running.
 
Requesting you to assist us to resolve the issue.
 
 
Thanks in advance.
 
 
 
-Original Message-
From: "Andrew Smalley" 
Sent: Thursday, October 6, 2016 7:43pm
To: "vi...@abinnovative.com" 
Cc: "Lukas Tribus" , "HAProxy" , 
malreddy.t...@abinnovative.com
Subject: Re: HaProxy for SFTP load balancing




If you want a connect to port check you can use the below example


listen sftpbind 192.168.100.100:8022 transparentmode httpbalance 
leastconnoption forwardfor if-nonestick on hdr(X-Forwarded-For,-1)
stick on srcstick-table type string len 64 size 10240k expire 30m peers 
loadbalancer_replicationserver backup 127.0.0.1:9081 backup  non-stick
option http-keep-aliveoption redispatchoption abortonclosemaxconn 
4server RIP_ 192.168.100.0:80  weight RIP_Name  check port 8022 inter 
4000  rise 2  fall 2  minconn 100  maxconn 0  on-marked-down shutdown-sessions


Of if you wish to define an external file to use then something like below will 
work. 

listen sftpbind 192.168.100.100:8022 transparentmode httpbalance 
leastconnoption forwardfor if-nonestick on hdr(X-Forwarded-For,-1)
stick on srcstick-table type string len 64 size 10240k expire 30m peers 
loadbalancer_replicationserver backup 127.0.0.1:9081 backup  non-stick
option external-checkexternal-check command /var/lib/[ 
loadbalancer.org/check/sftp_check.sh ]( 
http://loadbalancer.org/check/sftp_check.sh )option http-keep-alive
option redispatchoption abortonclosemaxconn 4server RIP_ 
192.168.100.0:80  weight RIP_Name  check  inter 4000  rise 2  fall 2  minconn 
100  maxconn 0  on-marked-down shutdown-sessions



I hope this helps?









Regards

Andrew Smalley

Loadbalancer.org Ltd.


On 6 October 2016 at 15:07, [ vi...@abinnovative.com ]( 
mailto:vi...@abinnovative.com ) <[ vi...@abinnovative.com ]( 
mailto:vi...@abinnovative.com )> wrote:


We gave, that doesn't works.


 Vijay



 On 06-10-2016 07:34 PM, Lukas Tribus wrote:

Hi Vijay, 


 enable health-checks, by adding the "check" keyword to both your server 
configuration lines. 


 Lukas 






-- 

Thanks
Vijay

tcp-check send string GET %hostname%_STATE

[Sylvain LEBON]

Hello everybody,

I am trying to implement a custom tcp check on a redis database.

What I am sure that this is pretty simple.
It should be something like :
"tcp-check send GET ${hostname/servername/...}_STATE"

Is there a way to have a variable with the server name or hostname in 
the tcp-check query ?


Thank you for your time.

Regards.

Sylvain Lebon

RE: HAProxy reloads lets old and outdated processes

[Pierre Cheynier]

Hi,
Any updates/findings on that issue ?

Many thanks,

Pierre

> From : Pierre Cheynier
> To: Lukas Tribus; haproxy@formilux.org
> Sent: Friday, October 14, 2016 12:54 PM
> Subject: RE: HAProxy reloads lets old and outdated processes
>     
> Hi Lukas,
> 
> > I did not meant no-reuseport to workaround or "solve" the problem 
> definitely, but rather to see if the problems can still be triggered, 
> since you can reproduce the problem easily.
> 
> This still happens using snapshot 20161005 with no-reuseport set, a bit less 
> probably because reload is faster.
> 
> Here is what I observe after reloading 50 times, waiting 0.1 sec between 
> each: 
> 
> $ ps fauxww | tail -9
> root 50253  0.1  0.0  46340  1820 ?    Ss   10:43   0:00 
> /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p 
> /run/haproxy.pid
> haproxy  51003  0.0  0.0  78256  9144 ?    S    10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51000
> haproxy  51025  0.3  0.0  78256  9208 ?    Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51000
> haproxy  51777  0.0  0.0  78256  9144 ?    S    10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51771
> haproxy  51834  0.3  0.0  78256  9208 ?    Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51771
> haproxy  51800  0.0  0.0  78256  9140 ?    S    10:44   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51785
> haproxy  51819  0.3  0.0  78256  9204 ?    Ss   10:44   0:00  |   \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 51785
> haproxy  52083  0.0  0.0  78256  9144 ?    S    10:47   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 52076
> haproxy  52084  0.3  0.0  78256  3308 ?    Ss   10:47   0:00  \_ 
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 
> 52076
> 
> $ sudo ss -tanp |grep -i listen | grep 80
> LISTEN 0  128  *:80   *:* 
>   users:(("haproxy",pid=52084,fd=8))
> LISTEN 0  128  *:8080 *:* 
>   users:(("haproxy",pid=52084,fd=6))
> LISTEN 0  128    10.5.6.7:8000 *:*
>    users:(("haproxy",pid=52084,fd=7))
> 
> $ head -12 /etc/haproxy/haproxy.cfg
> global
>  log 127.0.0.1 local0 warning
>  log 127.0.0.1 local1 notice
>  maxconn 262144
>  user haproxy
>  group haproxy
>  nbproc 1
>  chroot /var/lib/haproxy
>  pidfile /var/run/haproxy.pid
>  stats socket /var/lib/haproxy/stats
>  noreuseport
> 
> Definitely, some instances seems to be "lost" (not referenced by another) and 
> will never be stopped.
> 
> In that case it will not impact the config consistency as only one is bound 
> to the socket, but the reload is far less transparent from a network point of 
> view.
> 
> Pierre

Latency spikes

[Dmitry Maslov]

Hello,

I'm experience latency spikes during the bussiness hours which affects all
services behind single HAProxy instance (both tcp and http).

Here's example lines from log:

Oct 18 14:29:34 localhost haproxy[173339]: 192.168.4.130:53568
 [18/Oct/2016:14:29:31.248] server_tcp
b_server_tcp/tcps0 654/633/3371 1049 -- 72186/700/700/162/0 0/0
Oct 18 14:31:56 localhost haproxy[173339]: 192.168.4.130:53630
 [18/Oct/2016:14:31:54.715] server_tcp
b_server_tcp/tcps3 352/444/2268 1049 -- 73099/701/701/128/0 0/0
Oct 18 14:31:59 localhost haproxy[173339]: 192.168.4.130:53632
 [18/Oct/2016:14:31:57.053] server_tcp
b_server_tcp/tcps4 337/456/2431 1049 -- 72798/702/701/143/0 0/0
Oct 18 14:32:01 localhost haproxy[173339]: 192.168.4.130:53634
 [18/Oct/2016:14:31:59.461] server_tcp
b_server_tcp/tcps0 291/314/2405 1049 -- 72294/701/701/163/0 0/0

If i do the same request directly to backend (they are on the same machine
with HAProxy), I'm getting response in 150-200 ms.

As you can see from logs, Tw and Tc is very high despite that backend and
server queues are empty. Spikes starts whenever actconn is going over 50k.

I'm using HAProxy 1.5.11, here is a part of my haproxy.conf (related to log
lines above):

frontend server_tcp
bind :80
bind :443
mode tcp
log global
option tcplog
maxconn 5000
timeout client 150s
option clitcpka
default_backend b_server_tcp

backend b_server_tcp
mode tcp
option tcplog
log global
option srvtcpka
timeout server 1m
balance roundrobin
server tcps0 192.168.2.200:9090 
maxconn 1000 send-proxy-v2
server tcps1 192.168.2.200:9091 
maxconn 1000 send-proxy-v2
server tcps2 192.168.2.200:9092 
maxconn 1000 send-proxy-v2
server tcps3 192.168.2.200:9093 
maxconn 1000 send-proxy-v2
server tcps4 192.168.2.200:9094 
maxconn 1000 send-proxy-v2

Any suggestion on how to investigate this problem?

Tanks a lot,
Dmitry.