Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections

[Shishir Kumar Yadav]

Hi All,

I am using haproxy 1.8.3 as a front end for a webserver. haproxy listens on
port 80 while the webserver listen on port 8080. Our webserver can handle
large read/write requests (HTTP GET). I am testing a setup where a bunch of
clients are generating number of GET requests and when number of connection
reaches somewhere around 250, the clients intermittently get 503 Service
Unavailable error. I have not put any connection limit or timeout in my
haproxy.cfg. I am puzzled about what is going on here. The GET requests can
read upto 128 MB of data. If someone can points me to a possible reason
then that would be great. I have verified that haproxy is sending the 503
error and not the webserver. No HTTPS connections are involved.

One thing that I have noticed is that during the workload Recv-Q of the
Haproxy - Webserver connection and also Send-Q of haproxy - client
connection has large number of buffers (upto 5 MB) but not sure if it has
any role.

#

global
# paths setup
log /dev/loglocal0
log /dev/loglocal1 notice
chroot /var/lib/haproxy

# stats setup
# Can use this socket to check the stats of the first process.
stats socket /run/haproxy/admin.sock mode 660 level admin
stats bind-process 1
stats timeout 30s

# protocols setup
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

# mode setup
# master-worker mode, introduced in 1.8, is the only mode suitable
to our use case.
# For more details, see
https://www.haproxy.com/blog/whats-new-haproxy-1-8/.
master-worker

nbproc 6

# DH key setup
# Sets the maximum size of the Diffie-Hellman parameters used for
generating
# the ephemeral/temporary Diffie-Hellman key in case of DHE key
exchange.
tune.ssl.default-dh-param 2048

defaults
modehttp
log global
option  httplog # Enable logging of HTTP request, session state
and timers
option  dontlognull # Disable logging of null connections

frontend haproxy-frontend
bind :::80 v4v6
bind :::443 v4v6 ssl crt /ssd/net/ca/gui.pem
default_backend ir-http-server-backend

backend ir-http-server-backend
server server1 127.0.0.1:8080



Here is the Haproxy build that I am using -

##

HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Re: HA Proxy Source IP Issue

[Dave Cottlehuber]

On Mon, 17 Sep 2018, at 13:04, Damen Barker wrote:
> Hi There
>
> We are running 1.6, the issue we are facing is that my backend servers
> are seeing the incoming IP address of the HAProxy server and not the
> client IP address and our application needs to see this. Please see
> below our configuration and if you can offer any advice that would be
> greatly received.

Welcome Damen.

See 
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-option%20forwardfor

option forwardfor

and adjust your application accordingly. Sometimes x-real-ip is used or 
sometimes the application can support the PROXY protocol, you'll need to check 
whats possible -- https://www.haproxy.com/blog/haproxy/proxy-protocol/ was 
invented IIRC by Willy for haproxy, but it's really widespread now in other 
applications, as a generic non-HTTP-specific way of providing inbound IP 
address to proxied applications.

A+
Dave

Re: OpenSSL and per-context option problem

[Thierry Fournier]

—
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

> On 17 Sep 2018, at 12:45, Emmanuel Hocdet  wrote:
> 
> 
> Hi Thierry,
> 
>> Le 15 sept. 2018 à 18:06, Thierry Fournier > > a écrit :
>> 
>> Hi,
>> 
>> I tried to use per-context options, in order to enable HTTP2 for a short
>> list of SNI. I just add lines like this:
>> 
>>   /certif1.pem [alpn h2,http/1.1] my-h2-host.com 
>>   /certif2.pem my-other-host.com 
>> 
>> This configuration works fine on debian 8 with OpenSSL 1.0.2g, and doesn’t
>> work on Ubuntu 16.04 with OpenSSL 1.0.2l.
>> 
>> I compile the OpenSSL debian package 1.0.2g on Ubuntu, and the feature is
>> enabled.
>> 
>> My conclusion, is that some version of OpenSSL doesn’t support all 
>> per-context
>> options. 
>> 
>> Do you have an opinion ?
>> 
> 
> Are you sure it's not the opposite: doesn't work with 1.0.2g?
> 
> "Major changes between OpenSSL 1.0.2g and OpenSSL 1.0.2h [3 May 2016]
> 
> Modify behavior of ALPN to invoke callback after SNI/servername callback, 
> such that updates to the SSL_CTX affect ALPN. »
> 
Sorry, I mix versions. The following is the reality:

OpenSSL 1.0.2l  25 May 2017 => works
OpenSSL 1.0.2g  1 Mar 2016  => doesn’t work.

The change explain the observed behavior !

br,
Thierry

Re: OpenSSL and per-context option problem

[Emmanuel Hocdet]

Hi Thierry,

> Le 15 sept. 2018 à 18:06, Thierry Fournier  a écrit 
> :
> 
> Hi,
> 
> I tried to use per-context options, in order to enable HTTP2 for a short
> list of SNI. I just add lines like this:
> 
>   /certif1.pem [alpn h2,http/1.1] my-h2-host.com
>   /certif2.pem my-other-host.com
> 
> This configuration works fine on debian 8 with OpenSSL 1.0.2g, and doesn’t
> work on Ubuntu 16.04 with OpenSSL 1.0.2l.
> 
> I compile the OpenSSL debian package 1.0.2g on Ubuntu, and the feature is
> enabled.
> 
> My conclusion, is that some version of OpenSSL doesn’t support all per-context
> options. 
> 
> Do you have an opinion ?
> 

Are you sure it's not the opposite: doesn't work with 1.0.2g?

"Major changes between OpenSSL 1.0.2g and OpenSSL 1.0.2h [3 May 2016]

Modify behavior of ALPN to invoke callback after SNI/servername callback, such 
that updates to the SSL_CTX affect ALPN. »

++
Manu