haproxy stopped balancing after about 2 weeks

2019-05-10 Thread ericr
resending my entire message because I didn't get subscribed in time...

A couple of weeks ago I installed haproxy on our server running FreeBSD
11.0-RELEASE-p16. (yes, I know it's an old version of the OS, I'm going to
upgrade it as soon as I solve my haproxy problem.)

Haproxy is supposed to load balance between 2 web servers running apache.
haproxy ran fine and balanced well for about 2 weeks, and then it stopped
sending client connections to the second web server.

It still works fine for the first server.  Why it persists across reboots
is a mystery.

Once haproxy stopped balancing, it's never used the second server, even
after a restart/reboot.

It still does health checks to both servers just fine, and reports L7OK/200
at every check for both servers. I've tried using both roundrobin and
leastconn, with no luck.  I've restarted haproxy several times, and
rebooted the server it's running on, and it the behavior doesn't change.
I'm out of ideas, does anyone have suggestions for fixing this (or
improving my config in general)?

Here's my config file:


# global holds defaults, global variables, etc.
global
daemon
user haproxy
group haproxy
log /dev/log local0
stats socket /var/run/haproxy/admin.sock user haproxy group haproxy
mode 660 level admin

# https://www.haproxy.com/blog/multithreading-in-haproxy/
maxconn 2048 # max connections we handle at once
nbproc 1 # number of haproxy processes to start
nbthread 4 # max threads, 1 per CPU core

# cpu map = number of cpu cores
cpu-map all 0-3

ssl-default-bind-ciphers "EECDH+ECDSA+AESGCM ECDH+aRSA+AESGCM
EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256
EECDH+aRSA+RC4 EECDH EDH+aRSA RC4"
ssl-default-bind-options ssl-min-ver TLSv1.2

defaults
timeout connect 30s
timeout client 600s
timeout server 30s
log global
mode http

stats enable
stats uri /haproxy?stats
stats realm Statistics
stats auth REMOVED
stats refresh 10s

# frontend holds info about the public face of the site
frontend vi-gate2.docbasedirect.com
bind XXX.XX.XX.XXX:80
bind XXX.XX.XX.XXX:443 ssl crt
"/usr/local/etc/2019-www-prod-SSL.crt"
http-request redirect scheme https if !{ ssl_fc }
default_backend web_servers
option httplog

# info about backend servers
backend web_servers
balance leastconn
cookie phpsessid insert indirect nocache
option httpchk HEAD /

default-server check maxconn 2048

server vi-www3 10.3.3.10:8080 cookie phpsessid inter 120s
server vi-www4 10.3.3.11:8080 cookie phpsessid inter 120s

email-alert mailers vi-mailer
email-alert from REMOVED
email-alert to REMOVED

mailers vi-mailer
mailer localhost 127.0.0.1:25
mailer vi-backup2 10.3.3.100:25


Version info:

 haproxy -vv
HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/
Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-ignored-qualifiers
-Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits
-Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1
USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2o-freebsd  27 Mar 2018
Running on OpenSSL version : OpenSSL 1.0.2j-freebsd  26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with zlib version : 1.2.11
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTTP   side=FE
  h2 : mode=HTXside=FE|BE
: mode=HTXside=FE|BE
: mode=TCP|HTTP   side=FE|BE

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

Thanks!

-- ericr


haproxy stopped balancing after about 2 weeks

2019-05-09 Thread ericr
A couple of weeks ago I installed haproxy on our server running FreeBSD
11.0-RELEASE-p16. (yes, I know it's an old version of the OS, I'm going to
upgrade it as soon as I solve my haproxy problem.)

Haproxy is supposed to load balance between 2 web servers running apache.
haproxy ran fine and balanced well for about 2 weeks, and then it stopped
sending client connections to the second web server.

It still does health checks to both servers just fine, and reports L7OK/200
at every check for both servers. I've tried using both roundrobin and
leastconn, with no luck.  I've restarted haproxy several times, and
rebooted the server it's running on, and it the behavior doesn't change.
I'm out of ideas, does anyone have suggestions for fixing this (or
improving my config in general)?

Here's my config file:


# global holds defaults, global variables, etc.
global
daemon
user haproxy
group haproxy
log /dev/log local0
stats socket /var/run/haproxy/admin.sock user haproxy group haproxy
mode 660 level admin

# https://www.haproxy.com/blog/multithreading-in-haproxy/
maxconn 2048 # max connections we handle at once
nbproc 1 # number of haproxy processes to start
nbthread 4 # max threads, 1 per CPU core

# cpu map = number of cpu cores
cpu-map all 0-3

ssl-default-bind-ciphers "EECDH+ECDSA+AESGCM ECDH+aRSA+AESGCM
EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256
EECDH+aRSA+RC4 EECDH EDH+aRSA RC4"
ssl-default-bind-options ssl-min-ver TLSv1.2

defaults
timeout connect 30s
timeout client 600s
timeout server 30s
log global
mode http

stats enable
stats uri /haproxy?stats
stats realm Statistics
stats auth REMOVED
stats refresh 10s

# frontend holds info about the public face of the site
frontend vi-gate2.docbasedirect.com
bind XXX.XX.XX.XXX:80
bind XXX.XX.XX.XXX:443 ssl crt
"/usr/local/etc/2019-www-prod-SSL.crt"
http-request redirect scheme https if !{ ssl_fc }
default_backend web_servers
option httplog

# info about backend servers
backend web_servers
balance leastconn
cookie phpsessid insert indirect nocache
option httpchk HEAD /

default-server check maxconn 2048

server vi-www3 10.3.3.10:8080 cookie phpsessid inter 120s
server vi-www4 10.3.3.11:8080 cookie phpsessid inter 120s

email-alert mailers vi-mailer
email-alert from REMOVED
email-alert to REMOVED

mailers vi-mailer
mailer localhost 127.0.0.1:25
mailer vi-backup2 10.3.3.100:25


Thanks!

-- ericr