Hi all,
We're deploying HAproxy and we're experiencing what apprears to be a memory
leak. After couple of days, HAproxy is consuming gigs of RAM.
Running a ps command returns:
# ps -u nobody u
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nobody 29960 0.2 1.0 387220 343472 ? Rs Jan27 2:40
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy
Yesterday, I've been tracking the VSZ memory and haproxy leaks ~655 bytes
per 30 minutes. Today, we're at 387220 bytes.
14:00 128396
14:10 131624
14:20 134868
14:30 138100
What is totally strange is that we have the same version of HAproxy running
in another environment, and there is no leak there... However, there is
just one difference: In the environment that leaks, there is a firewall
pinging HAproxy every second (TCP open/close). I see that HAproxy is
reporting those ping as request errors (ereq). Here is a trace when running
haproxy with the -d flag:
00000000:web.accept(0005)=0009 from [10.90.19.3:25611]
00000000:web.clicls[0009:ffffffff]
00000000:web.closed[0009:ffffffff]
00000002:web.accept(0005)=000a from [10.90.19.2:52213]
00000002:web.clicls[000a:ffffffff]
00000002:web.closed[000a:ffffffff]
00000004:web.accept(0005)=000a from [10.90.19.2:52357]
00000004:web.clicls[000a:ffffffff]
00000004:web.closed[000a:ffffffff]
00000007:web.accept(0005)=000a from [10.90.19.2:52502]
00000007:web.clicls[000a:ffffffff]
00000007:web.closed[000a:ffffffff]
Anyway idea what could be wrong?
Thanks!
----
Our configuration is pretty simple:
global
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
maxconn 50000
uid 99
gid 99
daemon
tune.ssl.default-dh-param 1024
stats socket :1935 level admin
stats timeout 2m
defaults
log global
mode http
retries 3
timeout connect 5s
timeout client 60s
timeout server 120s
timeout queue 60s
timeout http-request 15s
timeout http-keep-alive 15s
balance roundrobin
option http-keep-alive
option forwardfor
option httpchk OPTIONS /
option httplog clf
option dontlognull
frontend web
bind *:80
bind *:443 ssl crt /etc/pki/tls/wildcardssl.pem
use_backend app_auth if { path_dir app-authnz }
use_backend app_stats if { path_dir app-stats }
use_backend app_search if { path_dir app-search }
use_backend app_settings if { path_dir app-settings }
backend app_auth
reqrep ^([^\ :]*)\ /[^/]*/(.*) \1\ /\2
server fe01 127.0.0.1:40000 check
server fe02 10.2.127.144:40000 check backup
server fe03 10.2.127.145:40000 check backup
server fe04 10.2.127.146:40000 check backup
Here is some additional details:
# haproxy -vv
HA-Proxy version 1.5.9 2014/11/25
Copyright 2000-2014 Willy Tarreau <[email protected]>
Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
------
# telnet localhost 1935
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
show info;show stat;show pools
Name: HAProxy
Version: 1.5.9
Release_date: 2014/11/25
Nbproc: 1
Process_num: 1
Pid: 29960
Uptime: 0d 0h12m25s
Uptime_sec: 745
Memmax_MB: 0
Ulimit-n: 100033
Maxsock: 100033
Maxconn: 50000
Hard_maxconn: 50000
CurrConns: 1
CumConns: 1630
CumReq: 1684
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 856
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 4
ConnRateLimit: 0
MaxConnRate: 9
SessRate: 4
SessRateLimit: 0
MaxSessRate: 9
SslRate: 2
SslRateLimit: 0
MaxSslRate: 4
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 1
SslFrontendSessionReuse_pct: 100
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 10
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 18
Run_queue: 1
Idle_pct: 100
node: toro63nsfe01.pf.spop.ca
description:
#
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
web,FRONTEND,,,0,16,2000,838,47616,183700,0,0,796,,,,,OPEN,,,,,,,,,1,2,0,,,,0,2,0,7,,,,0,60,0,799,0,0,,2,14,859,,,0,0,0,0,,,,,,,,
app_authnz,fe01,0,0,0,2,,4,3886,6145,,0,,0,0,0,0,no
check,1,1,0,,,,,,1,3,1,,4,,2,0,,1,,,,0,4,0,0,0,0,0,,,,0,0,,,,,44,,,0,1,58,64,
app_authnz,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,3,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_authnz,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,3,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_authnz,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,3,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_authnz,BACKEND,0,0,0,2,200,4,3886,6145,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,3,0,,4,,1,0,,1,,,,0,4,0,0,0,0,,,,,0,0,0,0,0,0,44,,,0,1,58,64,
app_search,fe01,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,1,0,,,,,,1,8,1,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_search,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,8,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_search,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,8,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_search,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,8,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_search,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
app_stats,fe01,0,0,0,1,,2,5883,594,,0,,0,0,0,0,no
check,1,1,0,,,,,,1,9,1,,2,,2,0,,1,,,,0,2,0,0,0,0,0,,,,0,0,,,,,45,,,0,1,1,2,
app_stats,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,9,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_stats,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,9,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_stats,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,9,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_stats,BACKEND,0,0,0,1,200,2,5883,594,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,9,0,,2,,1,0,,1,,,,0,2,0,0,0,0,,,,,0,0,0,0,0,0,45,,,0,1,1,2,
app_settings,fe01,0,0,0,2,,8,6620,3565,,0,,0,0,0,0,no
check,1,1,0,,,,,,1,14,1,,8,,2,0,,7,,,,0,7,0,1,0,0,0,,,,0,0,,,,,699,,,0,0,1,2,
app_settings,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,14,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_settings,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,14,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_settings,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no
check,1,0,1,,,,,,1,14,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
app_settings,BACKEND,0,0,0,2,200,8,6620,3565,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,14,0,,8,,1,0,,7,,,,0,7,0,1,0,0,,,,,0,0,0,0,0,0,699,,,0,0,1,2,
stats,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,15,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,795,0,,1,15,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Dumping pools usage. Use SIGQUIT to flush them.
- Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
- Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users
[SHARED]
- Pool channel (80 bytes) : 28 allocated (2240 bytes), 4 used, 1 users
[SHARED]
- Pool task (112 bytes) : 32 allocated (3584 bytes), 18 used, 1 users
[SHARED]
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users
[SHARED]
- Pool connection (320 bytes) : 29 allocated (9280 bytes), 4 used, 1
users [SHARED]
- Pool hdr_idx (416 bytes) : 14 allocated (5824 bytes), 1 used, 1 users
[SHARED]
- Pool session (864 bytes) : 16 allocated (13824 bytes), 2 used, 1 users
[SHARED]
- Pool requri (1024 bytes) : 2 allocated (2048 bytes), 0 used, 1 users
[SHARED]
- Pool buffer (16416 bytes) : 28 allocated (459648 bytes), 4 used, 1
users [SHARED]
Total: 10 pools, 496608 bytes allocated, 71584 used.
-- Georges-Etienne
