Hi all, We're deploying HAproxy and we're experiencing what apprears to be a memory leak. After couple of days, HAproxy is consuming gigs of RAM.
Running a ps command returns: # ps -u nobody u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nobody 29960 0.2 1.0 387220 343472 ? Rs Jan27 2:40 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy Yesterday, I've been tracking the VSZ memory and haproxy leaks ~655 bytes per 30 minutes. Today, we're at 387220 bytes. 14:00 128396 14:10 131624 14:20 134868 14:30 138100 What is totally strange is that we have the same version of HAproxy running in another environment, and there is no leak there... However, there is just one difference: In the environment that leaks, there is a firewall pinging HAproxy every second (TCP open/close). I see that HAproxy is reporting those ping as request errors (ereq). Here is a trace when running haproxy with the -d flag: 00000000:web.accept(0005)=0009 from [10.90.19.3:25611] 00000000:web.clicls[0009:ffffffff] 00000000:web.closed[0009:ffffffff] 00000002:web.accept(0005)=000a from [10.90.19.2:52213] 00000002:web.clicls[000a:ffffffff] 00000002:web.closed[000a:ffffffff] 00000004:web.accept(0005)=000a from [10.90.19.2:52357] 00000004:web.clicls[000a:ffffffff] 00000004:web.closed[000a:ffffffff] 00000007:web.accept(0005)=000a from [10.90.19.2:52502] 00000007:web.clicls[000a:ffffffff] 00000007:web.closed[000a:ffffffff] Anyway idea what could be wrong? Thanks! ---- Our configuration is pretty simple: global log 127.0.0.1 local1 info chroot /usr/share/haproxy maxconn 50000 uid 99 gid 99 daemon tune.ssl.default-dh-param 1024 stats socket :1935 level admin stats timeout 2m defaults log global mode http retries 3 timeout connect 5s timeout client 60s timeout server 120s timeout queue 60s timeout http-request 15s timeout http-keep-alive 15s balance roundrobin option http-keep-alive option forwardfor option httpchk OPTIONS / option httplog clf option dontlognull frontend web bind *:80 bind *:443 ssl crt /etc/pki/tls/wildcardssl.pem use_backend app_auth if { path_dir app-authnz } use_backend app_stats if { path_dir app-stats } use_backend app_search if { path_dir app-search } use_backend app_settings if { path_dir app-settings } backend app_auth reqrep ^([^\ :]*)\ /[^/]*/(.*) \1\ /\2 server fe01 127.0.0.1:40000 check server fe02 10.2.127.144:40000 check backup server fe03 10.2.127.145:40000 check backup server fe04 10.2.127.146:40000 check backup Here is some additional details: # haproxy -vv HA-Proxy version 1.5.9 2014/11/25 Copyright 2000-2014 Willy Tarreau <w...@1wt.eu> Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. ------ # telnet localhost 1935 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. show info;show stat;show pools Name: HAProxy Version: 1.5.9 Release_date: 2014/11/25 Nbproc: 1 Process_num: 1 Pid: 29960 Uptime: 0d 0h12m25s Uptime_sec: 745 Memmax_MB: 0 Ulimit-n: 100033 Maxsock: 100033 Maxconn: 50000 Hard_maxconn: 50000 CurrConns: 1 CumConns: 1630 CumReq: 1684 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 856 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 4 ConnRateLimit: 0 MaxConnRate: 9 SessRate: 4 SessRateLimit: 0 MaxSessRate: 9 SslRate: 2 SslRateLimit: 0 MaxSslRate: 4 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 1 SslFrontendSessionReuse_pct: 100 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 10 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 18 Run_queue: 1 Idle_pct: 100 node: toro63nsfe01.pf.spop.ca description: # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, web,FRONTEND,,,0,16,2000,838,47616,183700,0,0,796,,,,,OPEN,,,,,,,,,1,2,0,,,,0,2,0,7,,,,0,60,0,799,0,0,,2,14,859,,,0,0,0,0,,,,,,,, app_authnz,fe01,0,0,0,2,,4,3886,6145,,0,,0,0,0,0,no check,1,1,0,,,,,,1,3,1,,4,,2,0,,1,,,,0,4,0,0,0,0,0,,,,0,0,,,,,44,,,0,1,58,64, app_authnz,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,3,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_authnz,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,3,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_authnz,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,3,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_authnz,BACKEND,0,0,0,2,200,4,3886,6145,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,3,0,,4,,1,0,,1,,,,0,4,0,0,0,0,,,,,0,0,0,0,0,0,44,,,0,1,58,64, app_search,fe01,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,1,0,,,,,,1,8,1,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_search,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,8,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_search,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,8,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_search,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,8,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_search,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0, app_stats,fe01,0,0,0,1,,2,5883,594,,0,,0,0,0,0,no check,1,1,0,,,,,,1,9,1,,2,,2,0,,1,,,,0,2,0,0,0,0,0,,,,0,0,,,,,45,,,0,1,1,2, app_stats,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,9,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_stats,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,9,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_stats,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,9,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_stats,BACKEND,0,0,0,1,200,2,5883,594,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,9,0,,2,,1,0,,1,,,,0,2,0,0,0,0,,,,,0,0,0,0,0,0,45,,,0,1,1,2, app_settings,fe01,0,0,0,2,,8,6620,3565,,0,,0,0,0,0,no check,1,1,0,,,,,,1,14,1,,8,,2,0,,7,,,,0,7,0,1,0,0,0,,,,0,0,,,,,699,,,0,0,1,2, app_settings,fe02,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,14,2,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_settings,fe03,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,14,3,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_settings,fe04,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,0,1,,,,,,1,14,4,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0, app_settings,BACKEND,0,0,0,2,200,8,6620,3565,0,0,,0,0,0,0,UP,1,1,3,,0,795,0,,1,14,0,,8,,1,0,,7,,,,0,7,0,1,0,0,,,,,0,0,0,0,0,0,699,,,0,0,1,2, stats,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,15,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,, stats,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,795,0,,1,15,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0, Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool channel (80 bytes) : 28 allocated (2240 bytes), 4 used, 1 users [SHARED] - Pool task (112 bytes) : 32 allocated (3584 bytes), 18 used, 1 users [SHARED] - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (320 bytes) : 29 allocated (9280 bytes), 4 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 14 allocated (5824 bytes), 1 used, 1 users [SHARED] - Pool session (864 bytes) : 16 allocated (13824 bytes), 2 used, 1 users [SHARED] - Pool requri (1024 bytes) : 2 allocated (2048 bytes), 0 used, 1 users [SHARED] - Pool buffer (16416 bytes) : 28 allocated (459648 bytes), 4 used, 1 users [SHARED] Total: 10 pools, 496608 bytes allocated, 71584 used. -- Georges-Etienne