HAproxy constant memory leak
Hi all,
We're deploying HAproxy and we're experiencing what apprears to be a memory
leak. After couple of days, HAproxy is consuming gigs of RAM.
Running a ps command returns:
# ps -u nobody u
USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND
nobody 29960 0.2 1.0 387220 343472 ? Rs Jan27 2:40
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy
Yesterday, I've been tracking the VSZ memory and haproxy leaks ~655 bytes
per 30 minutes. Today, we're at 387220 bytes.
14:00 128396
14:10 131624
14:20 134868
14:30 138100
What is totally strange is that we have the same version of HAproxy running
in another environment, and there is no leak there... However, there is
just one difference: In the environment that leaks, there is a firewall
pinging HAproxy every second (TCP open/close). I see that HAproxy is
reporting those ping as request errors (ereq). Here is a trace when running
haproxy with the -d flag:
:web.accept(0005)=0009 from [10.90.19.3:25611]
:web.clicls[0009:]
:web.closed[0009:]
0002:web.accept(0005)=000a from [10.90.19.2:52213]
0002:web.clicls[000a:]
0002:web.closed[000a:]
0004:web.accept(0005)=000a from [10.90.19.2:52357]
0004:web.clicls[000a:]
0004:web.closed[000a:]
0007:web.accept(0005)=000a from [10.90.19.2:52502]
0007:web.clicls[000a:]
0007:web.closed[000a:]
Anyway idea what could be wrong?
Thanks!
Our configuration is pretty simple:
global
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
maxconn 5
uid 99
gid 99
daemon
tune.ssl.default-dh-param 1024
stats socket :1935 level admin
stats timeout 2m
defaults
log global
modehttp
retries 3
timeout connect 5s
timeout client 60s
timeout server 120s
timeout queue 60s
timeout http-request 15s
timeout http-keep-alive 15s
balance roundrobin
option http-keep-alive
option forwardfor
option httpchk OPTIONS /
option httplog clf
option dontlognull
frontend web
bind *:80
bind *:443 ssl crt /etc/pki/tls/wildcardssl.pem
use_backend app_auth if { path_dir app-authnz }
use_backend app_stats if { path_dir app-stats }
use_backend app_search if { path_dir app-search }
use_backend app_settings if { path_dir app-settings }
backend app_auth
reqrep ^([^\ :]*)\ /[^/]*/(.*) \1\ /\2
server fe01 127.0.0.1:4 check
server fe02 10.2.127.144:4 check backup
server fe03 10.2.127.145:4 check backup
server fe04 10.2.127.146:4 check backup
Here is some additional details:
# haproxy -vv
HA-Proxy version 1.5.9 2014/11/25
Copyright 2000-2014 Willy Tarreau w...@1wt.eu
Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
--
# telnet localhost 1935
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
show info;show stat;show pools
Name: HAProxy
Version: 1.5.9
Release_date: 2014/11/25
Nbproc: 1
Process_num: 1
Pid: 29960
Uptime: 0d 0h12m25s
Uptime_sec: 745
Memmax_MB: 0
Ulimit-n: 100033
Maxsock: 100033
Maxconn: 5
Hard_maxconn: 5
CurrConns: 1
CumConns: 1630
CumReq: 1684
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 856
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 4
ConnRateLimit: 0
MaxConnRate: 9
SessRate: 4
SessRateLimit: 0
MaxSessRate: 9
SslRate: 2
SslRateLimit: 0
MaxSslRate: 4
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 1
SslFrontendSessionReuse_pct: 100
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 10
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 18
Run_queue: 1
Idle_pct: 100
node: toro63nsfe01.pf.spop.ca
description:
#
[PATCH] BUG/MINOR: parse: check the validity of size string in a more strict way
Hi Willy,
Attached is a patch for parse_size_err().
If a stick table is defined as below:
stick-table type ip size 50ka expire 300s
HAProxy will stop parsing size after passing through 50k and return
the value directly. But such format string of size should not be valid
in my opinion. So a further check is needed, that is this patch does.
With this patch, we will get the error message when start HAProxy with
the above configuration of stick table:
[ALERT] 027/175100 (22532) : parsing [h.cfg:53] : stick-table:
unexpected character 'a' in argument of 'size'.
If you think it is necessary to apply this patch, both 1.6 and 1.5 need it.
--
Best Regards,
Godbach
From 174943fb20fb3b45f186a6536b53151bdf00fee7 Mon Sep 17 00:00:00 2001
From: Godbach nylzhao...@gmail.com
Date: Wed, 28 Jan 2015 17:36:16 +0800
Subject: [PATCH] BUG/MINOR: parse: check the validity of size string in a
more strict way
If a stick table is defined as below:
stick-table type ip size 50ka expire 300s
HAProxy will stop parsing size after passing through 50k and return the value
directly. But such format string of size should not be valid. The patch checks
the next character to report error if any.
Signed-off-by: Godbach nylzhao...@gmail.com
---
src/standard.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/standard.c b/src/standard.c
index 93c44bb..f28825f 100644
--- a/src/standard.c
+++ b/src/standard.c
@@ -1656,6 +1656,9 @@ const char *parse_size_err(const char *text, unsigned
*ret) {
return text;
}
+ if (*text != '\0' *++text != '\0')
+ return text;
+
*ret = value;
return NULL;
}
--
1.7.11.7
Re: [PATCH] BUG/MINOR: parse: check the validity of size string in a more strict way
Hi Godbach, On Wed, Jan 28, 2015 at 05:57:13PM +0800, Godbach wrote: Hi Willy, Attached is a patch for parse_size_err(). If a stick table is defined as below: stick-table type ip size 50ka expire 300s HAProxy will stop parsing size after passing through 50k and return the value directly. But such format string of size should not be valid in my opinion. So a further check is needed, that is this patch does. Yes, good point. We have the same issue in many statements in the config parser, as well as with extra arguments that are silently ignored and that tend to confuse people. That's inline with what we want to change in the 1.6 parser. I've applied it to 1.6, do you want it into 1.5 as well ? Willy
Re: possible bug with CumReq info stat
BTW, the patch in the previous mail was based on the master branch of the haproxy-1.5 repo if that matters. Thanks, wt On Tue, Jan 27, 2015 at 3:04 PM, Warren Turkal w...@signalfuse.com wrote: The definition of the global.req_count at include/types/global.h line 109 is an unsigned int. The print code it treating it as a signed int. The attached commit fixes that. Also, is there an SSL protected location for fetching the haproxy git repo whose cert is signed by a widespread CA? The haproxy.org site also seems to be pretty slow for git cloning. wt -- Warren Turkal -- Warren Turkal
Soldes : dernière démarque, tout à moins 60 pourcent !
Derniegrave;re deacute;marque : tout agrave; -60 pourcent sur www.ikks.com [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] Visualisez la version en ligne. [http://link.openwebb.fr/Interspire/display.php?M=1046288114C=fd8dcd21c8b87c14fd9f061fb6634ad3S=1249L=83N=707] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22115F=T [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22116F=T validant, j'accepte de recevoir par e-mail les offres promotionnelles, avant-premiegrave;res et actualiteacute;s de IKKS. [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22114F=T] [http://link.openwebb.fr/Interspire/link.php?M=1046288114N=1249L=22117F=T IKKS valables sur les vecirc;tements de la collection Automne-Hiver 2014 signaleacute;s par une remise (Hors nouvelle collection Eteacute; 2015, parfums, cartes cadeaux, et articles non signaleacute;s par une remise). Offre non cumulable avec d'autres offres ou promotions en cours. Se deacute;sinscrire [http://link.openwebb.fr/Interspire/unsubscribe.php?M=1046288114C=fd8dcd21c8b87c14fd9f061fb6634ad3L=83N=1249]
Option no-sslv3 no being honoured with wildcard certs
Hi, I have a situation where the no-sslv3 is being ignored using version 1.5.10 on centos 6.6 and my test backend Java Rest api test servers are rejecting SSL handshakes with : DEBUG [2015-01-28 15:45:40,755] org.eclipse.jetty.server.HttpConnection: ! javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 not enabled or not supported Having had a google I suspect the X509 self-signed cert might be the issue as the backend hosts are: server web01 na.web.be.msm.internal:7445 check ssl verify none port 7445 maxconn 500 server web02 nb.web.be.msm.internal:7446 check ssl verify none port 7446 maxconn 500 server web03 nc.web.be.msm.internal:7447 check ssl verify none port 7447 maxconn 500 But the certificate CN on the host side is *.na.web.be.msm.internal, *.nb.web.be.msm.internal and *.nc.web.be.msm.internal for example. Could this be possible ? any ideas why we are still sending SSLv3 handshake requests? Regards, Paul - The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information, or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Although this e-mail and any attachments are believed to be free of any virus, or other defect which might affect any computer or system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Moneysupermarket.com Financial Group Limited for any loss or damage from receipt or use thereof. The views expressed are of the individual, and do not necessarily reflect the views of Moneysupermarket.com Financial Group Limited. Moneysupermarket.com Limited is an appointed representative of Moneysupermarket.com Financial Group Limited, which is authorised and regulated by the Financial Services Authority (FSA FRN 303190). Moneysupermarket.com Financial Group Limited, registered in England No. 3157344. Registered Office: Moneysupermarket House, St. David’s Park, Ewloe, CH5 3UZ. Telephone 01244 665700.
Re: [PATCH] BUG/MINOR: parse: check the validity of size string in a more strict way
Hi Willy, On 2015/1/28 18:28, Willy Tarreau wrote: Hi Godbach, On Wed, Jan 28, 2015 at 05:57:13PM +0800, Godbach wrote: Hi Willy, Attached is a patch for parse_size_err(). If a stick table is defined as below: stick-table type ip size 50ka expire 300s HAProxy will stop parsing size after passing through 50k and return the value directly. But such format string of size should not be valid in my opinion. So a further check is needed, that is this patch does. Yes, good point. We have the same issue in many statements in the config parser, as well as with extra arguments that are silently ignored and that tend to confuse people. That's inline with what we want to change in the 1.6 parser. I've applied it to 1.6, do you want it into 1.5 as well ? Willy Thanks. Since it's not an important issue and will not bring much side effect, just apply to 1.6 is OK. -- Best Regards, Godbach
Re: [PATCH] BUG/MINOR: parse: check the validity of size string in a more strict way
Hi Godbach, On Thu, Jan 29, 2015 at 10:29:49AM +0800, Godbach wrote: Since it's not an important issue and will not bring much side effect, just apply to 1.6 is OK. OK I prefer this as well, as anything related to config file format can uncover config bugs resulting in a non-starting setup. Thanks, Willy
Confirmation d'inscription
Title: IKKS Bienvenue et merci pour votre inscription et profitez de tout à -60 pourcent.Si ce message ne s'affiche pas correctement, visualisez la version en ligne. MERCI.Bonjour, Nous vous remercions pour votre inscription. Pour vous souhaiter la bienvenue, profitez de notre dernire dmarque : Soldes IKKS valables sur les vtements de la collection Automne-Hiver 2014 signals par une remise (Hors nouvelle collection Et 2015, parfums, cartes cadeaux, et articles non signals par une remise). Offre non cumulable avec d'autres offres ou promotions en cours.
