On 13 окт. 2014 г., at 14:37, Lukas Tribus <[email protected]> wrote:
> Hi Dmitry,
>
>
>
>> I am using haproxy-1.5.4 on FreeBSD-10.
>>
>> Upon startup, it looks like this:
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>> 8459 www 1 37 0 86376K 28824K CPU16 16 0:16 26.56% haproxy
>>
>> (about 80MB RES)
>
> Its 80MB SIZE and 28M RES here.
>
>
>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>> 82720 www 1 36 0 244M 108M CPU29 29 29.2H 26.95% haproxy
>>
>> (244MB RES).
>
> Its 244M SIZE and 108M RES. So 108M of real RAM used here.
>
>
Yes, I am sorry, I meant SIZE.
>
>> When I do reload, I see that old process is in swread state for some time,
>> and
>> swap usage decreases for about 150MB when old process finishes.
>>
>> Does it mean memory leak is somewhere? Any additional information I could
>> provide will be useful?
>
> Share you configuration, especially maxconn related stuff, the output of
defaults
log global
mode tcp
balance roundrobin
maxconn 10000
option abortonclose
option allbackups
# option dontlog-normal
# option dontlognull
option redispatch
option tcplog
# option log-separate-errors
option socket-stats
retries 4
timeout check 500ms
timeout client 15s
timeout connect 100ms
timeout http-keep-alive 3s
timeout http-request 5s
timeout queue 1s
timeout server 15s
fullconn 3000
default-server inter 5s downinter 1s fastinter 500ms fall 3 rise 1 slowstart
60s maxqueue 1 minconn 5 maxconn 150
I can send you full config in private e-mail if necessary.
> "haproxy -vv"
HA-Proxy version 1.5.4 2014/09/02
Copyright 2000-2014 Willy Tarreau <[email protected]>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -pipe -pipe -g -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PC1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1i-freebsd 6 Aug 2014
Running on OpenSSL version : OpenSSL 1.0.1i-freebsd 6 Aug 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.33 2013-05-28
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
> a
> nd possibly "show info;show stat;show pools"
> from the unix admin socket.
>
show info:
Name: HAProxy
Version: 1.5.4
Release_date: 2014/09/02
Nbproc: 1
Process_num: 1
Pid: 32459
Uptime: 4d 6h09m46s
Uptime_sec: 367786
Memmax_MB: 0
Ulimit-n: 131218
Maxsock: 131218
Maxconn: 65500
Hard_maxconn: 65500
CurrConns: 508
CumConns: 517986272
CumReq: 602369265
MaxSslConns: 0
CurrSslConns: 16
CumSslConns: 452700
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 2611
ConnRateLimit: 0
MaxConnRate: 3965
SessRate: 2611
SessRateLimit: 0
MaxSessRate: 3965
SslRate: 4
SslRateLimit: 0
MaxSslRate: 33
SslFrontendKeyRate: 2
SslFrontendMaxKeyRate: 34
SslFrontendSessionReuse_pct: 50
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 74867
SslCacheMisses: 60826
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 1550
Run_queue: 1
Idle_pct: 55
show pools on freshly started process:
Dumping pools usage. Use SIGQUIT to flush them.
- Pool pipe (32 bytes) : 19 allocated (608 bytes), 5 used, 3 users [SHARED]
- Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool channel (80 bytes) : 766 allocated (61280 bytes), 672 used, 1 users
[SHARED]
- Pool task (112 bytes) : 1426 allocated (159712 bytes), 1378 used, 1 users
[SHARED]
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool connection (320 bytes) : 424 allocated (135680 bytes), 360 used, 1
users [SHARED]
- Pool hdr_idx (416 bytes) : 383 allocated (159328 bytes), 335 used, 1 users
[SHARED]
- Pool session (864 bytes) : 385 allocated (332640 bytes), 337 used, 1 users
[SHARED]
- Pool requri (1024 bytes) : 51 allocated (52224 bytes), 22 used, 1 users
[SHARED]
- Pool buffer (32800 bytes) : 766 allocated (25124800 bytes), 672 used, 1
users [SHARED]
Total: 10 pools, 26026272 bytes allocated, 22818112 used.
show pools after few days of uptime:
Dumping pools usage. Use SIGQUIT to flush them.
- Pool pipe (32 bytes) : 961 allocated (30752 bytes), 5 used, 3 users [SHARED]
- Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool channel (80 bytes) : 4136 allocated (330880 bytes), 648 used, 1 users
[SHARED]
- Pool task (112 bytes) : 3109 allocated (348208 bytes), 1367 used, 1 users
[SHARED]
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool connection (320 bytes) : 2537 allocated (811840 bytes), 343 used, 1
users [SHARED]
- Pool hdr_idx (416 bytes) : 2068 allocated (860288 bytes), 323 used, 1 users
[SHARED]
- Pool session (864 bytes) : 2068 allocated (1786752 bytes), 326 used, 1
users [SHARED]
- Pool requri (1024 bytes) : 1491 allocated (1526784 bytes), 16 used, 1 users
[SHARED]
- Pool buffer (32800 bytes) : 4136 allocated (135660800 bytes), 648 used, 1
users [SHARED]
Total: 10 pools, 141356304 bytes allocated, 22001680 used.
(compare the last line)
I can send you "show stat" in private if necessary.
> I don't think that its a memory leak, but haproxy its just allocating
> what it can according to its maxconn values and doesn't necessarily
> free all those buffers when there aren't used anymore.
>
Problem is that during reload I often get a bunch of similar errors in the log:
Oct 8 18:23:42 hostname haproxy[992]: IP:42908 [08/Oct/2014:18:23:38.947]
ONE-front ONE-back/backend 93/176/-1/-1/3067 503 212 - - CQ-- 2055/759/480/20/0
0/163 "POST /some/url HTTP/1.1"
Oct 8 18:23:44 hostname haproxy[992]: IP:23920 [08/Oct/2014:18:23:39.041]
ONE-front ONE-back/<NOSRV> 75/841/-1/-1/4997 503 212 - - CQ-- 2560/1023/589/0/0
0/203 "POST /some/url HTTP/1.1"
There are a lot of similar errors and only during reload.
