RE: Strange memory usage
Hi Dmitry, show pools after few days of uptime: Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 961 allocated (30752 bytes), 5 used, 3 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool channel (80 bytes) : 4136 allocated (330880 bytes), 648 used, 1 users [SHARED] - Pool task (112 bytes) : 3109 allocated (348208 bytes), 1367 used, 1 users [SHARED] - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (320 bytes) : 2537 allocated (811840 bytes), 343 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 2068 allocated (860288 bytes), 323 used, 1 users [SHARED] - Pool session (864 bytes) : 2068 allocated (1786752 bytes), 326 used, 1 users [SHARED] - Pool requri (1024 bytes) : 1491 allocated (1526784 bytes), 16 used, 1 users [SHARED] - Pool buffer (32800 bytes) : 4136 allocated (135660800 bytes), 648 used, 1 users [SHARED] Total: 10 pools, 141356304 bytes allocated, 22001680 used. Ok, this is after 4 days of uptime, correct? I would monitor this for another 5 - 10 days without reloading or SIGQUIT'ing and check the results then, to understand if the buffer allocations increase linearly with the uptime. Regards, Lukas
Re: Strange memory usage
Hi Dmitry, On Tue, Oct 14, 2014 at 11:35:40AM +0200, Lukas Tribus wrote: Hi Dmitry, show pools after few days of uptime: Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 961 allocated (30752 bytes), 5 used, 3 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool channel (80 bytes) : 4136 allocated (330880 bytes), 648 used, 1 users [SHARED] - Pool task (112 bytes) : 3109 allocated (348208 bytes), 1367 used, 1 users [SHARED] - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (320 bytes) : 2537 allocated (811840 bytes), 343 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 2068 allocated (860288 bytes), 323 used, 1 users [SHARED] - Pool session (864 bytes) : 2068 allocated (1786752 bytes), 326 used, 1 users [SHARED] - Pool requri (1024 bytes) : 1491 allocated (1526784 bytes), 16 used, 1 users [SHARED] - Pool buffer (32800 bytes) : 4136 allocated (135660800 bytes), 648 used, 1 users [SHARED] Total: 10 pools, 141356304 bytes allocated, 22001680 used. Ok, this is after 4 days of uptime, correct? I would monitor this for another 5 - 10 days without reloading or SIGQUIT'ing and check the results then, to understand if the buffer allocations increase linearly with the uptime. It simply looks to me that you've reached at least 2068 concurrent connections at least once (2 buffers per connection), which explains the 135 MB of buffers allocated with your 32kB buffers. Unfortunately I'm realizing that we don't have in the stats the maximum total number of connections reached, we only have it per frontend. But the fact that you reached about 4k sessions/s tends to validate this observation. Best regards, Willy
RE: Strange memory usage
Hi Dmitry, I am using haproxy-1.5.4 on FreeBSD-10. Upon startup, it looks like this: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 8459 www 1 37 0 86376K 28824K CPU16 16 0:16 26.56% haproxy (about 80MB RES) Its 80MB SIZE and 28M RES here. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 82720 www 1 36 0 244M 108M CPU29 29 29.2H 26.95% haproxy (244MB RES). Its 244M SIZE and 108M RES. So 108M of real RAM used here. When I do reload, I see that old process is in swread state for some time, and swap usage decreases for about 150MB when old process finishes. Does it mean memory leak is somewhere? Any additional information I could provide will be useful? Share you configuration, especially maxconn related stuff, the output of haproxy -vv and possibly show info;show stat;show pools from the unix admin socket. I don't think that its a memory leak, but haproxy its just allocating what it can according to its maxconn values and doesn't necessarily free all those buffers when there aren't used anymore. Regards, Lukas
Re: Strange memory usage
On 13 окт. 2014 г., at 14:37, Lukas Tribus luky...@hotmail.com wrote: Hi Dmitry, I am using haproxy-1.5.4 on FreeBSD-10. Upon startup, it looks like this: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 8459 www 1 37 0 86376K 28824K CPU16 16 0:16 26.56% haproxy (about 80MB RES) Its 80MB SIZE and 28M RES here. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 82720 www 1 36 0 244M 108M CPU29 29 29.2H 26.95% haproxy (244MB RES). Its 244M SIZE and 108M RES. So 108M of real RAM used here. Yes, I am sorry, I meant SIZE. When I do reload, I see that old process is in swread state for some time, and swap usage decreases for about 150MB when old process finishes. Does it mean memory leak is somewhere? Any additional information I could provide will be useful? Share you configuration, especially maxconn related stuff, the output of defaults log global modetcp balance roundrobin maxconn 1 option abortonclose option allbackups #option dontlog-normal #option dontlognull option redispatch option tcplog #option log-separate-errors option socket-stats retries 4 timeout check 500ms timeout client 15s timeout connect 100ms timeout http-keep-alive 3s timeout http-request 5s timeout queue 1s timeout server 15s fullconn 3000 default-server inter 5s downinter 1s fastinter 500ms fall 3 rise 1 slowstart 60s maxqueue 1 minconn 5 maxconn 150 I can send you full config in private e-mail if necessary. haproxy -vv HA-Proxy version 1.5.4 2014/09/02 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -pipe -g -DFREEBSD_PORTS OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PC1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1i-freebsd 6 Aug 2014 Running on OpenSSL version : OpenSSL 1.0.1i-freebsd 6 Aug 2014 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.33 2013-05-28 PCRE library supports JIT : yes Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. a nd possibly show info;show stat;show pools from the unix admin socket. show info: Name: HAProxy Version: 1.5.4 Release_date: 2014/09/02 Nbproc: 1 Process_num: 1 Pid: 32459 Uptime: 4d 6h09m46s Uptime_sec: 367786 Memmax_MB: 0 Ulimit-n: 131218 Maxsock: 131218 Maxconn: 65500 Hard_maxconn: 65500 CurrConns: 508 CumConns: 517986272 CumReq: 602369265 MaxSslConns: 0 CurrSslConns: 16 CumSslConns: 452700 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 2611 ConnRateLimit: 0 MaxConnRate: 3965 SessRate: 2611 SessRateLimit: 0 MaxSessRate: 3965 SslRate: 4 SslRateLimit: 0 MaxSslRate: 33 SslFrontendKeyRate: 2 SslFrontendMaxKeyRate: 34 SslFrontendSessionReuse_pct: 50 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 74867 SslCacheMisses: 60826 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 1550 Run_queue: 1 Idle_pct: 55 show pools on freshly started process: Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 19 allocated (608 bytes), 5 used, 3 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool channel (80 bytes) : 766 allocated (61280 bytes), 672 used, 1 users [SHARED] - Pool task (112 bytes) : 1426 allocated (159712 bytes), 1378 used, 1 users [SHARED] - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool connection (320 bytes) : 424 allocated (135680 bytes), 360 used, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 383 allocated (159328 bytes), 335 used, 1 users [SHARED] - Pool session (864 bytes) : 385 allocated (332640 bytes), 337 used, 1 users [SHARED] - Pool requri (1024 bytes) : 51 allocated (52224 bytes), 22 used, 1 users [SHARED] - Pool buffer (32800 bytes) : 766 allocated (25124800 bytes), 672 used, 1 users [SHARED] Total: 10 pools, 26026272 bytes allocated, 22818112 used. show pools after few days of uptime: Dumping pools usage. Use SIGQUIT to flush them. - Pool pipe (32 bytes) : 961 allocated (30752 bytes), 5 used, 3 users [SHARED] - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] - Pool channel (80 bytes) : 4136 allocated (330880 bytes), 648 used, 1 users [SHARED] - Pool task (112 bytes) : 3109 allocated (348208 bytes), 1367
Strange memory usage
Hello! I am using haproxy-1.5.4 on FreeBSD-10. Upon startup, it looks like this: PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 8459 www 1 370 86376K 28824K CPU16 16 0:16 26.56% haproxy (about 80MB RES) After few days of running, it looks like this: PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 82720 www 1 360 244M 108M CPU29 29 29.2H 26.95% haproxy (244MB RES). When I do reload, I see that old process is in swread state for some time, and swap usage decreases for about 150MB when old process finishes. Does it mean memory leak is somewhere? Any additional information I could provide will be useful? Thanks!
