Hi All, Responding to myself. It seems that increasing vm.max_map_count to e.g. 262120 (instead of default 65530) and restart helps. (3.0.2 on Debian Squeeze running fine on default 65530).
WBR, Denis 2014-10-25 9:51 GMT+02:00 Denis Zhdanov <[email protected]>: > Hi All, > > We are trying to upgrade our varnish from pretty old 3.0.2 to 3.0.5 (3.0.6 > was just released, we'll try it too, but now we are working with 3.0.5) > We are using only RAM storage, and before upgrade it consume about 90G > after 8 hours and stay like that. But it seems there's some leak in 3.0.6 - > it doesn't stop in RAM consuming until its ran out. It's not fast process - > can took a week, but after that it starts swapping and we are forced to > restart it (and lost all cache). > I tried to compare varnishstat from "bad" and "good" node but didn't find > anything strange - at least SMA.ram.g_bytes is approximately same for both > nodes. > Could someone please take a look? > > Good varnishstat -1 > --------------- > epsvarnish_good: > client_conn 210291617 5.38 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 19264242207 493.21 Client requests received > cache_hit 17890026068 458.02 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 1373994639 35.18 Cache misses > backend_conn 13241605 0.34 Backend conn. success > backend_unhealthy 926862 0.02 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 47 0.00 Backend conn. failures > backend_reuse 1359807120 34.81 Backend conn. reuses > backend_toolate 13870 0.00 Backend conn. was closed > backend_recycle 1359883733 34.82 Backend conn. recycles > backend_retry 44845 0.00 Backend conn. retry > fetch_head 0 0.00 Fetch head > fetch_length 1373044190 35.15 Fetch with Length > fetch_chunked 950 0.00 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 0 0.00 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 100547 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 0 0.00 Fetch no body (204) > fetch_304 0 0.00 Fetch no body (304) > n_sess_mem 3722 . N struct sess_mem > n_sess 1870 . N struct sess > n_object 4007618 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 4009072 . N struct objectcore > n_objecthead 4023957 . N struct objecthead > n_waitinglist 6689775 . N struct waitinglist > n_vbc 20 . N struct vbc > n_wrk 1600 . N worker threads > n_wrk_create 1617 0.00 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 1703 0.00 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 3 . N backends > n_expired 25190657 . N expired objects > n_lru_nuked 1343746322 . N LRU nuked objects > n_lru_moved 16549646438 . N LRU moved objects > losthdr 1771 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 19157759781 490.48 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 210297216 5.38 Total Sessions > s_req 19264242207 493.21 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 0 0.00 Total pass > s_fetch 1372944593 35.15 Total fetch > s_hdrbytes 7493165988804 191841.77 Total header bytes > s_bodybytes 230757381423391 5907903.79 Total body bytes > sess_closed 7877791 0.20 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 19261784636 493.14 Session Linger > sess_herd 12270879099 314.16 Session herd > shm_records 796058319688 20380.87 SHM records > shm_writes 34949444335 894.78 SHM writes > shm_flushes 3 0.00 SHM flushes due to overflow > shm_cont 213814789 5.47 SHM MTX contention > shm_cycles 343350 0.01 SHM cycles through buffer > sms_nreq 1271021 0.03 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 578078958 . SMS bytes allocated > sms_bfree 578078958 . SMS bytes freed > backend_req 1373106504 35.15 Backend requests made > n_vcl 8 0.00 N vcl total > n_vcl_avail 8 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 19230468606 492.34 HCB Lookups without lock > hcb_lock 1371994951 35.13 HCB Lookups with lock > hcb_insert 1371991264 35.13 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 39059096 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 1 . Loaded VMODs > n_gzip 7 0.00 Gzip operations > n_gunzip 0 0.00 Gunzip operations > VBE.autojournaal(10.32.230.188,,8484).vcls 5 . VCL > references > VBE.autojournaal(10.32.230.188,,8484).happy18446744073709551615 . > Happy health probes > VBE.beslist(10.32.230.188,,8484).vcls 1 . VCL > references > VBE.beslist(10.32.230.188,,8484).happy 18446744073709551615 . > Happy health probes > LCK.sms.creat 1 0.00 > Created locks > LCK.sms.destroy 0 0.00 > Destroyed locks > LCK.sms.locks 3813063 0.10 Lock > Operations > LCK.sms.colls 0 0.00 > Collisions > LCK.smp.creat 0 0.00 > Created locks > LCK.smp.destroy 0 0.00 > Destroyed locks > LCK.smp.locks 0 0.00 Lock > Operations > LCK.smp.colls 0 0.00 > Collisions > LCK.sma.creat 2 0.00 > Created locks > LCK.sma.destroy 0 0.00 > Destroyed locks > LCK.sma.locks 6898140689 176.61 Lock > Operations > LCK.sma.colls 0 0.00 > Collisions > LCK.smf.creat 0 0.00 > Created locks > LCK.smf.destroy 0 0.00 > Destroyed locks > LCK.smf.locks 0 0.00 Lock > Operations > LCK.smf.colls 0 0.00 > Collisions > LCK.hsl.creat 0 0.00 > Created locks > LCK.hsl.destroy 0 0.00 > Destroyed locks > LCK.hsl.locks 0 0.00 Lock > Operations > LCK.hsl.colls 0 0.00 > Collisions > LCK.hcb.creat 1 0.00 > Created locks > LCK.hcb.destroy 0 0.00 > Destroyed locks > LCK.hcb.locks 2740196312 70.16 Lock > Operations > LCK.hcb.colls 0 0.00 > Collisions > LCK.hcl.creat 0 0.00 > Created locks > LCK.hcl.destroy 0 0.00 > Destroyed locks > LCK.hcl.locks 0 0.00 Lock > Operations > LCK.hcl.colls 0 0.00 > Collisions > LCK.vcl.creat 1 0.00 > Created locks > LCK.vcl.destroy 0 0.00 > Destroyed locks > LCK.vcl.locks 16514901 0.42 Lock > Operations > LCK.vcl.colls 0 0.00 > Collisions > LCK.stat.creat 1 0.00 > Created locks > LCK.stat.destroy 0 0.00 > Destroyed locks > LCK.stat.locks 3722 0.00 Lock > Operations > LCK.stat.colls 0 0.00 > Collisions > LCK.sessmem.creat 1 0.00 > Created locks > LCK.sessmem.destroy 0 0.00 > Destroyed locks > LCK.sessmem.locks 210376778 5.39 Lock > Operations > LCK.sessmem.colls 0 0.00 > Collisions > LCK.wstat.creat 1 0.00 > Created locks > LCK.wstat.destroy 0 0.00 > Destroyed locks > LCK.wstat.locks 154004420 3.94 Lock > Operations > LCK.wstat.colls 0 0.00 > Collisions > LCK.herder.creat 1 0.00 > Created locks > LCK.herder.destroy 0 0.00 > Destroyed locks > LCK.herder.locks 31 0.00 Lock > Operations > LCK.herder.colls 0 0.00 > Collisions > LCK.wq.creat 16 0.00 > Created locks > LCK.wq.destroy 0 0.00 > Destroyed locks > LCK.wq.locks 25196944400 645.10 Lock > Operations > LCK.wq.colls 0 0.00 > Collisions > LCK.objhdr.creat 1371962780 35.13 > Created locks > LCK.objhdr.destroy 1367969515 35.02 > Destroyed locks > LCK.objhdr.locks 79782749795 2042.62 Lock > Operations > LCK.objhdr.colls 0 0.00 > Collisions > LCK.exp.creat 1 0.00 > Created locks > LCK.exp.destroy 0 0.00 > Destroyed locks > LCK.exp.locks 2780936064 71.20 Lock > Operations > LCK.exp.colls 0 0.00 > Collisions > LCK.lru.creat 2 0.00 > Created locks > LCK.lru.destroy 0 0.00 > Destroyed locks > LCK.lru.locks 2716690775 69.55 Lock > Operations > LCK.lru.colls 0 0.00 > Collisions > LCK.cli.creat 1 0.00 > Created locks > LCK.cli.destroy 0 0.00 > Destroyed locks > LCK.cli.locks 2227237 0.06 Lock > Operations > LCK.cli.colls 0 0.00 > Collisions > LCK.ban.creat 1 0.00 > Created locks > LCK.ban.destroy 0 0.00 > Destroyed locks > LCK.ban.locks 2781138078 71.20 Lock > Operations > LCK.ban.colls 0 0.00 > Collisions > LCK.vbp.creat 1 0.00 > Created locks > LCK.vbp.destroy 0 0.00 > Destroyed locks > LCK.vbp.locks 9013627 0.23 Lock > Operations > LCK.vbp.colls 0 0.00 > Collisions > LCK.vbe.creat 1 0.00 > Created locks > LCK.vbe.destroy 0 0.00 > Destroyed locks > LCK.vbe.locks 26485466 0.68 Lock > Operations > LCK.vbe.colls 0 0.00 > Collisions > LCK.backend.creat 3 0.00 > Created locks > LCK.backend.destroy 0 0.00 > Destroyed locks > LCK.backend.locks 2773680373 71.01 Lock > Operations > LCK.backend.colls 0 0.00 > Collisions > SMA.ram.c_req 4109962205 105.22 > Allocator requests > SMA.ram.c_fail 52455807295423 1342985.70 > Allocator failures > SMA.ram.c_bytes 27203969692807 696482.32 > Bytes allocated > SMA.ram.c_freed 27128807778988 694558.00 > Bytes freed > SMA.ram.g_alloc 8016209 . > Allocations outstanding > SMA.ram.g_bytes 75161913819 . Bytes > outstanding > SMA.ram.g_space 13861 . Bytes > available > SMA.Transient.c_req 47750773 1.22 > Allocator requests > SMA.Transient.c_fail 0 0.00 > Allocator failures > SMA.Transient.c_bytes 58471006951 1496.99 Bytes > allocated > SMA.Transient.c_freed 58470979290 1496.99 Bytes > freed > SMA.Transient.g_alloc 22 . > Allocations outstanding > SMA.Transient.g_bytes 27661 . Bytes > outstanding > SMA.Transient.g_space 0 . Bytes > available > VBE.proxy(10.x.x.x,,80).vcls 8 . VCL references > VBE.proxy(10.x.x.x,,80).happy 18446744073709551615 . Happy > health probes > --------------- > Bad varnishstat -1 > --------------- > epsvarnish_bad: > client_conn 2661600 6.43 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 238438255 575.65 Client requests received > cache_hit 213261383 514.87 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 25173891 60.78 Cache misses > backend_conn 397844 0.96 Backend conn. success > backend_unhealthy 18139 0.04 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 2675 0.01 Backend conn. failures > backend_reuse 24825052 59.93 Backend conn. reuses > backend_toolate 45 0.00 Backend conn. was closed > backend_recycle 24825811 59.94 Backend conn. recycles > backend_retry 70506 0.17 Backend conn. retry > fetch_head 0 0.00 Fetch head > fetch_length 25137933 60.69 Fetch with Length > fetch_chunked 10 0.00 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 0 0.00 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 530 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 0 0.00 Fetch no body (204) > fetch_304 0 0.00 Fetch no body (304) > n_sess_mem 4732 . N struct sess_mem > n_sess 148 . N struct sess > n_object 4003646 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 4004774 . N struct objectcore > n_objecthead 4015924 . N struct objecthead > n_waitinglist 1588 . N struct waitinglist > n_vbc 22 . N struct vbc > n_wrk 1600 . N worker threads > n_wrk_create 3068 0.01 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 13255 0.03 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 3 . N backends > n_expired 263264 . N expired objects > n_lru_nuked 20870512 . N LRU nuked objects > n_lru_moved 200584889 . N LRU moved objects > losthdr 8 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 236130810 570.08 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 2661694 6.43 Total Sessions > s_req 238438255 575.65 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 0 0.00 Total pass > s_fetch 25137413 60.69 Total fetch > s_hdrbytes 117632270881 283995.98 Total header bytes > s_bodybytes 2917193607304 7042890.96 Total body bytes > sess_closed 169195 0.41 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 238353273 575.45 Session Linger > sess_herd 148971811 359.66 Session herd > shm_records 10788946827 26047.42 SHM records > shm_writes 446371290 1077.66 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 2328506 5.62 SHM MTX contention > shm_cycles 4750 0.01 SHM cycles through buffer > sms_nreq 39446 0.10 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 18106011 . SMS bytes allocated > sms_bfree 18106011 . SMS bytes freed > backend_req 25222980 60.90 Backend requests made > n_vcl 2 0.00 N vcl total > n_vcl_avail 2 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_gone 1 . N total gone bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 238293148 575.30 HCB Lookups without lock > hcb_lock 25165922 60.76 HCB Lookups with lock > hcb_insert 25165889 60.76 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 414204 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 1 . Loaded VMODs > n_gzip 0 0.00 Gzip operations > n_gunzip 0 0.00 Gunzip operations > sess_pipe_overflow 0 . Dropped sessions due to > session pipe overflow > LCK.sms.creat 1 0.00 Created locks > LCK.sms.destroy 0 0.00 Destroyed locks > LCK.sms.locks 118338 0.29 Lock Operations > LCK.sms.colls 0 0.00 Collisions > LCK.smp.creat 0 0.00 Created locks > LCK.smp.destroy 0 0.00 Destroyed locks > LCK.smp.locks 0 0.00 Lock Operations > LCK.smp.colls 0 0.00 Collisions > LCK.sma.creat 2 0.00 Created locks > LCK.sma.destroy 0 0.00 Destroyed locks > LCK.sma.locks 114658606 276.82 Lock Operations > LCK.sma.colls 0 0.00 Collisions > LCK.smf.creat 0 0.00 Created locks > LCK.smf.destroy 0 0.00 Destroyed locks > LCK.smf.locks 0 0.00 Lock Operations > LCK.smf.colls 0 0.00 Collisions > LCK.hsl.creat 0 0.00 Created locks > LCK.hsl.destroy 0 0.00 Destroyed locks > LCK.hsl.locks 0 0.00 Lock Operations > LCK.hsl.colls 0 0.00 Collisions > LCK.hcb.creat 1 0.00 Created locks > LCK.hcb.destroy 0 0.00 Destroyed locks > LCK.hcb.locks 46330459 111.85 Lock Operations > LCK.hcb.colls 0 0.00 Collisions > LCK.hcl.creat 0 0.00 Created locks > LCK.hcl.destroy 0 0.00 Destroyed locks > LCK.hcl.locks 0 0.00 Lock Operations > LCK.hcl.colls 0 0.00 Collisions > LCK.vcl.creat 1 0.00 Created locks > LCK.vcl.destroy 0 0.00 Destroyed locks > LCK.vcl.locks 175044 0.42 Lock Operations > LCK.vcl.colls 0 0.00 Collisions > LCK.stat.creat 1 0.00 Created locks > LCK.stat.destroy 0 0.00 Destroyed locks > LCK.stat.locks 2666278 6.44 Lock Operations > LCK.stat.colls 0 0.00 Collisions > LCK.sessmem.creat 1 0.00 Created locks > LCK.sessmem.destroy 0 0.00 Destroyed locks > LCK.sessmem.locks 2667059 6.44 Lock Operations > LCK.sessmem.colls 0 0.00 Collisions > LCK.wstat.creat 1 0.00 Created locks > LCK.wstat.destroy 0 0.00 Destroyed locks > LCK.wstat.locks 1316488 3.18 Lock Operations > LCK.wstat.colls 0 0.00 Collisions > LCK.herder.creat 1 0.00 Created locks > LCK.herder.destroy 0 0.00 Destroyed locks > LCK.herder.locks 11717 0.03 Lock Operations > LCK.herder.colls 0 0.00 Collisions > LCK.wq.creat 16 0.00 Created locks > LCK.wq.destroy 0 0.00 Destroyed locks > LCK.wq.locks 305048412 736.47 Lock Operations > LCK.wq.colls 0 0.00 Collisions > LCK.objhdr.creat 25167014 60.76 Created locks > LCK.objhdr.destroy 21151796 51.07 Destroyed locks > LCK.objhdr.locks 996006829 2404.63 Lock Operations > LCK.objhdr.colls 0 0.00 Collisions > LCK.exp.creat 1 0.00 Created locks > LCK.exp.destroy 0 0.00 Destroyed locks > LCK.exp.locks 46685330 112.71 Lock Operations > LCK.exp.colls 0 0.00 Collisions > LCK.lru.creat 2 0.00 Created locks > LCK.lru.destroy 0 0.00 Destroyed locks > LCK.lru.locks 46007928 111.08 Lock Operations > LCK.lru.colls 0 0.00 Collisions > LCK.cli.creat 1 0.00 Created locks > LCK.cli.destroy 0 0.00 Destroyed locks > LCK.cli.locks 88654 0.21 Lock Operations > LCK.cli.colls 0 0.00 Collisions > LCK.ban.creat 1 0.00 Created locks > LCK.ban.destroy 0 0.00 Destroyed locks > LCK.ban.locks 46686411 112.71 Lock Operations > LCK.ban.colls 0 0.00 Collisions > LCK.vbp.creat 1 0.00 Created locks > LCK.vbp.destroy 0 0.00 Destroyed locks > LCK.vbp.locks 125227 0.30 Lock Operations > LCK.vbp.colls 0 0.00 Collisions > LCK.vbe.creat 1 0.00 Created locks > LCK.vbe.destroy 0 0.00 Destroyed locks > LCK.vbe.locks 801030 1.93 Lock Operations > LCK.vbe.colls 0 0.00 Collisions > LCK.backend.creat 3 0.00 Created locks > LCK.backend.destroy 0 0.00 Destroyed locks > LCK.backend.locks 50871308 122.82 Lock Operations > LCK.backend.colls 0 0.00 Collisions > SMA.ram.c_req 71969321 173.75 Allocator requests > SMA.ram.c_fail 22016345 53.15 Allocator failures > SMA.ram.c_bytes 504148551835 1217150.37 Bytes allocated > SMA.ram.c_freed 428986641816 1035689.28 Bytes freed > SMA.ram.g_alloc 8008174 . Allocations outstanding > SMA.ram.g_bytes 75161910019 . Bytes outstanding > SMA.ram.g_space 17661 . Bytes available > SMA.Transient.c_req 368754 0.89 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 463851431 1119.86 Bytes allocated > SMA.Transient.c_freed 463818676 1119.78 Bytes freed > SMA.Transient.g_alloc 26 . Allocations outstanding > SMA.Transient.g_bytes 32755 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > VBE.proxy(10.x.x.x,,80).vcls 2 . VCL references > VBE.proxy(10.x.x.x,,80).happy18446744073709551615 . Happy > health probes > --------------- > > Command line > /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f > /etc/varnish/varnish.vcl -T :6082 -S /etc/varnish/secret -s ram=malloc,70G > -p thread_pool_add_delay=2 -p http_gzip_support=on -p thread_pools=16 -p > default_ttl=7200 -p thread_pool_min=100 -p thread_pool_max=4000 -p > session_linger=50 -p sess_workspace=1048576 -p shm_workspace=8192 > > Server is PowerEdge M610, 96G RAM, 2xXeon E5620+HT (16 cores). > Good system running on Debian Squeeze, Bad one - on Ubuntu Precise (yep, I > know, it was a bad idea to change varnish and OS in same time *sigh*) > > With best regards, > Denis >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
