Hi together, I have an old varnish 3.0.5 running. (I know its outdated, but my issue is more understanding what happens here then fixing the problem on the actual system). While running, the systems takes up more and more memory.
I already switched to file storage, but still, after one day running and serving about 2 million requests, the varnish process takes more 20GB of ram. The first thing I am confused is, that as far as I see it varnishstats does not show the memory consumed (I will attach an output at the bottom). Tools like top and pmap show the output. pmap -x ... total kB 25368872 16712056 15197372 Beside some VCL-Code, the varnish uses the vmods redis, digest, and std. Any idea how the find out what is causing the memory leak? Thanks in advance, Tobias Here is my varnishstat -1 output client_conn 2115657 18.63 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 5176344 45.58 Client requests received cache_hit 2207169 19.43 Cache hits cache_hitpass 18635 0.16 Cache hits for pass cache_miss 508733 4.48 Cache misses backend_conn 590509 5.20 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 2391162 21.05 Backend conn. reuses backend_toolate 581 0.01 Backend conn. was closed backend_recycle 2392558 21.07 Backend conn. recycles backend_retry 142 0.00 Backend conn. retry fetch_head 604 0.01 Fetch head fetch_length 1493458 13.15 Fetch with Length fetch_chunked 1277164 11.25 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 9969 0.09 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 528 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 186330 1.64 Fetch no body (204) fetch_304 14773 0.13 Fetch no body (304) n_sess_mem 307 . N struct sess_mem n_sess 59 . N struct sess n_object 59401 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 59439 . N struct objectcore n_objecthead 60591 . N struct objecthead n_waitinglist 44 . N struct waitinglist n_vbc 2 . N struct vbc n_wrk 44 . N worker threads n_wrk_create 1296 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 7886 0.07 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 2 . N backends n_expired 449200 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 1917928 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 4825557 42.49 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 2115659 18.63 Total Sessions s_req 5176344 45.58 Total Requests s_pipe 1304 0.01 Total pipe s_pass 2473728 21.78 Total pass s_fetch 2981770 26.25 Total fetch s_hdrbytes 2065942876 18190.28 Total header bytes s_bodybytes 133824929675 1178306.04 Total body bytes sess_closed 418834 3.69 Session Closed sess_pipeline 63774 0.56 Session Pipeline sess_readahead 7618 0.07 Session Read Ahead sess_linger 4929655 43.40 Session Linger sess_herd 4533770 39.92 Session herd shm_records 397316593 3498.31 SHM records shm_writes 22134819 194.89 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 83004 0.73 SHM MTX contention shm_cycles 173 0.00 SHM cycles through buffer sms_nreq 693 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 472858 . SMS bytes allocated sms_bfree 472858 . SMS bytes freed backend_req 2982527 26.26 Backend requests made n_vcl 2 0.00 N vcl total n_vcl_avail 2 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 2729927 24.04 HCB Lookups without lock hcb_lock 470290 4.14 HCB Lookups with lock hcb_insert 470044 4.14 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 113574 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 3 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 2590700 22.81 Gunzip operations sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 2079 0.02 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 1 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 11567121 101.85 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 1 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 2137363 18.82 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 881563 7.76 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 10574 0.09 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 2115908 18.63 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 2129936 18.75 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 239816 2.11 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 6034 0.05 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 10137930 89.26 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 471022 4.15 Created locks LCK.objhdr.destroy 410522 3.61 Destroyed locks LCK.objhdr.locks 11873362 104.54 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 1086349 9.57 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 523780 4.61 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 877 0.01 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 1071376 9.43 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 1183870 10.42 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 2 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 6560980 57.77 Lock Operations LCK.backend.colls 0 0.00 Collisions SMF.s0.c_req 943045 8.30 Allocator requests SMF.s0.c_fail 0 0.00 Allocator failures SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated SMF.s0.c_freed 54163292160 476898.69 Bytes freed SMF.s0.g_alloc 129160 . Allocations outstanding SMF.s0.g_bytes 3356336128 . Bytes outstanding SMF.s0.g_space 938631168 . Bytes available SMF.s0.g_smf 138281 . N struct smf SMF.s0.g_smf_frag 9118 . N small free smf SMF.s0.g_smf_large 3 . N large free smf SMA.Transient.c_req 4780332 42.09 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed SMA.Transient.g_alloc 85 . Allocations outstanding SMA.Transient.g_bytes 1054912 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.default(127.0.0.1,,8080).vcls 2 . VCL references VBE.default(127.0.0.1,,8080).happy 0 . Happy health probes VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health probes -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agb/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : [email protected] Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
