I'm curious what the Virtual Memory vs Resident Set Size from top show. Most likely, the RSS will be stable and small. Virtual memory is almost valueless as a measure.
ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish" -Jason On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbrönner < [email protected]> wrote: > Hi together, > > I have an old varnish 3.0.5 running. (I know its outdated, but my issue > is more understanding what happens here then fixing the problem on the > actual system). While running, the systems takes up more and more memory. > > I already switched to file storage, but still, after one day running and > serving about 2 million requests, the varnish process takes more 20GB of > ram. > > The first thing I am confused is, that as far as I see it varnishstats > does not show the memory consumed (I will attach an output at the > bottom). Tools like top and pmap show the output. > > pmap -x ... > total kB 25368872 16712056 15197372 > > Beside some VCL-Code, the varnish uses the vmods redis, digest, and std. > > Any idea how the find out what is causing the memory leak? > > Thanks in advance, > > Tobias > > Here is my varnishstat -1 output > > client_conn 2115657 18.63 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 5176344 45.58 Client requests received > cache_hit 2207169 19.43 Cache hits > cache_hitpass 18635 0.16 Cache hits for pass > cache_miss 508733 4.48 Cache misses > backend_conn 590509 5.20 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 0 0.00 Backend conn. failures > backend_reuse 2391162 21.05 Backend conn. reuses > backend_toolate 581 0.01 Backend conn. was closed > backend_recycle 2392558 21.07 Backend conn. recycles > backend_retry 142 0.00 Backend conn. retry > fetch_head 604 0.01 Fetch head > fetch_length 1493458 13.15 Fetch with Length > fetch_chunked 1277164 11.25 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 9969 0.09 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 528 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 186330 1.64 Fetch no body (204) > fetch_304 14773 0.13 Fetch no body (304) > n_sess_mem 307 . N struct sess_mem > n_sess 59 . N struct sess > n_object 59401 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 59439 . N struct objectcore > n_objecthead 60591 . N struct objecthead > n_waitinglist 44 . N struct waitinglist > n_vbc 2 . N struct vbc > n_wrk 44 . N worker threads > n_wrk_create 1296 0.01 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 7886 0.07 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 2 . N backends > n_expired 449200 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_moved 1917928 . N LRU moved objects > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 4825557 42.49 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 2115659 18.63 Total Sessions > s_req 5176344 45.58 Total Requests > s_pipe 1304 0.01 Total pipe > s_pass 2473728 21.78 Total pass > s_fetch 2981770 26.25 Total fetch > s_hdrbytes 2065942876 18190.28 Total header bytes > s_bodybytes 133824929675 1178306.04 Total body bytes > sess_closed 418834 3.69 Session Closed > sess_pipeline 63774 0.56 Session Pipeline > sess_readahead 7618 0.07 Session Read Ahead > sess_linger 4929655 43.40 Session Linger > sess_herd 4533770 39.92 Session herd > shm_records 397316593 3498.31 SHM records > shm_writes 22134819 194.89 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 83004 0.73 SHM MTX contention > shm_cycles 173 0.00 SHM cycles through buffer > sms_nreq 693 0.01 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 472858 . SMS bytes allocated > sms_bfree 472858 . SMS bytes freed > backend_req 2982527 26.26 Backend requests made > n_vcl 2 0.00 N vcl total > n_vcl_avail 2 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_gone 1 . N total gone bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 2729927 24.04 HCB Lookups without lock > hcb_lock 470290 4.14 HCB Lookups with lock > hcb_insert 470044 4.14 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 113574 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 3 . Loaded VMODs > n_gzip 0 0.00 Gzip operations > n_gunzip 2590700 22.81 Gunzip operations > sess_pipe_overflow 0 . Dropped sessions due to > session pipe overflow > LCK.sms.creat 1 0.00 Created locks > LCK.sms.destroy 0 0.00 Destroyed locks > LCK.sms.locks 2079 0.02 Lock Operations > LCK.sms.colls 0 0.00 Collisions > LCK.smp.creat 0 0.00 Created locks > LCK.smp.destroy 0 0.00 Destroyed locks > LCK.smp.locks 0 0.00 Lock Operations > LCK.smp.colls 0 0.00 Collisions > LCK.sma.creat 1 0.00 Created locks > LCK.sma.destroy 0 0.00 Destroyed locks > LCK.sma.locks 11567121 101.85 Lock Operations > LCK.sma.colls 0 0.00 Collisions > LCK.smf.creat 1 0.00 Created locks > LCK.smf.destroy 0 0.00 Destroyed locks > LCK.smf.locks 2137363 18.82 Lock Operations > LCK.smf.colls 0 0.00 Collisions > LCK.hsl.creat 0 0.00 Created locks > LCK.hsl.destroy 0 0.00 Destroyed locks > LCK.hsl.locks 0 0.00 Lock Operations > LCK.hsl.colls 0 0.00 Collisions > LCK.hcb.creat 1 0.00 Created locks > LCK.hcb.destroy 0 0.00 Destroyed locks > LCK.hcb.locks 881563 7.76 Lock Operations > LCK.hcb.colls 0 0.00 Collisions > LCK.hcl.creat 0 0.00 Created locks > LCK.hcl.destroy 0 0.00 Destroyed locks > LCK.hcl.locks 0 0.00 Lock Operations > LCK.hcl.colls 0 0.00 Collisions > LCK.vcl.creat 1 0.00 Created locks > LCK.vcl.destroy 0 0.00 Destroyed locks > LCK.vcl.locks 10574 0.09 Lock Operations > LCK.vcl.colls 0 0.00 Collisions > LCK.stat.creat 1 0.00 Created locks > LCK.stat.destroy 0 0.00 Destroyed locks > LCK.stat.locks 2115908 18.63 Lock Operations > LCK.stat.colls 0 0.00 Collisions > LCK.sessmem.creat 1 0.00 Created locks > LCK.sessmem.destroy 0 0.00 Destroyed locks > LCK.sessmem.locks 2129936 18.75 Lock Operations > LCK.sessmem.colls 0 0.00 Collisions > LCK.wstat.creat 1 0.00 Created locks > LCK.wstat.destroy 0 0.00 Destroyed locks > LCK.wstat.locks 239816 2.11 Lock Operations > LCK.wstat.colls 0 0.00 Collisions > LCK.herder.creat 1 0.00 Created locks > LCK.herder.destroy 0 0.00 Destroyed locks > LCK.herder.locks 6034 0.05 Lock Operations > LCK.herder.colls 0 0.00 Collisions > LCK.wq.creat 2 0.00 Created locks > LCK.wq.destroy 0 0.00 Destroyed locks > LCK.wq.locks 10137930 89.26 Lock Operations > LCK.wq.colls 0 0.00 Collisions > LCK.objhdr.creat 471022 4.15 Created locks > LCK.objhdr.destroy 410522 3.61 Destroyed locks > LCK.objhdr.locks 11873362 104.54 Lock Operations > LCK.objhdr.colls 0 0.00 Collisions > LCK.exp.creat 1 0.00 Created locks > LCK.exp.destroy 0 0.00 Destroyed locks > LCK.exp.locks 1086349 9.57 Lock Operations > LCK.exp.colls 0 0.00 Collisions > LCK.lru.creat 2 0.00 Created locks > LCK.lru.destroy 0 0.00 Destroyed locks > LCK.lru.locks 523780 4.61 Lock Operations > LCK.lru.colls 0 0.00 Collisions > LCK.cli.creat 1 0.00 Created locks > LCK.cli.destroy 0 0.00 Destroyed locks > LCK.cli.locks 877 0.01 Lock Operations > LCK.cli.colls 0 0.00 Collisions > LCK.ban.creat 1 0.00 Created locks > LCK.ban.destroy 0 0.00 Destroyed locks > LCK.ban.locks 1071376 9.43 Lock Operations > LCK.ban.colls 0 0.00 Collisions > LCK.vbp.creat 1 0.00 Created locks > LCK.vbp.destroy 0 0.00 Destroyed locks > LCK.vbp.locks 0 0.00 Lock Operations > LCK.vbp.colls 0 0.00 Collisions > LCK.vbe.creat 1 0.00 Created locks > LCK.vbe.destroy 0 0.00 Destroyed locks > LCK.vbe.locks 1183870 10.42 Lock Operations > LCK.vbe.colls 0 0.00 Collisions > LCK.backend.creat 2 0.00 Created locks > LCK.backend.destroy 0 0.00 Destroyed locks > LCK.backend.locks 6560980 57.77 Lock Operations > LCK.backend.colls 0 0.00 Collisions > SMF.s0.c_req 943045 8.30 Allocator requests > SMF.s0.c_fail 0 0.00 Allocator failures > SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated > SMF.s0.c_freed 54163292160 476898.69 Bytes freed > SMF.s0.g_alloc 129160 . Allocations outstanding > SMF.s0.g_bytes 3356336128 . Bytes outstanding > SMF.s0.g_space 938631168 . Bytes available > SMF.s0.g_smf 138281 . N struct smf > SMF.s0.g_smf_frag 9118 . N small free smf > SMF.s0.g_smf_large 3 . N large free smf > SMA.Transient.c_req 4780332 42.09 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated > SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed > SMA.Transient.g_alloc 85 . Allocations outstanding > SMA.Transient.g_bytes 1054912 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > VBE.default(127.0.0.1,,8080).vcls 2 . VCL references > VBE.default(127.0.0.1,,8080).happy 0 . Happy health > probes > VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references > VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health > probes > > -- > LAMP solutions GmbH > Gostenhofer Hauptstrasse 35 > 90443 Nuernberg > > Amtsgericht Nuernberg: HRB 22366 > Geschaeftsfuehrer: Heiko Schubert > > Es gelten unsere allgemeinen Geschaeftsbedingungen. > http://www.lamp-solutions.de/agb/ > > Telefon : 0911 / 376 516 0 > Fax : 0911 / 376 516 11 > E-Mail : [email protected] > Web : www.lamp-solutions.de > Facebook : http://www.facebook.com/LAMPsolutions > Twitter : http://twitter.com/#!/lampsolutions > > _______________________________________________ > varnish-misc mailing list > [email protected] > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
