>> >> The system is currently fully using available physical memory and has burned >> through about 15% of the swap space. The varnishstat was taken after the >> physical memory was exhausted and with just a few Gigs of swap in use. Here >> is a current snapshot from top and varnishstat –1 if that helps. > > Could it be that your memory allocator is leaking memory? The varnishstat > looks decent, there are no counters that are off the charts. Last time we saw > behaviour like that was when we were seeing seeing leaks with malloc on Linux, > before we switched to jemalloc. > > I think Varnish will try to link with jemalloc if it is available on your > system. Is it? As you can see there are quite a few allocations happening and > if that memory is not reclaimed and reused you'll be leaking quite a bit. >
My understanding is that the base malloc in freebsd 9.2 is jemalloc — the code was ported back into the base malloc.c. Between FreeBSD 9.2 and FreeBSD 10.0, they changed how it is done, and now is using the pure jemalloc.c code that should be the same as in Linux. https://svnweb.freebsd.org/base/release/9.2.0/lib/libc/stdlib/malloc.c?revis ion=255898&view=markup Vs https://svnweb.freebsd.org/base/release/10.0.0/contrib/jemalloc/src/jemalloc .c?revision=260789&view=markup That said, we have a freebsd 10 server running the same version of varnishd that has the same problem (the reason I had discounted freebsd as the culprit initially). This server’s hardware is much newer and as you can see has quite a bit more physical memory as well. It is running with a few more threads as you can see, because it has more cpu cores and was configured with more thread_pools. Otherwise, it is the same as our 9.2 boxes. >From our FreeBSD 10 box running varnish: CPU: 4.0% user, 0.0% nice, 0.7% system, 0.2% interrupt, 95.1% idle Mem: 4509M Active, 111G Inact, 6229M Wired, 73M Cache, 2757M Free ARC: 1911M Total, 464M MFU, 730M MRU, 400K Anon, 390M Header, 326M Other Swap: 32G Total, 8353M Used, 24G Free, 25% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1311 www 1291 20 0 354G 116G uwait 8 476.0H 21.19% varnishd $ uname -v > FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 [email protected]:/usr/obj/usr/src/sys/GENERIC varnishstat –1: client_conn 317914959 133.36 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 507413109 212.85 Client requests received cache_hit 236222630 99.09 Cache hits cache_hitpass 21505999 9.02 Cache hits for pass cache_miss 213793146 89.68 Cache misses backend_conn 111736980 46.87 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 8690564 3.65 Backend conn. too many backend_fail 516882 0.22 Backend conn. failures backend_reuse 265640872 111.43 Backend conn. reuses backend_toolate 14139104 5.93 Backend conn. was closed backend_recycle 279780619 117.36 Backend conn. recycles backend_retry 25784 0.01 Backend conn. retry fetch_head 1770 0.00 Fetch head fetch_length 121268619 50.87 Fetch with Length fetch_chunked 254268690 106.66 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 1590881 0.67 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 167507 0.07 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 172230 0.07 Fetch no body (304) n_sess_mem 6946 . N struct sess_mem n_sess 593 . N struct sess n_object 992149 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 993155 . N struct objectcore n_objecthead 988241 . N struct objecthead n_waitinglist 1273 . N struct waitinglist n_vbc 71 . N struct vbc n_wrk 1280 . N worker threads n_wrk_create 1595 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 4530 0.00 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 7 . N backends n_expired 135419385 . N expired objects n_lru_nuked 76410300 . N LRU nuked objects n_lru_moved 128024378 . N LRU moved objects losthdr 28 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 355076718 148.95 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 317934238 133.37 Total Sessions s_req 507413109 212.85 Total Requests s_pipe 0 0.00 Total pipe s_pass 165619470 69.47 Total pass s_fetch 377135515 158.20 Total fetch s_hdrbytes 189267454297 79393.14 Total header bytes s_bodybytes 3732756334613 1565801.44 Total body bytes sess_closed 35770111 15.00 Session Closed sess_pipeline 24574 0.01 Session Pipeline sess_readahead 7384 0.00 Session Read Ahead sess_linger 476097250 199.71 Session Linger sess_herd 494921706 207.61 Session herd shm_records 45611717569 19133.02 SHM records shm_writes 2842104094 1192.19 SHM writes shm_flushes 2794 0.00 SHM flushes due to overflow shm_cont 5861694 2.46 SHM MTX contention shm_cycles 19426 0.01 SHM cycles through buffer sms_nreq 2859746 1.20 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 8118447295 . SMS bytes allocated sms_bfree 8118447295 . SMS bytes freed backend_req 377315426 158.27 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 459 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 2383927 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 257811067 108.15 Gzip operations n_gunzip 276663353 116.05 Gunzip operations sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 8579238 3.60 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 2111376064 885.67 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 0 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 0 0.00 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 250007 0.10 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 942040132 395.16 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 4406502 1.85 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 317940634 133.37 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 318157037 133.46 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 6160352 2.58 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1734 0.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 32 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 1140526679 478.42 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 183579053 77.01 Created locks LCK.objhdr.destroy 182591336 76.59 Destroyed locks LCK.objhdr.locks 1584172928 664.52 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 426984689 179.11 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 289232005 121.33 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 1180095 0.50 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 427224495 179.21 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 1809670 0.76 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 224508313 94.18 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 7 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 904980852 379.62 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 578846458 242.81 Allocator requests SMA.s0.c_fail 76630001 32.14 Allocator failures SMA.s0.c_bytes 25682079523063 10773014.24 Bytes allocated SMA.s0.c_freed 25675638298667 10770312.30 Bytes freed SMA.s0.g_alloc 2023035 . Allocations outstanding SMA.s0.g_bytes 6441224396 . Bytes outstanding SMA.s0.g_space 1226548 . Bytes available SMA.Transient.c_req 343584190 144.13 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 21240906064202 8910048.87 Bytes allocated SMA.Transient.c_freed 21240900389430 8910046.49 Bytes freed SMA.Transient.g_alloc 5520 . Allocations outstanding SMA.Transient.g_bytes 5674772 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
