Re: Memory usage
In message 24a219a51001261923n40146083yb221aac2cadbc...@mail.gmail.com, Marti n Goldman writes: 1. How can you tell whether your Varnish objects fit in RAM? If you start seeing disk-activity, they do not fit. 2. If I have objects residing in virtual memory, to what extent will my performance be adversely affected? If I want my site to be fast, do I basically need to go out and buy as much RAM as it will take so that virtual memory isn't needed? Well, the impact is the necessary disk-I/O to bring the object into RAM. Getting more RAM is one solution, but if your working set is much larger than 4G, getting a SSD disk instead might be a better investment. 3. I noticed tonight that my machine was using a few hundred megs of swap space, Yes, varnish will force inactive programs (inetd, getty, sendmail etc) out to swap so it can get at the RAM. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory usage
My backing store is file-based. In any case, thanks to you both -- I have ordered some more RAM and will see what happens. Martin On Wed, Jan 27, 2010 at 12:23 AM, Michael Fischer mich...@dynamine.netwrote: On Tue, Jan 26, 2010 at 7:23 PM, Martin Goldman m...@mgoldman.com wrote: I'm running Varnish on a box with 4GB RAM. There are hundreds of thousands of objects being served, and I'm certain that they don't all fit in that relatively meager amount of RAM. I understand that Varnish's model dictates that the kernel will be trusted to use virtual memory as necessary if the cached objects don't fit in RAM. I have a few questions about this: 1. How can you tell whether your Varnish objects fit in RAM? You can't guarantee that they will unless you set your cache size at or below the amount of RAM you have installed. 2. If I have objects residing in virtual memory, to what extent will my performance be adversely affected? If I want my site to be fast, do I basically need to go out and buy as much RAM as it will take so that virtual memory isn't needed? Technically, it's go out and buy as much RAM as it will take to avoid being swamped by paging. But yes. 3. I noticed tonight that my machine was using a few hundred megs of swap space, which I've never seen happen before. Varnish is the only non-system service running on this box. My understanding was that Varnish would get only as much RAM as was available and then send the overflow into the file-backed virtual memory. If that's the case, though, then why is swap space being used? Is this just a side effect of how the kernel allocates memory, or is something else going on here? Is your backing store file-based, or malloc-based? If the latter, that would explain the swap space being consumed. Or, as Darryl said, the housekeeping overhead of a VERY large file-backed cache could make the Varnish process very large. --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Memory usage
Hello all, I'm running Varnish on a box with 4GB RAM. There are hundreds of thousands of objects being served, and I'm certain that they don't all fit in that relatively meager amount of RAM. I understand that Varnish's model dictates that the kernel will be trusted to use virtual memory as necessary if the cached objects don't fit in RAM. I have a few questions about this: 1. How can you tell whether your Varnish objects fit in RAM? 2. If I have objects residing in virtual memory, to what extent will my performance be adversely affected? If I want my site to be fast, do I basically need to go out and buy as much RAM as it will take so that virtual memory isn't needed? 3. I noticed tonight that my machine was using a few hundred megs of swap space, which I've never seen happen before. Varnish is the only non-system service running on this box. My understanding was that Varnish would get only as much RAM as was available and then send the overflow into the file-backed virtual memory. If that's the case, though, then why is swap space being used? Is this just a side effect of how the kernel allocates memory, or is something else going on here? Many thanks, Martin ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory usage
Thanks Darryl! One follow-up about the VIRT size (mine is 55.2GB). I thought the VIRT size includes the entire amount of VARNISH_STORAGE_SIZE (50GB in my case), regardless of how much of the virtual memory is actually being used to store cached objects. This seems to be the case based on a few minutes of experimenting with that setting. So I'm not sure I understand how to determine the amount of virtual memory I'm actually using -- in other words, the amount of RAM I need to add for optimal performance -- from the VIRT and RES numbers alone. Any chance I could ask you to please fill in what I'm missing? Thanks again, Martin On Tue, Jan 26, 2010 at 10:34 PM, Darryl Dixon - Winterhouse Consulting darryl.di...@winterhouseconsulting.com wrote: Hi Martin, I'm running Varnish on a box with 4GB RAM. There are hundreds of thousands of objects being served, and I'm certain that they don't all fit in that relatively meager amount of RAM. I understand that Varnish's model dictates that the kernel will be trusted to use virtual memory as necessary if the cached objects don't fit in RAM. I have a few questions about this: 1. How can you tell whether your Varnish objects fit in RAM? In short, `top` - the VIRT column tells you total virtual process size, the RES column then tells you which portion of that is currently resident in physical memory 2. If I have objects residing in virtual memory, to what extent will my performance be adversely affected? If I want my site to be fast, do I basically need to go out and buy as much RAM as it will take so that virtual memory isn't needed? Pretty much. 3. I noticed tonight that my machine was using a few hundred megs of swap space, which I've never seen happen before. Varnish is the only non-system service running on this box. My understanding was that Varnish would get only as much RAM as was available and then send the overflow into the file-backed virtual memory. If that's the case, though, then why is swap space being used? Is this just a side effect of how the kernel allocates memory, or is something else going on here? Two things; 1) The varnish process itself requires memory (eg, to hold the ban list etc), which is not part of the file-backed object cache. 2) Even if the above usage were minimal, it is still entirely possibly that your VMM (the OS) has decided that the memory being used to cache objects is more important that some other system processes that have now been shunted out to swap. Once again, `top` VIRT versus RES will give you a good clue regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory usage
Thanks Darryl! One follow-up about the VIRT size (mine is 55.2GB). I thought the VIRT size includes the entire amount of VARNISH_STORAGE_SIZE (50GB in my case), regardless of how much of the virtual memory is actually being used to store cached objects. This seems to be the case based on a few minutes of experimenting with that setting. That is correct. I was assuming from your initial post (probably wrongly) that your cache was 'full' ie, that there were in your case 50GB of objects cached already. You can see how much of the allocated cache is used with varnishstat; the bytes allocated and bytes free row will tell you this. Unfortunately, what you want is the intersection of bytes allocated and bytes not currently in physical memory and if the entire cache is not full, then it gets pretty tough to know, because Varnish *by design* neither knows nor cares about the OSes swapping arrangements. o I'm not sure I understand how to determine the amount of virtual memory I'm actually using -- in other words, the amount of RAM I need to add for optimal performance -- from the VIRT and RES numbers alone. Any chance I could ask you to please fill in what I'm missing? Probably the two entries from the varnishstat output I mentioned above will give you a better idea. Obviously, if you really have 50GB of items to cache, then you will need a lot of RAM. Perhaps if you monitor the bytes allocated row over time you will see it stabilise - this is the amount of RAM that you would need to keep all your cacheable items in physical memory. Of course, just because something ends up in the cache once, doesn't mean there is necessarily any value in keeping it in physical memory; if no-one ever requests it again, then it makes sense for it to end up in swap... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory usage
On Tue, Jan 26, 2010 at 7:23 PM, Martin Goldman m...@mgoldman.com wrote: I'm running Varnish on a box with 4GB RAM. There are hundreds of thousands of objects being served, and I'm certain that they don't all fit in that relatively meager amount of RAM. I understand that Varnish's model dictates that the kernel will be trusted to use virtual memory as necessary if the cached objects don't fit in RAM. I have a few questions about this: 1. How can you tell whether your Varnish objects fit in RAM? You can't guarantee that they will unless you set your cache size at or below the amount of RAM you have installed. 2. If I have objects residing in virtual memory, to what extent will my performance be adversely affected? If I want my site to be fast, do I basically need to go out and buy as much RAM as it will take so that virtual memory isn't needed? Technically, it's go out and buy as much RAM as it will take to avoid being swamped by paging. But yes. 3. I noticed tonight that my machine was using a few hundred megs of swap space, which I've never seen happen before. Varnish is the only non-system service running on this box. My understanding was that Varnish would get only as much RAM as was available and then send the overflow into the file-backed virtual memory. If that's the case, though, then why is swap space being used? Is this just a side effect of how the kernel allocates memory, or is something else going on here? Is your backing store file-based, or malloc-based? If the latter, that would explain the swap space being consumed. Or, as Darryl said, the housekeeping overhead of a VERY large file-backed cache could make the Varnish process very large. --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
I experienced this same issue under x64. Varnish seemed great but once I put some real traffic on it under x64 the memory leaks began and it would eventually crash/restart. Ended up putting Varnish on the back burner and have been waiting for it to stabilize before even trying to present it to upper management again. Varnish has great potential but until it can run stable under x64 it's got a long fight ahead of itself. (I do want to note that my comments are based mainly on varnish 1 and not varnish 2.0) --cripy On Wed, Oct 21, 2009 at 8:22 AM, Henry Paulissen h.paulis...@qbell.nlwrote: We encounter the same problem. Its seems to occur only on x64 platforms. We decided to take a different approach and installed vmware to the machine. Next we did a setup of 6 guests with x32 PAE software. No strange memory leaks occurred since then at the price of small storage (3.5G max) and limited worker threads (256 max). Opened a ticket for the problem, but the wont listen until I buy a support contract (á €8K). Seems they don’t want to know there is some kind of memory issue in their software. Anyway... Varnish is running stable now with some few tricks. Regards, -Original Message- From: varnish-misc-boun...@projects.linpro.no [mailto: varnish-misc-boun...@projects.linpro.no] On Behalf Of Kristian Lyngstol Sent: woensdag 21 oktober 2009 13:34 To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
Hopefully your upper management allows you to install contemporary software and distributions. Otherwise memory leaks and x86_64 would be the least of your concerns. Honestly, you're waiting for Varnish to stabilize and you're running v1? My data point: 5 months and over 100PB of transfers, and 2.0.4 is stable and has never leaked in our pure x86_64 production environment. Its memory use can be precisely monitored and controlled between Varnish configuration and the OS environment by any competent sysadmin, IMHO. We actually can't use Squid at all because it really does leak like a sieve. pmap does not lie. I just hope that people that have problems with any software are taking on the responsibility of diagnosing their own environments as much as they expect any OSS project to diagnose its code -- the former is just as often the problem as the latter. -- Ken On Nov 5, 2009, at 12:22 PM, cripy wrote: I experienced this same issue under x64. Varnish seemed great but once I put some real traffic on it under x64 the memory leaks began and it would eventually crash/restart. Ended up putting Varnish on the back burner and have been waiting for it to stabilize before even trying to present it to upper management again. Varnish has great potential but until it can run stable under x64 it's got a long fight ahead of itself. (I do want to note that my comments are based mainly on varnish 1 and not varnish 2.0) --cripy ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
On Thu, Oct 22, 2009 at 6:04 AM, Henry Paulissen h.paulis...@qbell.nl wrote:
I will report back.
Did this solve the problem?
Removing this?
if (req.http.Cache-Control == no-cache || req.http.Pragma ==
no-cache) {
purge_url(req.url);
}
Cheers
Att,
--
Rogério Schneider
MSN: stoc...@hotmail.com
GTalk: stoc...@gmail.com
Skype: stockrt
http://stockrt.github.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
No, varnishd still usages way more than allowed.
The only solutions I found at the moment are:
Run on x64 linux and restart varnish every 4 hours (crontab).
Run on x32 linux (all is working as expected but you cant allocate more as
4G each instance).
I hope linpro will find this issue and address it.
Again @ linpro: if you need a machine (with live traffic) to run some tests,
please contact me.
We have multiple machines in high availability, so testing and rebooting a
instance wouldnt hurt us.
Regards.
-Oorspronkelijk bericht-
Van: Rogério Schneider [mailto:stoc...@gmail.com]
Verzonden: woensdag 4 november 2009 22:04
Aan: Henry Paulissen
CC: Scott Wilson; varnish-misc@projects.linpro.no
Onderwerp: Re: Varnish virtual memory usage
On Thu, Oct 22, 2009 at 6:04 AM, Henry Paulissen h.paulis...@qbell.nl
wrote:
I will report back.
Did this solve the problem?
Removing this?
if (req.http.Cache-Control == no-cache || req.http.Pragma ==
no-cache) {
purge_url(req.url);
}
Cheers
Att,
--
Rogério Schneider
MSN: stoc...@hotmail.com
GTalk: stoc...@gmail.com
Skype: stockrt
http://stockrt.github.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Our load balancer transforms all connections from keep-alive to close. So keep-alive connections arent the issue here. Also, if I limit the thread count I still see the same behavior. -Oorspronkelijk bericht- Van: Ken Brownfield [mailto:k...@slide.com] Verzonden: donderdag 5 november 2009 0:31 Aan: Henry Paulissen CC: varnish-misc@projects.linpro.no Onderwerp: Re: Varnish virtual memory usage Looks like varnish is allocating ~1.5GB of RAM for pure cache (which may roughly match your -s file option) but 1,610 threads with your 1MB stack limit will use 1.7GB of RAM. Pmap is reporting the footprint of this instance as roughly 3.6GB, and I'm assuming top/ps agree with that number. Unless your -s file option is significantly less than 1-1.5GB, the sheer thread count explains your memory usage: maybe using a stacksize of 512K or 256K could help, and/or disable keepalives on the client side? Also, if you happen to be using a load balancer, TCP Buffering (NetScaler) or Proxy Buffering? (BigIP) or the like can drastically reduce the thread count (and they can handle the persistent keepalives as well). But IMHO, an event-based (for example) handler for idle or slow threads is probably the next important feature, just below persistence. Without something like TCP buffering, the memory available for actual caching is dwarfed by the thread stacksize alloc overhead. Ken On Nov 4, 2009, at 3:18 PM, Henry Paulissen wrote: I attached the memory dump. Child processes count gives me 1610 processes (on this instance). Currently the server isnt so busy (~175 requests / sec). Varnishstat -1: = = = = = = == == = = = = = = == == uptime 3090 . Child uptime client_conn435325 140.88 Client connections accepted client_drop 0 0.00 Connection dropped, no sess client_req 435294 140.87 Client requests received cache_hit 4574014.80 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 12644540.92 Cache misses backend_conn 355277 114.98 Backend conn. success backend_unhealthy0 0.00 Backend conn. not attempted backend_busy0 0.00 Backend conn. too many backend_fail0 0.00 Backend conn. failures backend_reuse 3433111.11 Backend conn. reuses backend_toolate 690 0.22 Backend conn. was closed backend_recycle 3502111.33 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 0 0.00 Fetch head fetch_length 384525 124.44 Fetch with Length fetch_chunked2441 0.79 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 2028 0.66 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed0 0.00 Fetch failed n_sess_mem989 . N struct sess_mem n_sess 94 . N struct sess n_object89296 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore89640 . N struct objectcore n_objecthead25379 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 26 . N struct vbe_conn n_wrk1600 . N worker threads n_wrk_create 1600 0.52 N worker threads created n_wrk_failed0 0.00 N worker threads not created n_wrk_max1274 0.41 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 1342 0.43 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 5 . N backends n_expired1393 . N expired objects n_lru_nuked 35678 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 20020 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr11 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent
Re: Varnish virtual memory usage
In message 003201ca5da9$57ae7e30$070b7a...@paulissen@qbell.nl, Henry Pauliss en writes: Our load balancer transforms all connections from keep-alive to close. That is a bad idea really, it increases the amount of work varnish has to do significantly. but 1,610 threads with your 1MB stack limit will use 1.7GB of RAM. It is very important to keep Virtual Address Space and RAM out from each other. The stacks will use 1.7G of VM-space, but certainly not as much RAM as most of the stacks are not accessed. The number you care about is the resident size, the _actual_ amount of RAM used. Only on 32bit systems is there any reason to be concerned about VM-space used. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
Hmm, well the memory adds up to a 1.5G -s option (can you confirm what you use with -s?) and memory required to run the number of threads you're running. Unless your -s is drastically smaller than 1.5GB, the pmap you sent is of a normal, non-leaking process. Ken On Nov 4, 2009, at 3:48 PM, Henry Paulissen wrote: Our load balancer transforms all connections from keep-alive to close. So keep-alive connections aren’t the issue here. Also, if I limit the thread count I still see the same behavior. -Oorspronkelijk bericht- Van: Ken Brownfield [mailto:k...@slide.com] Verzonden: donderdag 5 november 2009 0:31 Aan: Henry Paulissen CC: varnish-misc@projects.linpro.no Onderwerp: Re: Varnish virtual memory usage Looks like varnish is allocating ~1.5GB of RAM for pure cache (which may roughly match your -s file option) but 1,610 threads with your 1MB stack limit will use 1.7GB of RAM. Pmap is reporting the footprint of this instance as roughly 3.6GB, and I'm assuming top/ps agree with that number. Unless your -s file option is significantly less than 1-1.5GB, the sheer thread count explains your memory usage: maybe using a stacksize of 512K or 256K could help, and/or disable keepalives on the client side? Also, if you happen to be using a load balancer, TCP Buffering (NetScaler) or Proxy Buffering? (BigIP) or the like can drastically reduce the thread count (and they can handle the persistent keepalives as well). But IMHO, an event-based (for example) handler for idle or slow threads is probably the next important feature, just below persistence. Without something like TCP buffering, the memory available for actual caching is dwarfed by the thread stacksize alloc overhead. Ken On Nov 4, 2009, at 3:18 PM, Henry Paulissen wrote: I attached the memory dump. Child processes count gives me 1610 processes (on this instance). Currently the server isn’t so busy (~175 requests / sec). Varnishstat -1: = = = = = = = = == = = = = = = = = == uptime 3090 . Child uptime client_conn435325 140.88 Client connections accepted client_drop 0 0.00 Connection dropped, no sess client_req 435294 140.87 Client requests received cache_hit 4574014.80 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 12644540.92 Cache misses backend_conn 355277 114.98 Backend conn. success backend_unhealthy0 0.00 Backend conn. not attempted backend_busy0 0.00 Backend conn. too many backend_fail0 0.00 Backend conn. failures backend_reuse 3433111.11 Backend conn. reuses backend_toolate 690 0.22 Backend conn. was closed backend_recycle 3502111.33 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 0 0.00 Fetch head fetch_length 384525 124.44 Fetch with Length fetch_chunked2441 0.79 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 2028 0.66 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed0 0.00 Fetch failed n_sess_mem989 . N struct sess_mem n_sess 94 . N struct sess n_object89296 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore89640 . N struct objectcore n_objecthead25379 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 26 . N struct vbe_conn n_wrk1600 . N worker threads n_wrk_create 1600 0.52 N worker threads created n_wrk_failed0 0.00 N worker threads not created n_wrk_max1274 0.41 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 1342 0.43 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 5 . N backends n_expired1393 . N expired objects n_lru_nuked 35678 . N LRU nuked objects
Re: Varnish virtual memory usage
;
} else {
set resp.http.X-Cache = MISS;
}
set resp.http.Server = static;
return (deliver);
}
sub vcl_error {
set obj.http.Content-Type = text/html; charset=utf-8;
synthetic {
?xml version=1.0 encoding=utf-8?
!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Strict//EN
http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;
html
head
title} obj.status obj.response {/title
/head
body
h1Error } obj.status obj.response {/h1
p} obj.response {/p
h3Guru Meditation:/h3
pXID: } req.xid {/p
/body
/html
};
return (deliver);
}
#
#
For further details see my ticket:
http://varnish.projects.linpro.no/ticket/546
@Kristian:
When the programmers / engineers have some spare time over, they are always
welcome to see it in live action.
-Oorspronkelijk bericht-
Van: Ken Brownfield [mailto:k...@slide.com]
Verzonden: woensdag 21 oktober 2009 21:57
Aan: Henry Paulissen
CC: varn...@projects.linpro.no
Onderwerp: Re: Varnish virtual memory usage
Small comments:
1) We're running Linux x86_64 exclusively here under significant load,
with no memory issues.
2) Why don't you compile a 32-bit version of Varnish; wouldn't this
have the same effect without the RAM and performance hit of VMs?
3) Do you make heavy use of purges?
--
kb
On Oct 21, 2009, at 6:22 AM, Henry Paulissen wrote:
We encounter the same problem.
Its seems to occur only on x64 platforms.
We decided to take a different approach and installed vmware to the
machine.
Next we did a setup of 6 guests with x32 PAE software.
No strange memory leaks occurred since then at the price of small
storage (3.5G max) and limited worker threads (256 max).
Opened a ticket for the problem, but the wont listen until I buy a
support contract (á €8K).
Seems they don’t want to know there is some kind of memory issue in
their software.
Anyway...
Varnish is running stable now with some few tricks.
Regards,
-Original Message-
From: varnish-misc-boun...@projects.linpro.no [mailto:varnish-misc-
boun...@projects.linpro.no] On Behalf Of Kristian Lyngstol
Sent: woensdag 21 oktober 2009 13:34
To: Roi Avinoam
Cc: varnish-misc@projects.linpro.no
Subject: Re: Varnish virtual memory usage
On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote:
At Metacafe we're testing the integration with Varnish, and I was
tasked with benchmarking our Varnish setup. I intentionally
over-flooded the server with requests, in an attempt to see how the
system will behave under extensive traffic. Surprisingly, the server
ran out of swap and crashed.
That seems mighty strange. What sort of tests did you do?
In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,
1G.
Does it mean Varnish shouldn't use more than 1GB of the virtual
memory?
Is there any other way to limit the memory/storage usage?
If you are using -s file and you have 4GB of memory, you are telling
Varnish to create a _file_ of 1GB, and it's up to the kernel what it
keeps in memory or not. If you actually run out of memory with this
setup, you've either hit a bug (need more details first), or you're
doing something strange like having the mmaped file (/var/lib/
varnish/) in tmpfs with a sizelimit less than 1GB or something along
those lines. But I need more details to say anything for certain.
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Awh,
Thank you for your comment.
I'll make a test case of it tomorrow (or else after the weekend).
I will report back.
-Original Message-
From: Scott Wilson [mailto:sc...@idealist.org]
Sent: donderdag 22 oktober 2009 8:52
To: Henry Paulissen
Cc: varnish-misc@projects.linpro.no; k...@slide.com
Subject: Re: Varnish virtual memory usage
We had a similar problem where varnish would fill all swap and crash
every couple of weeks. The trick that seems to have solved the problem
was to remove purge.url from our VCL (a lot of badly behaved clients
send a lot more no-cache headers than necessary).
We replaced purge.url with an approach that sets the object's ttl to
zero and restarts the request. The details are here:
http://varnish.projects.linpro.no/wiki/VCLExampleEnableForceRefresh
In our case we're using FreeBSD 7.2 64-bit.
All that said, it doesn't seem that this solution jives with Roi's
random url test unless purge.url figured in his vcl / testing script.
cheers,
scott
2009/10/22 Henry Paulissen h.paulis...@qbell.nl:
We ran CentOS 5.3 X64 when we noticed this strange behavior. Later on we
moved to Fedora core 11 X64. But we where still noticing the same memory
allocation problems. Later on we reinstalled the server with vmware to run a
couple of (half live a.k.a. beta) tests and noticed it isn’t happening under
fedora core 11 x32.
We do about 3000 connections/sec for static content (smaller images). For
large images ( 200kb), javascript and css we have another instances running
(all having the same issues, but im going to tell you about the static
content instance).
Hitrate is close to the 100% (99-100%).
Server core's: 16
Memory: 24GB (VM host server is upgraded to 64GB ram and only doing varnish
guests on malloc, so I doubt there's a real performance impact)
Tried changing the number of thread_pools and workers, nothing helped.
Did the sysctl recommended settings. Disabled conntrack filter in iptables.
All incoming requests are with the connection: close close header (we have
a high availability server above it, who doesn’t allow keep-alive
connections. So he transforms every connection to close).
Both storage type's where used.
I did noticed something when I changed the lru_interval to 60. The reserved
memory was keeping within his limits (before this changing this setting it
grow way above max limit). But virtual memory is still way above memory the
limit.
If we didn’t restart varnish every few hours it grow above the physical
memory limit and starts using the swap space. If the varnish server was
restarted it freed up the memory.
Tried both stable and svn versions.
My VCL for static:
#
#
director staticbackend round-robin {
{
.backend = {
.host = 192.168.x.x;
.port = x;
.connect_timeout = 2s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
}
{
.backend = {
.host = 192.168.x.x;
.port = x;
.connect_timeout = 2s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
}
}
sub vcl_recv {
set req.backend = staticbackend;
if (req.request != GET req.request != HEAD req.request !=
PUT req.request != POST req.request != TRACE req.request !=
OPTIO
NS req.request != DELETE) {
/*
Non-RFC2616 or CONNECT which is weird.
Shoot this client, but first go in pipeline to the
webserver.
Maybe he knows what to do with this request.
*/
return (pipe);
}
remove req.http.X-Forwarded-For;
remove req.http.Accept-Encoding;
remove req.http.Accept-Charset;
remove req.http.Accept-Language;
remove req.http.Referer;
remove req.http.Accept;
remove req.http.Cookie;
return (lookup);
}
sub vcl_pipe {
set req.http.connection = close;
return (pipe);
}
sub vcl_pass {
return (pass);
}
sub vcl_hash {
set req.hash += req.url;
if (req.http.Cache-Control == no-cache || req.http.Pragma ==
no-cache) {
purge_url(req.url);
}
return (hash);
}
sub vcl_hit {
if (!obj.cacheable) {
return(pass);
}
return (deliver);
}
sub vcl_miss {
return (fetch);
}
sub vcl_fetch {
/*
I hate it when varnish cashes my redirects.
Some of them are dynamic
Re: Varnish virtual memory usage
On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgphXG7YeoQJX.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Thanks for your reply. 1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'): a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish. b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio. What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. 2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM? Thanks again :) -- Roi -Original Message- From: Kristian Lyngstol [mailto:krist...@redpill-linpro.com] Sent: Wednesday, October 21, 2009 1:34 PM To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
We encounter the same problem. Its seems to occur only on x64 platforms. We decided to take a different approach and installed vmware to the machine. Next we did a setup of 6 guests with x32 PAE software. No strange memory leaks occurred since then at the price of small storage (3.5G max) and limited worker threads (256 max). Opened a ticket for the problem, but the wont listen until I buy a support contract (á €8K). Seems they don’t want to know there is some kind of memory issue in their software. Anyway... Varnish is running stable now with some few tricks. Regards, -Original Message- From: varnish-misc-boun...@projects.linpro.no [mailto:varnish-misc-boun...@projects.linpro.no] On Behalf Of Kristian Lyngstol Sent: woensdag 21 oktober 2009 13:34 To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote: Thanks for your reply. 1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'): a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish. b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio. What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. When did it fill up and how fast? 2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM? The -s parameter only applies to storage, which is to say that it's an approximiation and there will be some additional overhead which is not stored there, ie datastructures for threads, sessions, backends etc. So with -s ..., 1G, you should expect a little bit over 1GB, but not nearly as much as you describe. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpBkeZKEmnEc.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Maybe it was a bit rough to say indeed... My apologies for that one. What im saying a longer time for now (also sayed to Paul by phone). I'm simply not making enough profit for €8k, but im not saying I don’t want to pay anything for service / support. It’s simply not within my reach and for me it's cheaper to run 6 guests in a big vmware server as paying €8K and get the problem (maybe) solved. Anyway, this is way to offtopic for this thread. I shared my experiences and solutions Roi can consider it as a solution for him. Regards, Henry Paulissen -Original Message- From: 'Kristian Lyngstol' [mailto:krist...@redpill-linpro.com] Sent: woensdag 21 oktober 2009 15:43 To: Henry Paulissen Cc: varn...@redpill-linpro.com Subject: Re: Varnish virtual memory usage (ticket #546) On Wed, Oct 21, 2009 at 02:57:34PM +0200, Henry Paulissen wrote: Opened a ticket for the problem, but the wont listen until I buy a support contract (á €8K). Seems they don’t want to know there is some kind of memory issue in their software. The ticket is not closed, we have, however, not been able to reproduce this as we point out in the ticket. Until we can either reproduce this ourself or get more data (more reports of the same issue, for instance), there really isn't much we can do. For a service agreement customer, we would most likely use their system to reproduce the issue and take it from there. You will understand if we do not log into your system for free to solve a problem which so far has been reported by two people. We do take the issue seriously, but memory leaks that only occur on a specific setup that we do not have access to is nearly impossible to track down. We could read through and verify our code for a year and still not find the bug. Service agreements help sponsor the development of Varnish, in return you get priority on bugs - even the ones that are difficult to track down. We do not require anyone to pay for our service agreements to use Varnish or report bugs, and we do not ignore bug reports from non-paying Varnish users. As you may notice, we (Tollef, Poul-Henning and myself) offer a great deal of support for free on the mailing lists and on IRC, so I think it's a bit unfair to state that we do not care unless you pay for a service agreement, even if we weren't able to help in your specific case. The offer of a service agreement in the ticket was not meant to be an entry-fee to the bug tracker, but rather a means to make us prioritize a complicated bug that we would otherwise have to put on hold. I'm sorry if that didn't come across clearly. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Can you paste your VCL please? Do you use the purge_url function in it? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com I don't remember how long it took exactly, but it filled up rather quickly. Probably a couple of hours. I also noticed the LRU-moved stat was pretty constant all along, no large spikes towards the end of the swap. Anyways, it was WAY over 1GB, we managed to fill more around 6-7GB. Just for kicks, I tried changing the storage to malloc, and limit it to 1GB, to see the difference. We have live traffic now, so I don't wanna run the load testing scripts, but we should hit the 1GB limit within a day or two. -Original Message- From: Kristian Lyngstol [mailto:krist...@redpill-linpro.com] Sent: Wednesday, October 21, 2009 3:28 PM To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote: Thanks for your reply. 1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'): a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish. b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio. What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. When did it fill up and how fast? 2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM? The -s parameter only applies to storage, which is to say that it's an approximiation and there will be some additional overhead which is not stored there, ie datastructures for threads, sessions, backends etc. So with -s ..., 1G, you should expect a little bit over 1GB, but not nearly as much as you describe. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
); } # # For further details see my ticket: http://varnish.projects.linpro.no/ticket/546 @Kristian: When the programmers / engineers have some spare time over, they are always welcome to see it in live action. -Oorspronkelijk bericht- Van: Ken Brownfield [mailto:k...@slide.com] Verzonden: woensdag 21 oktober 2009 21:57 Aan: Henry Paulissen CC: varn...@projects.linpro.no Onderwerp: Re: Varnish virtual memory usage Small comments: 1) We're running Linux x86_64 exclusively here under significant load, with no memory issues. 2) Why don't you compile a 32-bit version of Varnish; wouldn't this have the same effect without the RAM and performance hit of VMs? 3) Do you make heavy use of purges? -- kb On Oct 21, 2009, at 6:22 AM, Henry Paulissen wrote: We encounter the same problem. Its seems to occur only on x64 platforms. We decided to take a different approach and installed vmware to the machine. Next we did a setup of 6 guests with x32 PAE software. No strange memory leaks occurred since then at the price of small storage (3.5G max) and limited worker threads (256 max). Opened a ticket for the problem, but the wont listen until I buy a support contract (á €8K). Seems they don’t want to know there is some kind of memory issue in their software. Anyway... Varnish is running stable now with some few tricks. Regards, -Original Message- From: varnish-misc-boun...@projects.linpro.no [mailto:varnish-misc- boun...@projects.linpro.no] On Behalf Of Kristian Lyngstol Sent: woensdag 21 oktober 2009 13:34 To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin, 1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/ varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish inside openvz: memory usage issue
Isaac Grant skrev: (..) I (think I) understand the varnish's design, but should varnish eat ALL the memory like that ? Varnish uses the VM to differentiate between what's cached in memory and what is put on disk. If the VM doesn't have any other use for the memory it will give it to varnish. The problem is that you are running VM inside a box (the openvz container) which artificially limits the memory usage. If this was a sane VM it would probably page out a lot of Varnish cache and not run out. I guess this is a bug or a weakness of openvz, first and foremost. You could work around the problem by limiting the cache size. I don't think you will encounter the same problem with Xen or any other virtualization solution where you have a separate VM for your VPS. Per. signature.asc Description: OpenPGP digital signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
varnish inside openvz: memory usage issue
Hi, I'm running varnish inside an openvz's container. Recently, I started to see problems with varnish using all available memory. It began to happen when we started delivering bigger files: videos (10 to 50MB). Inside a virtual machine having a total 4GB of virtual memory, varnish will eat all memory in less than one hour, at 20-30 req/s. When all memory is used by varnishd, it stops to deliver content. No more process can fork, the only solution is to kill varnishd from outside the VM.. I use version 1.1.2 and tried 2.0-tp2 with similar result. I tested with a malloc and file storage, same result again. I (think I) understand the varnish's design, but should varnish eat ALL the memory like that ? ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Memory Usage when using malloc
I need some help to understand this I have a 64bit server with 32GB of RAM and 60GB SWAP And varnish is running with these parameters /usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t 120 -u varnish -g varnish -p thread_pool_add_delay 100 -p thread_pool_timeout 600 -p client_http11 on -p lru_interval 3600 -s malloc,30G -P /var/run/varnish.pid malloc'ed 30G of memory.. So in theory I should not even need swap. But [EMAIL PROTECTED] ~]# free -m total used free sharedbuffers cached Mem: 32242 31735507 0 0 92 -/+ buffers/cache: 31642600 Swap:59871 51322 8549 The machine is using 51GB of swap and is swapping in pages frequently [EMAIL PROTECTED] ~]# vmstat 1 procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 0 0 52549748 446992780 96284 29 1148 15711 4 2 92 1 0 0 0 52549708 530360780 96348 2800 280 0 9654 6935 4 8 87 1 0 0 0 52549708 528160780 9642000 0 0 6785 7560 2 2 96 0 0 0 0 52549664 526668780 96316 3160 316 0 6570 7350 2 2 95 1 0 0 0 52549648 524932780 96396 1040 104 0 6300 7456 2 1 96 0 0 0 0 52549644 523372780 96400 28028 0 6013 7099 2 2 96 0 0 0 0 52549640 521884788 96424 2802828 5649 7343 2 2 96 0 0 [EMAIL PROTECTED] ~]# varnishstat -1 client_conn2089633996 362.23 Client connections accepted client_req 8345199749 1446.62 Client requests received cache_hit 8289156139 1436.90 Cache hits cache_hitpass1610 0.00 Cache hits for pass cache_miss 53623537 9.30 Cache misses backend_conn 53621117 9.30 Backend connections success backend_fail 4031 0.00 Backend connections failures backend_reuse 0 0.00 Backend connections reuses backend_recycle 0 0.00 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr7689 . N struct srcaddr n_srcaddr_act4164 . N active struct srcaddr n_sess_mem 59601 . N struct sess_mem n_sess 64719 . N struct sess n_object 2025596 . N struct object n_objecthead 2025576 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn106 . N struct vbe_conn n_bereq96 . N struct bereq n_wrk 78 . N worker threads n_wrk_create15968 0.00 N worker threads created n_wrk_failed0 0.00 N worker threads not created n_wrk_max 19191968933.27 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 21012861936.43 N overflowed work requests n_wrk_drop 31202342 5.41 N dropped work requests n_backend 1 . N backends n_expired31600718 . N expired objects n_lru_nuked 20008124 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 779599235 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 196 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 6674989721 1157.09 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 2073648637 359.46 Total Sessions s_req 8345234372 1446.62 Total Requests s_pipe 0 0.00 Total pipe s_pass 0 0.00 Total pass s_fetch 53620083 9.29 Total fetch s_hdrbytes 2924438007698506943.34 Total header bytes s_bodybytes 13839780950697 2399088.22 Total body bytes sess_closed 54123498 9.38 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 42322579 7.34 Session Read Ahead sess_herd 8270331676 1433.64 Session herd shm_records 347828089203 60295.05 SHM records shm_writes14631506809 2536.33 SHM writes shm_flushes908314 0.16 SHM flushes due to overflow shm_cont 14245861 2.47 SHM MTX contention sm_nreq 0 0.00 allocator requests sm_nobj
