Hi Lasse, thanks for the prompt reply. I've done a little bit more
investigation on this issue.
1. I tried using smart bans but I saw no change in performance. I tried change
the ban from a regex to a straight match on the URL (i.e. from
ban("obj.http.x-url ~ " + [header containing regex]) to ban("obj.http.x-url ==
" + [header containing url]) ) and I still observed the same performance
issues. I also tried changing from ban() to ban_url() - no performance
improvement. Removing that line from the VCL brings the CPU right back down, so
I've concluded the problem is adding the ban to the ban list. Is this an
inherently CPU intensive operation, is there going to be no way around this?
2. I've also gone down the route of trying purges because they are
significantly faster and look like they can handle the load okay, but my only
issue is matching variations of a page. My vcl_hash function looks like this:
sub vcl_hash {
hash_data(req.http.Accept);
hash_data(req.http.X-Forwarded-Proto);
}
So the problem I'm having is purging all variations of the file based on
"Accept" and "X-Forwarded-Proto" headers, as well as the host. Is this
something that is even possible? I personally wouldn't have thought so but
after seeing in the documentation
(https://www.varnish-cache.org/docs/trunk/tutorial/purging.html): "The purge
in vcl_miss is necessary to purge all variants in the cases where you hit an
object, but miss a particular variant." Seems to imply that this functionality
is present but I can't find any more documentation on how it works. Do I need
to remove the hash_data() calls in vcl_hash and instead include "Accept" and
"X-Forwarded-Proto" in the "Vary" header coming from the backend server?
Thanks,
Connor Walls
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Lasse Karstensen
Sent: 26 July 2012 10:36
To: [email protected]
Subject: Re: High CPU usage when using ban with HTTP PURGE
Connor Walls:
> So, I've been having an issue with CPU usage recently. We run a
> cluster of
> 3 varnish boxes, running 3.0.2, serving content on a particular port,
> and accepting HTTP PURGE request on another (:82). This is used so
> whenever a user on the site changes something on their profile page,
> one of the backend servers will send a HTTP PURGE request to each box
> in the cluster, containing a regex in the header, which is then used
> to ban based on a particular URL - the relevant VCL is as follows:
[..]
> Now what we've been seeing is very high CPU usage with this
> functionality, which has been slowly growing over time with increased
> traffic on the website, and is now averaging above 70%, peaking at
> 100% at times and causing some user requests to fail. The CPU usage
> correlates very closely with timeout exceptions we get when making the
> HTTP PURGE requests. This became an issue yesterday to the point where
> we disabled the HTTP PURGE - CPU usage has now fallen from averaging >
> 70% to averaging around 8%, so clearly the purges are the issue. What
> I'm wondering is whether or not banning like this is going to be
> inherently heavy and I need to find a way to throttle the purge
> requests, or if I could rework the VCL so it will be less CPU heavy to
> process these requests? Will the CPU usage depend on what the actual regex is?
There are two methods for cache invalidation in Varnish 3.0: Purges and bans.
Bans are flexible and cool, and purges are less flexible but very fast. Use
purges if you can.
If you are using a high amount of bans, you should use what Kristian calls
"Smart bans" in the Varnish Book:
https://www.varnish-software.com/static/book/Cache_invalidation.html#smart-bans
--
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/
_______________________________________________
varnish-misc mailing list
[email protected]
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
_______________________________________________
varnish-misc mailing list
[email protected]
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
