Re: Not seeing a successful purge
]] John Norman
| I think it's the backend's (Apache/Passenger) header:
|
| Vary: Accept-Encoding,User-Agent
|
| Which seems to prevent (???) this from working in my vcl_hash:
|
| if (req.http.Accept-Encoding ~ gzip) {
| set req.hash += gzip;
| } // etc
No, that works fine, but is useless. Varnish does Vary handling
automatically.
| The Varnish doc says: But by default, Varnish will perform no
| transforms on the headers singled out by Vary: for comparison
| (http://varnish-cache.org/wiki/ArchitectureVary).
Yes, this means that if you know your backend only cares about
gzip/non-gzip, you should normalise the header in vcl_recv. Varnish
can't know this, as it requires knowledge of your backend and
application.
However, even if you changed that, you still have a Vary on
user-agent. This is generally a really bad idea, and you should try to
see if you can vary on something else. If you don't, you are going to
see poor hit rates. (Assuming that you're not just caching for a
corporate intranet where everybody has _exactly_ the same versions of
browser, etc.)
| So . . . I'm not sure what I should do. If the backend says Vary for
| Accept-Encoding, does that mean that I should or should not be able to
| access that header for the purposes of setting the hash?
Normalise Accept-Encoding, stop adding it to the hash.
| What I am observing is:
|
| browser makes request with Accept-Encoding: gzip,deflate
|
| When I try to purge, the request with the purge says: Accept-Encoding:
| gzip,identity
|
| Even though gzip is in the request for both the browser
| Accept-Encoding, and for the purge, they seem to be getting hashed
| differently.
No, they hash to the same object but as the Vary-ed string differs the
Vary handling makes it out to be different objects.
--
Tollef Fog Heen
Redpill Linpro -- Changing the game!
t: +47 21 54 41 73
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Not seeing a successful purge
Hi,
Your PURGE request is getting a different hash than your browser
requests because there is no Accept-Encoding header on the PURGE. (You
see the same problem when using Vary on the response). See
http://varnish-cache.org/wiki/Purging. You can either use
purge(req.url ~ req.url); in vcl_recv, or send multiple PURGE
requests with each of the relavant Accept-Encoding values.
Laurence
On 11 February 2010 23:25, John Norman j...@7fff.com wrote:
Hi, folks.
I'm trying to purge with the pseudo HTTP PURGE method Varnish supports.
I do seem to have a cached page, but the PURGE response suggests that
it's missing.
So . . . any idea why the PURGE isn't working?
In my VCL, my vcl_hash looks like this (I intend it to only hash on
the request URL and compression):
sub vcl_hash {
set req.hash += req.url;
if (req.http.Accept-Encoding ~ gzip) {
set req.hash += gzip;
} else if (req.http.Accept-Encoding ~ deflate) {
set req.hash += deflate;
}
return (hash);
}
And the checks for PURGE look like this (full VCL way below):
sub vcl_hit {
if (req.request == PURGE) {
set obj.ttl = 0s;
error 200 Purged.;
}
if (!obj.cacheable) {
pass;
}
deliver;
}
sub vcl_miss {
if (req.request == PURGE) {
error 404 Not in cache.;
}
}
And in vcl_recv:
if (req.request == PURGE) {
lookup;
}
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Not seeing a successful purge
Here's a bit more on my purge problem -- a comparison of a purge that works on my development machine, vs. one that doesn't work on my staging system. On both, the browser request goes to haproxy, then to varnish. The VCL is identical, as quoted in a prior e-mail. The backends are different: on my local, it's the Ruby webrick server; on staging, it's Apache+Passenger. Again, I'm not purging through varnishadm: This is using the pseudo-http-method PURGE. One thing I can say about the staging environment if that if I do a non-browser request using mechanize from the staging system itself, then the later purge DOES work. In that case, the two differences are: The user-agent is the same for both the get and the purge; and requesting IP would be the same for both the get and the purge. Here are the log details. DEVELOPMENT -- first the browser request, showing the hit; then the purge, showing the hit. Awesome! 11 ReqStart c 127.0.0.1 50808 1691945259 11 RxRequestc GET 11 RxURLc /products/sillyputty 11 RxProtocol c HTTP/1.1 11 RxHeader c Host: localhost 11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0 11 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 11 RxHeader c Accept-Language: en-us,en;q=0.5 11 RxHeader c Accept-Encoding: gzip,deflate 11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 11 RxHeader c Keep-Alive: 300 11 RxHeader c Connection: close 11 RxHeader c Referer: http://localhost/ 11 RxHeader c Cookie: remember_token=f27172bfab54dc47d20b6d8c853afb8677fa2d11 11 RxHeader c X-Forwarded-For: 127.0.0.1 11 VCL_call c recv 11 VCL_return c lookup 11 VCL_call c hash 11 VCL_return c hash 11 Hit c 1691945214 11 VCL_call c hit 11 VCL_return c deliver 11 Length c 201518 11 VCL_call c deliver 11 VCL_return c deliver 11 TxProtocol c HTTP/1.1 11 TxStatus c 200 11 TxResponse c OK 11 TxHeader c Cache-Control: max-age=8280, public 11 TxHeader c X-Runtime: 818 11 TxHeader c Content-Type: text/html; charset=utf-8 11 TxHeader c Etag: f29fbc0160d276fb97a298bf5bce8ff3 11 TxHeader c Server: WEBrick/1.3.1 (Ruby/1.9.1/2009-07-16) 11 TxHeader c Content-Length: 201518 11 TxHeader c Date: Fri, 12 Feb 2010 16:19:33 GMT 11 TxHeader c X-Varnish: 1691945259 1691945214 11 TxHeader c Age: 14 11 TxHeader c Via: 1.1 varnish 11 TxHeader c Connection: close 11 ReqEnd c 1691945259 1265991573.541040897 1265991573.547173977 0.77009 0.55075 0.006078005 11 ReqStart c ::1 51006 1691945309 11 RxRequestc PURGE 11 RxURLc /products/sillyputty 11 RxProtocol c HTTP/1.1 11 RxHeader c Accept: */* 11 RxHeader c User-Agent: WWW-Mechanize/1.0.0 (http://rubyforge.org/projects/mechanize/) 11 RxHeader c Connection: keep-alive 11 RxHeader c Keep-Alive: 300 11 RxHeader c Accept-Encoding: gzip,identity 11 RxHeader c Accept-Language: en-us,en;q=0.5 11 RxHeader c Host: localhost:8000 11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 11 VCL_call c recv 11 VCL_return c lookup 11 VCL_call c hash 11 VCL_return c hash 11 Hit c 1691945214 11 VCL_call c hit 11 TTL c 1691945214 VCL 0 1265991617 0 Debug- VCL_error(200, Purged.) 11 VCL_return c error 11 VCL_call c error 11 VCL_return c deliver 11 Length c 322 11 VCL_call c deliver 11 VCL_return c deliver 11 TxProtocol c HTTP/1.1 11 TxStatus c 200 11 TxResponse c Purged. 11 TxHeader c Server: Varnish 11 TxHeader c Retry-After: 0 11 TxHeader c Content-Type: text/html; charset=utf-8 11 TxHeader c Content-Length: 322 11 TxHeader c Date: Fri, 12 Feb 2010 16:20:16 GMT 11 TxHeader c X-Varnish: 1691945309 11 TxHeader c Age: 0 11 TxHeader c Via: 1.1 varnish 11 TxHeader c Connection: close 11 ReqEnd c 1691945309 1265991616.884471893 1265991616.884622097 0.000173807 0.99182 0.51022 --- Now my problematic STAGING system -- first the GET with the hit, then the purge that fails to hit. 4 ReqStart c 10.253.191.95 45944 904319331 4 RxRequestc GET 4 RxURLc /products/sillyputty 4 RxProtocol c HTTP/1.1 4 RxHeader c Host: staging1.example.com 4 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0 4 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 4 RxHeader c Accept-Language: en-us,en;q=0.5 4 RxHeader c Accept-Encoding: gzip,deflate 4 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 4 RxHeader c Keep-Alive: 300 4 RxHeader c Connection: keep-alive 4 RxHeader c Referer: http://staging1.example.com/ 4 RxHeader c Cookie: cehq-id=10.252.66.194.1263417178788050;
Re: Not seeing a successful purge
I think it's the backend's (Apache/Passenger) header:
Vary: Accept-Encoding,User-Agent
Which seems to prevent (???) this from working in my vcl_hash:
if (req.http.Accept-Encoding ~ gzip) {
set req.hash += gzip;
} // etc
The Varnish doc says: But by default, Varnish will perform no
transforms on the headers singled out by Vary: for comparison
(http://varnish-cache.org/wiki/ArchitectureVary).
So . . . I'm not sure what I should do. If the backend says Vary for
Accept-Encoding, does that mean that I should or should not be able to
access that header for the purposes of setting the hash?
What I am observing is:
browser makes request with Accept-Encoding: gzip,deflate
When I try to purge, the request with the purge says: Accept-Encoding:
gzip,identity
Even though gzip is in the request for both the browser
Accept-Encoding, and for the purge, they seem to be getting hashed
differently.
If when I do a purge, I force Accept-Encoding: gzip,deflate, then it
matches what the browser did exactly, and I am able to purge
successfully.
On Fri, Feb 12, 2010 at 11:54 AM, John Norman j...@7fff.com wrote:
Here's a bit more on my purge problem -- a comparison of a purge
that works on my development machine, vs. one that doesn't work on my
staging system.
On both, the browser request goes to haproxy, then to varnish. The VCL
is identical, as quoted in a prior e-mail. The backends are different:
on my local, it's the Ruby webrick server; on staging, it's
Apache+Passenger.
Again, I'm not purging through varnishadm: This is using the
pseudo-http-method PURGE.
One thing I can say about the staging environment if that if I do a
non-browser request using mechanize from the staging system itself,
then the later purge DOES work. In that case, the two differences are:
The user-agent is the same for both the get and the purge; and
requesting IP would be the same for both the get and the purge.
Here are the log details.
DEVELOPMENT -- first the browser request, showing the hit; then the
purge, showing the hit. Awesome!
11 ReqStart c 127.0.0.1 50808 1691945259
11 RxRequest c GET
11 RxURL c /products/sillyputty
11 RxProtocol c HTTP/1.1
11 RxHeader c Host: localhost
11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS
X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0
11 RxHeader c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
11 RxHeader c Accept-Language: en-us,en;q=0.5
11 RxHeader c Accept-Encoding: gzip,deflate
11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
11 RxHeader c Keep-Alive: 300
11 RxHeader c Connection: close
11 RxHeader c Referer: http://localhost/
11 RxHeader c Cookie:
remember_token=f27172bfab54dc47d20b6d8c853afb8677fa2d11
11 RxHeader c X-Forwarded-For: 127.0.0.1
11 VCL_call c recv
11 VCL_return c lookup
11 VCL_call c hash
11 VCL_return c hash
11 Hit c 1691945214
11 VCL_call c hit
11 VCL_return c deliver
11 Length c 201518
11 VCL_call c deliver
11 VCL_return c deliver
11 TxProtocol c HTTP/1.1
11 TxStatus c 200
11 TxResponse c OK
11 TxHeader c Cache-Control: max-age=8280, public
11 TxHeader c X-Runtime: 818
11 TxHeader c Content-Type: text/html; charset=utf-8
11 TxHeader c Etag: f29fbc0160d276fb97a298bf5bce8ff3
11 TxHeader c Server: WEBrick/1.3.1 (Ruby/1.9.1/2009-07-16)
11 TxHeader c Content-Length: 201518
11 TxHeader c Date: Fri, 12 Feb 2010 16:19:33 GMT
11 TxHeader c X-Varnish: 1691945259 1691945214
11 TxHeader c Age: 14
11 TxHeader c Via: 1.1 varnish
11 TxHeader c Connection: close
11 ReqEnd c 1691945259 1265991573.541040897 1265991573.547173977
0.77009 0.55075 0.006078005
11 ReqStart c ::1 51006 1691945309
11 RxRequest c PURGE
11 RxURL c /products/sillyputty
11 RxProtocol c HTTP/1.1
11 RxHeader c Accept: */*
11 RxHeader c User-Agent: WWW-Mechanize/1.0.0
(http://rubyforge.org/projects/mechanize/)
11 RxHeader c Connection: keep-alive
11 RxHeader c Keep-Alive: 300
11 RxHeader c Accept-Encoding: gzip,identity
11 RxHeader c Accept-Language: en-us,en;q=0.5
11 RxHeader c Host: localhost:8000
11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
11 VCL_call c recv
11 VCL_return c lookup
11 VCL_call c hash
11 VCL_return c hash
11 Hit c 1691945214
11 VCL_call c hit
11 TTL c 1691945214 VCL 0 1265991617
0 Debug - VCL_error(200, Purged.)
11 VCL_return c error
11 VCL_call c error
11 VCL_return c deliver
11 Length c 322
11 VCL_call c deliver
11 VCL_return c deliver
11 TxProtocol c HTTP/1.1
11 TxStatus c 200
11 TxResponse c Purged.
11 TxHeader c Server: Varnish
11 TxHeader c Retry-After: 0
11 TxHeader c Content-Type: text/html; charset=utf-8
11 TxHeader
Re: Not seeing a successful purge
For any cache, not just Varnish, the Accept-Encoding header used for the purge
(or a regular cache hit) must match the request header /exactly/. If you use
anything other than exactly Accept-Encoding: gzip,deflate your purge will
miss. So this is the expected behavior, AFAIK.
Any other header specified in Vary must also match exactly for a hit (and
optional purge) to occur.
You could do some hashing hacks to augment the standard functionality, but it
might become a bit of a mess. I haven't needed to find out, though.
--
Ken
On Feb 12, 2010, at 10:44 AM, John Norman wrote:
I think it's the backend's (Apache/Passenger) header:
Vary: Accept-Encoding,User-Agent
Which seems to prevent (???) this from working in my vcl_hash:
if (req.http.Accept-Encoding ~ gzip) {
set req.hash += gzip;
} // etc
The Varnish doc says: But by default, Varnish will perform no
transforms on the headers singled out by Vary: for comparison
(http://varnish-cache.org/wiki/ArchitectureVary).
So . . . I'm not sure what I should do. If the backend says Vary for
Accept-Encoding, does that mean that I should or should not be able to
access that header for the purposes of setting the hash?
What I am observing is:
browser makes request with Accept-Encoding: gzip,deflate
When I try to purge, the request with the purge says: Accept-Encoding:
gzip,identity
Even though gzip is in the request for both the browser
Accept-Encoding, and for the purge, they seem to be getting hashed
differently.
If when I do a purge, I force Accept-Encoding: gzip,deflate, then it
matches what the browser did exactly, and I am able to purge
successfully.
On Fri, Feb 12, 2010 at 11:54 AM, John Norman j...@7fff.com wrote:
Here's a bit more on my purge problem -- a comparison of a purge
that works on my development machine, vs. one that doesn't work on my
staging system.
On both, the browser request goes to haproxy, then to varnish. The VCL
is identical, as quoted in a prior e-mail. The backends are different:
on my local, it's the Ruby webrick server; on staging, it's
Apache+Passenger.
Again, I'm not purging through varnishadm: This is using the
pseudo-http-method PURGE.
One thing I can say about the staging environment if that if I do a
non-browser request using mechanize from the staging system itself,
then the later purge DOES work. In that case, the two differences are:
The user-agent is the same for both the get and the purge; and
requesting IP would be the same for both the get and the purge.
Here are the log details.
DEVELOPMENT -- first the browser request, showing the hit; then the
purge, showing the hit. Awesome!
11 ReqStart c 127.0.0.1 50808 1691945259
11 RxRequestc GET
11 RxURLc /products/sillyputty
11 RxProtocol c HTTP/1.1
11 RxHeader c Host: localhost
11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS
X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0
11 RxHeader c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
11 RxHeader c Accept-Language: en-us,en;q=0.5
11 RxHeader c Accept-Encoding: gzip,deflate
11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
11 RxHeader c Keep-Alive: 300
11 RxHeader c Connection: close
11 RxHeader c Referer: http://localhost/
11 RxHeader c Cookie:
remember_token=f27172bfab54dc47d20b6d8c853afb8677fa2d11
11 RxHeader c X-Forwarded-For: 127.0.0.1
11 VCL_call c recv
11 VCL_return c lookup
11 VCL_call c hash
11 VCL_return c hash
11 Hit c 1691945214
11 VCL_call c hit
11 VCL_return c deliver
11 Length c 201518
11 VCL_call c deliver
11 VCL_return c deliver
11 TxProtocol c HTTP/1.1
11 TxStatus c 200
11 TxResponse c OK
11 TxHeader c Cache-Control: max-age=8280, public
11 TxHeader c X-Runtime: 818
11 TxHeader c Content-Type: text/html; charset=utf-8
11 TxHeader c Etag: f29fbc0160d276fb97a298bf5bce8ff3
11 TxHeader c Server: WEBrick/1.3.1 (Ruby/1.9.1/2009-07-16)
11 TxHeader c Content-Length: 201518
11 TxHeader c Date: Fri, 12 Feb 2010 16:19:33 GMT
11 TxHeader c X-Varnish: 1691945259 1691945214
11 TxHeader c Age: 14
11 TxHeader c Via: 1.1 varnish
11 TxHeader c Connection: close
11 ReqEnd c 1691945259 1265991573.541040897 1265991573.547173977
0.77009 0.55075 0.006078005
11 ReqStart c ::1 51006 1691945309
11 RxRequestc PURGE
11 RxURLc /products/sillyputty
11 RxProtocol c HTTP/1.1
11 RxHeader c Accept: */*
11 RxHeader c User-Agent: WWW-Mechanize/1.0.0
(http://rubyforge.org/projects/mechanize/)
11 RxHeader c Connection: keep-alive
11 RxHeader c Keep-Alive: 300
11 RxHeader c Accept-Encoding: gzip,identity
11 RxHeader c Accept-Language: en-us,en;q=0.5
11 RxHeader c Host: localhost:8000
11 RxHeader c Accept-Charset:
