Re: obj.cacheable vs expires headers?
Hi!
It's seems to be correct. Be careful with cacheable and ttl=0 answers.
Some simple debuging:
sub vcl_fetch {
# Varnish determined the object was not cacheable
if (!obj.cacheable) {
set obj.http.X-Cacheable = "No";
} elsif (obj.ttl > 0s) {
set obj.http.X-Cacheable = "Yes";
} else {
set obj.http.X-Cacheable = "Yes: ttl=0";
}
[...]
}
Good luck!
On dl, feb 01, 2010 at 09:42:58 +0100, Luc Stroobant wrote:
> Hello list,
>
> I have a vcl-config with a snippet like the one on:
> http://varnish-cache.org/wiki/VCLExampleLongerCaching
>
> sub vcl_fetch {
> if (obj.cacheable) {
>unset obj.http.expires;
>set obj.http.cache-control = "max-age = 900";
>set obj.ttl = 2w;
>set obj.http.magicmarker = "1";
> }
> }
>
> sub vcl_deliver {
> if (resp.http.magicmarker) {
>/* unset marker and serve it for upstream as new */
>unset resp.http.magicmarker;
>set resp.http.age = "0";
> }
> }
>
> We used that in an attempt to override the expires headers from static
> files for a site and keep them in Varnish cache. We don't want to cache
> an dynamic (PHP) page.
>
> At first sight, everything went fine. But once the load rised a bit, we
> noticed that dynamic pages are sometimes cached.
> I've been checking the headers and it looks like the expires header for
> dynamic pages is also removed. I don't think the headers from the
> (first) request below should be considerd obj.cacheable according to
> the definition on http://varnish-cache.org/wiki/VCL
> Or did I miss something? Is this a mistake in my config or a possible
> Varnish issue?
>
> Test without the vcl snippet.
> PHP page: unmodified headers and not cached, as expected:
>
> HTTP/1.1 200 OK
> Server: Apache/2.2.3 (CentOS)
> X-Powered-By: PHP/5.2.12
> Set-Cookie: SESSbc5f9ce1c97eee1824d1ab6sdfsdfssf70k2hqq7mgo6;
> expires=Wed, 24-Feb-2010 22:58:21 GMT; path=/; domain=removed
> Expires: Sun, 19 Nov 1978 05:00:00 GMT
> Last-Modified: Mon, 01 Feb 2010 19:25:01 GMT
> Cache-Control: store, no-cache, must-revalidate
> Cache-Control: post-check=0, pre-check=0
> Content-Type: text/html; charset=utf-8
> Content-Length: 56194
> Date: Mon, 01 Feb 2010 19:25:02 GMT
> X-Varnish: 963899298
> Age: 0
> Via: 1.1 varnish
> Connection: close
>
>
> With the config:
> PHP page, Expires header removed and cache-control is set
>
> HTTP/1.1 200 OK
> Server: Apache/2.2.3 (CentOS)
> X-Powered-By: PHP/5.2.12
> Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ce3057b=91dpgvghvkmonso1;
> expires=Wed, 24-Feb-2010 23:21:04 GMT; path=/; domain=removed
> Last-Modified: Mon, 01 Feb 2010 19:47:44 GMT
> Content-Type: text/html; charset=utf-8
> Content-Length: 59418
> cache-control: max-age = 900
> Date: Mon, 01 Feb 2010 19:47:48 GMT
> X-Varnish: 1790685114
> Via: 1.1 varnish
> Connection: close
> age: 0
>
> I used the same VCL-code in vcl_fetch for both examples and for static
> files, everything works as expected. Any ideas?
> (Running on Varnish 2.0.6)
>
> Luc
>
> sub vcl_recv {
>
> # Add a unique header containing the client address
> remove req.http.X-Forwarded-For;
> setreq.http.X-Forwarded-For = client.ip;
>
> /* Don't cache accept */
> if (req.http.host ~ "^some.host$") {
>return (pass);
> }
>
> if (req.url ~ "\.(css|js|jpg|jpeg|gif|ico|png)$") {
> unset req.http.cookie;
> lookup;
> }
>
> /* Never cache php files... */
> if (req.url ~ ".*\.php") {
> return (pass);
> }
>
> /* ... or Apache server status */
> if (req.url ~ ".*/server-status$") {
> return (pass);
> }
> }
> ___
> varnish-misc mailing list
> varnish-misc@projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
--
Antoni Villalonga i Noceras / root / New Technologies / Dorna Sports S.L.
Tel. +34 934 702 885 / Fax. +34 934 737 779
Narcís Monturiol 2, 08960, Sant Just Desvern - Spain
http://www.motogp.com
http://www.dorna.com
* DISCLAIMER *
This message is intended exclusively for the named person. It may
contain confidential, propietary or legally privileged information. No
confidentiality or privilege is waived or lost by any mistransmission.
If you receive this message in error, please immediately delete it and
all copies of it from your system, destroy any hard copies of it an
notify the sender. Your must not, directly or indirectly, use,
disclose, distribute, print, or copy any part of this message if you
are not the intended recipient.
Please note that internet e-mail neither guarantees the
confidentiality nor the proper receipt of the message sent. If the
addressee of this message does not consent to the use of internet
e-mail, please communicate it to us immediately.
AVISO LEGAL ***
Re: obj.cacheable vs expires headers?
Hello,
Antoni Villalonga wrote:
> It's seems to be correct. Be careful with cacheable and ttl=0 answers.
>
> Some simple debuging:
> sub vcl_fetch {
> # Varnish determined the object was not cacheable
> if (!obj.cacheable) {
> set obj.http.X-Cacheable = "No";
> } elsif (obj.ttl > 0s) {
> set obj.http.X-Cacheable = "Yes";
> } else {
> set obj.http.X-Cacheable = "Yes: ttl=0";
> }
> [...]
> }
Thanks for your reply. I enabled the extra debugging on a test-instance,
like you proposed.
obj.ttl seems to be zero indeed, but I still don't get how his would
make the request cacheable. At least it's not the behaviour one would
expect?
Secondly: I also thought that Varnish never caches requests with a
Set-cookie header?
Or is it just the header that causes problems with intermediate proxies
between our server and the users (and/or browser cache of the clients)?
Request sent through Varnish:
HTTP/1.1 200 OK
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.12
Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ce3057b14; expires=Wed, 03-Mar-
2010 23:24:42 GMT; path=/; domain=removed
Last-Modified: Mon, 08 Feb 2010 19:49:49 GMT
ETag: "8060981009417cf0496649018d6535fa"
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Content-Length: 12955
X-Cacheable: Yes: ttl=0
cache-control: max-age = 900
Date: Mon, 08 Feb 2010 19:51:22 GMT
X-Varnish: 1349734797
Via: 1.1 varnish
Connection: close
age: 0
Direct request:
HTTP/1.1 200 OK
Date: Mon, 08 Feb 2010 19:53:56 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.12
Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ebj735pffubr1e86; expires=Wed,
03-Mar-2010 23:27:16 GMT; path=/; domain=removed
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Mon, 08 Feb 2010 19:53:56 GMT
Cache-Control: store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Connection: close
Content-Type: text/html; charset=utf-8
If this is wanted behaviour, I would propose to add a warning on
http://varnish-cache.org/wiki/VCLExampleLongerCaching
The config can cause serious privacy problems and other weird things on
sites with logged in users.
Luc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
In message <4b707314.5090...@gmail.com>, Luc Stroobant writes: >obj.ttl seems to be zero indeed, but I still don't get how his would >make the request cacheable. At least it's not the behaviour one would >expect? We distinguish between "can be cached" and "how long should it be cached" because they are very different questions. "can be cached" is a matter of correctness, whereas "how long" is just a performance issue. >Secondly: I also thought that Varnish never caches requests with a >Set-cookie header? Varnish has a special kind of cache entries called "hit-for-pass". This is a cache entry that says that the object can not be cached, that solves a pile-up issue on busy objects. The fact that you see zero TTL, can be indicative of the clock on the varnish-host and the clock on the backend not agreeing what time it is. Check your ntpd. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
Poul-Henning Kamp wrote: > In message <4b707314.5090...@gmail.com>, Luc Stroobant writes: > >> obj.ttl seems to be zero indeed, but I still don't get how his would >> make the request cacheable. At least it's not the behaviour one would >> expect? > > We distinguish between "can be cached" and "how long should it be cached" > because they are very different questions. > > "can be cached" is a matter of correctness, whereas "how long" is > just a performance issue. So you mean: the object /can/ be cached, but it would only be cached for zero seconds? >> Secondly: I also thought that Varnish never caches requests with a >> Set-cookie header? > > Varnish has a special kind of cache entries called "hit-for-pass". > This is a cache entry that says that the object can not be cached, > that solves a pile-up issue on busy objects. > > The fact that you see zero TTL, can be indicative of the clock on > the varnish-host and the clock on the backend not agreeing what > time it is. Check your ntpd. I don't think it's a clock issue. Varnish and the webserver are running on the same host. We wanted to use Varnish to cache some static files on an overloaded prefork Apache + mod-php (and it did a great job, till we started to see weird session issues). Luc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
In message <4b707680.6020...@gmail.com>, Luc Stroobant writes: >Poul-Henning Kamp wrote: >> "can be cached" is a matter of correctness, whereas "how long" is >> just a performance issue. > >So you mean: the object /can/ be cached, but it would only be cached for >zero seconds? Yes, that is a valid, but silly configuration. >I don't think it's a clock issue. Varnish and the webserver are running >on the same host. We wanted to use Varnish to cache some static files on >an overloaded prefork Apache + mod-php (and it did a great job, till we >started to see weird session issues). Ok, no, it's not a clock issue then. Check your varnishlog output, it shows all headers sent/received and look for what cache-control the backend gives varnish... Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
Poul-Henning Kamp wrote: > Check your varnishlog output, it shows all headers sent/received > and look for what cache-control the backend gives varnish... The cache-control headers of the backend seem to be ok? (btw, the Varnish test-instance is running on my laptop, but the behaviour is exactly the same as it was on the production server) Luc First request: 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1265664137 1.0 8 SessionOpen c 127.0.0.1 48239 127.0.0.1:80 8 ReqStart c 127.0.0.1 48239 858012427 8 RxRequestc GET 8 RxURLc / 8 RxProtocol c HTTP/1.0 8 RxHeader c Host: www.removed 8 RxHeader c Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01 8 RxHeader c Accept-Encoding: gzip, compress, bzip2 8 RxHeader c Accept-Language: en 8 RxHeader c User-Agent: Lynx/2.8.7pre.6 libwww-FM/2.14 SSL-MM/1.4.1 8 VCL_call c recv 8 VCL_return c lookup 8 VCL_call c hash 8 VCL_return c hash 8 VCL_call c miss 8 VCL_return c fetch 9 BackendOpen b default 192.168.1.13 41812 80 8 Backend c 9 default default 9 TxRequestb GET 9 TxURLb / 9 TxProtocol b HTTP/1.1 9 TxHeader b Host: www.removed 9 TxHeader b Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01 9 TxHeader b Accept-Encoding: gzip, compress, bzip2 9 TxHeader b Accept-Language: en 9 TxHeader b User-Agent: Lynx/2.8.7pre.6 libwww-FM/2.14 SSL-MM/1.4.1 9 TxHeader b X-Forwarded-For: 127.0.0.1 9 TxHeader b X-Varnish: 858012427 9 TxHeader b X-Forwarded-For: 127.0.0.1 9 RxProtocol b HTTP/1.1 9 RxStatus b 200 9 RxResponse b OK 9 RxHeader b Date: Mon, 08 Feb 2010 21:22:19 GMT 9 RxHeader b Server: Apache/2.2.3 (CentOS) 9 RxHeader b X-Powered-By: PHP/5.2.12 9 RxHeader b Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ce3057b=aar9ormanugvueruqlpr327ka5; expires=Thu, 04-Mar-2010 00:55:39 GMT; path=/; domain=.removed 9 RxHeader b Last-Modified: Mon, 08 Feb 2010 21:19:29 GMT 9 RxHeader b ETag: "8f386ae81f34f492ba4abde82d8ef425" 9 RxHeader b Expires: Sun, 19 Nov 1978 05:00:00 GMT 9 RxHeader b Cache-Control: must-revalidate 9 RxHeader b Content-Encoding: gzip 9 RxHeader b Transfer-Encoding: chunked 9 RxHeader b Content-Type: text/html; charset=utf-8 8 ObjProtocol c HTTP/1.1 8 ObjStatusc 200 8 ObjResponse c OK 8 ObjHeaderc Date: Mon, 08 Feb 2010 21:22:19 GMT 8 ObjHeaderc Server: Apache/2.2.3 (CentOS) 8 ObjHeaderc X-Powered-By: PHP/5.2.12 8 ObjHeaderc Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ce3057b=aar9ormanugvueruqlpr327ka5; expires=Thu, 04-Mar-2010 00:55:39 GMT; path=/; domain=.removed 8 ObjHeaderc Last-Modified: Mon, 08 Feb 2010 21:19:29 GMT 8 ObjHeaderc ETag: "8f386ae81f34f492ba4abde82d8ef425" 8 ObjHeaderc Expires: Sun, 19 Nov 1978 05:00:00 GMT 8 ObjHeaderc Cache-Control: must-revalidate 8 ObjHeaderc Content-Encoding: gzip 8 ObjHeaderc Content-Type: text/html; charset=utf-8 9 BackendReuse b default 8 TTL c 858012427 RFC 0 1265664139 1265664139 280299600 0 0 8 VCL_call c fetch 8 TTL c 858012427 VCL 1209600 1265664139 8 VCL_return c deliver 8 Length c 10987 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 200 8 TxResponse c OK 8 TxHeader c Server: Apache/2.2.3 (CentOS) 8 TxHeader c X-Powered-By: PHP/5.2.12 8 TxHeader c Set-Cookie: SESSbc5f9ce1c97eee1824d1ab670ce3057b=aar9ormanugvueruqlpr327ka5; expires=Thu, 04-Mar-2010 00:55:39 GMT; path=/; domain=.removed 8 TxHeader c Last-Modified: Mon, 08 Feb 2010 21:19:29 GMT 8 TxHeader c ETag: "8f386ae81f34f492ba4abde82d8ef425" 8 TxHeader c Content-Encoding: gzip 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c Content-Length: 10987 8 TxHeader c X-Cacheable: Yes: ttl=0 8 TxHeader c cache-control: max-age = 900 8 TxHeader c Date: Mon, 08 Feb 2010 21:22:19 GMT 8 TxHeader c X-Varnish: 858012427 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Connection: close 8 TxHeader c age: 0 8 ReqEnd c 858012427 1265664139.376405954 1265664139.577864408 0.000237465 0.201274157 0.000184298 8 SessionClose c not HTTP/1.1 8 StatSess c 127.0.0.1 48239 0 1 1 0 0 1 556 10987 0 StatAddr - 127.0.0.1 0 0 1 1 0 0 1 556 10987 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1265664140 1.0 A second request: 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1265664528 1.0 0 CLI - Rd ping 0 CLI
Re: obj.cacheable vs expires headers?
In message <4b708469.9070...@gmail.com>, Luc Stroobant writes: >Poul-Henning Kamp wrote: > >> Check your varnishlog output, it shows all headers sent/received >> and look for what cache-control the backend gives varnish... > >The cache-control headers of the backend seem to be ok? >(btw, the Varnish test-instance is running on my laptop, but the >behaviour is exactly the same as it was on the production server) > 9 RxHeader b Expires: Sun, 19 Nov 1978 05:00:00 GMT This is your problem: the backend says that the object is already expired, so obviously we cannot cache it for any amount of time. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
- "Poul-Henning Kamp" wrote: > > 9 RxHeader b Expires: Sun, 19 Nov 1978 05:00:00 GMT > > This is your problem: the backend says that the object is already > expired, so obviously we cannot cache it for any amount of time. I'm not sure we're still talking about the same problem. :-) Just to be sure: we don't want to cache that dynamic page, but I'm wondering why (with those) headers, it's still seen as obj.cacheable by Varnish. If you check the log of the second request in my previous mail, you even see a message " 8 VCL_call c hit" which seems to suggest the object is served from cache? (cache-control: max-age = 900 is also set) Luc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.cacheable vs expires headers?
In message <15261481.301265709947078.javamail@syntop>, lstroob...@gmail.com writes: >I'm not sure we're still talking about the same problem. :-) >Just to be sure: we don't want to cache that dynamic page, but I'm >wondering why (with those) headers, it's still seen as obj.cacheable >by Varnish. As I said, the cacheability is a different decision from how long time to cache. The logic of the code is in the sourcefile "rfc2616.c", try to walk your request through it by hand... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
