Re: Strange different behavior
In message 20100114215025.gb9...@kjeks.kristian.int, Kristian Lyngstol writes
:
Vary on User-Agent is generally bad, and you should Just Fix That [tm].
Apart from the compatibility issue, a secondary reason it is a bad
idea, is that User-Agent is practically unique for every single PC
in the world, so you will cache up to hundreds of copies of the pages
for no good reason.
If your site is running live on Varnish, try running:
varnishtop -i rxheader -I User-Agent
and see how many different strings your clients send you...
In all likelyhood, your backend looks at only one or two of the bits
in User-Agent (MSIE or Mozilla ?) but Varnish has to look at the
entire string, since it has no way of knowing what your backend
looks at.
One workaround, is to do what we call User-Agent-Washing, where
Varnish rewrites the Useragent to the handfull of different variants
your backend really cares about, along the lines of:
sub vcl_recv {
if (req.http.user-agent ~ MSIE) {
set req.http.user-agent = MSIE;
} else {
set req.http.user-agent = Mozilla;
}
}
So that you only cache the relevant number of copies.
But as Kristian says: The best thing, is to not Vary on User-Agent
in the first place, that's how the InterNet is supposed to work.
Poul-Henning
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
OK. But if your application backend really doesn't do anything different for different user agents, then one should probably remove the user-agent? On Fri, Jan 15, 2010 at 7:52 AM, Poul-Henning Kamp p...@phk.freebsd.dk wrote: In message b6b8b6b71001150449v40d1b9c3s5b86dbd27cc59...@mail.gmail.com, John Norman writes: Sorry to be so obtuse: So with the default setup, there will be a cached copy of a page for every single user agent? Yes, unless you do something about the Vary: User-Agent header returned from the backend. If so, does anyone have a good number of user agents that should be supported for calculating the size of the cache? E.g., if I've guessed 64M for my pages, and I imagine that there are 10 user agents (I know it's more) then I'd want to multiply that 64M x 10. You really need to find out what bit of user-agent your backend cares about. We are talking a multiplication factor of 100-1000 here. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
Poul-Henning Kamp wrote: You really need to find out what bit of user-agent your backend cares about. We are talking a multiplication factor of 100-1000 here Very slightly off-topic, but is it possible to vary based on a cookie? I'd rather leave one of our applications to process the user-agent, login credentials etc, than to move that logic into Varnish. Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
In message b6b8b6b71001150646w7f3ba876y30401d85f1813...@mail.gmail.com, John Norman writes: OK. But if your application backend really doesn't do anything different for different user agents, then one should probably remove the user-agent? yes, by all means do so. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
2010/1/15 Rob S rtshils...@gmail.com:
Poul-Henning Kamp wrote:
You really need to find out what bit of user-agent your backend
cares about. We are talking a multiplication factor of 100-1000 here
Very slightly off-topic, but is it possible to vary based on a cookie?
I'd rather leave one of our applications to process the user-agent,
login credentials etc, than to move that logic into Varnish.
You can do this by extracting the value of a cookie into its own
header, then vary on that. e.g:
sub vcl_recv {
...
# We only care about the language in the I18N_LANGUAGE cookie
if (req.http.Cookie ~ (^|.*; )I18N_LANGUAGE=) {
set req.http.Accept-Language = regsub(req.http.Cookie, (^|.*;
)I18N_LANGUAGE=([^;]*)(; .*|$), \2);
# XXX need to work out the proper way to match here, e.g. en
set req.http.Accept-Language = regsub(req.http.Accept-Language,
^.(.*).$, \1);
} else {
set req.http.Accept-Language = en;
}
...
}
Laurence
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
On Wed, Jan 13, 2010 at 06:45:26PM +0100, Bernardf FRIT wrote: Hi, I'm facing a pretty strange issue. URLs not supposed to be cached are in cache when requested by firefox but not by lwp. (...) Vary: Accept-Encoding,User-Agent This is why. Vary: means this page will look different depending on the following client headers. Your backend is telling Varnish to expect different variants of the page depending on what Accept-Encoding AND User-Agent header the client provides. Accept-Encoding is normal and sane (ie: if one client supports gzip and an other does not, one client will get a compressed variant of the page and the other client the uncompressed one). Vary on User-Agent is generally bad, and you should Just Fix That [tm]. Due to Vary: Accept-Encoding, lwp-request and Firefox will still get different variants, but you can supply the Accept-Encoding header to lwp-request, but if you want the content, you'll need to pipe it through gunzip (headers are still readable). -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpvY82yV0SD3.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Strange different behavior
Hi,
I'm facing a pretty strange issue. URLs not supposed to be cached are in
cache when
requested by firefox but not by lwp.
I'm suspecting cookie management... but I don't realy know how to avoid
caching
these URLs. Any help would be apreciated.
URL like XX.html are XXX.php rewrited.
In vcl I have :
if (req.request == GET req.url ~ \.(html|php)) {
pass;
}
And when requesting pages :
1. Under Linux/GET
# GET -m HEAD
_*http://XXX/annonces-immobilier/appartement-roanne/11.html*_
200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Connection: close
_*Date: Wed, 16 Dec 2009 07:26:46 GMT*_
Pragma: no-cache
Via: 1.1 varnish
Age: 0
Server: Apache-NSCA
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Wed, 16 Dec 2009 07:26:20 GMT
Client-Response-Num: 1
Set-Cookie: PHPSESSID=36e4f91a2e8ae9b1aa00d819c06e03a8; path=/
Set-Cookie: DYNSRV=s0; path=/
_*X-Cache: MISS*_
X-Served-By: Server 203
X-Varnish: 1411446250
2. Under Windows/Firefox
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Type: text/html
Content-Length: 11758
X-Cacheable: YES
_*Date: Wed, 16 Dec 2009 07:30:21 GMT*_
X-Varnish: 1411446251 1411446175
Age: 570
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: Server 203
_*X-Cache: HIT*_
X-Cache-Hits: 1
Server: Apache-NSCA
3. Under Linux/GET
# GET -m HEAD
_*http://XXX/annonces-immobilier/appartement-roanne/11.html*_
200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Connection: close
_*Date: Wed, 16 Dec 2009 07:31:34 GMT*_
Pragma: no-cache
Via: 1.1 varnish
Age: 0
Server: Apache-NSCA
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Wed, 16 Dec 2009 07:31:08 GMT
Client-Response-Num: 1
Set-Cookie: PHPSESSID=fd043b9a9207d7137f79777970a54735; path=/
Set-Cookie: DYNSRV=s0; path=/
_*X-Cache: MISS*_
X-Served-By: Server 203
Regards
--
Bernard FRIT
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Strange different behavior
Hi,
I'm facing a pretty strange issue. URLs not supposed to be cached are in
cache when
requested by firefox but not by lwp.
I'm suspecting cookie management... but I don't realy know how to avoid
caching
these URLs. Any help would be apreciated.
URL like XX.html are XXX.php rewrited.
In vcl I have :
if (req.request == GET req.url ~ \.(html|php)) {
pass;
}
And when requesting pages :
1. Under Linux/GET
# GET -m HEAD
_*http://XXX/annonces-immobilier/appartement-roanne/11.html*_
200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Connection: close
_*Date: Wed, 16 Dec 2009 07:26:46 GMT*_
Pragma: no-cache
Via: 1.1 varnish
Age: 0
Server: Apache-NSCA
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Wed, 16 Dec 2009 07:26:20 GMT
Client-Response-Num: 1
Set-Cookie: PHPSESSID=36e4f91a2e8ae9b1aa00d819c06e03a8; path=/
Set-Cookie: DYNSRV=s0; path=/
_*X-Cache: MISS*_
X-Served-By: Server 203
X-Varnish: 1411446250
2. Under Windows/Firefox
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Type: text/html
Content-Length: 11758
X-Cacheable: YES
_*Date: Wed, 16 Dec 2009 07:30:21 GMT*_
X-Varnish: 1411446251 1411446175
Age: 570
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: Server 203
_*X-Cache: HIT*_
X-Cache-Hits: 1
Server: Apache-NSCA
3. Under Linux/GET
# GET -m HEAD
_*http://XXX/annonces-immobilier/appartement-roanne/11.html*_
200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Connection: close
_*Date: Wed, 16 Dec 2009 07:31:34 GMT*_
Pragma: no-cache
Via: 1.1 varnish
Age: 0
Server: Apache-NSCA
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Wed, 16 Dec 2009 07:31:08 GMT
Client-Response-Num: 1
Set-Cookie: PHPSESSID=fd043b9a9207d7137f79777970a54735; path=/
Set-Cookie: DYNSRV=s0; path=/
_*X-Cache: MISS*_
X-Served-By: Server 203
Regards
--
Bernard FRIT
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
Ooouups, sorry for the double... -- BF ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
