Re: X-Vary-Options patch
On lör, 2008-06-07 at 10:43 +0800, Adrian Chadd wrote: I think some of their stuff was backed out of Squid-2.7 before release. The Vary invalidation patch was backed out from 2.7 as it's incomplete and broke things. But this X-Vary-Options patch never got committed. Thread stops after your request for him to file a bugzilla entry, and it got lost in the noise until found again by Mark. I have concerns about the completeness about this patch, for example if it handles q values properly. It's not such abig deal on Accept-Encoding, but can get quite messy if applying this to Accept or Accept-Language. For Accept* heaers I think it needs to be extended with an option instructing caches to parse the Accept* header to a limited degree, which means the cache needs to know the list of available choices for the header at the server and their order of priority. Also, for cookie it needs to be a little more elaborate as most often one wants to match on cookie names, not their vaule.. and sometimes a value of a specific cookie. Regards Henrik
Re: X-Vary-Options patch
+1 to what H says. I'm definitely interested in this area, but want to think through it a bit more. We can get a certain amount of functionality without any extension; e.g., canonicalising selecting headers to take care of whitespace and case issues, and perhaps even ordering (this doesn't work for generic headers, but where we know the semantics and ordering isn't significant, it isn't a problem). That doesn't take care of the qval problem, but it helps in the accept- encoding case, which is the most common (I don't often see qvals on a- e; anybody?). One of my concerns about X-Vary-Options (can we please drop the 'X-'?) is that there are non-obvious corner cases; it forces the origin server admin to think very carefully about all the different variants that they're going to issue, and the request headers that will match them. If we can get rid of the common cases by canonicalisation, they will have less opportunity to mess things up. Also, it's important to realise that at some point it's more worthwhile to take an approach like TCN and describe the available variants, rather than match against selecting headers. Cheers, On 07/06/2008, at 5:19 PM, Henrik Nordstrom wrote: On lör, 2008-06-07 at 10:43 +0800, Adrian Chadd wrote: I think some of their stuff was backed out of Squid-2.7 before release. The Vary invalidation patch was backed out from 2.7 as it's incomplete and broke things. But this X-Vary-Options patch never got committed. Thread stops after your request for him to file a bugzilla entry, and it got lost in the noise until found again by Mark. I have concerns about the completeness about this patch, for example if it handles q values properly. It's not such abig deal on Accept-Encoding, but can get quite messy if applying this to Accept or Accept-Language. For Accept* heaers I think it needs to be extended with an option instructing caches to parse the Accept* header to a limited degree, which means the cache needs to know the list of available choices for the header at the server and their order of priority. Also, for cookie it needs to be a little more elaborate as most often one wants to match on cookie names, not their vaule.. and sometimes a value of a specific cookie. Regards Henrik -- Mark Nottingham [EMAIL PROTECTED]
Re: X-Vary-Options patch
Squid devs: Did this make its way into 2.7? Tim: AIUI, the following header: X-Vary-Options: Accept-Encoding; list-contains=gzip will bucket the cache for this URI into two entries; those whose Accept-Encoding contains the list value gzip, and those that don't. Is that correct? Also, I'm assuming the origin would still need to send Vary: Accept-Encoding to work properly with downstream caches. Cheers, On 26/02/2008, at 3:30 PM, Adrian Chadd wrote: G'day, I'm happy to commit this to Squid-2.HEAD as-is. Can you throw it in a Bugzilla report and spit me the number? Thanks, Adrian On Fri, Feb 08, 2008, Tim Starling wrote: There are two major sources of suboptimal hit rate on Wikipedia which relate to the Vary header: * In Accept-Encoding, we only care whether gzip is present or not, but IE and Firefox use different whitespace conventions and so each get separate entries in the cache * We only care whether the user is logged in or not. Other cookies, such as pure-JavaScript cookies used by client-side code to store preferences, unnecessarily degrade our hit rate. There have been other patches related to this problem, but as far as I'm aware, they're all special-case, site-specific hacks. My patch adds an X-Vary-Options response header (hereafter XVO), and thus gives the origin server fine control over cache variance. In the patch, the XVO header overrides the Vary header, so the Vary header can still be sent as usual for compatibility with caches that don't support this feature. The format of the XVO header is inspired by the format of the Accept header. As in Vary, XVO is separated by commas into parts which relate to different request headers. Then those parts are further separated by semicolons. The first semicolon-separated part is the request header name, and subsequent parts give name/value pairs separated by equals signs, defining options relating to the variance of that header. Two option names are currently defined: list-contains: splits the request header into comma-separated parts and varies depending on whether the resulting list contains the option value string-contains: performs a simple search of the request header and varies depending on whether it matches. Multiple such options per header are allowed. So for example: X-Vary-Options: Cookie; string-contains=UserID; string-contains=_session, Accept-Encoding; list-contains=gzip This would vary the cache on three tests: * whether the Cookie header contains the string UserID * whether the Cookie header contains the string _session * whether the Accept-Encoding header, interpreted as a comma- separated list, contains the item gzip The patch refactors all references to the Vary and X-Accelerator-Vary headers into the functions httpHeaderHasVary() and httpHeaderGetVary() in HttpHeader.c. It then adds X-Vary-Options to these functions, interpreting it as a string rather than a list to avoid inappropriate splitting on whitespace. It puts the resulting combined header into X-Vary-Options instead of Vary in the base vary marker object, again to avoid inappropriate list-style interpretation. httpMakeVaryMark() then interprets this combined header in the way described above. The added features of the patch are conditional, and are enabled by the configure option --enable-vary-options. Autoconf and automake will need to be run after applying this patch. The interpretation of some non-standards-compliant Vary headers (those containing semicolons) is changed slightly by this patch regardless of --enable-vary-options. The patch is attached and also available at: http://noc.wikimedia.org/~tstarling/patches/vary_options_upstream.patch For your review and consideration. -- Tim Starling diff -Xdiffx -ru squid-2.6.18.orig/configure.in squid-2.6.18/ configure.in --- squid-2.6.18.orig/configure.in2008-01-10 23:34:23.0 +1100 +++ squid-2.6.18/configure.in 2008-02-07 19:43:23.0 +1100 @@ -1507,6 +1507,16 @@ fi ]) +dnl Enable vary options +AC_ARG_ENABLE(vary_options, +[ --enable-vary-options + Enable support for the X-Vary-Options header.], +[ if test $enableval = yes ; then +echo Enabling support for vary options +AC_DEFINE(VARY_OPTIONS, 1, [Enable support for the X-Vary- Options header]) + fi +]) + AC_ARG_ENABLE(follow-x-forwarded-for, [ --enable-follow-x-forwarded-for Enable support for following the X- Forwarded-For diff -Xdiffx -ru squid-2.6.18.orig/src/client_side.c squid-2.6.18/ src/client_side.c --- squid-2.6.18.orig/src/client_side.c 2008-02-07 19:28:38.0 +1100 +++ squid-2.6.18/src/client_side.c2008-02-08 14:39:38.0 +1100 @@ -735,10 +735,7 @@ request_t *request = http-request; const char *etag = httpHeaderGetStr(mem-reply-header, HDR_ETAG); const char *vary = request-vary_headers; - int has_vary =
Re: X-Vary-Options patch
I think some of their stuff was backed out of Squid-2.7 before release. What we -should- do is create a wiki page to document all the crazy stuff about Vary: and coordinate things a little better. I'd really like to see more sensible vary handling go into Squid and there certainly seems like there is enough interest in it now to get work done. Adrian On Sat, Jun 07, 2008, Mark Nottingham wrote: Squid devs: Did this make its way into 2.7? Tim: AIUI, the following header: X-Vary-Options: Accept-Encoding; list-contains=gzip will bucket the cache for this URI into two entries; those whose Accept-Encoding contains the list value gzip, and those that don't. Is that correct? Also, I'm assuming the origin would still need to send Vary: Accept-Encoding to work properly with downstream caches. Cheers, On 26/02/2008, at 3:30 PM, Adrian Chadd wrote: G'day, I'm happy to commit this to Squid-2.HEAD as-is. Can you throw it in a Bugzilla report and spit me the number? Thanks, Adrian On Fri, Feb 08, 2008, Tim Starling wrote: There are two major sources of suboptimal hit rate on Wikipedia which relate to the Vary header: * In Accept-Encoding, we only care whether gzip is present or not, but IE and Firefox use different whitespace conventions and so each get separate entries in the cache * We only care whether the user is logged in or not. Other cookies, such as pure-JavaScript cookies used by client-side code to store preferences, unnecessarily degrade our hit rate. There have been other patches related to this problem, but as far as I'm aware, they're all special-case, site-specific hacks. My patch adds an X-Vary-Options response header (hereafter XVO), and thus gives the origin server fine control over cache variance. In the patch, the XVO header overrides the Vary header, so the Vary header can still be sent as usual for compatibility with caches that don't support this feature. The format of the XVO header is inspired by the format of the Accept header. As in Vary, XVO is separated by commas into parts which relate to different request headers. Then those parts are further separated by semicolons. The first semicolon-separated part is the request header name, and subsequent parts give name/value pairs separated by equals signs, defining options relating to the variance of that header. Two option names are currently defined: list-contains: splits the request header into comma-separated parts and varies depending on whether the resulting list contains the option value string-contains: performs a simple search of the request header and varies depending on whether it matches. Multiple such options per header are allowed. So for example: X-Vary-Options: Cookie; string-contains=UserID; string-contains=_session, Accept-Encoding; list-contains=gzip This would vary the cache on three tests: * whether the Cookie header contains the string UserID * whether the Cookie header contains the string _session * whether the Accept-Encoding header, interpreted as a comma- separated list, contains the item gzip The patch refactors all references to the Vary and X-Accelerator-Vary headers into the functions httpHeaderHasVary() and httpHeaderGetVary() in HttpHeader.c. It then adds X-Vary-Options to these functions, interpreting it as a string rather than a list to avoid inappropriate splitting on whitespace. It puts the resulting combined header into X-Vary-Options instead of Vary in the base vary marker object, again to avoid inappropriate list-style interpretation. httpMakeVaryMark() then interprets this combined header in the way described above. The added features of the patch are conditional, and are enabled by the configure option --enable-vary-options. Autoconf and automake will need to be run after applying this patch. The interpretation of some non-standards-compliant Vary headers (those containing semicolons) is changed slightly by this patch regardless of --enable-vary-options. The patch is attached and also available at: http://noc.wikimedia.org/~tstarling/patches/vary_options_upstream.patch For your review and consideration. -- Tim Starling diff -Xdiffx -ru squid-2.6.18.orig/configure.in squid-2.6.18/ configure.in --- squid-2.6.18.orig/configure.in2008-01-10 23:34:23.0 +1100 +++ squid-2.6.18/configure.in 2008-02-07 19:43:23.0 +1100 @@ -1507,6 +1507,16 @@ fi ]) +dnl Enable vary options +AC_ARG_ENABLE(vary_options, +[ --enable-vary-options + Enable support for the X-Vary-Options header.], +[ if test $enableval = yes ; then +echo Enabling support for vary options +AC_DEFINE(VARY_OPTIONS, 1, [Enable support for the X-Vary- Options header]) + fi +]) + AC_ARG_ENABLE(follow-x-forwarded-for, [ --enable-follow-x-forwarded-for Enable support
Re: X-Vary-Options patch
G'day, I'm happy to commit this to Squid-2.HEAD as-is. Can you throw it in a Bugzilla report and spit me the number? Thanks, Adrian On Fri, Feb 08, 2008, Tim Starling wrote: There are two major sources of suboptimal hit rate on Wikipedia which relate to the Vary header: * In Accept-Encoding, we only care whether gzip is present or not, but IE and Firefox use different whitespace conventions and so each get separate entries in the cache * We only care whether the user is logged in or not. Other cookies, such as pure-JavaScript cookies used by client-side code to store preferences, unnecessarily degrade our hit rate. There have been other patches related to this problem, but as far as I'm aware, they're all special-case, site-specific hacks. My patch adds an X-Vary-Options response header (hereafter XVO), and thus gives the origin server fine control over cache variance. In the patch, the XVO header overrides the Vary header, so the Vary header can still be sent as usual for compatibility with caches that don't support this feature. The format of the XVO header is inspired by the format of the Accept header. As in Vary, XVO is separated by commas into parts which relate to different request headers. Then those parts are further separated by semicolons. The first semicolon-separated part is the request header name, and subsequent parts give name/value pairs separated by equals signs, defining options relating to the variance of that header. Two option names are currently defined: list-contains: splits the request header into comma-separated parts and varies depending on whether the resulting list contains the option value string-contains: performs a simple search of the request header and varies depending on whether it matches. Multiple such options per header are allowed. So for example: X-Vary-Options: Cookie; string-contains=UserID; string-contains=_session, Accept-Encoding; list-contains=gzip This would vary the cache on three tests: * whether the Cookie header contains the string UserID * whether the Cookie header contains the string _session * whether the Accept-Encoding header, interpreted as a comma-separated list, contains the item gzip The patch refactors all references to the Vary and X-Accelerator-Vary headers into the functions httpHeaderHasVary() and httpHeaderGetVary() in HttpHeader.c. It then adds X-Vary-Options to these functions, interpreting it as a string rather than a list to avoid inappropriate splitting on whitespace. It puts the resulting combined header into X-Vary-Options instead of Vary in the base vary marker object, again to avoid inappropriate list-style interpretation. httpMakeVaryMark() then interprets this combined header in the way described above. The added features of the patch are conditional, and are enabled by the configure option --enable-vary-options. Autoconf and automake will need to be run after applying this patch. The interpretation of some non-standards-compliant Vary headers (those containing semicolons) is changed slightly by this patch regardless of --enable-vary-options. The patch is attached and also available at: http://noc.wikimedia.org/~tstarling/patches/vary_options_upstream.patch For your review and consideration. -- Tim Starling diff -Xdiffx -ru squid-2.6.18.orig/configure.in squid-2.6.18/configure.in --- squid-2.6.18.orig/configure.in2008-01-10 23:34:23.0 +1100 +++ squid-2.6.18/configure.in 2008-02-07 19:43:23.0 +1100 @@ -1507,6 +1507,16 @@ fi ]) +dnl Enable vary options +AC_ARG_ENABLE(vary_options, +[ --enable-vary-options + Enable support for the X-Vary-Options header.], +[ if test $enableval = yes ; then +echo Enabling support for vary options +AC_DEFINE(VARY_OPTIONS, 1, [Enable support for the X-Vary-Options header]) + fi +]) + AC_ARG_ENABLE(follow-x-forwarded-for, [ --enable-follow-x-forwarded-for Enable support for following the X-Forwarded-For diff -Xdiffx -ru squid-2.6.18.orig/src/client_side.c squid-2.6.18/src/client_side.c --- squid-2.6.18.orig/src/client_side.c 2008-02-07 19:28:38.0 +1100 +++ squid-2.6.18/src/client_side.c2008-02-08 14:39:38.0 +1100 @@ -735,10 +735,7 @@ request_t *request = http-request; const char *etag = httpHeaderGetStr(mem-reply-header, HDR_ETAG); const char *vary = request-vary_headers; - int has_vary = httpHeaderHas(entry-mem_obj-reply-header, HDR_VARY); -#if X_ACCELERATOR_VARY - has_vary |= httpHeaderHas(entry-mem_obj-reply-header, HDR_X_ACCELERATOR_VARY); -#endif + int has_vary = httpHeaderHasVary(entry-mem_obj-reply-header); if (has_vary) vary = httpMakeVaryMark(request, mem-reply); @@ -4948,10 +4945,7 @@ varyEvaluateMatch(StoreEntry * entry, request_t *
Re: X-Vary-Options patch
can you please send the TAGs what i made change in squid.conf Robert Collins wrote: On Fri, 2008-02-08 at 16:26 +1100, Tim Starling wrote: The added features of the patch are conditional, and are enabled by the configure option --enable-vary-options. Unless there is non-trivial process required for regular vary headers with this enabled, I don't think it needs to be optional. -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. -- View this message in context: http://www.nabble.com/X-Vary-Options-patch-tp15349937p15495250.html Sent from the Squid - Development mailing list archive at Nabble.com.
Re: X-Vary-Options patch
On Fri, 2008-02-08 at 16:26 +1100, Tim Starling wrote: The added features of the patch are conditional, and are enabled by the configure option --enable-vary-options. Unless there is non-trivial process required for regular vary headers with this enabled, I don't think it needs to be optional. -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part
