Re: Vary:User-Agent, best practices, and making the web faster.
On Sun, Jun 5, 2011 at 21:37, Joshua Marantz jmara...@google.com wrote: Does Magento actually vary the content of CSS JS based on user-agent? Or does it only vary the content of HTML? I don't know. I'm by no means a Magento expert, I only run into it from time to time. That site I broke? That was in the summer of 2009 while beefing up the security and performance of a large retailer's web shop, mostly by putting stuff behind reverse proxies.
Re: Vary:User-Agent, best practices, and making the web faster.
On Sun, Jun 5, 2011 at 13:42, Joshua Marantz jmara...@google.com wrote: This is a case where the content varies based on user-agent. The recommendation on the mod_deflate doc page is add vary:user-agent for any non-image. Can you think of a case where the absence of a vary:user-agent header causes broken behavior when the content doesn't vary? I'm not objecting to setting vary:user-agent when content varies: that's what it's for. I'm objecting to setting vary:user-agent when content does *not* vary. The mod_deflate documentation unambiguously recommends setting vary:user-agent, and my feeling is that this is to work around a bug that exists only in IE5 or pre-2007 patch of IE6. Sorry, Joshua, we're conflating things. You raised two issues in your original post: 1. Updating the mod_deflate documentation. Seems reasonable. The Vary: UA recommendation was added in 2002 during a general clean-up of the mod_deflate documentation and the commit log doesn't tell why. You could open a bugzilla issue or raise it on the httpd-dev mailing list (the former is the proper channel but the bugzilla is something of a graveyard). 2. mod_pagespeed second-guessing the user's intent. That still seems like an unambiguously bad idea. To touch on Magento again, its documentation links (or linked) directly to that section of the mod_deflate docs and people are using that. If your module scans for and neutralizes that Header directive, you will break someone's site.
Re: Vary:User-Agent, best practices, and making the web faster.
Thanks for the feedback, Ben! You are omniscient in the ways of Apache. I'll try to lobby for an update to the mod_deflate page. Your concerns about Magento are interesting -- my impression from our forums and Twitter is that mod_pagespeed is successfully accelerating many Magento sites now, since releasing the mod_rewrite workarounds, which we were able to do thanks to your help a few months ago. The reason that we don't have problems with vary:user-agent on HTML is that we don't ever cache HTML under any circumstance. We assume HTML generally varies on user-agent, cookies, locale-of-client, and is generally updated frequently. Does Magento actually vary the content of CSS JS based on user-agent? Or does it only vary the content of HTML? You've dissuaded me from trying to infer the intent of the site administrator, and we will find another way to phase in vary:user-agent compliance without falling off a performance cliff. -Josh On Sun, Jun 5, 2011 at 2:54 PM, Ben Noordhuis i...@bnoordhuis.nl wrote: On Sun, Jun 5, 2011 at 13:42, Joshua Marantz jmara...@google.com wrote: This is a case where the content varies based on user-agent. The recommendation on the mod_deflate doc page is add vary:user-agent for any non-image. Can you think of a case where the absence of a vary:user-agent header causes broken behavior when the content doesn't vary? I'm not objecting to setting vary:user-agent when content varies: that's what it's for. I'm objecting to setting vary:user-agent when content does *not* vary. The mod_deflate documentation unambiguously recommends setting vary:user-agent, and my feeling is that this is to work around a bug that exists only in IE5 or pre-2007 patch of IE6. Sorry, Joshua, we're conflating things. You raised two issues in your original post: 1. Updating the mod_deflate documentation. Seems reasonable. The Vary: UA recommendation was added in 2002 during a general clean-up of the mod_deflate documentation and the commit log doesn't tell why. You could open a bugzilla issue or raise it on the httpd-dev mailing list (the former is the proper channel but the bugzilla is something of a graveyard). 2. mod_pagespeed second-guessing the user's intent. That still seems like an unambiguously bad idea. To touch on Magento again, its documentation links (or linked) directly to that section of the mod_deflate docs and people are using that. If your module scans for and neutralizes that Header directive, you will break someone's site.
Re: Vary:User-Agent, best practices, and making the web faster.
This encourages all site owners to add Vary:User-Agent to all css and js files, whether they actually vary in content or not. Does anyone know the history of this recommendation? Surely that is an inappropriate recommendation for mod_deflate. Vary:Accept-Encoding make sense in the context of that filter, but not Vary:User-Agent. It's because of the other (dated) canned exceptions that set/unset no-gzip/gzip-only-text/html based on the User-Agent, to second-guess browsers that send AE:gzip but can't properly deal with it.
Re: Vary:User-Agent, best practices, and making the web faster.
On Sat, Jun 4, 2011 at 21:26, Joshua Marantz jmara...@google.com wrote: I think what we'd do is basically let mod_pagespeed ignore Vary:User-Agent if we saw that it was inserted per this exact pattern. This would, to be This seems like a stupendously bad idea. Warn about it in your docs, complain about it in the logs but don't willy-nilly override people's settings.
Re: Vary:User-Agent, best practices, and making the web faster.
It was with some reluctance that I brought this up. It occurs to me that this idea propagates the sort of spec violations that led to this issue (inappropriate user of Vary:User-Agent) in the first place. However, I'm trying to figure out how to improve compliance to support legitimate uses of Vary:User-Agent without causing mod_pagespeed to become significantly less ineffective across a broad range of sites. We have found that putting complaints in Apache logs mostly causes disks to fill and servers to crash -- although that does get it noticed :). The problem, put another way, is that mod_pagespeed cannot distinguish legitimate uses of Vary:User-Agent, so it really has no business complaining in logs. Complaining in docs is fine; but some existing mod_pagespeed users that simply type sudo yum update will later notice a performance-drop and may not consult the docs to figure out why. I'm also trying to grok the first response from Eric: It's because of the other (dated) canned exceptions that set/unset no-gzip/gzip-only-text/html based on the User-Agent, to second-guess browsers that send AE:gzip but can't properly deal with it. Going backwards: which browsers send AE:gzip but can't properly deal with it? Does IE6 have that issue or is it only true of IE5? I know that IE6 has had issues with compression in the past but they appear to be addressed by patches issued by Microsoft four and a half years ago: http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312496. Moreover IE6 is shrinking in market sharehttp://arstechnica.com/web/news/2011/05/web-browser-market-share-upgrade-analysis.ars(~ 10%) and IE5 does not appear in the pie-chart at all. And I still don't understand how that relates to Vary:User-Agent. What's really at issue here seems more related to proxies; is that right? That proxies were not respecting Accept-Encoding, but sending gzipped content to browsers that did not want it? Is that still a problem? Which proxies were broken? Are they still broken? And, while I understand the reluctance to help me figure out from our module what values were passed to SetEnvIfNoCase and Header, I would like to see whether there's agreement that the Apache 2.2 docs for mod_deflate are no longer appropriate -- and in fact harmful. -Josh On Sat, Jun 4, 2011 at 5:03 PM, Ben Noordhuis i...@bnoordhuis.nl wrote: On Sat, Jun 4, 2011 at 21:26, Joshua Marantz jmara...@google.com wrote: I think what we'd do is basically let mod_pagespeed ignore Vary:User-Agent if we saw that it was inserted per this exact pattern. This would, to be This seems like a stupendously bad idea. Warn about it in your docs, complain about it in the logs but don't willy-nilly override people's settings.
Re: Vary:User-Agent, best practices, and making the web faster.
On Sun, Jun 5, 2011 at 00:34, Joshua Marantz jmara...@google.com wrote: It was with some reluctance that I brought this up. It occurs to me that this idea propagates the sort of spec violations that led to this issue (inappropriate user of Vary:User-Agent) in the first place. However, I'm trying to figure out how to improve compliance to support legitimate uses of Vary:User-Agent without causing mod_pagespeed to become significantly less ineffective across a broad range of sites. We have found that putting complaints in Apache logs mostly causes disks to fill and servers to crash -- although that does get it noticed :). The problem, put another way, is that mod_pagespeed cannot distinguish legitimate uses of Vary:User-Agent, so it really has no business complaining in logs. Complaining in docs is fine; but some existing mod_pagespeed users that simply type sudo yum update will later notice a performance-drop and may not consult the docs to figure out why. I'm also trying to grok the first response from Eric: It's because of the other (dated) canned exceptions that set/unset no-gzip/gzip-only-text/html based on the User-Agent, to second-guess browsers that send AE:gzip but can't properly deal with it. Going backwards: which browsers send AE:gzip but can't properly deal with it? Does IE6 have that issue or is it only true of IE5? I know that IE6 has had issues with compression in the past but they appear to be addressed by patches issued by Microsoft four and a half years ago: http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312496. Moreover IE6 is shrinking in market sharehttp://arstechnica.com/web/news/2011/05/web-browser-market-share-upgrade-analysis.ars(~ 10%) and IE5 does not appear in the pie-chart at all. This was indeed a (since fixed) problem with IE6. I haven't seen the gzip issue crop up since but that is purely anecdotal. And I still don't understand how that relates to Vary:User-Agent. What's really at issue here seems more related to proxies; is that right? That proxies were not respecting Accept-Encoding, but sending gzipped content to browsers that did not want it? Is that still a problem? Which proxies were broken? Are they still broken? Some popular OSS packages depend on Vary: User-Agent to make downstream proxies (reverse or forward) do the right thing. And, while I understand the reluctance to help me figure out from our module what values were passed to SetEnvIfNoCase and Header, I would like to see whether there's agreement that the Apache 2.2 docs for mod_deflate are no longer appropriate -- and in fact harmful. I've been mulling it over for 10 minutes and I can't decide. It's harmful because it leads to a proliferation of cached objects (bad) but removing it from the documentation will break things for someone somewhere (also bad).
