On Mon, 2016-10-24 at 19:03 +1300, Amos Jeffries wrote: > On 24/10/2016 6:28 a.m., gar...@comnet.uz wrote: > > > > On 2016-10-23 18:31, Amos Jeffries wrote: > > > > > > On 23/10/2016 2:32 a.m., garryd wrote: > > > > > > > > Since I started use Squid, it's configuration always RFC > > > > compliant by > > > > default, _but_ there were always knobs for users to make it > > > > HTTP > > > > violent. It was in hands of users to decide how to handle a web > > > > resource. Now it is not always possible, and the topic is an > > > > evidence. > > > > For example, in terms of this topic, users can't violate this > > > > RFC > > > > statement [1]: > > > > > > > > A Vary field value of "*" signals that anything about the > > > > request > > > > might play a role in selecting the response representation, > > > > possibly > > > > including elements outside the message syntax (e.g., the > > > > client's > > > > network address). A recipient will not be able to determine > > > > whether > > > > this response is appropriate for a later request without > > > > forwarding > > > > the request to the origin server. A proxy MUST NOT generate > > > > a Vary > > > > field with a "*" value. > > > > > > > > [1] https://tools.ietf.org/html/rfc7231#section-7.1.4 > > > > > > > > > Please name the option in any version of Squid which allowed > > > Squid to > > > cache those "Vary: *" responses. > > > > > > No such option ever existed. For the 20+ years Vary has existed > > > Squid > > > has behaved in the same way it does today. For all that time you > > > did not > > > notice these responses. > > > > You are absolutely right, but there were not such abuse vector in > > the > > past (at least in my practice). There were tools provided by devs > > to > > admins to protect against trending abuse cases. > > What trend? There is exactly one mentioned URL that I'm aware of, the > Chrome browser download URL. I've posted two reasons why Chrome uses > the > Vary:* header. Just opinions of mine, but formed after actual > discussions with the Chrome developers some years back. > > > [I very much dislike writing this. But you seem to have been sucked > in > and deserve to know the history.] > > All the fuss that is going on AFAICS was started by Yuri. His comment > history here and in bugzilla, and in private responses range from > non-compromising "cache everything no matter what - do what I say, > now!" > (repeatedy in unrelated bugzilla reports), "f*ck the RFCs and anyone > following them, just store everything I dont care about what happens" > (this mornings post), to personal attacks against anyone who mentions > the previous stance might have problems (all the "Squid developers > believe/say/..." comments - none of which match what the team we have > actually said to him or believe). > > There is one other email address which changes its name occasionally > and > posts almost exactly the same words as Yuri's. So it looks to me as > Yuri > and some sock puppets performing a campaign to spread lies and FUD > about > Squid and hurt the people doing work on it. > > Not exactly a good way to get people to do things for free. But it > seems > to have worked on getting you and a few others now doing the coding > part > for him at no cost, and I have now wasted time responding to you and > thinking of a solution for it that might get accepted for merge. > > > This particular topic is not the first to have such behaviour by > Yuri. > There have been other things where someone made a mistake (overlooked > something) and all hell full of insults broke loose at them. And > several > other cases where missing features in Squid did not get instant > obedience to quite blunt and insulting demands. Followed by weeks of > insults until the bug was fixed by other people - then suddenly > polite > Yuri comes back overnight. > > > As a developer, I personally decided not to write the requested code. > Not in the way demanded. This seems to have upset Yuri who has taken > to > insulting me and the rest of the dev team as a whole. I'm not sure if > he > is trolling to intentionally cause the above mentioned effects, or > really in need of medical assistance to deal with work related > stress. > > [/history] > > > > > > So, the question arised, > > what changed in Squid development policy? > > In policy: Nothing I'm aware of in the past 10 years. > > What changed on the Internet? a new bunch of RFCs came out, the > server > and clients Squid talks to all got updated to follow those documents > more closely. > > What changed in Squid? the dev team have been slowly adding the new > abilities to Squid. One by one, its only ~90% (maybe less) compliant > withe the MUST conditions, not even close to that on the SHOULDs, > MAYs, > and implied processing abilities. > > > What do you think should happen to Squid when all the software it > talks > to speaks and expects what the RFCs say they should expect from > recipients/Squid ? > > Consider the extreme this leads towards: You could easily write a > piece > of software that took HTTP requests and sent back random data that > looked vaguely like HTTP responses. But how useless that is when used > as > a proxy. So much for ignoring the HTTP specs. > > > > > > Why there is no configuration > > option like 'ignore_vary [acl]', so highly demanded by many users > > in the > > list? > > AFAIK nobody has ever even proposed adding one until these past few > weeks. > > A proposals to ignore "Vary: *" has come up every few years, but when > the proposer was made aware of the server expectations as stated in > the > RFC 2616 they went away, no attempt to go any further. So I assume > that > means it wasn't really a problem for them, not a serious one. > > In RFC 2616 (or older) compliant web there is no benefit to caching > these objects. The revalidate clause did not exist so the server can > be > expected to always produce a new one. > So what is gained by caching it? a waste of space other HIT's could > have > used better. Negative benefits really, not even zero gain. > > The middle sentence of that part of RFC 7231 has only existed for the > past few years. Even I had overlooked it until today. Sorry, but I > have > been concentrating on getting Cache-Control going properly (no-cache > and > side effects are still a hot topic 4 years after it went in). Vary is > much broken in other serious ways, '*' storage seems a minor issue to > me. > > Also be aware that Squid is still in the process of moving from > HTTP/1.0 > behaviour to HTTP/1.1 revalidation. We (Eduard, Alex and myself) have > not magically made everything work perfectly in one release. This is > one > of probably many cases where revalidation is potentially usable - > just > that nobody has coded it yet. Or cases which were previously > forbidden > and HTTPbis discussions (recent 7-ish years) shown that other vendors > are widely okay with it so its been allowed now by the 723x RFC > series. > > > As maintainer its my responsibility to point out the issues to anyone > even proposing a violation gets added. To make sure they are aware of > the expectations the servers and clients may have of Squid. > > All violation proposals get the same treatment (heck all patches get > this same treatment too, I'm just a little more lenient on compliance > increasing patches): > - make the proposer aware of the problems they will cause, > ** my posts in reply to this and other threads. > > - mention any alternatives (where available), > ** others have mentioned eCAP/ICAP as better suited to such > violations. > That is exactly what those interfaces are designed for. > ** re-reading the section you quoted above, the middle sentence adds > a > possibility I overlooked earlier. You now have a potential direction > to > go without even doing any violation. The audit process shows its > worth > right there :-). > > - and let them decide if it is worth the trouble to go forward. > ** so far mostly whats happened is a lot of complaints that "Squid > dev > team" are not providing things for free, on demand, to service a very > abusive and extremist person. Or just outright abuse. Seems like the > individual who started the discussions does not care about getting it > to > happen. You care more, and as you say below ... > > > > > > Personally, I'm no affected by the Vary abuse, but I suppose there > > will be increasing number of abuse cases in the future. > > Me neither (for seeing the problem in my sysadmin roles), so for both > of > us and many others it seems (to me) not to be a major problem. Or at > least not worth facing the consequences the proposed change risks > causing. > > We can guess at what the future holds. But until we get there we > really > dont know. > > My view of things includes years of app developer queries sent to the > IETF HTTPbis WG. There is an increasing number of applications that > *need* proxies to follow the spec requirements. I am not seeing them > much in the wild yet, but those applications are definitely around > and > breaking them could lead people to a lot of trouble. > > > FWIW: The current spec for HTTP/1.1 is extremely tollerant. There > are a > very few places where it comes to an absolute inviolate rule. Those > are > cases which have very clear use-case mentioned in the spec and > problems > being avoided mentioned as well. > > For example the quoted section above about Vary. I overlooked that > middle sentance that has been added since 2616. That creates the > leeway > we need to cache the object within compliance. We are thusly allowed > to > treat "Vary:*" as another of the must-revalidate caching cases, just > like Cc:private or Authenticated responses. > (From the use-cases that have been mentioned in IETF WG about this > header, and cases people have been advised to use it I very much > doubt > any server will respond with 304 to such revalidation. But we might > get > lucky, or future servers might do so.) > > NP: all previous discussions and proposals of Vary:* handling came up > when 2616 was the spec to follow, and Squid did not do 1.1 much > anyway - > so using revalidation was not even on the horizon so to speak. > > > Anyhow, back to me rant ;-P > > The current specs were written by representatives standing for *all* > the > major browsers, *all* the major web servers, *all* the major > language/library frameworks, almost all the major command-line tools, > and many application developers as well. > > So it is not written in isolation by some few "ivory tower types" > Yuri > would have you believe (yes some specs are - HTTP is not, or at least > those types are outnumbered for HTTP). The RFC 723x set is explicitly > a > collection of details about how the current HTTP applications > actually > found around the world *do* talk to each other today and how best to > exchange messages so the processing operations are understood at both > ends of the network connection. > > The HTTP RFC's are effectively a description of current best > practice. > Violating is only useful if you are fixing some broken application > your > particular proxy has to deal with. All other cases are just causing > inefficiency and varying amounts of nasty depending on the violation. > > This latter point is part of why we require a clear use-case before > accepting violation patches - the need for it should be well thought > out. People who choose to violate that type of spec without a good > reason are just being stupid. > > > The three persons (well, 2 use aliased email addresses - I suspect > are > actually one or two persons from the same region of the world) > providing > the most noise about Vary keep demanding it be provided for free. > > > > > > One of your > > answers confirmed my assumption regarding the question: > > > > > > > > - there is a very high risk of copy-and-paste sysadmin spreading > > > the > > > problems without realising what they are doing. Particularly > > > since those > > > proposing it are so vocal about how great it *seems* for them. > > > > When someone else does the coding I act as both auditor and > maintainer. > The policies checked during audit do not go into good/bad judgements > - > just: code style, quality (no identifiable bugs) and a clear > statement > of the need behind the addition (for the merge repo comment to guide > future maintenance of that code). > > As maintainer I merge patches myself only if I'm happy to take on > code > maintenance for them, or supplier explicitly takes it on themselves > (ie > helper authors). Sometimes I get patches up to scratch in audit then > let > others merge, or advise they just be used as custom patches. In the > latter case we at least know that patch is not full of obvious bug > behaviours. > > But note so far *nobody* has submitted patches to audit about this. > It > has all just been a rather heated discussion in various unrelated bug > reports and some threads in this users list. > > Amos > > _______________________________________________ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users
Hi Amos, Thank you very much for so detailed explanation. I've made conclusions from presented information. I deeply regret, that the topic took so many time from you. I believe, information presented here will be helpful for the community. Nevertheless, the topic surfaced new details regarding the Vary and I tried conditional requests on same URL (Google Chrome) from different machines/IPs. Here results: $ curl --head --header "If-Modified-Since: Thu, 22 Oct 2016 08:29:09 GMT" https://dl.google.com/linux/direct/google-chrome-stable_current_am d64.deb HTTP/1.1 304 Not Modified Etag: "101395" Server: downloads Vary: * X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block Date: Mon, 24 Oct 2016 08:53:44 GMT Alt-Svc: quic=":443"; ma=2592000; v="36,35,34" ---- $ curl --head --header 'If-None-Match: "101395"' https://dl.google.com/ linux/direct/google-chrome-stable_current_amd64.deb HTTP/1.1 304 Not Modified Etag: "101395" Last-Modified: Thu, 20 Oct 2016 08:29:09 GMT Server: downloads Vary: * X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block Date: Mon, 24 Oct 2016 08:54:18 GMT Alt-Svc: quic=":443"; ma=2592000; v="36,35,34" Garri _______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
