On 05/10/2011 08:05 AM, John Craws wrote: > "This would be http://bugs.squid-cache.org/show_bug.cgi?id=3153 yes? > Neither Alex nor Christos who are working on it read this mailing > list. Contact them via bugzilla or squid-dev please." > > I don't think this looks like 3153. The client is not disconnecting > and the server is not aborting. > Please see below for the details and differences. > > Thanks a lot! > > --- > > Hello, > > I have an ICAP server that I am using with squid (3.1.12.1 but have > tried others) in order to support RESPMOD. It usually works fine but I > am now having problems trying to implement what I think should be a > simple service. > > I am not including full logs in this initial email as I'm not sure > exactly what would help (I have not seen error messages) and I don't > want to just dump everything. I will gladly sent whatever could help. > > The problem: > > A RESPMOD service which returns a 302 (I have also tried a 404 but the > results were the same) following a GET request from a client (wget). > > I have no problem downloading the file without going through ICAP. > All caching is disabled. > I have tried with different files on different servers. > There are no visible network / routing issues. > The ICAP service is respmod_precache and bypass on. > Strange: For some reason, it has worked on very few occasions (maybe > 5/6 on dozens of attempts). > The ICAP exchange appears to be completed without problems (the socket > is closed by the ICAP server). > The ICAP server implements other services (both REQMOD and RESPMOD) > with squid and they work well. > > Case A - The file is locally hosted > Case B - The file is hosted on an external, third party website > > Case A: > > 1. Squid receives the GET request for the file. > 2. Squid starts fetching the file from the origin server. > 3. Squid sends (in parallel) the RESPMOD request and receives an > answer from the ICAP server (302/Redirect). > 4. Squid continues downloading data from the origin server. > 5. At some point (> preview size, < total size) Squid sends a > RST/ACK to the origin server and closes the connection. > 6. Squid sends the 302 to the client. > > Case B: > > 1. Squid receives the GET request for the file. > 2. Squid starts fetching the file from the origin server. > 3. Squid sends (in parallel) the RESPMOD request and receives an > answer from the ICAP server (302/Redirect). > 4. Squid continues downloading data from the origin server. > 5. At some point (> preview size, < total size) Squid sends a "TCP > Window Full" to the origin server. > 6. Squid and the origin server enter a TCP Zero Window Probe cycle > that they never get out of. > 7. Squid never gets done with the connection to the origin server > and therefore does not send the 302 to the client. > 8. The client is stuck waiting for a reply to its original GET > while Squid is stuck downloading data that it does not need in order > to send the reply. > > As noted above, it seems that squid starts fetching the original > requested media as soon as it receives the GET request even though the > ICAP RESPMOD modifies the response to just send a 302.
The Squid behavior you describe seems correct and reasonable. The origin server does not know about your ICAP service and should not be punished with connection resets and other nasties just because your service does not care about the response. What alternative behavior would you prefer in this case? If you would like to prevent useless response transfers and do not really care about the response itself, consider moving your service to precache REQMOD vectoring point and using ICAP request satisfaction instead. There may be other, more complex options as well. > It also appears that it does not return an answer to the client until > it is done with the origin server from which it is getting the file. This should not be happening. If it does, it is a Squid bug, IMO. From client point of view, your ICAP RESPMOD service is, essentially, the HTTP origin server. However, your HTTP adapted response is kind of malformed, which may cause symptoms similar to the above. See below for details. > The reason why Case B leads to a Window Full is still unclear. The > window is full before it has downloaded the number of bytes at which > it closes the connection itself in Case A. That suggests that it > should have enough buffer space to continue storing the response. > > I'm not sure why squid, after it has received the response from the > ICAP server needs to continue downloading from the origin server, Squid just wants to complete the HTTP transaction nicely. HTTP does not have a way of saying "I changed my mind, do not send me the rest of this response". See above for one of the options to prevent this waste. > why > it needs this to end in order to send the ICAP response to the client, > and why it gets in a situation where it stops reading from the origin > server (window full) and therefore blocks the request. Is it waiting > for something to consume the original file? If so, what should? Why > does local hosting make a difference? Does your ICAP RESPMOD service consume the ICAP request with the embedded HTTP response body after returning an ICAP 200 response? It must. See also ICAP 206 extension that was recently added to Squid, IIRC. The local-vs-remove hosting of the response probably changes some TCP stack timings/buffering and is not really relevant to the core problem. > The ICAP exchange (the log below is with returning a 404 but it's the > same with a 302 + Location header (the original problem)). > > ---- > RESPMOD icap://127.0.0.1:1344/df ICAP/1.0 > > Host: 127.0.0.1:1344 > > Date: Mon, 09 May 2011 14:43:09 GMT > > Encapsulated: req-hdr=0, res-hdr=128, res-body=374 > > Preview: 64 > > X-Client-IP: 127.0.0.1 > > > > GET http://172.16.199.35/sanity/media/PrinceOfPersia.flv HTTP/1.0 > > User-Agent: Wget/1.11.4 > > Accept: */* > > Host: 172.16.199.35 > > > > HTTP/1.1 200 OK > > Date: Mon, 09 May 2011 14:43:09 GMT > > Server: Apache/2.2.3 (Red Hat) > > Last-Modified: Mon, 04 Oct 2010 14:30:31 GMT > > ETag: "74c0111-5bffc8-491cb62ee0fc0" > > Accept-Ranges: bytes > > Content-Length: 6029256 > > Content-Type: video/x-flv > > > > 40 > > FLV.............K......... > onMetaData.......duration.@b.|..hs..s > > 0 > > > > ICAP/1.0 200 OK > > Encapsulated: res-hdr=0, null-body=26 > > Connection: close > > Date: Mon, 09 May 2011 14:43:09 GMT > > Server: Optimizer > > ISTag: "757795719" > > > > HTTP/1.1 404 Not Found > > > ---- The above embedded adapted HTTP 404 response must have a (possibly zero-length) body (from HTTP rules point of view). Yet, your ICAP service says there is no HTTP response body. I do not know how Squid will handle this contradiction, but you should change the response to have a body or, at the very least, add a "Content-Length: 0" HTTP response header. Same for the HTTP 302 response. HTH, Alex.
