Re: [squid-users] Directives ignore-private and override-expire not working Squid 3.2 and 3.3
Thank you, I saw the problem. So now I have to deal with Cache-Control: private header sent from IIS7.5 Don't know why IIS 7.5 always return private, Google show some bugs of this. Thank you again Mr Jeffries. On Tue, Nov 26, 2013 at 2:14 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 26/11/2013 6:06 p.m., Le Trung, Kien wrote: Hi, Eliezer Croitoru I already sent the header in the first email. Is this the information you want ? = Squid 3.3.x HTTP/1.1 200 OK Cache-Control: private Content-Length: 117991 Content-Type: text/html; charset=utf-8 Expires: Thu, 21 Nov 2013 03:12:14 GMT Server: Microsoft-IIS/7.5 Date: Thu, 21 Nov 2013 03:12:15 GMT X-Cache: MISS from localhost.localdomain Connection: close And after Amos's reply I check again the header of Squid-3.1 = Squid 3.1.x HTTP/1.0 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Expires: Tue, 26 Nov 2013 05:00:03 GMT Server: Microsoft-IIS/7.5 Date: Tue, 26 Nov 2013 05:00:04 GMT Content-Length: 117904 Age: 64 Warning: 110 squid/3.1.23 Response is stale (confused here too !) X-Cache: HIT from localhost.localdomain Connection: close In both case I used the same directives ignore-private and override-expire and same origin server. Squids also built in same server, the difference is only http service ports. Still don't know why squid 3.3 and 3.2 can't ignore-private and override-expire header. I still think you are misunderstanding what is happening here. Ignoring private simply means that Squid will store it instead of discarding immediately as required by RFC 2616 (and by Law in many countries). For safe use of privileged information we consider this content to expire the instant it was received. * The handling of that content once it is in cache still goes ahead in full accordance with HTTP/1.1 requirements had the private not been there to prevent caching. override-expires means that when the Expires: header is present the value inside it is replaced (overridden with) with the values in refresh_pattern header. * The calculation of how fresh/stale the object is still happens - just without the HTTP response header value for Expires. 3.1.20 are HTTP/1.0 proxies and do not perform HTTP/1.1 protocol validation perfectly. The headers still contain the Squid Warning: about the object coming out of cache (HIT) and being stale. 3.2+ are HTTP/1.1 proxies and are more strictly following RFC2616 requirements about revalidating stale content before use. It just happened that the server presented a new copy for delivery. NOTE: private *was* ignored. Expires *was* overridden. There was new content to deliver regardless of the values you changed them to. ALSO NOTE: The X-Cache header does not display REFRESH states. It displays MISS usually in the event of REFRESH_MODIFIED and HIT usually in the event of REFRESH_UNMODIFIED. You can get a better test of the private/Expires caching by causing the server those objects came from to be disconnected/unavailable when accessed from your Squid. In which case you should see the same headers as present in 3.1 indicating a HIT with stale object returned. Amos -- Best Regards, Kiên Lê
[squid-users] ##palin AW: [squid-users] #Can't access certain webpages
Hi Kinkie, yes i made a capture but don't see the cause. I send you my traces. Kind regards. Marc -Ursprüngliche Nachricht- Von: Kinkie [mailto:gkin...@gmail.com] Gesendet: Montag, 25. November 2013 15:45 An: Grooz, Marc (regio iT) Cc: squid-users@squid-cache.org Betreff: Re: [squid-users] #Can't access certain webpages On Mon, Nov 25, 2013 at 3:21 PM, Grooz, Marc (regio iT) marc.gr...@regioit.de wrote: Hi, Currently I use Squid 3.3.8 and I can't use/access two webservers thru squid. If I bypass squid this websites work great. One of this websites is a fileupload/download website with a generated downloadlink. When I upload a file I receive the following Squidlog Entrys: TCP_MISS/200 398 GET http://w.y.x.z/cgi-bin/upload_status.cgi? . . TCP_MISS_ABORTED/000 0 GET http:// w.y.x.z/cgi-bin/upload_status.cgi? TCP_MISS/200 398 GET http://w.y.x.z/cgi-bin/upload_status.cgi? And the downloadlink never gets generated. In the second case you never get a webpage back from squid. If I use lynx from the commandline of the squid system the Webpage gets loaded. With a tcpdump I see that if squid makes the request then the Webserver didn't answer. Well, this is consistent with the behavior in squid's logs. Have you tried accessing the misbehaving server from a client running on the squid box, and comparing the differences in the network traces? -- /kinkie smime.p7s Description: S/MIME cryptographic signature
[squid-users] CLOSE_WAIT state in Squid leads to bandwidth drop
Hi All, I am doing a small test for bandwidth measurement of my test setup while squid is running. I am running a script to pump the traffic from client browser to Web-server via Squid box. The script creates around 50 user sessions and tries to do wget of randomly selected dynamic URL's. After some time , I m observing a drop in bandwidth of the link, which is connecting the webserver even there is no HIT in the squid cache. I analyzed the netstat output during the problem scenario, I could see Recv-q gets piled up in CLOSE_WAIT tcp state of squid and also squid stays in CLOSE_WAIT state for more than a minute. The number of squid sessions to webserver are getting dropped to 5 from 70, but still tcp sessions from client to squid are around 80. Without Squid, there is no drop in the bandwidth with the same load. Why bandwidth is getting dropped when squid is running? Please provide your suggestions on this. Logs Squid version : 2.6.STABLE14 2013-11-25 10:17:53 Collecting netstat statistics... tcp 248352 0 172.19.134.2:51439 194.50.177.163:80 CLOSE_WAIT 5477/(squid) tcp77229 0 172.19.134.2:41998 64.15.157.134:80 CLOSE_WAIT 5477/(squid) tcp15853 0 172.19.134.2:55344 64.136.20.39:80 CLOSE_WAIT 5477/(squid) tcp30022 0 172.19.134.2:47485 50.56.161.66:80 CLOSE_WAIT 5477/(squid) tcp30202 0 172.19.134.2:59213 198.90.22.194:80 CLOSE_WAIT 5477/(squid) tcp 9787 0 172.19.134.2:52761 184.26.136.73:80 CLOSE_WAIT 5477/(squid) tcp 106892 0 172.19.134.2:55109 184.26.136.115:80 CLOSE_WAIT 5477/(squid) 2013-11-25 10:18:42 Collecting netstat statistics... tcp 248352 0 172.19.134.2:51439 194.50.177.163:80 CLOSE_WAIT 5477/(squid) tcp95558 0 172.19.134.2:42559 67.192.29.225:80 CLOSE_WAIT 5477/(squid) tcp77229 0 172.19.134.2:41998 64.15.157.134:80 CLOSE_WAIT 5477/(squid) tcp15853 0 172.19.134.2:55344 64.136.20.39:80 CLOSE_WAIT 5477/(squid) tcp30022 0 172.19.134.2:47485 50.56.161.66:80 CLOSE_WAIT 5477/(squid) tcp30202 0 172.19.134.2:59213 198.90.22.194:80 CLOSE_WAIT 5477/(squid) tcp 9787 0 172.19.134.2:52761 184.26.136.73:80 CLOSE_WAIT 5477/(squid) tcp 106892 0 172.19.134.2:55109 184.26.136.115:80 CLOSE_WAIT 5477/(squid) Squid info : --- Connection information for squid: Number of clients accessing cache: 3 Number of HTTP requests received: 257549 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 1443.2 Average ICP messages per minute since start:0.0 Select loop called: 4924570 times, 2.174 ms avg Cache information for squid: Request Hit Ratios: 5min: 0.0%, 60min: 0.0% Byte Hit Ratios:5min: -0.0%, 60min: 3.2% Request Memory Hit Ratios: 5min: 0.0%, 60min: 0.0% Request Disk Hit Ratios:5min: 0.0%, 60min: 0.0% Storage Swap size: 107524 KB Storage Mem size: 8408 KB Mean Object Size: 20.69 KB Requests given to unlinkd: 0 Regards, Saravanan N
Re: [squid-users] CLOSE_WAIT state in Squid leads to bandwidth drop
On Tuesday 26 November 2013 at 11:37, SaRaVanAn wrote: Hi All, I am doing a small test for bandwidth measurement of my test setup while squid is running. I am running a script to pump the traffic from client browser to Web-server via Squid box. Er, do you really mean you are sending data from the browser to the server? The script creates around 50 user sessions and tries to do wget of randomly selected dynamic URL's. That sounds more standard - wget will fetch data from the server to the browser. What do you mean by dynamic URLs? Where / how is the content actually being generated? After some time, Please define. I'm observing a drop in bandwidth of the link, Please define - what network setup are you using - what bandwidth are you getting at the start. what level does it drop to, does it return to the previous level? Squid version : 2.6.STABLE14 That is rather old (the last release of the 2.6 branch was STABLE23 September 2009). Is there any reason you have not upgraded to a current version? Regards, Antony. -- Behind the counter a boy with a shaven head stared vacantly into space, a dozen spikes of microsoft protruding from the socket behind his ear. - William Gibson, Neuromancer (1984) http://www.Open.Source.ITPlease reply to the list; The Open Source IT forum please don't CC me.
[squid-users] ##palin AW: [squid-users] #Can't access certain webpages
In my first case: Squid request: -MGET /cgi-bin/upload_status.cgi?uid=060950223627files=:iso-27001-router-security-audit-checklist.xlsok=1 HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Referer: http://xyz/ Accept-Language: de-DE User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko Accept-Encoding: gzip, deflate Host: xyz X-Forwarded-For: unknown, unknown Cache-Control: max-age=0 Connection: keep-alive Webserver answer: [-MHTTP/1.1 200 OK Date: Mon, 25 Nov 2013 12:48:57 GMT Server: Apache/2.2.22 (Linux/SUSE) Expires: Mon, 26 Jul 1997 05:00:00 GMT Pragma: no-cache Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html Squid send the first request again and again. Direct request without squid: Gm/GET /cgi-bin/upload_status.cgi?uid=318568766743files=:aukirche.JPGok=1 HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Referer: http://xyz/ Accept-Language: de-DE User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko Accept-Encoding: gzip, deflate Host: xyz DNT: 1 Connection: Keep-Alive Webserver answer: GmHTTP/1.1 200 OK Date: Tue, 26 Nov 2013 10:36:25 GMT Server: Apache/2.2.22 (Linux/SUSE) Expires: Mon, 26 Jul 1997 05:00:00 GMT Pragma: no-cache Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html Website gets displayed. In my second case: Squid request: SGET / HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Accept-Language: de-DE User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko Accept-Encoding: gzip, deflate If-Modified-Since: Tue, 26 Nov 2013 10:52:01 GMT DNT: 1 Host: xyz Pragma: no-cache X-Forwarded-For: unknown, unknown Cache-Control: max-age=259200 Connection: keep-alive No answer from Host Direct request without squid: S GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Accept-Language: de-DE User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko Accept-Encoding: gzip, deflate Host: xyz If-Modified-Since: Tue, 26 Nov 2013 10:52:01 GMT DNT: 1 Connection: Keep-Alive successful answer from Webserver. Kind regards marc -Ursprüngliche Nachricht- Von: Grooz, Marc (regio iT) [mailto:marc.gr...@regioit.de] Gesendet: Dienstag, 26. November 2013 11:55 An: Kinkie Cc: squid-users@squid-cache.org Betreff: [squid-users] ##palin AW: [squid-users] #Can't access certain webpages Hi Kinkie, yes i made a capture but don't see the cause. I send you my traces. Kind regards. Marc -Ursprüngliche Nachricht- Von: Kinkie [mailto:gkin...@gmail.com] Gesendet: Montag, 25. November 2013 15:45 An: Grooz, Marc (regio iT) Cc: squid-users@squid-cache.org Betreff: Re: [squid-users] #Can't access certain webpages On Mon, Nov 25, 2013 at 3:21 PM, Grooz, Marc (regio iT) marc.gr...@regioit.de wrote: Hi, Currently I use Squid 3.3.8 and I can't use/access two webservers thru squid. If I bypass squid this websites work great. One of this websites is a fileupload/download website with a generated downloadlink. When I upload a file I receive the following Squidlog Entrys: TCP_MISS/200 398 GET http://w.y.x.z/cgi-bin/upload_status.cgi? . . TCP_MISS_ABORTED/000 0 GET http:// w.y.x.z/cgi-bin/upload_status.cgi? TCP_MISS/200 398 GET http://w.y.x.z/cgi-bin/upload_status.cgi? And the downloadlink never gets generated. In the second case you never get a webpage back from squid. If I use lynx from the commandline of the squid system the Webpage gets loaded. With a tcpdump I see that if squid makes the request then the Webserver didn't answer. Well, this is consistent with the behavior in squid's logs. Have you tried accessing the misbehaving server from a client running on the squid box, and comparing the differences in the network traces? -- /kinkie smime.p7s Description: S/MIME cryptographic signature
Re: [squid-users] What do you recommend?
Hi, CentOS / RHEL 6.4 runs natively on the Hyper-V platform. Just keep in mind that i've never done an install with a desktop manager running as i generally just with the console / ssh. I manage several web filtering servers based on squid running that distro (usually squid 3.3.9/10 on CentOS 6.4), it works fine and is rock stable. Other than that, i use a really basic squid / squidGuard solution, log parsing is done by sarg. 2013/11/25 alamb200 alamb...@hotmail.com: Hi, I have several aborted attempts to get what I want to do to work and have failed miserably every time, so I thought I would ask you for advice. My plan is simple (in my head anyway) I want to set in place a device to run squid proxy so that I can the reduce bandwidth usage and also so i can see what users are doing on the web. So far I have tried a Windows solution but could not sort out the syslog bit and a linux solution which I struggled with and had to give up. My plan is to host squid on a virtual server hosted in Hyper V, on my previous attempts with linux I tried to use the gui desktop but could not get it to display properley so I am going to have to work around the command line to get it working. Can anyone help with this? Which OS should I use? What monitoring software would you recommend? I am trying to keep costs to a minimum while doing this while managing to have a reasonable solution. Thanks in advance for any advice you can pass on. alamb200 -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/What-do-you-recommend-tp4663512.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] ##palin AW: [squid-users] #Can't access certain webpages
On 27/11/2013 1:00 a.m., Grooz, Marc (regio iT) wrote: In my first case: Squid request: -MGET /cgi-bin/upload_status.cgi?uid=060950223627files=:iso-27001-router-security-audit-checklist.xlsok=1 HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Webserver answer: [-MHTTP/1.1 200 OK Date: Mon, 25 Nov 2013 12:48:57 GMT Squid send the first request again and again. Direct request without squid: Gm/GET /cgi-bin/upload_status.cgi?uid=318568766743files=:aukirche.JPGok=1 HTTP/1.1 Webserver answer: GmHTTP/1.1 200 OK Website gets displayed. Are those -M Gm/ cgaracters really in front of the GET method name and the HTTP/1.1 response version label? It looks like you may be receiving SOCKS protocol traffic. Amos
Re: [squid-users] CLOSE_WAIT state in Squid leads to bandwidth drop
On Tue, Nov 26, 2013 at 5:16 PM, Antony Stone antony.st...@squid.open.source.it wrote: On Tuesday 26 November 2013 at 11:37, SaRaVanAn wrote: Hi All, I am doing a small test for bandwidth measurement of my test setup while squid is running. I am running a script to pump the traffic from client browser to Web-server via Squid box. Er, do you really mean you are sending data from the browser to the server? The script creates around 50 user sessions and tries to do wget of randomly selected dynamic URL's. That sounds more standard - wget will fetch data from the server to the browser. = The script randomly picks the URL from the list of URL's defined in a file and tries to fetch that URL. What do you mean by dynamic URLs? Where / how is the content actually being generated? == Its a standard list of URL's with question mark in the end to avoid Squid caching. For example : www.espncricinfo.com? After some time, Please define. == After 15-20 minutes from the time of execution of script. I'm observing a drop in bandwidth of the link, Please define - what network setup are you using - what bandwidth are you getting at the start. what level does it drop to, does it return to the previous level? eth0 eth1 Windows Laptop - Linux machine(Squid Running) - Internet We are measuring the outgoing traffic in the link(eth1), which leads to the internet in order to calculate the bandwidth usage. Eth1 link bandwidth capability is around 10 Mbps. we are able utilize a maximum of 7-8 Mbps when squid is running. After 15 minutes, there is a sudden drop in bandwidth from 8Mbps to 6.5 Mbps and it comes back to 8Mbps after 2 -3 min. Squid version : 2.6.STABLE14 That is rather old (the last release of the 2.6 branch was STABLE23 September 2009). Is there any reason you have not upgraded to a current version? = There are some practical difficulties(our side) in upgrading to newer version. Regards, Antony. -- Behind the counter a boy with a shaven head stared vacantly into space, a dozen spikes of microsoft protruding from the socket behind his ear. - William Gibson, Neuromancer (1984) http://www.Open.Source.ITPlease reply to the list; The Open Source IT forum please don't CC me.
[squid-users] ##palin AW: [squid-users] #Can't access certain webpages
I've got it. I set the option forwared-for from off to delete and now both website gets displayed thru squid. Kind regrads Marc -Ursprüngliche Nachricht- Von: Amos Jeffries [mailto:squ...@treenet.co.nz] Gesendet: Dienstag, 26. November 2013 13:11 An: squid-users@squid-cache.org Betreff: Re: [squid-users] ##palin AW: [squid-users] #Can't access certain webpages On 27/11/2013 1:00 a.m., Grooz, Marc (regio iT) wrote: In my first case: Squid request: -MGET /cgi-bin/upload_status.cgi?uid=060950223627files=:iso-27001-router-se curity-audit-checklist.xlsok=1 HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Webserver answer: [-MHTTP/1.1 200 OK Date: Mon, 25 Nov 2013 12:48:57 GMT Squid send the first request again and again. Direct request without squid: Gm/GET /cgi-bin/upload_status.cgi?uid=318568766743files=:aukirche.JPGok=1 HTTP/1.1 Webserver answer: GmHTTP/1.1 200 OK Website gets displayed. Are those -M Gm/ cgaracters really in front of the GET method name and the HTTP/1.1 response version label? It looks like you may be receiving SOCKS protocol traffic. Amos smime.p7s Description: S/MIME cryptographic signature
[squid-users] is SPDY supported by squid ?
Hi, I found http://wiki.squid-cache.org/Features/HTTP2 and I wonder if it is the actual state, that SPDY is planned for squid 3.5, or is it allready implemented in the actual version. -- Regards Dieter -- I do not get viruses because I do not use MS software. If you use Outlook then please do not put my email address in your address-book so that WHEN you get a virus it won't use my address in the From field.
Re: [squid-users] is SPDY supported by squid ?
Hi, as I understand from several messages on the squid-dev mailing list, SPDY is not going to be supported. The first HTTP/2.0-related code is being debated and worked on in these weeks. If you are interested, you may want to join the squid-dev mailing list. Contributions are always welcome :) On Tue, Nov 26, 2013 at 4:20 PM, Dieter Bloms sq...@bloms.de wrote: Hi, I found http://wiki.squid-cache.org/Features/HTTP2 and I wonder if it is the actual state, that SPDY is planned for squid 3.5, or is it allready implemented in the actual version. -- Regards Dieter -- I do not get viruses because I do not use MS software. If you use Outlook then please do not put my email address in your address-book so that WHEN you get a virus it won't use my address in the From field. -- /kinkie
Re: [squid-users] is SPDY supported by squid ?
On 2013-11-27 04:20, Dieter Bloms wrote: Hi, I found http://wiki.squid-cache.org/Features/HTTP2 and I wonder if it is the actual state, that SPDY is planned for squid 3.5, or is it allready implemented in the actual version. SPDY is not planned at all. Unless the SPDY people re-write their spec to drop all the features IETF WG determined to be incompatible with real world HTTP requirements. I am working on HTTP/2 support now and have some very basic code which should be sufficient to let an interception proxy operate over port 443 or port 80 with direct HTTP/2 connections happening. Sponsorship and/or assistance welcome. PS. Thanks for the reminder. I've updated the page. Amos
Re: [squid-users] Squid with PHP Apache
On Tue, Nov 26, 2013 at 5:30 AM, Amos Jeffries squ...@treenet.co.nz wrote: On 26/11/2013 10:13 a.m., Ghassan Gharabli wrote: Hi, I have built a PHP script to cache HTTP 1.X 206 Partial Content like WindowsUpdates Allow seeking through Youtube many websites . Ah. So you have written your own HTTP caching proxy in PHP. Well done. Did you read RFC 2616 several times? your script is expected to to obey all the MUST conditions and clauses in there discussing proxy or cache. Yes , I have read it and I will read it again , but the reason i am building such a script is because internet here in Lebanon is really expensive and scarce. As you know Youtube is sending dynamic chunks for each video . For example , if you watch a video on Youtube more than 10 times , then Squid fill up the cache with more than 90 chunks per video , that is why allowing to seek at any position of the video using my script would save me the headache . NOTE: the easy way to do this is to upgrade your Squid to the current series and use ACLs on the range_offset_limit directive. That way Squid will convert Range requests to normal fetch requests and cache the object before sending the requested pieces of it back to the client. http://www.squid-cache.org/Doc/config/range_offset_limit/ I have successfully supported HTTP/206, if the object is cached and my target is to enable Range headers, as I can see that iPhones or Google Chrome check if the server has a header Accept-Ranges: Bytes then they send a request bytes=x-y or multiple bytes like bytes=x-y,x-y . I am willing to move from PHP to C++ hopefully after a while. The script is almost finished , but I have several question, I have no idea if I should always grab the HTTP Response Headers and send them back to the borwsers. The response headers you get when receiving the object are meta data describing that object AND the transaction used to fetch it AND the network conditions/pathway used to fetch it. The cachs job is to store those along with the object itself and deliver only the relevant headers when delivering a HIT. 1) Does Squid still grab the HTTP Response Headers, even if the object is already in cache or Squid has already a cached copy of the HTTP Response header . If Squid caches HTTP Response Headers then how do you deal with HTTP CODE 302 if the object is already cached . I am asking this question because I have already seen most websites use same extensions such as .FLV including Location Header. Yes. All proxies on the path are expected to relay the end-to-end headers, drop the hop-by-hop headers, and MUST update/generate the feature negotiation and state information headers to match its capabilities in each direction. Do you mean by Yes , for grabbing the Http Response Headers even if the object is already in cache, so therefore latency of network is always added even if MISS or HIT situation?. I have tested Squid and I have noticed that reading HIT objects from Squid takes about 0.x ms, which I believe objects are always offline until expiry occurs.Right? Till now I am using $http_response_headers as it is the fastest method by far , but I still have an issue with latency as for each request the function takes about 0.30s, which is really high, even if my network latency is 100~150 ms. That is why I have thought that I could possibly grab the HTTP Response Headers for the first time and store them, so if the URI was called for a second time, then I would send them the cached Headers instead of grabbing them again , to eliminate the network latency. But I still have an issue ... How am i going to know if the website sends HTTP/302 (because some websites send HTTP/302 for the same requested file name ), if I am not grabbing the header again in a HIT situation just to improve the latency. Second issue is Saving headers of CDN. 2) Do you also use mime.conf to send the Content-Type to the browser in case of FTP/HTTP or only FTP ? Only FTP and Gopher *if* Squid is translating from the native FTP/Gopher connection to HTTP. HTTP and protocols relayed using HTTP message format are expected to supply the correct header. 3) Does squid compare the length of the local cached copy with the remote file if you already have the object file or you use refresh_pattern?. Content-Length is a declaration of how many payload bytes are following the response headers. It has no relation to the servers object except in the special case where the entire object is being delivered as payload without any encoding. I am only caching objects that have Content-Length header, if the size was greater than 0 and I have noticed that there are some files like XML , CSS , JS, which I believe I should save, but do you think I must follow if-modified header to see if there is a fresh copy?. 4) What happens if the user modies a refresh_pattern to cache an object, for example .xml which does not have [Content-Length] header. Do you
Re: [squid-users] Squid with PHP Apache
Hey Ghassan, Moving from PHP to C++ is a nice idea. I do not know the size of the cache or it's limits but couple things to consider while implementing the cache: * clients latency * server overload * total cost * efficiency of the cache Bandwidth can cost lots of money in some cases and which some are willing to pay for. Youtube by itself is a beast since the number of visits per video might not be worth all the efforts that are being invested only in one video file\chunk. Specifically on youtube you need to grab the response headers and in some cases even filter couple of them. If you are caching and you are 99.5% sure that this chunk or file is ok as it is and as an object the headers can be considered as a side effect but in some cases are important. A compromise between Response Headers from a file to from source is that in a case that the headers file or container is deleted to fetch new ones or in a case the expiration headers are out-of-date then fetch new Headers\object. The main issue with 302 is the concept behind it. I have seen that in the past the usage of 302 was in order to give enough time for the upstream proxy\cdn node to fetch more data but in some cases it was a honest redirection towards the best origin server. In a case you know that uses 302 responses handle them by the site rather then in a Global way. The Content-Type is used from the origin server headers since this is probably what the client application expects. On a web-server you would see that by the file extension the Content-Type can be decided but this is not how squid handles http requests at all. Squid algorithm are pretty simple while considering the basic shape of the object from the headers. It is indeed an overhead to fetch from the web couple headers and there are some cases which it can be avoided but a re-validation of the integrity of the object\file is kind of important. Back to the beginning of the Email: If you do know that the object as it is now will not be changed for example as the owner of the web-service you can even serve the client stale content. There is no force in the world that limits you to do that. I can say that for example for youtube I was thinking about using another approach which would rank videos and will consider removing videos that was used once or twice per two weeks(which is depends on the size of the storage and load). If you do have a strong server that can run PHP you can try to take for a spin squid with StoreID that can help you to use only squid for youtube video caching. The only thing you will need to take care off is 302 response with an ICAP service for example. I do know how tempting it is to use PHP and it can be in many cases better for a network to use another solution then only squid. I do not know if you have seen this article: http://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator The article shows couple aspect of youtube caching. There was some PHP code at: http://code.google.com/p/yt-cache/ Which I have seen long time ago.(2011-12) StoreID is at the 3.4 branch of squid and is still on the Beta stage: http://wiki.squid-cache.org/Features/StoreID StoreID code by itself is very well tested and I am using it on a daily basis not even once restarting\reloading my local server for a very long time. I have not heard about a very big production environment(clustered) reports in my email yet. The basic idea of StoreID is to take the current existing internals of squid and to unleash them in a way that they can be exploited\used by external helper. StoreID is not here to replace the PHP or any other methods that might fit any network, it comes to allow the admin and see the power of squid caching even in this dead-end case which requires acrobatics. You can try to just test it in a small testing environment and to see if it fits to you. One of the benefits that Apache+PHP has is the Threading which allows one service such as apache to utilize as much horse power as the machine has as a metal. Since squid is already there the whole internal traffic between the apache and squid can be spared while using StoreID. Note that fetching the headers *only* from the origin server can still help you to decide if you want to fetch the whole object from it. A fetch of a whole headers set which will not exceed 1KB is worth for even a 200KB file size in many cases. I have tried to not miss somethings but I do not want to write a whole Scroll about yet so if there is more interest in it I will add more later. Regards, Eliezer On 25/11/13 23:13, Ghassan Gharabli wrote: Hi, I have built a PHP script to cache HTTP 1.X 206 Partial Content like WindowsUpdates Allow seeking through Youtube many websites . I am willing to move from PHP to C++ hopefully after a while. The script is almost finished , but I have several question, I have no idea if I should always grab the HTTP Response
[squid-users] Squid accel only after logon
Hi, I want to use Squid as a reverse proxy (accel) to my main website but only if they've authenticated - something like a captive portal (not sure if that's the right phrase). By authenticated, I don't mean basic or digest etc. I want to provide my own logon page (say php) - I can host another authentication website to host that. How do I go about achieving that? Splash page functionality is something that looks promising in squid but I can't get my head around how to force squid to reverse proxy my site only after users have authenticated on my php splash page. Also I need to terminate their session after 3 hours. http://wiki.squid-cache.org/ConfigExamples/Portal/Splash I can do something like this: #Show auth.php external_acl_type splash_page ttl=60 concurrency=100 %SRC /usr/local/sbin/squid/ext_session_acl -t 7200 -b /var/lib/squid/session.db acl existing_users external splash_page http_access deny !existing_users # Deny page to display deny_info 511:https://myauthserver/auth.php?url=%s existing_users #end authphp #reverse proxy https_port 443 cert=/path/to/x_domain_com.pem key=/path/to/x_domain_com.pem accel cache_peer 1.1.1.1 parent 443 0 no-query originserver ssl sslflags=DONT_VERIFY_PEER name=x_domain_com acl sites_server_x_domain_com dstdomain x.domain.com cache_peer_access x_domain_com allow sites_server_x_domain_com http_access allow sites_server_x_domain_com # end reverse proxy But how is this going to work? I can present a username/password on my auth.php and present a submit button to validate. But how do I tell squid that it is OK to serve x.domain.com? Also is there a better way of achieving my purpose? Thanks. Please help.
