[squid-users] Squid 3.4 reverse proxy. Cache peers on alternate ports (444) ?
I'm running squid as a reverse proxy with many web servers behind it, using host headers to determine which site the user is trying to access. One of these web servers runs IIS, with multiple websites on different ports: 443, 444, 445, etc. I would like squid to be able to connect to this server on a different port based on which address the user tries to access. https://help.mydomain.com:443 - 10.0.1.4:443 https://app.mydomain.com:443 - 10.0.1.4:444 I have tried the following to achieve this: cache_peer help.mydomain.com parent 443 0 no-query originserver ssl sslversion=3 connect-timeout=8 connect-fail-limit=2 sslflags=DONT_VERIFY_PEER front-end-https=on name=help login=PASSTHRU cache_peer app.mydomain.com parent 444 0 no-query originserver ssl sslversion=3 connect-timeout=8 connect-fail-limit=2 sslflags=DONT_VERIFY_PEER front-end-https=on name=app login=PASSTHRU unfortunately, accessing both these sites connects to the site running on port 443 on the IIS server :( How can I achieve what I'm trying to? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-3-4-reverse-proxy-Cache-peers-on-alternate-ports-444-tp4665167.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Issue with HTTP 1.0 when used with Squid
Hi Amos, Thanks for your reply. As you have mentioned, Server does not send the Content-length in the response therefore Squid sends a Connection: Close to the client. Whereas in the hit case squid knows the content-length therefore we send content-length along with keep-alive in the response to client. The Client which is sending FIN is MediaFoundation Framework(embedded within windows). The video object is rendered fine in the case of a miss but in the case of hit we see a FIN and a blank screen from the client. If the same URL is accessed using DirectShow Framework(embedded within Windows), in the case of miss the video is rendered absolutely fine. In the case of hit the video is rendered fine 4 out of 10 times. Rest of the times we see a blank screen, but there isn't any FIN from the client and wee see a complete cache hit in squid. Therefore I am guessing it might be an issue in the Client side only and not Squid. Thanks, Aditya On Thursday, 6 March 2014 2:25 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 6/03/2014 5:32 p.m., aditya agarwal wrote: Hi, I wanted a clarification if the following is an expected behavior from Squid. Our setup is as follows: Client Squid --- Server (Squid 2.7) (Windows Media Player) Miss case: 1. Client issues a HTTP 1.0 request with Connection: keep-alive in request header 2. Squid sends a HTTP 1.1 request to Server with Connection: Keep-Alive in request header 3. Server responds with HTTP 1.0 with no Connection in response header 4. Squid responds to the client with HTTP 1.1 with Connection: Close in response header HIT case: 1. Client issues a HTTP 1.0 request with Connection: keep-alive in request header 2. Squid responds with a HTTP 1.1 with Connection: Keep-Alive in response header Our problem is that in the HIT case the client is sending a FIN to Squid while we are serving the video!! So we are trying to figure out is this an expected behavior from Squid and could this behavior be our issue? Yes, for several reasons... 1) The client, Squid or server are all free to abandon the connection at any time for any reason. So while its not exactly desirable to get a FIN mid-transaction it is one of the behaviour cases that is expected. 2) If Squid does not know the length of the reply being delivered up-front or cannot send it using chunked encoding. Then Squid is required to send Connection:close and a FIN to signal completion. 2b) HTTP/1.1 chunked encoding is forbidden when sending to a HTTP/1.0 remote end (ie your client). From that behaviour I assume the video is delivered from the server without a Content-Length header. So when forwarding the MISS Squid is forced to use connection closure to signal completion. However with HIT, Squid should know the size and be able to send that. Resulting in keep-alive being possible. Why the FIN is coming from the client is not clear. Amos
[squid-users] Pinning objects in Squid Cache
Hi, I wanted to know if there is a way to PIN certain objects in Squid's cache, so that they are not removed or subjected to eviction because of the LRU policy running in squid. Thanks, Aditya
Re: [squid-users] Squid 3.4 reverse proxy. Cache peers on alternate ports (444) ?
On 13/03/2014 7:09 p.m., PSA wrote: I'm running squid as a reverse proxy with many web servers behind it, using host headers to determine which site the user is trying to access. One of these web servers runs IIS, with multiple websites on different ports: 443, 444, 445, etc. I would like squid to be able to connect to this server on a different port based on which address the user tries to access. https://help.mydomain.com:443 - 10.0.1.4:443 https://app.mydomain.com:443 - 10.0.1.4:444 One the requests are received Squid is perfectly able to route them to different cache_peer. That is done somewhat like you tried already. This configuration depends more on getting the requests at all though. Since HTTPS certificate validation hinges very much on matching the domain certified with the domain fetched when the server cert is delivered to the client before any of the HTTPS requests are sent to tell the server what domain is being fetched. This big problem can be avoided in two ways: 1) using a wildcard certificate for *.mydomain.com that can validate both sub-dmains on your listening https_port. 2) SNI (Server Name Indication) extension feature of SSL/TLS. Where the client tells the server which domain is being accessed so the server can send out a certificate for that domain. Squid does not yet support method #2 IIRC. Any contributions towards that are welcome. I have tried the following to achieve this: cache_peer help.mydomain.com parent 443 0 no-query originserver ssl sslversion=3 connect-timeout=8 connect-fail-limit=2 sslflags=DONT_VERIFY_PEER front-end-https=on name=help login=PASSTHRU cache_peer app.mydomain.com parent 444 0 no-query originserver ssl sslversion=3 connect-timeout=8 connect-fail-limit=2 sslflags=DONT_VERIFY_PEER front-end-https=on name=app login=PASSTHRU unfortunately, accessing both these sites connects to the site running on port 443 on the IIS server :( How can I achieve what I'm trying to? For the cache_peer side you are on the right track but there are a few issues that need to be fixed: 1) use cache_peer_access to determine for each request which peer server it is going to: acl help dstdomain help.mydomain.com cache_peer_access help.mydomain.com allow help cache_peer_access help.mydomain.com deny all ... etc 2) remove sslflags=DONT_VERIFY_PEER. It is okay for initial testing that the SSL/TLS connections will work. But before this goes into production you should setup Squid with the CA appropriate to verify the peers certififcates. Verifying peers prevents hijacking of the HTTPS connections between your proxy and the master servers. 3) you may need the forceddomain= option on app.mydomain.com to deliver the port change. Amos
Re: [squid-users] Issue with HTTP 1.0 when used with Squid
On 13/03/2014 9:19 p.m., aditya agarwal wrote: Hi Amos, Thanks for your reply. As you have mentioned, Server does not send the Content-length in the response therefore Squid sends a Connection: Close to the client. Whereas in the hit case squid knows the content-length therefore we send content-length along with keep-alive in the response to client. The Client which is sending FIN is MediaFoundation Framework(embedded within windows). The video object is rendered fine in the case of a miss but in the case of hit we see a FIN and a blank screen from the client. If the same URL is accessed using DirectShow Framework(embedded within Windows), in the case of miss the video is rendered absolutely fine. In the case of hit the video is rendered fine 4 out of 10 times. Rest of the times we see a blank screen, but there isn't any FIN from the client and wee see a complete cache hit in squid. Therefore I am guessing it might be an issue in the Client side only and not Squid. Yes. No need to guess. Squid is very well tested in this area with a very wide range of client software which all interoperates just fine. The odd one out being this one framework fingers it as being buggy. Amos
Re: [squid-users] Pinning objects in Squid Cache
On 13/03/2014 9:22 p.m., aditya agarwal wrote: Hi, I wanted to know if there is a way to PIN certain objects in Squid's cache, so that they are not removed or subjected to eviction because of the LRU policy running in squid. Thanks, Aditya That depends on what the objects are ... so what exactly are you trying to achieve? Meanwhile ... objects locally served up by Squid using the /squid-internal-static/ well-known URL path prefix have it. Such things as icons for the error pages and FTP directory listings. See the mime.conf file installed with your Squid on how to configure those URL objects. However, if you are wanting this for arbitrary objects served up elsewhere: * the best way is not to bother. Cache is a _temporary_ storage area (a type of buffer) not a long term archive. Correctly following HTTPP protocol ensures up to date reliable content at all times. * the second-best way is to simply poll your proxy with a request for it before the replacement policy removes it. This works on the same principle as prefetching and has all the same problems with generating correct client headers. Amos
Re: [squid-users] Squid-3.3.8: Cant force to cache-hit on this url
On 12 March 2014 20:06, Amos Jeffries squ...@treenet.co.nz wrote: On 12/03/2014 7:14 p.m., Mark Williams wrote: Greetings all, Im having problems with cache hit rates im trying to work out. I have a complete random URL im concentrating on: http://performanceforums.com/forums/mgc_cb_evo/clientscript/dojo/dijit/themes/claro/layout/TabContainer.css No matter what i do, i cannot get this URL to be served from the cache, it's always a MISS. Can anyone help me work out why? Perobably teh Vary object cachign but in 3.3. Please try 3.3.12 or 3.4.4 which just came out. They both have a fix for that amongst the other security bugs. It appears you were correct. The problem has been fixed by moving to 3.4.4. Thankyou :)
Re: [squid-users] Re: ICP and HTCP and StoreID
On Wed, Mar 12, 2014 at 1:27 AM, Alex Rousskov rouss...@measurement-factory.com wrote: On 02/14/2014 04:38 AM, Nikolai Gorchilov wrote: On Fri, Feb 14, 2014 at 7:22 AM, Alex Rousskov wrote: snip Would using ICP reqnum field as a cache key or adding StoreID to ICP/HTCP requests work for your use cases? I have not fully checked whether the former is possible, but I think it is. The latter is possible, but is more difficult to implement (and will bump into UDP packet size limits more often?). Yep. Both will do. I personally prefer the second option - StoreID URL normalization on incoming ICP/HTCP request, in order to avoid packet size bumps as much as possible. Just to make sure we are on the same page, here is a list of options I recall being discussed: 1. Using ICP reqnum field as a cache key. I don't understand how this option is going to work. AFAIK regnum is just 4 octets long - how is it supposed to accommodate the StoreID? 2. Adding StoreID to ICP/HTCP requests as an optional field. 3. Computing StoreID upon receiving a regular ICP/HTCP request. Out of those three, do you prefer #3? Note that #1 is a little hackish, but may be a easier to implement (and is a lot cheaper CPU-wise) than #3. Neither #1 nor #3 make the ICP packets bigger, unlike #2. Option 3 is the only universal solution that works in all scenarios. Sharing the a StoreID string or a derivative of it (checksum/hash/digest/whatever) will do only for peers using same StoreID rewriting logic. Best, Niki
[squid-users] Re: Automatic StoreID ?
What about a learning mechanism . For example ObjectX is at urlA , urlB , urlC . It is no problem if squid downloads ObjectX from all of them for ONE time but after that it should delete it from cache storage and serve all of them from one file . And then squid should never download them again until they changed ( based on mechanisms which currently using to check that an object should redownload or not ) Also it would be more powerful if squid could learn some relation between urlA , urlB , urlC and if ObjectY requested from one of them , then squid GUESS the same behavior . I know i am dreaming Utopia but discussion is better than silence about it . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Automatic-StoreID-tp4665140p4665174.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Re: ICP and HTCP and StoreID
On 03/13/2014 07:24 AM, Nikolai Gorchilov wrote: On Wed, Mar 12, 2014 at 1:27 AM, Alex Rousskov wrote: Just to make sure we are on the same page, here is a list of options I recall being discussed: 1. Using ICP reqnum field as a cache key. I don't understand how this option is going to work. AFAIK regnum is just 4 octets long - how is it supposed to accommodate the StoreID? By using StoreIDs that are 31 bits long. Recall that you control the StoreID map and, in most cases, there are fewer than 2^31 mapped/altered URLs in the cache, so one could use positive reqnums as regular reqnums and negative reqnums as this is my special StoreID reqnums. There are other caveats or optimizations that may make sense with this scheme. And, as I said earlier, this is a hack (that may work well in some environments). 2. Adding StoreID to ICP/HTCP requests as an optional field. 3. Computing StoreID upon receiving a regular ICP/HTCP request. Out of those three, do you prefer #3? Note that #1 is a little hackish, but may be a easier to implement (and is a lot cheaper CPU-wise) than #3. Neither #1 nor #3 make the ICP packets bigger, unlike #2. Option 3 is the only universal solution that works in all scenarios. Sharing the a StoreID string or a derivative of it (checksum/hash/digest/whatever) will do only for peers using same StoreID rewriting logic. Yes, of course. And with a StoreID cache or, in the worst case, a loaded module computing Store IDs, it will be fast enough too. Cheers, Alex.
Re: [squid-users] Pinning objects in Squid Cache
Hi Amos, The objects that we are trying to cache are video files. We want to cache these videos for a particular time frame and insure that they are not evicted from cache during this time frame. We had already thought of the second option to fetch the objects at regular intervals so that they are always at the head of the queue in cache, but it doesn't seem to be very scalable as we can have 100s of videos which the client might want to pin to cache. I wanted to know if there is any modification that can be done in squid to support pinning of objects. Thanks, Aditya On Thursday, 13 March 2014 3:05 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 13/03/2014 9:22 p.m., aditya agarwal wrote: Hi, I wanted to know if there is a way to PIN certain objects in Squid's cache, so that they are not removed or subjected to eviction because of the LRU policy running in squid. Thanks, Aditya That depends on what the objects are ... so what exactly are you trying to achieve? Meanwhile ... objects locally served up by Squid using the /squid-internal-static/ well-known URL path prefix have it. Such things as icons for the error pages and FTP directory listings. See the mime.conf file installed with your Squid on how to configure those URL objects. However, if you are wanting this for arbitrary objects served up elsewhere: * the best way is not to bother. Cache is a _temporary_ storage area (a type of buffer) not a long term archive. Correctly following HTTPP protocol ensures up to date reliable content at all times. * the second-best way is to simply poll your proxy with a request for it before the replacement policy removes it. This works on the same principle as prefetching and has all the same problems with generating correct client headers. Amos
[squid-users] SquidGuard redirect to parent proxy (Off-Topic)
Hi, I know that my question is a little bit off-topic but nevertheless I hope that some can help me :-) I've configured squid3 with squidguard and one parent-proxy. In the case of access violation squidguard redirects the user to a customized block page hosted by the proxy himself. Unfortunately the proxy tries to access the local blockpage over his parent proxy. Does some have an idea why? Even if I use the hostname or the ip address... the result is the same Here is the part from my sqzidguard.conf: [...] default { pass white !ads !aggressive !audio-video !drugs !gambling !hacking !mail !porn !proxy !redirector spyware !suspect !violence !warez !webradio all redirect http://proxyname.localsuffix/blockpage/index.cgi?%a+%i+%t+%u } Best regards Christian
Re: [squid-users] Pinning objects in Squid Cache
On 03/13/2014 09:52 AM, aditya agarwal wrote: We had already thought of the second option to fetch the objects at regular intervals so that they are always at the head of the queue in cache, but it doesn't seem to be very scalable as we can have 100s of videos which the client might want to pin to cache. Please also keep in mind that not all cache_dir types support LRU. For example, frequently requesting URLs in Rock storage would not help much. I wanted to know if there is any modification that can be done in squid to support pinning of objects. Yes, it would be possible to add such support. I can think of several options: * A cache manager interface to pin and unpin individual cached objects. It will not be simple if you want pinning to last across Squid restarts or if you want to pin using regular expressions and such. * An extension HTTP request method for the same purpose, but cache manager may be an overall better approach, especially from access control point of view. * It is also possible to add an eCAP/ICAP (or even a new helper) interface to mark misses for pinning. Adaptation makes pinning using regular expressions easy, but it will add performance overheads unless you are already using an adaptation service. * Finally, one could make a pinning set configurable via squid.conf ACLs. For mostly static sets that can be stored in a few MB or RAM (thousands of URLs, not millions), this is probably the most efficient and simple option. If you decide to work on any of this, please consider discussing specifics on squid-dev first. There are caveats related to each option and the choice of the best option is not obvious IMO. Cheers, Alex. On Thursday, 13 March 2014 3:05 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 13/03/2014 9:22 p.m., aditya agarwal wrote: Hi, I wanted to know if there is a way to PIN certain objects in Squid's cache, so that they are not removed or subjected to eviction because of the LRU policy running in squid. Thanks, Aditya That depends on what the objects are ... so what exactly are you trying to achieve? Meanwhile ... objects locally served up by Squid using the /squid-internal-static/ well-known URL path prefix have it. Such things as icons for the error pages and FTP directory listings. See the mime.conf file installed with your Squid on how to configure those URL objects. However, if you are wanting this for arbitrary objects served up elsewhere: * the best way is not to bother. Cache is a _temporary_ storage area (a type of buffer) not a long term archive. Correctly following HTTPP protocol ensures up to date reliable content at all times. * the second-best way is to simply poll your proxy with a request for it before the replacement policy removes it. This works on the same principle as prefetching and has all the same problems with generating correct client headers. Amos
Re: [squid-users] Re: Automatic StoreID ?
On 2014-03-14 02:37, Omid Kosari wrote: What about a learning mechanism . For example ObjectX is at urlA , urlB , urlC . It is no problem if squid downloads ObjectX from all of them for ONE time but after that it should delete it from cache storage and serve all of them from one file . And then squid should never download them again until they changed ( based on mechanisms which currently using to check that an object should redownload or not ) Also it would be more powerful if squid could learn some relation between urlA , urlB , urlC and if ObjectY requested from one of them , then squid GUESS the same behavior . I know i am dreaming Utopia but discussion is better than silence about it . You just described how Store-ID feature works today. The map of urlA == urlB == urlC is inside the helper. You can make it a static list of regex patterns like the original Squid-2 helpers, a DB text file of patterns like the bundled Squid-3 helper, or anything else you like inside the helper. Squid learns the mappings by asking the helper about each URL. There is a helper response cache on these lookups same as other helpers and prevent complex/slow mappings having much impact on hot objects. Amos
Re: [squid-users] SquidGuard redirect to parent proxy (Off-Topic)
On 2014-03-14 05:21, Christian Scholz wrote: Hi, I know that my question is a little bit off-topic but nevertheless I hope that some can help me :-) I've configured squid3 with squidguard and one parent-proxy. In the case of access violation squidguard redirects the user to a customized block page hosted by the proxy himself. Unfortunately the proxy tries to access the local blockpage over his parent proxy. Does some have an idea why? 1) this is a re-write, not a redirect. HTTP redirects have a 3xx status code prefixing the URL in squidguard config. redirect 302:http://example.com/ # redirect client to example.com redirect http://example.com/ # re-write URL to http://example.com and fetch 2) you probably also have no cache_peer_access rules preventing the parent from being a source for these ttp://proxyname.localsuffix/... URLs. Amos
[squid-users] squid 3.4.4 and squidGuard 1.4 redirection 404 error
Hi, I have upgraded my squid proxy from 3.1.23 to 3.4.4 with squidGuard 1.4. For some reasons, the redirection from the result of squidGuard throws out the 404 error. However, I switch back 3.1.23 and the issue goes away. Any suggestion? Thanks.
[squid-users] Delay Pools
Testing delay_pools on 3.3.11 (and other versions too) delay_pools 1 delay_class 1 3 delay_parameters 1 -1/-1 -1/-1 128000/256000 delay_access 1 allow all This works as expected. However, when requests goes a bit higher, say 20 req/s, the CPU shoots up to 100%, and strace shows epoll calls: epoll_ctl(6, EPOLL_CTL_MOD, 7251, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7251, u64=15832364667084217427}}) = 0 epoll_ctl(6, EPOLL_CTL_MOD, 8068, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=8068, u64=14612651267063816068}}) = 0 epoll_wait(6, {{EPOLLOUT, {u32=46, u64=3639753988997382190}}, {EPOLLOUT, {u32=7111, u64=16029163435841297351}}, {EPOLLOUT,.. [snip] Is this normal? Next, I also tried client_delay_pools (3.3.11/3.4.3) client_delay_pools 1 client_delay_access 1 allow all client_delay_parameters 1 128000 256000 This gets connection reset straightaway. What am I missing?
Re: [squid-users] Re: Automatic StoreID ?
On Thu, Mar 13, 2014 at 9:37 PM, Omid Kosari omidkos...@yahoo.com wrote: What about a learning mechanism . For example ObjectX is at urlA , urlB , urlC . For static objects/URLs, that should work. For others, every time you access urlA, you get something different.
Re: [squid-users] Re: Automatic StoreID ?
On 14/03/2014 4:56 p.m., csn233 wrote: On Thu, Mar 13, 2014 at 9:37 PM, Omid Kosari omidkos...@yahoo.com wrote: What about a learning mechanism . For example ObjectX is at urlA , urlB , urlC . For static objects/URLs, that should work. For others, every time you access urlA, you get something different. The concept behind Store-ID is to de-duplicate *identical* objects at different URI. When the URL is dynamic like that it is not even a candidate for Store-ID. Amos
Re: [squid-users] Pinning objects in Squid Cache
Hi Alex, Thanks for you response. Please also keep in mind that not all cache_dir types support LRU. For example, frequently requesting URLs in Rock storage would not help much. What do you mean by Rock Storage here? Thanks, Aditya On Thursday, 13 March 2014 11:13 PM, Alex Rousskov rouss...@measurement-factory.com wrote: On 03/13/2014 09:52 AM, aditya agarwal wrote: We had already thought of the second option to fetch the objects at regular intervals so that they are always at the head of the queue in cache, but it doesn't seem to be very scalable as we can have 100s of videos which the client might want to pin to cache. Please also keep in mind that not all cache_dir types support LRU. For example, frequently requesting URLs in Rock storage would not help much. I wanted to know if there is any modification that can be done in squid to support pinning of objects. Yes, it would be possible to add such support. I can think of several options: * A cache manager interface to pin and unpin individual cached objects. It will not be simple if you want pinning to last across Squid restarts or if you want to pin using regular expressions and such. * An extension HTTP request method for the same purpose, but cache manager may be an overall better approach, especially from access control point of view. * It is also possible to add an eCAP/ICAP (or even a new helper) interface to mark misses for pinning. Adaptation makes pinning using regular expressions easy, but it will add performance overheads unless you are already using an adaptation service. * Finally, one could make a pinning set configurable via squid.conf ACLs. For mostly static sets that can be stored in a few MB or RAM (thousands of URLs, not millions), this is probably the most efficient and simple option. If you decide to work on any of this, please consider discussing specifics on squid-dev first. There are caveats related to each option and the choice of the best option is not obvious IMO. Cheers, Alex. On Thursday, 13 March 2014 3:05 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 13/03/2014 9:22 p.m., aditya agarwal wrote: Hi, I wanted to know if there is a way to PIN certain objects in Squid's cache, so that they are not removed or subjected to eviction because of the LRU policy running in squid. Thanks, Aditya That depends on what the objects are ... so what exactly are you trying to achieve? Meanwhile ... objects locally served up by Squid using the /squid-internal-static/ well-known URL path prefix have it. Such things as icons for the error pages and FTP directory listings. See the mime.conf file installed with your Squid on how to configure those URL objects. However, if you are wanting this for arbitrary objects served up elsewhere: * the best way is not to bother. Cache is a _temporary_ storage area (a type of buffer) not a long term archive. Correctly following HTTPP protocol ensures up to date reliable content at all times. * the second-best way is to simply poll your proxy with a request for it before the replacement policy removes it. This works on the same principle as prefetching and has all the same problems with generating correct client headers. Amos
[squid-users] Re: SquidGuard redirect to parent proxy (Off-Topic)
You need to make sure, that something like this is in your squid.conf: acl local-server dstdomain .mydomain.com acl blockeddomains dstdomain blockeddomains.lst #file contains list of blocked domains http_access deny blockeddomains deny_info http://mydomain.com/blocked.html blockeddomains #mydomain.com is hosted on #local_host/same machine as squid . always_direct allow local-server #To access mydomain.com NOT via parent proxy never_direct allow all MfG :-) -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/SquidGuard-redirect-to-parent-proxy-Off-Topic-tp4665178p4665187.html Sent from the Squid - Users mailing list archive at Nabble.com.