[squid-users] Squid 2.7 working with reference to storeurl/caching?
Hi all, I request that please read my squid.conf file and guide me on the order of the directives and any issue as i am unable to cache a single thing. Does it matter the order of definition of the following? 1. storeurl program 2. refresh patterns 3 storeurl rewrite lists... I assume : 1. whenever a user open a page in his user agent squid very first of all check the refresh pattern for deciding whether to search in the cache or go to web. Am i right? 2. Now if the request match the storeurl rewrite lists then the request is forwarded to storeurl program who then see that if the object is available in the cache in that case it is returned. Otherwise object is fetched from the web and stored as store_url for future reference. Please guide i am totally unclear? 3. With the following squid.conf not a single object is being cached. I am not sure whats happening? # This is the configuration file for instance 2 which is doing all the caching. squid v 2.7 stable 9 is chosen for its store_url feature. visible_hostname squidlhr1 unique_hostname squidlhr1cache cache_effective_user proxy # Directives to enhance security. allow_underscore off httpd_suppress_version_string on forwarded_for off log_mime_hdrs on pid_filename /var/run/inst2squid.pid access_log /var/logs/inst2access.log squid cache_log /var/logs/inst2cache.log cache_store_log /var/logs/inst2store.log http_port 1975 icp_port 0 # This option must be supported through giving at compilation snmp_port 7172 #Explicit definition of all is must in squid 2.7 version acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl CONNECT method CONNECT # Only allow cachemgr access from localhost http_access allow manager localhost http_access deny manager # If peering with ISA then following two directives will be required. Otherwise not #cache_peer 10.1.82.205 parent 8080 0 default no-digest no-query no-delay #never_direct allow all cache_dir aufs /cachedisk1/var/spool/squid 5 128 256 coredump_dir /cachedisk1/var/spool/squid cache_swap_low 75 #should be 1/4 of the physical memory installed in the system cache_mem 1000 MB range_offset_limit -1 KB maximum_object_size 4194304 KB minimum_object_size 10 KB quick_abort_min -1 KB cache_replacement_policy heap LFUDA # This portion is not understood yet well # Let the clients favorite video site through with full caching # - they can come from any of a number of youtube.com subdomains. # - this is NOT ideal, the 'merging' of identical content is really needed here acl youtube dstdomain .youtube.com cache allow youtube #-Refresh Pattern Portion-- # Custom Refresh patterns will come first # Updates windows/debian etc.. refresh_pattern windowsupdate.com/.*.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims refresh_pattern update.microsoft.com/.*.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims refresh_pattern download.microsoft.com/.*.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims refresh_pattern download.windowsupdate.com/.*\.(cab|exe|dll|msi) 1440 100% 43200 reload-into-ims refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880 refresh_pattern .deb$ 518400 100% 518400 override-expire #specific for youtube custom refreshpatterns belowones refresh_pattern -i (get_video\?|videoplayback\?|videodownload\?) 5259487 % 5259487 override-expire ignore-reload # Break HTTP standard for flash videos. Keep them in cache even if asked not to. refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache override-expire ignore-private # Other long-lived items refresh_pattern -i .(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)(\?|$) 161280 3000% 525948 override-expire reload-into-ims #Trial/Test refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|mpg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-private refresh_pattern -i \.(deb|rpm|exe|ram|bin|pdf|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-private refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-private refresh_pattern -i \.(zip|gz|arj|lha|lzh|tar|tgz|cab|rar)$ 10080 95% 43200 override-expire ignore-no-cache ignore-private refresh_pattern -i \.(php|asp|aspx|cgi|html|htm|css|js) 1440 40% 40320 refresh_pattern ^http://*.gmail.*/.* 720 100% 4320 refresh_pattern ^http://*.twitter.*/.* 720 100% 4320 refresh_pattern ^http://*.yimg.*/.* 720 100% 4320 refresh_pattern ^http://*.ymail.*/.* 720 100% 4320 refresh_pattern ^http://*.hotmail.*/.* 720 100% 4320 refresh_pattern ^http://*.live.*/.* 720 100% 4320 refresh_pattern ^http://*.wikipedia.*/.* 720 100% 4320 refresh_pattern ^http://wiki.*.*/.* 720 100% 4320 refresh_pattern ^http://*.profile/.* 720 100% 4320 refresh_pattern ^http://*.yahoo.*/.* 720 100% 4320 refresh_pattern ^http://*.microsoft.*/.* 720 100% 4320
Re: [squid-users] Squid 3.1 rejecting connections after few thousands requests
Henrik Nordström wrote: lör 2010-05-22 klockan 15:08 +1200 skrev Amos Jeffries: Wow. Sure thats hits/sec and not hits/minute? The 'extreme' setups of Squid-2.7 only reached 990req/sec. 990 isn̈́'t the extreme.. but very high. I mean the ExtremeCarpFrontend configuration examples. 990rps was simply the limit reached on the testing hardware. There may be hardware able to go faster already. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
Re: [squid-users] Startup/shutdown script which was working perfactly alright for squid 3.0stable25 is not working for squid 2.7 stable9.0
GIGO . wrote: Hi all, I am able to run squid manually however whenever i try to run it through the startup/shutdown script it fails. This is the same script working for squid 3.0 stable 25 however i am not being able to figure out that why its failing on squid 2.7 stable 9? Neither of the instance starts with system startup. Please guide me i be thankful. My startup script and tail of cache.log for both instances is below. #!/bin/sh # #my script case $1 in start) /usr/sbin/squid -D -s -f /etc/squid/squidcache.conf /usr/sbin/squid -D -s -f /etc/squid/squid.conf #The below line is to automatically start apache with system startup /usr/sbin/httpd -k start #KRB5_KTNAME=/etc/squid/HTTP.keytab #export KRB5_KTNAME #KRB5RCACHETYPE=none #export KRB5RCACHETYPE ;; stop) /usr/sbin/squid -k shutdown -f /etc/squid/squidcache.conf echo Shutting down squid secondary process /usr/sbin/squid -k shutdown -f /etc/squid/squid.conf echo Shutting down squid main process # The below line is to automatically stop apache at system shutdown /usr/sbin/httpd -k stop ;; esac The script looks right to me. tail instance 2 cache file: 2010/05/22 06:05:18| Beginning Validation Procedure 2010/05/22 06:05:18| Completed Validation Procedure 2010/05/22 06:05:18| Validated 0 Entries 2010/05/22 06:05:18| store_swap_size = 0k 2010/05/22 06:05:18| storeLateRelease: released 0 objects 2010/05/22 06:09:28| Preparing for shutdown after 62 requests This message means the Squid instance has received the shutdown signal from some external process. Either kill or squid -k shutdown. 2010/05/22 06:09:28| Waiting 30 seconds for active connections to finish 2010/05/22 06:09:28| FD 16 Closing HTTP connection 2010/05/22 06:09:28| WARNING: store_rewriter #1 (FD 7) exited 2010/05/22 06:09:28| Too few store_rewriter processes are running 2010/05/22 06:09:28| Starting new helpers 2010/05/22 06:09:28| helperOpenServers: Starting 1 'storeurl.pl' processes That may be a bug, restarting helpers on shutdown looks wrong. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
Re: [squid-users] Squid 2.7 working with reference to storeurl/caching?
GIGO . wrote: Hi all, I request that please read my squid.conf file and guide me on the order of the directives and any issue as i am unable to cache a single thing. Does it matter the order of definition of the following? You seems to make a lot of queries asking what squid.conf directives mean. Are you aware of the online squid.conf manual? http://www.squid-cache.org/Doc/config/ 1. storeurl program no. 2. refresh patterns only in relation to other of these directives. 3 storeurl rewrite lists... only in relation to other of these directives. I assume : 1. whenever a user open a page in his user agent squid very first of all check the refresh pattern for deciding whether to search in the cache or go to web. Am i right? No. First check is whether security access is to be granted. then other security lookups, then possible cacheability of the reply, then what to send in the relayed request, then source for the reply, then how to fetch the reply, then whether the reply is acceptable. Thats a general overview, fine details are difficult to explain simply. 2. Now if the request match the storeurl rewrite lists then the request is forwarded to storeurl program who then see that if the object is available in the cache in that case it is returned. Otherwise object is fetched from the web and stored as store_url for future reference. Please guide i am totally unclear? No storeurl access lists determine whether the storeurl_program is allowed to alter the storage location for the reply. Whether to store is determined by the cache directive and reply cache control headers. 3. With the following squid.conf not a single object is being cached. I am not sure whats happening? # This is the configuration file for instance 2 which is doing all the caching. squid v 2.7 stable 9 is chosen for its store_url feature. snip # This portion is not understood yet well # Let the clients favorite video site through with full caching # - they can come from any of a number of youtube.com subdomains. # - this is NOT ideal, the 'merging' of identical content is really needed here acl youtube dstdomain .youtube.com cache allow youtube The default caching permission is to allow. UNLESS you specify otherwise. Also, ... allow/deny control lists terminate with an implicit rule inverting the permission of the previous configuration line. What that means is that your configuration above actually doing this: cache allow youtube cache deny all What you need to do is remove the cache allow youtube rule and let squid default allow all to work properly. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
Re: [squid-users] 401 Unauthorized
zongo saiba wrote: Hi, I am using Squid Version 2.7.STABLE7 with Ubuntu 10.04. Everything seems to be working fine but I have one issue which is to be able to pass on the credential windows to squid for one of my remote site. I have tried using cache_peer as such: cache_peer hostname sibling 80 0 login=PASS sibling ... you are passing the credentials to another proxy server that listens on port 80 instead of a proxy port? To pass to a web server use parent type and also the originserver option. But to no avail. I added this line -- cache_peer_access hostname allow all -- but to no avail permitting requests to MAYBE go through the peer... Finally I tried this -- never_direct deny all -- Again, to no avail. forcing all traffic to ALWAYS go through the peer... I decided to tried to create an acl for this particular domain as such: acl xxx dstdomain domainname.com http_access allow xxx But again all this to no avail. permitting the requests for this domain to go through your Squid. Then I realised that this might have something to do with HTT header. As per doc, basic authentication seems to use the WWW-Authenticate HTTP Header. I have that disable for security reasons. So I did find out why So for security reasons you have to prevent all authentication passing through your proxy? Please explain these security reasons which lead you to disabling remote websites security while still allowing general access to public websites which have no security? the credentials window is not passed on the my browser. I decided to try If by credentials window you mean the little popup box that browsers themselves create when they need to supply credentials to a website but do not have any stored for use already ... then, yes you are removing all versions of the authentication methods header. There are several of these headers added by web servers advertising the many protocols they can use for auth. Without any of them at all the browser does not know what type of credentials to send, so can't do anything but error out. Basic auth which you seem to be afraid of is only one protocol of many which use these headers. to create a customer HTTP header for this site as such: acl cdnCustomHeader req_header ^domainame.com http_access allow cdnCustomHeader WTF? This had the effect of reporting the 401 error on all the websites I was connecting. reporting as in how? you mean the browsers suddenly started showing error pages with 401 Authorization Required status messages? I did try with this as well but again to no avail header_access WWW-Authenticate allow hostname. WWW_Authenticate header does not contain a hostname of any such thing. It usually contains one word from these: Basic, NTLM, Negotiate, Digest, and maybe a custom auth protocol name for certain security agents. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
Re: [squid-users] 2.7 upstream parent (cache_peer) connection reset. Child how to handle?
James Tan wrote: Hi Amos, the PoC is for a project involving malware inspection, a personal project. I tried to chain 2 Squids as part of solution. The AV perform the check on the wire before actually allowing Parent Squid to get hold of it. I.e. Client -- ... ... - Parent Squid -- AV (inspects HTTP, it it is 'infected', do a TCP Disconnect as seen on Sysinternals Procmon) -- Website *There was no TCP Disconnect for 'clean' pages. From what I observe when the client is directly connected to the Parent Squid, I got the following message in Parent. I am OK with this message in Parent, but how can I let the Child also know that and display similar message when Parent got it instead of hung? I suspect you have something like the half_closed_clients setting turned on or that the child Squid is stuck in a period of re-tries looping to find a source which will supply the requested information. FWIW; you are better off using a Squid-3 as the parent with AV capabilities plugged in directly via the ICAP interface. Most AV software these days seems to have some form of ICAP server you can plug Squid into. This will let either the AV or the parent Squid supply the client with an nice explanation page about what and why the request was aborted. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
Re: [squid-users] Slowness in downloading files, but not web browsing
Kelly, Jack wrote: Hi everyone, We're running Squid 3.1.1 on a virtual Ubuntu x64 server sitting on a fiber LUN. It's been up for a couple of months without issue any issues until recently. Over the past week or so I've had users calling in to report that downloading files from the internet has been very slow. They'll start out with a fast download speed, but it will quickly go down to about 5kb/sec. If I circumvent the proxy server and connect to these sites directly, the download goes right through no problem. I've tried restarting the Squid service with no luck. Any suggestions? Thanks Jack You may want to check the FD usage and restart Squid to see if it resolves the issue. 3.1 has a known slow leak under some obscure circumstances we have not managed to track down yet. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.3
