Does this look reasonable? I still have the analysis to start with after this point but will use some linux tools to help with that...
auth_param basic realm P*****r ProxyServer auth_param basic credentialsttl 2 hours auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd authenticate_cache_garbage_interval 1 hour authenticate_ip_ttl 2 hours #acl all src 0.0.0.0/0.0.0.0 acl all src all acl manager proto cache_object acl localhost src 127.0.0.1 acl cacheadmin src 88.xxx.xxx.xxx 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 1863 # MSN messenger acl ncsa_users proxy_auth REQUIRED acl maxuser max_user_ip -s 2 acl CONNECT method CONNECT http_access allow manager cacheadmin http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny to_localhost http_access deny manager http_access allow ncsa_users http_access deny maxuser http_access deny all icp_access allow all http_port 8080 http_port 88.xxx.xxx.xxx:80 hierarchy_stoplist cgi-bin ? cache_mem 256MB maximum_object_size_in_memory 50 KB cache_replacement_policy heap LFUDA cache_dir aufs /var/spool/squid 40000 16 256 maximum_object_size 50 MB cache_swap_low 90 cache_swap_high 95 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log buffered_logs on refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 quick_abort_min 0 KB quick_abort_max 0 KB acl apache rep_header Server ^Apache broken_vary_encoding allow apache half_closed_clients off cache_mgr ***'***.com cachemgr_passwd ******** all visible_hostname P*****r ProxyServer log_icp_queries off dns_nameservers 208.67.222.222 208.67.220.220 hosts_file /etc/hosts memory_pools off forwarded_for off client_db off coredump_dir /var/spool/squid > ---------------------------------------- >> From: [email protected] >> To: [email protected]; [email protected] >> Date: Sat, 13 Feb 2010 16:35:29 +0000 >> Subject: RE: [squid-users] Cache manager analysis >> >> >> Thanks. >> A few questions on this: >> (a) when you said this all src all is that meant to be acl src all? >> (b) Hint 2: if possible, define an ACL or the network ranges where you >> accept logins. Use it like so >> The logins are accepted form IP addresses that I never know, it is an >> external proxy server for geo location so not sure I can do this? logins >> will only ever by directed to the 88.xxx.xxx.xxx server though? >> (c) cache_mem 100 MB >> Bump this up as high as you can go without risking memory swapping. >> Objects served from RAM are 100x faster than objects not. >> Where can I view if memeory swapping is happening? >> (D) maximum_object_size 50 MB >> Bump this up too. Holding full ISO CDs and windows service packs can >> boost performance when one is used from the cache. 40GB of disk can >> store a few. >> If I increase this, will the server ever try to store streamed video? I had >> an efficiency problem with the original configuration that came with squid, >> which meant that streamed video was buffering constantly. Not sure what >> caused it but with the current config it does not do that. >> If I increase the cache_mem and max object size do I also need to increase >> this? >> maximum_object_size_in_memory 50 KB >> (E) >> cache_swap_low 90 >> cache_swap_high 95 >> access_log /var/log/squid/access.log squid >> cache_log /var/log/squid/cache.log >> buffered_logs on >> acl QUERY urlpath_regex cgi-bin \? >> cache deny QUERY >> >> Drop the QUERY bits above. It's more than halving the things your Squid can >> store. >> Remove the acl and the cache deny? >> At present, does this stop the cache from storing anything with a ?, ie >> dynamic pages? >> What if the same request is made for a dynamic page, will it retrive it from >> the cache (old page) rather then fetch the new dynamic content? >> >> current conf redone below: >> ---------------------------- >> auth_param basic realm Proxy server >> auth_param basic credentialsttl 2 hours >> auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd >> authenticate_cache_garbage_interval 1 hour >> authenticate_ip_ttl 2 hours >> #acl all src 0.0.0.0/0.0.0.0 >> acl src all >> acl manager proto cache_object >> acl localhost src 127.0.0.1 >> acl cacheadmin src 88.xxx.xxx.xxx >> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >> acl SSL_ports port 443 >> acl Safe_ports port 80 # http >> acl Safe_ports port 21 # ftp >> acl Safe_ports port 443 # https >> acl Safe_ports port 70 # gopher >> acl Safe_ports port 210 # wais >> acl Safe_ports port 1025-65535 # unregistered ports >> acl Safe_ports port 280 # http-mgmt >> acl Safe_ports port 488 # gss-http >> acl Safe_ports port 591 # filemaker >> acl Safe_ports port 777 # multiling http >> acl Safe_ports port 1863 # MSN messenger >> acl ncsa_users proxy_auth REQUIRED >> acl maxuser max_user_ip -s 2 >> acl CONNECT method CONNECT >> http_access allow manager localhost >> http_access allow manager cacheadmin >> http_access deny !Safe_ports >> http_access deny CONNECT !SSL_ports >> http_access deny to_localhost >> http_access deny manager >> http_access allow ncsa_users >> http_access deny maxuser >> #http_access allow localhost >> http_access deny all >> icp_access allow all >> http_port 8080 >> http_port 88.xxx.xxx.xxx:80 >> hierarchy_stoplist cgi-bin ? >> cache_mem 100 MB >> maximum_object_size_in_memory 50 KB >> cache_replacement_policy heap LFUDA >> cache_dir aufs /var/spool/squid 40000 16 256 >> maximum_object_size 50 MB >> cache_swap_low 90 >> cache_swap_high 95 >> access_log /var/log/squid/access.log squid >> cache_log /var/log/squid/cache.log >> buffered_logs on >> #acl QUERY urlpath_regex cgi-bin \? >> #cache deny QUERY >> refresh_pattern ^ftp: 1440 20% 10080 >> refresh_pattern ^gopher: 1440 0% 1440 >> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >> refresh_pattern . 0 20% 4320 >> quick_abort_min 0 KB >> quick_abort_max 0 KB >> acl apache rep_header Server ^Apache >> broken_vary_encoding allow apache >> half_closed_clients off >> cache_mgr [email protected] >> cachemgr_passwd aaa all >> visible_hostname ProxyServer >> log_icp_queries off >> dns_nameservers 208.67.222.222 208.67.220.220 >> hosts_file /etc/hosts >> memory_pools off >> forwarded_for off >> client_db off >> coredump_dir /var/spool/squid >> >> ---------------------------------------- >>> Date: Sat, 13 Feb 2010 18:03:00 +1300 >>> From: [email protected] >>> To: [email protected] >>> Subject: Re: [squid-users] Cache manager analysis >>> >>> J. Webster wrote: >>>> What is the best place to start with in cache analysis? >>>> Would it be cache size, memory object size, IO, etc.? >>>> I'm looking to optimise the settings for my squid server. >>> >>> Step 0) migrate to the latest Squid 2.7 or 3.1 or if possible 2.HEAD >>> (that one is only nominally beta, it's very stable in reality) >>> >>> 1) Start by defining 'optimize' ... are you going to prioritize... >>> Faster service? >>> More bandwidth saving? >>> More client connections? >>> >>> 2a) For faster service, look at DNS delays, disk IO delays, maximizing >>> cacheable objects (dynamic objects etc). >>> >>> 2b) For pure bandwidth savings start with a look at object cacheablity. >>> Check dynamics are being cached, ranges are being fetched in full, etc >>> >>> 3) Then profile all the objects stored over a reasonably long period, >>> looking at size. compare with the age of objects being discarded. >>> >>> 3a) tune the storage limits to prioritize the storage locations. giving >>> priority to RAM, then COSS, then AUFS/diskd. >>> >>> 3b) set the storage limits as high as possible to maximize amount of >>> data stored. anywhere. >>> >>> 4) take a good long look at your access controls and in particular the >>> types speedy/fast/slow. You may get some speed benefits from fixing up >>> the ordering a bit. regex are killers, remote lookups (helpers, or DNS) >>> are second worst. >>> (some performance hints below) >>> >>> 5) repeat from (2b) as often as possible. concentrate traffic which >>> seems to logically be storeable but gets a TCP_MISS anyway. >>> >>> Objects served from cache lead to faster service ties for those objects, >>> so the speed vs bandwidth are inter-related somewhat. But there is a >>> tipping point somewhere where tuning one starts to impact the other. >>> >>> >>>> >>>> Server: about 220GB available for the cache, I'm only using 40000 MB at >>>> present as in the config below. >>>> system D2812-A2 >>>> /0 bus D2812-A2 >>>> /0/0 memory 110KiB BIOS >>>> /0/4 processor Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz >>>> /0/4/5 memory 64KiB L1 cache >>>> /0/4/6 memory 3MiB L2 cache >>>> /0/4/0.1 processor Logical CPU >>>> /0/4/0.2 processor Logical CPU >>>> /0/7 memory 3MiB L3 cache >>>> /0/2a memory 1GiB System Memory >>>> /0/2a/0 memory 1GiB DIMM DDR2 Synchronous 667 MHz (1.5 ns) >>>> /0/2a/1 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/2a/2 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/2a/3 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>> /0/1 processor >>>> /0/1/0.1 processor Logical CPU >>>> /0/1/0.2 processor Logical CPU >>>> >>>> >>>> Current squid.conf: >>>> --------------------- >>>> auth_param basic realm Proxy server >>>> auth_param basic credentialsttl 2 hours >>>> auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd >>>> authenticate_cache_garbage_interval 1 hour >>>> authenticate_ip_ttl 2 hours >>>> acl all src 0.0.0.0/0.0.0.0 >>> >>> all src all >>> >>>> acl manager proto cache_object >>>> acl localhost src 127.0.0.1/255.255.255.255 >>> >>> acl localhost src 127.0.0.1 >>> >>>> acl cacheadmin src 88.xxx.xxx.xxx >>>> acl to_localhost dst 127.0.0.0/8 >>> >>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >>> >>>> acl SSL_ports port 443 >>>> acl Safe_ports port 80 # http >>>> acl Safe_ports port 21 # ftp >>>> acl Safe_ports port 443 # https >>>> acl Safe_ports port 70 # gopher >>>> acl Safe_ports port 210 # wais >>>> acl Safe_ports port 1025-65535 # unregistered ports >>>> acl Safe_ports port 280 # http-mgmt >>>> acl Safe_ports port 488 # gss-http >>>> acl Safe_ports port 591 # filemaker >>>> acl Safe_ports port 777 # multiling http >>>> acl Safe_ports port 1863 # MSN messenger >>>> acl ncsa_users proxy_auth REQUIRED >>>> acl maxuser max_user_ip -s 2 >>>> acl CONNECT method CONNECT >>>> http_access allow manager localhost >>>> http_access allow manager cacheadmin >>> >>> Hint: add the localhost IP to the cacheadmin ACL and drop one full set >>> of "allow manager localhost" tests. >>> >>>> http_access deny manager >>>> http_access allow ncsa_users >>> >>> Hint: drop the authentication down ... >>> >>>> http_access deny !Safe_ports >>>> http_access deny CONNECT !SSL_ports >>>> http_access deny to_localhost >>> >>> ... to here. All the attacks against your proxy for bad ports and >>> sources will be dropped quickly by the security blanket settings. Load >>> on your auth server will reduce and may speed up it's response time. >>> >>> Hint 2: if possible, define an ACL or the network ranges where you >>> accept logins. Use it like so: >>> >>> http_access allow localnet ncsa_users >>> >>> ... once again that speeds up the rejections, and helps by reducing >>> the number of times the slow auth lookup needs checking. >>> >>>> http_access deny maxuser >>>> http_access allow localhost >>> >>> If localhost really is allowed to do anything, move it up above the >>> "to_localhost" one. >>> Otherwise drop this completely, having the correct auth login details >>> will permit links from localhost just as easily as from anywhere else. >>> >>>> http_access deny all >>>> icp_access allow all >>> >>> Define the networks where peer siblings are trusted. Allwo them and deny >>> everything else. >>> That will reduce a fair bit of load on your Squid trying to service >>> random ICP requests from the general Internet. >>> >>>> http_port 8080 >>>> http_port 88.xxx.xxx.xxx:80 >>>> hierarchy_stoplist cgi-bin ? >>>> cache_mem 100 MB >>> >>> Bump this up as high as you can go without risking memory swapping. >>> Objects served from RAM are 100x faster than objects not. >>> >>>> maximum_object_size_in_memory 50 KB >>>> cache_replacement_policy heap LFUDA >>>> cache_dir aufs /var/spool/squid 40000 16 256 >>> >>> If you pick 2.x squid to upgrade to, add a COSS directory as well. >>> See the recent threads on optimizing COSS for how to tune that. >>> >>>> maximum_object_size 50 MB >>> >>> Bump this up too. Holding full ISO CDs and windows service packs can >>> boost performance when one is used from the cache. 40GB of disk can >>> store a few. >>> >>>> cache_swap_low 90 >>>> cache_swap_high 95 >>>> access_log /var/log/squid/access.log squid >>>> cache_log /var/log/squid/cache.log >>>> buffered_logs on >>>> acl QUERY urlpath_regex cgi-bin \? >>>> cache deny QUERY >>> >>> Drop the QUERY bits above. It's more than halving the things your Squid >>> can store. >>> >>>> refresh_pattern ^ftp: 1440 20% 10080 >>>> refresh_pattern ^gopher: 1440 0% 1440 >>> >>> Add right here: >>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >>> >>>> refresh_pattern . 0 20% 4320 >>>> quick_abort_min 0 KB >>>> quick_abort_max 0 KB >>>> acl apache rep_header Server ^Apache >>>> broken_vary_encoding allow apache >>>> half_closed_clients off >>>> cache_mgr [email protected] >>>> cachemgr_passwd aaa all >>>> visible_hostname ProxyServer >>>> log_icp_queries off >>>> dns_nameservers 208.67.222.222 208.67.220.220 >>>> hosts_file /etc/hosts >>>> memory_pools off >>> >>> Might cause efficiency problems if the underlying malloc is not >>> optimized. but oh well, up to you. >>> >>>> forwarded_for off >>>> client_db off >>>> coredump_dir /var/spool/squid >>>> >>> >>> >>> Amos >>> -- >>> Please be using >>> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23 >>> Current Beta Squid 3.1.0.16 >> >> _________________________________________________________________ >> Send us your Hotmail stories and be featured in our newsletter >> http://clk.atdmt.com/UKM/go/195013117/direct/01/ > > _________________________________________________________________ > Send us your Hotmail stories and be featured in our newsletter > http://clk.atdmt.com/UKM/go/195013117/direct/01/ _________________________________________________________________ We want to hear all your funny, exciting and crazy Hotmail stories. Tell us now http://clk.atdmt.com/UKM/go/195013117/direct/01/
