Hi Amos, regrets, I am late.
On 21 April 2015 at 09:15, Amos Jeffries <squ...@treenet.co.nz> wrote: > On 20/04/2015 7:31 p.m., Jagannath Naidu wrote: > > Hi, > > > > I am having this issue very frequently. Please help on this. > > > > I get these errors randomly, mostly when usage is at very peak. (800 > users) > > > > > > /var/log/squid/cache.log > > > > 2015/04/20 12:37:40| externalAclLookup: 'wbinfo_group_helper' queue > > overload (ch=0x7fc99e2ce518) > > What do you think "overload" means? > The helper is unable to cope with the traffic load being passed to it. > > Here is the biggest hint: > > > > in /var/log/messages, I get the following errors > > > > pr 20 12:59:15 GGNPROXY01 winbindd[1910]: winbindd: Exceeding 200 > client > > connections, no idle connection found > > > > > > Then squid stops working. For squid to start work again, I have to dlete > > the cache and restart the squid "squid -k reconfigure", and then squid > > restart. > > What Squid version are you using? > > my squid version squid-3.1.10-19.el6_4.x86_64 > > > > squid.conf > > > > max_filedesc 17192 > > acl manager proto cache_object > > acl localhost src 172.16.50.61/24 > > changed to "acl localhost src 172.16.50.6*1*" already > You have an entire /24 (256 IPs) assigned to this machine? > > I think you need to remove that "/24" part if the *.61 is the local > machines *public* IP. > > > > http_access allow manager localhost > > dns_nameservers 172.16.3.34 10.1.2.91 > > acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63 > > 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157 > > http_access allow allowips > > > auth_param basic realm Squid proxy-caching web server > > auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0 > > children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl > > The above two very mangled config lines are useless. Remove them. > > > acl localnet src 172.16.0.0/24 > changed > Its a bit strange that none of the localhost machine IPs > (172.16.50.0-172.16.50.255) are part of the LAN its plugged into > 172.16.0.0-172.16.0.255. > > > > acl localnet src fc00::/7 # RFC 4193 local private network range > > acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) > machines > > auth_param ntlm program /usr/bin/ntlm_auth --diagnostics > > --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET > > Okay you have configured NTLM... > > > auth_param ntlm program /usr/bin/ntlm_auth > > --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET > > ... but twice. With different settings. Only these last ones will have > any effect. > > > > auth_param ntlm children 600 > > auth_param ntlm keep_alive off > > > auth_param negotiate children 150 > > auth_param negotiate keep_alive off > > visible_hostname GGNPROXY01.HTMEDIA.NET > > external_acl_type wbinfo_group_helper ttl=0 children=40 %LOGIN > > /usr/lib64/squid/wbinfo_group.pl -d > > auth_param negotiate keep_alive off > > You have several useless configuration lines for Negotiate auth which is > not being used in any way. Remove those. > > > > acl Safe_ports port 8080 #https > > acl SSL_ports port 443 > > acl Safe_ports port 80 # http > > acl Safe_ports port 21 # ftp > > acl Safe_ports port 443 # https > > acl Safe_ports port 70 # gopher > > acl Safe_ports port 210 # wais > > acl Safe_ports port 1025-65535 # unregistered ports > > acl Safe_ports port 280 # http-mgmt > > acl Safe_ports port 488 # gss-http > > acl Safe_ports port 591 # filemaker > > acl Safe_ports port 777 # multiling http > > acl CONNECT method CONNECT > > acl auth proxy_auth REQUIRED > > acl google dstdomain -i "/etc/squid/google_site.com" > > http_access allow google > > acl sq1 external wbinfo_group_helper "/etc/squid/HT/sq1" > > acl sq2 external wbinfo_group_helper "/etc/squid/HT/sq2" > > acl sq3 external wbinfo_group_helper "/etc/squid/HT/sq3" > > acl sq4 external wbinfo_group_helper "/etc/squid/HT/sq4" > > acl sq5 external wbinfo_group_helper "/etc/squid/HT/sq5" > > acl pro1 external wbinfo_group_helper "/etc/squid/HT/pro1" > > acl pro2 external wbinfo_group_helper "/etc/squid/HT/pro2" > > acl pro3 external wbinfo_group_helper "/etc/squid/HT/pro3" > > acl pro4 external wbinfo_group_helper "/etc/squid/HT/pro4" > > acl pro5 external wbinfo_group_helper "/etc/squid/HT/pro5" > > acl pro6 external wbinfo_group_helper "/etc/squid/HT/pro6" > > acl webvip external wbinfo_group_helper "/etc/squid/HT/webvip" > > acl allgroup external wbinfo_group_helper "/etc/squid/HT/allgreop" > > acl restricted external wbinfo_group_helper "/etc/squid/HT/restricted" > > acl ad_auth proxy_auth REQUIRE > > You already have an ACL named "auth" which performs authentication. > The above line is not useful. Remove it and replace all uses of > "ad_auth" ACL with "auth" ACL. > > > acl allowwebsites dstdomain -i "/blacklists/allowedwebsite/domains" > > acl allowwebsites_url url_regex -i "/blacklists/allowedwebsite/url" > > http_access allow allowwebsites > > http_access allow allowwebsites_url > > acl shopping dstdomain -i "/etc/squid/shopping.txt" > > acl social_networking dstdomain -i "/blacklists/social/social.networking" > > acl youtube dstdomain -i .youtube.com > > http_access allow Safe_ports pro1 pro2 pro3 pro4 pro5 pro6 webvip > > Incorrect use of "Safe_ports" security check. Correct usage is to deny > access to all *unsafe* ports. They are unsafe because HTTP can be > smuggled within the ports native protocol to attack your proxy. > > Once the correct security protections for Safe_port and CONNECT tunnels > have been moved up the top remove the "Safe_ports" check from this line. > > This line is also very odd in another way. ACL tests in a single line > are AND'ed together - so this means the request must be from a user who is: > authenticated AND a member of group pro1 AND pro2 AND pro3 AND pro4 > AND pro5 AND pro6 AND webvip > > This hints at what your main helper problem is. The above line requires > 7 group helper lookups *per request*. The winbind helper has a maximum > of 200 simultaneous connections. This line alone will limit your proxy > just under 30 new visitors per second (that becomes 60 lookups/sec > before queue overload). > The helper result caching will help a lot, but you also have a LOT of > other group checks being made and 800 users. > > > > http_access allow youtube pro5 > > http_access allow youtube pro6 > > http_access allow youtube webvip > > http_access deny youtube > > http_access allow shopping pro5 > > http_access allow shopping pro6 > > http_access allow shopping webvip > > http_access deny shopping > > Optimization hint: > "youtube" and "shopping" have the same allow/deny criteria. It would be > worth combining them into one ACL. > > > http_access allow social_networking pro2 > > http_access allow social_networking pro4 > > http_access allow social_networking pro6 > > http_access allow social_networking webvip > > http_access deny social_networking > > acl porn_site1 dstdomain "/etc/squid/blacklists/porn/domains.txt" > > acl porn_site2 dstdom_regex -i "/etc/squid/blacklists/porn/expressions" > > acl porn_site3 dstdom_regex -i "/etc/squid/blacklists/porn/urls.txt" > > acl audio_video1 dstdomain "/etc/squid/blacklists/audio-video/urls.txt" > > ###################### THERE ARE TOO MANY acls and http_access , so not > > bothering with vast linux > > I will bet a lot of those ACLs are also calling the group helper too yes? > > > http_access allow liquorinfo webvip > > http_access deny liquorinfo > > http_access allow ad_auth > > http_access allow auth > > Once you have removed ad_auth ACL, this becomes: > http_access allow auth > http_access allow auth > > I hope you can see how redundant that is. > > Also, its very likely that the "allow auth" is a useless operation after > a great many group checks have also performed authentication. That "TOO > MANY acls and https_access" list you omitted will be needed to determine > that. > > > > http_access allow sq1 sq2 > > acl NTLMUsers proxy_auth REQUIRED > > You already have an ACL named "auth" which performs authentication. > The above line is not being used in any way. Remove it. > > > http_access deny !Safe_ports > > http_access deny CONNECT !SSL_ports > > These are basic security protection against Denial of Service and other > types of protocol smuggling attacks. They only work when they are used > *above* your custom "allow" rules. > > Move these two lines above your "http_access allow google" line. > > > > > http_port 8080 > > hierarchy_stoplist cgi-bin ? > > The above line is not useful these days. Remove it. > > > cache_effective_user squid > > cache_dir aufs /var/spool/squid 20384 32 512 > > cache_mem 50 MB > > cache_replacement_policy heap LFUDA > > cache_swap_low 85 > > cache_swap_high 95 > > maximum_object_size 5 MB > > maximum_object_size_in_memory 50 KB > > ipcache_size 5240 > > ipcache_low 90 > > ipcache_high 95 > > cache_mgr amit > > acl SSL_ports port 443 > > The above is a duplicate config line. Remove it. > > > http_access allow CONNECT SSL_ports > > coredump_dir /var/spool/squid > > refresh_pattern ^ftp: 1440 20% 10080 > > refresh_pattern ^gopher: 1440 0% 1440 > > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > > refresh_pattern . 0 20% 4320 > > url_rewrite_program /usr/local/bin/squidGuard -c > > /usr/local/squidGuard/squidGuard.conf > > > > > Now, as to solving your problem: > > 1) Clean up your config. Reduce the amount of redundant or unused > things. I've mentioned a few above. > > 2) Run "squid -k parse" and fix any other problems it highlights. > > 3) optimize your ACls and http_access rules. I've mentioned a few, such > as moving the main security checks to the top so DoS traffic does not > put load on the helpers and other ACLs. > > I believe though that you will probably find Squid works much better > having the following access controls pattern: > " > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > > # if they are not authenticated, they will not be in a group > http_access deny !auth > > # assuming that webvip are the group with full access? > http_access allow webvip > > # your long list of per-site group check ACLs go here > ... > > # this is where defining the LAN ranges correctly comes in. > # note that users have authenticated simply to get near here > http_access allow localnet > http_access deny all > " > > > 4) consider an upgrade to Squid 3.4+. The "notes" ACL type offers much > more efficient ACL testing with a custom group lookup helper. The all-of > and any-of ACL types can also much reduce your http_access lines. > > HTH > Amos > _______________________________________________ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users > Thank you Amos, I will check and will update the list. -- Thanks & Regards B Jagannath Keen & Able Computers Pvt. Ltd.
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
