Re: [squid-users] Eui logging bug
There is now a patch in the bug report which seems to fix this issue and a few others. http://bugs.squid-cache.org/show_bug.cgi?id=3982 I would like confirmation (in the bug report please) from some others testing before this gets applied. Cheers Amos
[squid-users] cache allow deny precedence WAS: Re: Cache Windows Updates ONLY
On 13 April 2014 16:11, Nick Hill wrote: > To paraphrase the logic coded here: > Don't cache anything with a query string UNLESS it matches the ACL > wupdatecachablequery. > > another way to write this more succinctly might be: > cache deny QUERY > cache allow wupdatecachablequery > > But I am not certain whether the deny clause will take a higher > priority than the allow clause in cases where both ACLs match. The > fandangled logic avoids this. > If the 'cache' directive follows the same rules as http_access, then Squid will stop processing any cache allow/deny directives as soon as any cache allow/deny directives have a positive match. So I can replace my logic with: cache allow wupdatecachablequery cache deny QUERY
Re: [squid-users] squid sslbump server-first local loops?
On 04/13/2014 08:35 PM, Eliezer Croitoru wrote: Why https_port? and why ssl_bump on https_port ? it should run ontop of http_port as far as I can understand and know. https_port is needed when you intercept port 443 traffic. http_port intercepts port 80 and https_port intercepts port 443. There was an issue which I reported about and which is similar and I have used couple acls to block the access and the loop from the port to itself. Can you share acl? Because there is already default acl called Safe_ports. And it does not list port 8081. Only ports listed in Safe_ports should be allowed. But this sslbump still continues and cause infinite loop. Eliezer Amm.
Re: [squid-users] squid sslbump server-first local loops?
Why https_port? and why ssl_bump on https_port ? it should run ontop of http_port as far as I can understand and know. There was an issue which I reported about and which is similar and I have used couple acls to block the access and the loop from the port to itself. Eliezer On 04/13/2014 02:11 PM, Amm wrote: I dont have any "via" line, so that means default in "on" Again tested it. Very easy to crash squid. It just takes 2 seconds for squid to report: WARNING! Your cache is running out of filedescriptors And takes away 100% CPU too. Regards, Amm
Re: [squid-users] Re: Cache Windows Updates ONLY
Dear Amos Thank you for reviewing the config and giving your deeply considered comments. On 13 April 2014 09:56, Amos Jeffries wrote: > Did your tests find any actual benefits in these "override-lastmod > override-expire ignore-reload ignore-must-revalidate ignore-private" > settings ? > > My tests earlier showed the reload-into-ims option was all that was > needed to make update caching behave nicely. It is also the only one of > those options which produces RFC compliant behaviour by the proxy. Yes! Clients generate zillions of range requests. This creates loads of revalidation. I have adopted the assumption that exe, cab and such files on windows update servers are static. A different file will take a different URL. Perhaps there are border cases where this assumption would fail, and maybe this needs more thought. Although I think it is fair to guarantee URLs with an embedded SHA1 checksum will always deliver the same content. I might rewrite this part to use reload-inot-ims for URL patterns which don't include a checksum, and use the full override and never expire for those URLs which do embed a checksum. > NP: Squid understands byte units whenever you see "KB" being used in config. > > So: > maximum_object_size 200 MB > maximum_object_size 6 GB > > Which is the first "howler". That directive deoes not take an access > list and only last value set matters. So adding " windowsupdate" to the > 6GB line and setting the 200MB value are both just useless text in the > config file. Ok. I really would like to limit object size on ACL, but will have to live with that! > > >> >> #My internet connection is not just used for Squid. I want to leave >> #responsive bandwidth for other services. This limits D/L speed >> delay_pools 1 >> delay_class 1 1 >> delay_access 1 allow all >> delay_parameters 1 120/120 > > It is better to use QoS controls in the system network settings that > limit Squid (usually by PID number) than applying a class-1 delay pool > to everything. I do have an iptables firewall set up and will perhaps add that to the bottom of my to-do list, unless I find it ineffectual and problematic. > >> >> #We use the store_id helper to convert windows update file hashes to bare >> URLs. >> #This way, any fetch for a given hash embedded in the URL will deliver >> the same data >> #You must make your own /etc/squid3/storeid_rewrite instructiosn at end. >> #change the helper program location from >> /usr/local/squid/libexec/storeid_file_rewrite to wherever yours is >> #It is written in PERL, so on most Linux systems, put it somewhere >> convenient, chmod 755 filename >> store_id_program /usr/local/squid/libexec/storeid_file_rewrite >> /etc/squid3/storeid_rewrite >> store_id_children 10 startup=5 idle=3 concurrency=0 >> store_id_access allow windowsupdate >> store_id_access deny all >> > > concurrency=0 is bad. Although I see this is due to a lack of > concurrency in the helper. Thats a bug which should get fixed. > > >> #We want to cache windowsupdate URLs which include queries >> #but only those queries which act on an installable file. >> #we don't want to cache queries on asp files as this is a genuine server >> #side query as opposed to a cache breaker >> acl wupdatecachablequery urlpath_regex >> (cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx|appxbundle|esd)\? >> >> #Deny caching for URLs matching query but not windowsupdate >> cache deny QUERY !windowsupdate >> #Deny caching for URLs matching query and windowsupdate but not cachable >> updates >> cache deny QUERY windowsupdate !wupdatecachablequery > > What does this help with exactly? Current Squid are prefectly capable of > caching despite query-string presence. > In fact we recommend dropping acl QUERY entirely and adding this right > above the '.' refresh_pattern: > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 I have three classes. Any URL with a query string. Any URL to a windows update server. Any URL to a windows update server which is specifically cache-able To paraphrase the logic coded here: Don't cache anything with a query string UNLESS it matches the ACL wupdatecachablequery. another way to write this more succinctly might be: cache deny QUERY cache allow wupdatecachablequery But I am not certain whether the deny clause will take a higher priority than the allow clause in cases where both ACLs match. The fandangled logic avoids this. > > >> >> #Given windows update is un-cooperative towards third party >> #methods to reduce network bandwidth, it is safe to presume >> #cache-specific headers or dates significantly differing from >> #system date will be unhelpful >> reply_header_access Date deny windowsupdate >> reply_header_access Age deny windowsupdate > > The "given" actually is not true IME. So not a safe assumption. > > Bad behaviour in the HTTP/1.1 revalidation by clients is a common side > effect of the override-* and ignore-* options being used on refresh_pattern. > The overrides used above make Squid ignor
Re: [squid-users] squid sslbump server-first local loops?
On 04/13/2014 04:27 PM, Amos Jeffries wrote: On 12/04/2014 5:23 p.m., Amm wrote: So I ran this command: openssl s_client -connect 192.168.1.2:8081 where 8081 is https_port on which squid runs. (with sslbump) And BOOM, squid went in to infinite loop! And started running out of file descriptors. Is this happening with "via on" ? It is an expected vulnerability with "via off". Amos I dont have any "via" line, so that means default in "on" Again tested it. Very easy to crash squid. It just takes 2 seconds for squid to report: WARNING! Your cache is running out of filedescriptors And takes away 100% CPU too. Regards, Amm
Re: [squid-users] squid sslbump server-first local loops?
On 12/04/2014 5:23 p.m., Amm wrote: > Hello, > > I accidentally came across this. I was trying to test what TLS version > my squid reports. > > So I ran this command: > openssl s_client -connect 192.168.1.2:8081 > > where 8081 is https_port on which squid runs. (with sslbump) > > And BOOM, squid went in to infinite loop! And started running out of > file descriptors. > Is this happening with "via on" ? It is an expected vulnerability with "via off". Amos
Re: [squid-users] Re: Cache Windows Updates ONLY
On 13/04/2014 7:08 a.m., Nick Hill wrote: > I have been ironing out issues with my windows updates set-up for > Squid. I have been through my squid.conf file to de-cruft it. > > The following squid.conf should be self-documenting. I have found this > works well in a multi-computer environment where you can expect a lot > of Windows machines to perform updates. A computer shop is a good > example. Of course, you will want to configure a DHCP server with a > wpad.dat address so that your client machines will auto-configure to > use your proxy. > > The principle difference between this and other configurations is that > it will cache windows updates even where a query string operates on a > cab, exe, or other non-dynamic response. I find the query string does > not change the file contents. (I know - it is possible that it > could...) > > The other feature is that Microsoft conveniently include SHA1 hashes > in URLs for static content files. Often, these static content files > will be found at differing locations, and will often be called with > query strings! Web cache hell! This configuration represents the data > internally to squid based purely on the SHA1 hash where available. If > two content items really have a SHA1 match, then you can guarantee > they are identical. Any successive file accesses from any of the > windows update domains which match the general SHA1 pattern used in > windows updates will generate a cache HIT, even where the URL is quite > different, and irrespective of any cache-bashing query string. > > I will monitor the configurations over the next week. Empirically, so > far, it all works! > If anyone can see howlers, let me know. Thanks! > > #squid.conf file for Squid Cache: Version 3.4.4 > #compiled on Ubuntu with configure options: '--enable-async-io=8' > '--enable-storeio=ufs,aufs,diskd' '--enable-removal-policies=lru,heap' > #'--enable-delay-pools' '--enable-underscores' '--enable-icap-client' > '--enable-follow-x-forwarded-for' '--with-logdir=/var/log/squid3' > #'--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' > '--with-large-files' '--with-default-user=proxy' > #'--enable-linux-netfilter' '--enable-storeid-rewrite-helpers=file' > > #Recommendations: in full production, you may want to set debug > options from 2 to 1 or 0. > #You may also want to comment out strip_query_terms off for user privacy > > #Explicitly define logs for my compiled version > cache_store_log /var/log/squid3/store.log > access_log /var/log/squid3/access.log > cache_log /var/log/squid3/cache.log > > #Lets have a fair bit of debugging info > debug_options ALL,2 > #Include query strings in logs > strip_query_terms off > > acl all src all > acl windowsupdate dstdomain .windowsupdate.microsoft.com > acl windowsupdate dstdomain .c.microsoft.com > acl windowsupdate dstdomain .ws.microsoft.com > acl windowsupdate dstdomain .update.microsoft.com > acl windowsupdate dstdomain images.metaservices.microsoft.com > acl windowsupdate dstdomain .download.windowsupdate.com > acl windowsupdate dstdomain wustat.windows.com > acl windowsupdate dstdomain swcdn.apple.com > acl windowsupdate dstdomain data-cdn.mbupdates.com > acl QUERY urlpath_regex cgi-bin \? > > #I'm behind a NAT firewall, so I don't need to restrict access > http_access allow all > > #Uncomment these if you have web apps on the local server which auth > through local ip > #acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 > #http_access deny to_localhost > > visible_hostname myclient.hostname.com > http_port 3128 > > #Always optimise bandwidth over hits > cache_replacement_policy heap LFUDA > #200Mb max object if not windowsupdate > maximum_object_size 20 KB > #Set these according to your file system > cache_dir ufs /home/smb/squid/squid 7 16 256 > coredump_dir /home/smb/squid/squid > > refresh_pattern -i > microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx|esd) > 43200 80% 43200 override-lastmod override-expire ignore-reload > ignore-must-revalidate ignore-private > refresh_pattern -i > windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx|esd) > 43200 80% 43200 override-lastmod override-expire ignore-reload > ignore-must-revalidate ignore-private > refresh_pattern -i > windows.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx|esd) > 43200 80% 43200 override-lastmod override-expire ignore-reload > ignore-must-revalidate ignore-private Did your tests find any actual benefits in these "override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private" settings ? My tests earlier showed the reload-into-ims option was all that was needed to make update caching behave nicely. It is also the only one of those options which produces RFC compliant behaviour by the proxy. > #Default refresh patterns last if no others match > refresh_pattern ^ftp: 1440 20% 10080 > refresh_pattern ^gopher: 1440 0% 1440 > refresh_pattern . 0 20% 4320 > > #Directive sets I have been experim
Re: [squid-users] Eui logging bug
On 13/04/2014 7:10 p.m., Dan Charlesworth wrote: > I filed that bug and it seems to have been getting a fair bit of attention > from other squid users who are suffering from it, but still no traction from > anyone that's able to diagnose it for some reason. > > I’m not on the dev mailing list because I’m not a developer, but I would > welcome anyone else to kickstart the discussion over there. Hi guys, I suppose an update is due on this. In summary; Eliezer and I have both been looking into it as best we can with no useful outcome so far. Eliezer has replicated it and is still slowly narrowing things down symptoms towards the cause. As I understand it he is slowed by this also being his first encounter with many of the components where EUI is relevant. I am unable to replicate the result due to unrelated issues with ARP in my build virtual machine. I also suffer from this one on my production machine, but experimenting there is a just bit too tricky at present. (As of today I have new hardware in my hands to play with that may let me replicate it. Crossed fingers etc.) Discussion/feedback has been a bit sparse due to lack of ideas that need to be collaborated on. There has been a little in the bug report and on IRC. HTH Amos
Re: [squid-users] Eui logging bug
I filed that bug and it seems to have been getting a fair bit of attention from other squid users who are suffering from it, but still no traction from anyone that's able to diagnose it for some reason. I’m not on the dev mailing list because I’m not a developer, but I would welcome anyone else to kickstart the discussion over there. Dan On 13 Apr 2014, at 7:43 am, Eliezer Croitoru wrote: > There is a direction but I am not too familiar with the code to tell you I > can find the right solution in a sec. > For me it takes time to dig inside the code and find the right way to fix it. > > This bug should be discussed at the squid development mailing list more then > it should be here. > > Eliezer > > On 04/13/2014 12:27 AM, Theron ZORBAS wrote: >> Hi Squid-Users, >> >> I've tried logging mac address into access.log with several squid >> versions and operatings systems such as Centos, OpenBSD etc. All my attempts >> resulted with 00:00:00:00:00:00. Clients are always at the same LAN segment. >> >> After some google work i saw this is a bug located at >> http://bugs.squid-cache.org/show_bug.cgi?id=3982 >> Any current information about this bug? Is it resolved? Or should we try an >> earlier version like X.XX? >> Thanks for your answers. >> >> -- >> Theron >> >
