Re: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
On 26/08/2014 12:11 p.m., Ragheb Rustom wrote: Dear All, I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have configured it as a transparent SSL_BUMP proxy. All is working well I can browse all SSL websites successfully after I have imported my generated CA file. The problem is that no matter how many times I request the SSL websites I always get a TCP_MISS in the squid access log. Among other websites I am trying to cache yahoo.com, facebook and youtube but most websites are always being served directly from source nothing is being served for the squid proxy. Please find below my configuration files. I deeply value any help on this matter. For a start configure this and re-check: strip_query_terms off That will allow your logs to show the full URL Squid is considering for cache HIT/MISS. You may find that a few hundred seemingly identical log entris are in fact highly variable in the query string portion. Such requests cannot be combined/HIT. squid.conf file: acl snmppublic snmp_community public acl bamboe src 10.128.135.0/24 #uncomment noway url, if necessary. #acl noway url_regex -i /etc/squid/noway acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 1935 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT #http_access deny noway http_access allow manager localhost http_access allow bamboe http_access deny manager The above http_access bits... http_access deny !Safe_ports http_access deny CONNECT !SSL_ports ... should be in here. http_access allow localhost htcp_access deny all miss_access allow all That is the default, you should get faster operation removing miss_access entirely. # NETWORK OPTIONS http_port 8080 http_port 8082 intercept https_port 8081 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER Avoid DONT_VERIFY_PEER as much as possible. It is considered harmful for security. Also usually unnecessary if the machines trusted CA certificates are setup properly and up to date. sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv 192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com static.videoku.tv no_cache deny QUERY Aha! no_cache deny QEURY The no_ part is obsolete syntax. What this line actually does is force all URLs with a query string ('?') to never be cached. This is the source of your MISS log entries. Remove it to get at least a chance at some HITs. Also, hierachy_stoplist is not useful in your configuration. You can probably remove it entirely. If your squid complains when its missing, set it to the default: hierarchy_stoplist /cgi-bin/ \? # MEMORY CACHE OPTIONS cache_mem 6000 MB maximum_object_size_in_memory 16 KB memory_replacement_policy heap GDSF # DISK CACHE OPTIONS cache_replacement_policy heap LFUDA cache_dir aufs /cache1 30 64 256 store_dir_select_algorithm least-load minimum_object_size 16 KB maximum_object_size 2 GB Put these global default min/max size limits above the cache_dir lines. Recent but outdated Squid like yoru 3.3 had a bug where the maximum_object_size is ignored if configured after cache_dir. Position for it does not normally matter, so placing it first always works and avoids needless annoyance. cache_swap_low 97 cache_swap_high 99 #LOGFILE OPTIONS access_log stdio:/var/log/squid/access.log cache_log /var/log/squid/cache.log cache_store_log none cache_swap_log /cache1/swap.state logfile_rotate 5 log_icp_queries off buffered_logs off #OPTIONS FOR TUNING THE CACHE Since Squid-3.2 some of the override and ignore options have changed. * ignore-no-cache is obsolete. Traffic with Cache-Control:no-cache will be cached properly by default. - remove this option from your config file. * combining reload-into-ims and ignore-reload is harmful. - ignore-reload makes Squid either HIT or MISS, rendering the revalidate CLIENT_REFRSH performance optimizations enabled by reload-into-ims useless. * ignore-private is harmful. Traffic with Cache-Control:private has mandatory revalidation. What can be cached will be cached properly by default, this option only causes all private data to be stored - it is never used from cache. - remove this option from your config
RE: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
I'm not sure if this is right or not, but wouldn't your refresh patterns need to have the ignore-private to cache ssl? Amos may know better, but I don't see that option specified in your All Files refresh_patterns. -Original Message- From: Ragheb Rustom [mailto:rag...@smartelecom.org] Sent: Monday, August 25, 2014 5:12 PM To: squid-users@squid-cache.org Subject: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests Dear All, I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have configured it as a transparent SSL_BUMP proxy. All is working well I can browse all SSL websites successfully after I have imported my generated CA file. The problem is that no matter how many times I request the SSL websites I always get a TCP_MISS in the squid access log. Among other websites I am trying to cache yahoo.com, facebook and youtube but most websites are always being served directly from source nothing is being served for the squid proxy. Please find below my configuration files. I deeply value any help on this matter. Squid setup settings: Squid Cache: Version 3.3.11 configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-wccpv2' '--enable-esi' '--enable-zph-qos' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=65535' '--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' 'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig' squid.conf file: acl snmppublic snmp_community public acl bamboe src 10.128.135.0/24 #uncomment noway url, if necessary. #acl noway url_regex -i /etc/squid/noway acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 1935 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT #http_access deny noway http_access allow manager localhost http_access allow bamboe http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost htcp_access deny all miss_access allow all # NETWORK OPTIONS http_port 8080 http_port 8082 intercept https_port 8081 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv 192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com static.videoku.tv no_cache deny QUERY # MEMORY CACHE OPTIONS cache_mem 6000 MB maximum_object_size_in_memory 16 KB memory_replacement_policy heap GDSF # DISK CACHE OPTIONS cache_replacement_policy heap LFUDA cache_dir aufs /cache1 30 64 256 store_dir_select_algorithm least-load minimum_object_size 16 KB maximum_object_size 2 GB cache_swap_low 97 cache_swap_high 99 #LOGFILE
Re: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
On 26/08/2014 3:29 p.m., Lawrence Pingree wrote: I'm not sure if this is right or not, but wouldn't your refresh patterns need to have the ignore-private to cache ssl? Amos may know better, but I don't see that option specified in your All Files refresh_patterns. HTTPS is not particularly private in the HTTP sense. It is just regular HTTP traffic wrapped in underlying transport security encryption. It does have a security scope difference from HTTP as to though due to that encryption. That scope difference is handled by the URL scheme portion. For example Squid must not and will not HIT on a http:// URL in cache for https:// request of otherwise identical URL, and vice versa. From the administrative viewpoint there is a higher risk with HTTPS of application designers breaking things and making vulnerable software simply by not understanding the above. There is high pressure to get privacy protection right with insecure http:// but weak for secure https:// on things like OAuth traffic and eCommerce checkout pages where they should have sent Cache-Control:private or no-store regardless. Amos