[squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
Dear All, I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have configured it as a transparent SSL_BUMP proxy. All is working well I can browse all SSL websites successfully after I have imported my generated CA file. The problem is that no matter how many times I request the SSL websites I always get a TCP_MISS in the squid access log. Among other websites I am trying to cache yahoo.com, facebook and youtube but most websites are always being served directly from source nothing is being served for the squid proxy. Please find below my configuration files. I deeply value any help on this matter. Squid setup settings: Squid Cache: Version 3.3.11 configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-wccpv2' '--enable-esi' '--enable-zph-qos' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=65535' '--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' 'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig' squid.conf file: acl snmppublic snmp_community public acl bamboe src 10.128.135.0/24 #uncomment noway url, if necessary. #acl noway url_regex -i /etc/squid/noway acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 1935 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT #http_access deny noway http_access allow manager localhost http_access allow bamboe http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost htcp_access deny all miss_access allow all # NETWORK OPTIONS http_port 8080 http_port 8082 intercept https_port 8081 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv 192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com static.videoku.tv no_cache deny QUERY # MEMORY CACHE OPTIONS cache_mem 6000 MB maximum_object_size_in_memory 16 KB memory_replacement_policy heap GDSF # DISK CACHE OPTIONS cache_replacement_policy heap LFUDA cache_dir aufs /cache1 30 64 256 store_dir_select_algorithm least-load minimum_object_size 16 KB maximum_object_size 2 GB cache_swap_low 97 cache_swap_high 99 #LOGFILE OPTIONS access_log stdio:/var/log/squid/access.log cache_log /var/log/squid/cache.log cache_store_log none cache_swap_log /cache1/swap.state logfile_rotate 5 log_icp_queries off buffered_logs off #OPTIONS FOR TUNING THE CACHE refresh_pattern -i \.swf$ 20160 80% 20160 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth refresh_pattern -i \.gif$ 20160 80% 20160 override-expire
Re: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
On 26/08/2014 12:11 p.m., Ragheb Rustom wrote:
Dear All,
I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have
configured it as a transparent SSL_BUMP proxy. All is working well I can
browse all SSL websites successfully after I have imported my generated CA
file. The problem is that no matter how many times I request the SSL
websites I always get a TCP_MISS in the squid access log. Among other
websites I am trying to cache yahoo.com, facebook and youtube but most
websites are always being served directly from source nothing is being
served for the squid proxy. Please find below my configuration files. I
deeply value any help on this matter.
For a start configure this and re-check:
strip_query_terms off
That will allow your logs to show the full URL Squid is considering for
cache HIT/MISS. You may find that a few hundred seemingly identical log
entris are in fact highly variable in the query string portion. Such
requests cannot be combined/HIT.
squid.conf file:
acl snmppublic snmp_community public
acl bamboe src 10.128.135.0/24
#uncomment noway url, if necessary.
#acl noway url_regex -i /etc/squid/noway
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 1935 # http acl Safe_ports port 21 #
ftp acl Safe_ports port 443 # https acl Safe_ports port 70
# gopher acl Safe_ports port 210 # wais acl Safe_ports port
1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http acl Safe_ports port 591 #
filemaker acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#http_access deny noway
http_access allow manager localhost
http_access allow bamboe
http_access deny manager
The above http_access bits...
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
... should be in here.
http_access allow localhost
htcp_access deny all
miss_access allow all
That is the default, you should get faster operation removing
miss_access entirely.
# NETWORK OPTIONS
http_port 8080
http_port 8082 intercept
https_port 8081 intercept ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem
key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow
all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER
Avoid DONT_VERIFY_PEER as much as possible. It is considered harmful
for security. Also usually unnecessary if the machines trusted CA
certificates are setup properly and up to date.
sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB
sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv
192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex
cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com
static.videoku.tv no_cache deny QUERY
Aha!
no_cache deny QEURY
The no_ part is obsolete syntax. What this line actually does is force
all URLs with a query string ('?') to never be cached.
This is the source of your MISS log entries. Remove it to get at least a
chance at some HITs.
Also, hierachy_stoplist is not useful in your configuration. You can
probably remove it entirely. If your squid complains when its missing,
set it to the default:
hierarchy_stoplist /cgi-bin/ \?
# MEMORY CACHE OPTIONS
cache_mem 6000 MB
maximum_object_size_in_memory 16 KB
memory_replacement_policy heap GDSF
# DISK CACHE OPTIONS
cache_replacement_policy heap LFUDA
cache_dir aufs /cache1 30 64 256
store_dir_select_algorithm least-load
minimum_object_size 16 KB
maximum_object_size 2 GB
Put these global default min/max size limits above the cache_dir lines.
Recent but outdated Squid like yoru 3.3 had a bug where the
maximum_object_size is ignored if configured after cache_dir. Position
for it does not normally matter, so placing it first always works and
avoids needless annoyance.
cache_swap_low 97
cache_swap_high 99
#LOGFILE OPTIONS
access_log stdio:/var/log/squid/access.log cache_log
/var/log/squid/cache.log cache_store_log none cache_swap_log
/cache1/swap.state logfile_rotate 5 log_icp_queries off buffered_logs off
#OPTIONS FOR TUNING THE CACHE
Since Squid-3.2 some of the override and ignore options have changed.
* ignore-no-cache is obsolete. Traffic with Cache-Control:no-cache will
be cached properly by default.
- remove this option from your config file.
* combining reload-into-ims and ignore-reload is harmful.
- ignore-reload makes Squid either HIT or MISS, rendering the
revalidate CLIENT_REFRSH performance optimizations enabled by
reload-into-ims useless.
* ignore-private is harmful. Traffic with Cache-Control:private has
mandatory revalidation. What can be cached will be cached properly by
default, this option only causes all private data to be stored - it is
never used from cache.
- remove this option from your config
RE: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
I'm not sure if this is right or not, but wouldn't your refresh patterns need to have the ignore-private to cache ssl? Amos may know better, but I don't see that option specified in your All Files refresh_patterns. -Original Message- From: Ragheb Rustom [mailto:rag...@smartelecom.org] Sent: Monday, August 25, 2014 5:12 PM To: squid-users@squid-cache.org Subject: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests Dear All, I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have configured it as a transparent SSL_BUMP proxy. All is working well I can browse all SSL websites successfully after I have imported my generated CA file. The problem is that no matter how many times I request the SSL websites I always get a TCP_MISS in the squid access log. Among other websites I am trying to cache yahoo.com, facebook and youtube but most websites are always being served directly from source nothing is being served for the squid proxy. Please find below my configuration files. I deeply value any help on this matter. Squid setup settings: Squid Cache: Version 3.3.11 configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-wccpv2' '--enable-esi' '--enable-zph-qos' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=65535' '--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' 'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig' squid.conf file: acl snmppublic snmp_community public acl bamboe src 10.128.135.0/24 #uncomment noway url, if necessary. #acl noway url_regex -i /etc/squid/noway acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 1935 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT #http_access deny noway http_access allow manager localhost http_access allow bamboe http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost htcp_access deny all miss_access allow all # NETWORK OPTIONS http_port 8080 http_port 8082 intercept https_port 8081 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv 192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com static.videoku.tv no_cache deny QUERY # MEMORY CACHE OPTIONS cache_mem 6000 MB maximum_object_size_in_memory 16 KB memory_replacement_policy heap GDSF # DISK CACHE OPTIONS cache_replacement_policy heap LFUDA cache_dir aufs /cache1 30 64 256 store_dir_select_algorithm least-load minimum_object_size 16 KB maximum_object_size 2 GB cache_swap_low 97 cache_swap_high 99 #LOGFILE
Re: [squid-users] FW: squid 3.3.10 always gives TCP_MISS for SSL requests
On 26/08/2014 3:29 p.m., Lawrence Pingree wrote: I'm not sure if this is right or not, but wouldn't your refresh patterns need to have the ignore-private to cache ssl? Amos may know better, but I don't see that option specified in your All Files refresh_patterns. HTTPS is not particularly private in the HTTP sense. It is just regular HTTP traffic wrapped in underlying transport security encryption. It does have a security scope difference from HTTP as to though due to that encryption. That scope difference is handled by the URL scheme portion. For example Squid must not and will not HIT on a http:// URL in cache for https:// request of otherwise identical URL, and vice versa. From the administrative viewpoint there is a higher risk with HTTPS of application designers breaking things and making vulnerable software simply by not understanding the above. There is high pressure to get privacy protection right with insecure http:// but weak for secure https:// on things like OAuth traffic and eCommerce checkout pages where they should have sent Cache-Control:private or no-store regardless. Amos
