Re: [squid-users] Squid, Gmail.com and HSTS.
Sorry for the noise - I figured it out. HTTPS was completely dead which made me wonder if squid was working properly. It turns out I had some folder permission issues. I needed to chmod -R 777 /var/lib/ssl_db. I guess lack of permissions to that directory caused cert generation to fail and HTTPS to break..Thanks for reading - Original Message - From: Michael Monette mmone...@2keys.ca To: Amos Jeffries squ...@treenet.co.nz Cc: squid-users squid-users@lists.squid-cache.org Sent: Wednesday, June 10, 2015 10:25:21 AM Subject: Re: [squid-users] Squid, Gmail.com and HSTS. Hi again, I finally had some time to get back into this, been a busy couple weeks. I compiled squid with the --with-openssl --enable-ssl-crtd you mentioned, and now things seem to be working better with ssl::servername. But for some reason I can't get HTTPS traffic to get a cert from squid. All HTTPS traffic is getting their certificate from the real sites and I don't really know why because it's the same config as before. Here's a small capture of the logs: 1433945978.888 95 10.117.67.157 TCP_MISS/302 694 GET http://a.tribalfusion.com/z/i.match? - HIER_DIRECT/204.11.109.68 text/html 1433945978.918306 10.117.67.157 TCP_MISS/302 658 GET http://pixel.advertising.com/ups/50/sync? - HIER_DIRECT/149.174.67.72 - 1433945978.994 72 10.117.67.157 TCP_MISS/204 737 GET http://su.addthis.com/red/usync? - HIER_DIRECT/104.16.24.235 image/png 1433945979.147 65 10.117.67.157 TAG_NONE/200 0 CONNECT 104.236.7.74:443 - ORIGINAL_DST/104.236.7.74 - 1433945979.152 58 10.117.67.157 TAG_NONE/200 0 CONNECT 104.236.7.74:443 - ORIGINAL_DST/104.236.7.74 - 1433945979.972 1068 10.117.67.157 TCP_MISS/204 719 GET http://su.addthis.com/red/usync? - HIER_DIRECT/104.16.24.235 image/png 1433945981.527 50 10.117.67.157 TAG_NONE/200 0 CONNECT 104.236.7.74:443 - ORIGINAL_DST/104.236.7.74 - 1433945981.753 52 10.117.67.157 TAG_NONE/200 0 CONNECT 104.236.7.74:443 - ORIGINAL_DST/104.236.7.74 - 1433945982.006100 10.117.67.157 TCP_MISS/200 546 GET http://www.google.ca/ads/user-lists/1072396910/? - HIER_DIRECT/216.254.140.45 text/html 1433945983.769 55 10.117.67.157 TCP_MISS/200 546 GET http://www.google.ca/ads/user-lists/1072396910/? - HIER_DIRECT/216.254.140.45 text/html All the HTTPS traffic are just CONNECT's. I feel like I ran into this problem when I had been working on this a couple weeks and I was able to get myself out of it by messing with the bump steps, but I can't seem to figure it out this time(or I just can't remember). Hoping for some guidance or hints. Here's my log again: # cat /etc/squid/squid.conf ~ debug_options ALL,9 acl localnet src 10.0.0.0/8# RFC1918 possible internal network acl localnet src 172.16.0.0/12# RFC1918 possible internal network acl localnet src 192.168.0.0/16# RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80# http acl Safe_ports port 21# ftp acl Safe_ports port 443# https acl Safe_ports port 70# gopher acl Safe_ports port 210# wais acl Safe_ports port 1025-65535# unregistered ports acl Safe_ports port 280# http-mgmt acl Safe_ports port 488# gss-http acl Safe_ports port 591# filemaker acl Safe_ports port 777# multiling http acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager acl step1 at_step SslBump1 acl step2 at_step SslBump2 acl step3 at_step SslBump3 ssl_bump peek step1 all ssl_bump bump step2 all ssl_bump bump step3 all acl bl1 dstdomain gmail.com mail.google.com accounts.google.com moz.com #acl bl1 url_regex -i ^http(s)?://gmail.com #acl bl2 url_regex -i ^http(s)?://([a-zA-Z]+).gmail.com.* #acl bl3 url_regex -i ^http(s)?://moz.com.* #acl bl4 url_regex -i moz.com deny_info http://ask.com bl1 # I was testing redirecting stuff, but since the acl is not even picked up, this stuff is useless. http_reply_access deny bl1 # useless #http_access deny bl1 #http_access deny bl1 CONNECT http_access allow localnet http_access allow localhost http_access allow all http_port 3128 accel vhost allow-direct #https_port 3129 transparent ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslproxy_options NO_SSLv2 sslproxy_options NO_SSLv3 sslcrtd_program /usr/lib/squid
Re: [squid-users] Squid, Gmail.com and HSTS.
Yeah I don't know what I am doing wrong but I don't have these ACL types..Or I am somehow not copy pasting properly: FATAL: Invalid ACL type 'ssl::server_name' FATAL: Bungled /etc/squid/squid.conf line 54: acl nobumpsites ssl::server_name .google.com Squid Cache (Version 3.5.4): Terminated abnormally. CPU Usage: 0.005 seconds = 0.003 user + 0.002 sys Maximum Resident Size: 24096 KB Page faults with physical i/o: 0 Squid restarted [root@ottt-corp-paz-squid-1 squid-3.5.4]# squid -v Squid Cache: Version 3.5.4 Service Name: squid configure options: '--prefix=/usr' '--includedir=/usr/include' '--datadir=/usr/share' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--sysconfdir=/etc/squid' '--with-included-ltdl' --enable-ltdl-convenience There are also issues with at_step now: 2015/05/27 14:32:17| FATAL: Invalid ACL type 'at_step' FATAL: Bungled /etc/squid/squid.conf line 52: acl step1 at_step SslBump1 Squid Cache (Version 3.5.4): Terminated abnormally. CPU Usage: 0.005 seconds = 0.003 user + 0.002 sys Maximum Resident Size: 24080 KB Page faults with physical i/o: 0 Did I miss something when compiling? I just followed what was on the Squid wiki. I am all out of ideas.. Thanks, Mike - Original Message - From: Amos Jeffries squ...@treenet.co.nz To: squid-users squid-users@lists.squid-cache.org Sent: Wednesday, May 27, 2015 1:20:33 PM Subject: Re: [squid-users] Squid, Gmail.com and HSTS. On 28/05/2015 4:15 a.m., Michael Monette wrote: Has anyone been able to configure Squid in a way so that if you type https://gmail.com in your browser, you are NOT presented with the OMG HSTS I refuse to load anything page? When I go to https://gmail.com, I get an invalid certificate because the cert is for mail.google.com, issued by my CA. If I go to https://mail.google.com, the cert is beautifully green. Why can't squid detect that gmail.com is redirecting my browser to mail.google.com and generate the cert accordingly? That is *actually* what their server certificate contains. Ironic isn't it that their own certs do not comply with the restrictions they require of all others. Squid actually does obey HSTS requirements for secure handling of the reqeust. Its just the browser is incapable of detecting that, notices the custom CA and assumes the worst. Even configuring an acl for gmail.com doesn't work. It seems like even though I am punching https://gmail.com in my browser, Squid detects it as though I am typing https://mail.google.com; in my browser and is ignoring any ACLs I have setup specifically for gmail.com. I can't be the only one with this issue? I've also attempted to do: acl bl1 gmail.com moz.com always_direct allow bl1 - from what I understand this bypasses squid and tells my browser to get the cert right from the site. Maybe I am wrong. You are. squid.conf has nothing to do with your browser. That line tells Squid not to use any cache_peer connections when serving a request that matches ACL bl1. In the very first implementation way, way back in 3.1 decrypted requests could leak out over insecure cache_peer. So people were advised to use always_direct allow all to force it to work correctly. That bug was fixed long ago but the config still persists in the web. But certificates still come from Squid, so I don't see any effect from that line. Here's my config, lots of garbage in there since I have been trying everything i can think of to get this working. I want to add that for my acl called BL1, the only one that works is moz.com . They are part of the same ACL line, so if one works, they should all work. Except they do not. Thanks in advance. cat /etc/squid/squid.conf ~~ debug_options ALL,9 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12# RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80# http acl Safe_ports port 21# ftp acl Safe_ports port 443 # https acl Safe_ports port 70# gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535# unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager acl step1 at_step SslBump1 acl step2 at_step SslBump2 acl step3 at_step SslBump3 ssl_bump peek step1 all ssl_bump bump step2 all ssl_bump bump step3 all all at the end
Re: [squid-users] Squid, Gmail.com and HSTS.
I just thought of something else. First of all I'm new to squid and I am not aware of 10% of the things its capable of yet so I will ask. Is squid capable of adding custom SNIs? Like could I have it so gmail.com is added to the certificate as a subject alternate name EVEN though the original certificate doesn't contain it? If such a thing is possible I would love to know the term for it so I can do some searches. Appreciate it! On May 27, 2015 12:15:37 PM EDT, Michael Monette mmone...@2keys.ca wrote: Has anyone been able to configure Squid in a way so that if you type https://gmail.com in your browser, you are NOT presented with the OMG HSTS I refuse to load anything page? When I go to https://gmail.com, I get an invalid certificate because the cert is for mail.google.com, issued by my CA. If I go to https://mail.google.com, the cert is beautifully green. Why can't squid detect that gmail.com is redirecting my browser to mail.google.com and generate the cert accordingly? Even configuring an acl for gmail.com doesn't work. It seems like even though I am punching https://gmail.com in my browser, Squid detects it as though I am typing https://mail.google.com; in my browser and is ignoring any ACLs I have setup specifically for gmail.com. I can't be the only one with this issue? I've also attempted to do: acl bl1 gmail.com moz.com always_direct allow bl1 - from what I understand this bypasses squid and tells my browser to get the cert right from the site. Maybe I am wrong. But certificates still come from Squid, so I don't see any effect from that line. Here's my config, lots of garbage in there since I have been trying everything i can think of to get this working. I want to add that for my acl called BL1, the only one that works is moz.com . They are part of the same ACL line, so if one works, they should all work. Except they do not. Thanks in advance. cat /etc/squid/squid.conf ~~ debug_options ALL,9 acl localnet src 10.0.0.0/8# RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16# RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443# https acl Safe_ports port 70 # gopher acl Safe_ports port 210# wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280# http-mgmt acl Safe_ports port 488# gss-http acl Safe_ports port 591# filemaker acl Safe_ports port 777# multiling http acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager acl step1 at_step SslBump1 acl step2 at_step SslBump2 acl step3 at_step SslBump3 ssl_bump peek step1 all ssl_bump bump step2 all ssl_bump bump step3 all acl bl1 dstdomain gmail.com mail.google.com accounts.google.com moz.com #acl bl1 url_regex -i ^http(s)?://gmail.com #acl bl2 url_regex -i ^http(s)?://([a-zA-Z]+).gmail.com.* #acl bl3 url_regex -i ^http(s)?://moz.com.* #acl bl4 url_regex -i moz.com deny_info http://ask.com bl1 # I was testing redirecting stuff, but since the acl is not even picked up, this stuff is useless. http_reply_access deny bl1 # useless #http_access deny bl1 #http_access deny bl1 CONNECT http_access allow localnet http_access allow localhost http_access allow all http_port 3128 accel vhost allow-direct #https_port 3129 transparent ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslproxy_options NO_SSLv2 sslproxy_options NO_SSLv3 sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB sslcrtd_children 8 startup=1 idle=1 #cache_dir ufs /var/spool/squid 100 16 256 coredump_dir /var/spool/squid refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher: 14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 00% 0 refresh_pattern . 0 20% 4320 Mike ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid, Gmail.com and HSTS.
On 28/05/2015 4:15 a.m., Michael Monette wrote: Has anyone been able to configure Squid in a way so that if you type https://gmail.com in your browser, you are NOT presented with the OMG HSTS I refuse to load anything page? When I go to https://gmail.com, I get an invalid certificate because the cert is for mail.google.com, issued by my CA. If I go to https://mail.google.com, the cert is beautifully green. Why can't squid detect that gmail.com is redirecting my browser to mail.google.com and generate the cert accordingly? That is *actually* what their server certificate contains. Ironic isn't it that their own certs do not comply with the restrictions they require of all others. Squid actually does obey HSTS requirements for secure handling of the reqeust. Its just the browser is incapable of detecting that, notices the custom CA and assumes the worst. Even configuring an acl for gmail.com doesn't work. It seems like even though I am punching https://gmail.com in my browser, Squid detects it as though I am typing https://mail.google.com; in my browser and is ignoring any ACLs I have setup specifically for gmail.com. I can't be the only one with this issue? I've also attempted to do: acl bl1 gmail.com moz.com always_direct allow bl1 - from what I understand this bypasses squid and tells my browser to get the cert right from the site. Maybe I am wrong. You are. squid.conf has nothing to do with your browser. That line tells Squid not to use any cache_peer connections when serving a request that matches ACL bl1. In the very first implementation way, way back in 3.1 decrypted requests could leak out over insecure cache_peer. So people were advised to use always_direct allow all to force it to work correctly. That bug was fixed long ago but the config still persists in the web. But certificates still come from Squid, so I don't see any effect from that line. Here's my config, lots of garbage in there since I have been trying everything i can think of to get this working. I want to add that for my acl called BL1, the only one that works is moz.com . They are part of the same ACL line, so if one works, they should all work. Except they do not. Thanks in advance. cat /etc/squid/squid.conf ~~ debug_options ALL,9 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12# RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80# http acl Safe_ports port 21# ftp acl Safe_ports port 443 # https acl Safe_ports port 70# gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535# unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager acl step1 at_step SslBump1 acl step2 at_step SslBump2 acl step3 at_step SslBump3 ssl_bump peek step1 all ssl_bump bump step2 all ssl_bump bump step3 all all at the end of ACL lines has no meaning unless there is an authentication ACL that would otherwise be on the end and you dont want to trigger auth popups. acl bl1 dstdomain gmail.com mail.google.com accounts.google.com moz.com #acl bl1 url_regex -i ^http(s)?://gmail.com #acl bl2 url_regex -i ^http(s)?://([a-zA-Z]+).gmail.com.* #acl bl3 url_regex -i ^http(s)?://moz.com.* #acl bl4 url_regex -i moz.com deny_info http://ask.com bl1 # I was testing redirecting stuff, but since the acl is not even picked up, this stuff is useless. http_reply_access deny bl1 # useless Yes, why bother testing for request *URL* domain and blocking on the *reply*. #http_access deny bl1 #http_access deny bl1 CONNECT http_access allow localnet http_access allow localhost http_access allow all http_port 3128 accel vhost allow-direct #https_port 3129 transparent ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem options=NO_SSLv3 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER sslproxy_options NO_SSLv2 sslproxy_options NO_SSLv3 NOTE: Only the latest sslproxy_options line has any effect. So the NO_SSLv2 line is not obeyed. Use this instead:
Re: [squid-users] Squid, Gmail.com and HSTS.
On 28/05/2015 5:51 a.m., Michael Monette wrote: Thanks for the reply. I am compiling the latest squid now and going to give that a shot using ssl::server_name. The precompiled binary for CentOS was only at 3.5.04 and didn't have ssl::server_name support I guess. While I have you here, I was thinking of something else I could do. Tell me if you think it would work. You seem like an expert. I'm the Squid maintainer. On this feature I'm just an interested party, the real expert is Christos Tsantilas who authored it. Can I not make it so www.gmail.com resolves to (for example) 1.2.3.4 using DNS or a hosts file, then setup an acl like this: acl gml dst 1.2.3.4 # Redirect request for 1.2.3.4 to https://mail.google.com deny_info https://mail.google.com gml # Not even sure what this does, found it on a website, didn't really question it since things were working as they were supposed to. http_reply_access deny bl1 This acts on the HTTP messages inside the TLS wrapper after SSL-Bump has decrypted the connection and the request has been sent through to the server. By replacing whatever reply the server actually sent with a 302 status diverting the client to that URL. It has no effect on the bumping process itself or any of the TLS layer details. It may have an effect on HSTS since any headers the server might send are dropped by the 302 replacement. This way it doesn't have to monitor for a dstdomain, it watches for an IP. It wastes bandwidth sending a request to the server and fetching the reply. Given that the dst-IP address is what you have from the very first TCP packet onwards. About the only utility it has over http_access is that its positioning naturally restricts it to the decrypted traffic where the 302 will always work. But by that time you have the domain name from the encrypted HTTP message anyway. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
