Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-07 Thread Amm 




- Original Message -
> From: Amos Jeffries 
> To: squid-users@squid-cache.org
> Cc: 
> Sent: Friday, 8 March 2013 2:47 AM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 7/03/2013 10:54 p.m., Amm wrote:
>>  - Original Message -

>>  [%h{Host}] was giving error, so i changed it to [%{Host}>h]
>> 
>>  Here is output:
>>  ABCD.net.in/1.2.3.4:33307 -> 173.194.36.16:443 (-:8081) :::0 -> 
> www.google.com/2404:6800:4009:802::1011:443 [www.google.com]
>> 
>>  Notice :::0 - somewhere it thinks its IPv6??
>> 
>>  If domain has just IPv4 address and no IPv6 address:
>> 
>>  ABCD.net.in/1.2.3.4:58347 -> 174.122.92.66:443 (-:8081) 0.0.0.0:0 -> 
> www.bigrock.com/174.122.92.65:443 [www.bigrock.com]
>> 
>> 
>>  If i use dns_v4_first, it logs IPv4 address.
>> 
>>  ABCD.mtnl.net.in/1.2.3.4:33559
>>    -> 74.125.236.147:443 (-:8081) 0.0.0.0:0 ->
>>  www.google.com/74.125.236.146:443 [www.google.com]
>> 
>>  Notice the change in IP address though. But may be that is expected as 
> squid does its own DNS lookup and picks other IP.


> Okay that zero IP:port on the outbound confirmed my suspicion about what 
> the code was doing. When using a pinned connection it is not setting the 
> real connection details into the log.


>>  Applying and trying patch will take about a day. Will let you know once I 
>> do.

 
> Thanks.
> The above log entry implies it should be the fix, but I will still need 
> confirmation of that.
> 
> Amos


I just applied the patch and it now logs IPv4 address correctly.

But earlier it was showing word PINNED now it shows HIER_DIRECT. I am not sure 
if it is right or wrong.

1362709553.045    172 1.2.3.4 TCP_MISS/302 1138 GET https://www.google.com/ - 
HIER_DIRECT/74.125.236.146 text/html

test.log file just incase you want to have a look.
ABCD.net.in/1.2.3.4:33007 -> 74.125.236.145:443 (-:8081) 1.2.3.4:33008 -> 
www.google.com/74.125.236.145:443 [www.google.com]

Thanks for the patch.

Regards

Amm.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-07 Thread Amos Jeffries 

On 7/03/2013 10:54 p.m., Amm wrote:

- Original Message -


From: Amos Jeffries 
To: squid-users@squid-cache.org
Cc:
Sent: Thursday, 7 March 2013 1:11 PM
Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain

On 7/03/2013 7:22 p.m., Amm wrote:




  For testing, URL was accessed with curl (curl -k https://www.google.com/)

  Here is debug output: (Google IP has changed but in same subnet, 1.2.3.4 is

my public IP replaced)

  2013/03/07 11:40:46.326 kid1| client_side.cc(2325) parseHttpRequest: HTTP

Client local=173.194.36.18:443 remote=1.2.3.4:50145 FD 21 flags=33

  2013/03/07 11:40:46.326 kid1| client_side.cc(2326) parseHttpRequest: HTTP

Client REQUEST:

  -
  GET / HTTP/1.1
  User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7

NSS/3.13.5.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7

  Host: www.google.com
  Accept: */*


  --
  2013/03/07 11:40:46.326 kid1| http.cc(2177) sendRequest: HTTP Server

local=1.2.3.4:50146 remote=173.194.36.18:443 FD 23 flags=1

  2013/03/07 11:40:46.326 kid1| http.cc(2178) sendRequest: HTTP Server

REQUEST:

  -
  GET / HTTP/1.1
  User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7

NSS/3.13.5.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7

  Host: www.google.com
  Accept: */*
  Via: 1.1 a.b.c (squid/3.3.2)
  X-Forwarded-For: 1.2.3.4
  Cache-Control: max-age=259200
  Connection: keep-alive


  HTTP server REQUEST shows 173.194.36.18, but access.logs show IPv6 address:
  1362636646.416 90 1.2.3.4 TCP_MISS/302 1138 GET https://www.google.com/

- PINNED/2404:6800:4009:802::1011 text/html



This is a really *really* strange outcome. It is indeed looking like a
code bug somewhere.

The cache.log showed the TCP level details being apparently correct. So
I think we can ignore everyting up to the point of logging.
Just to cofirm that can you add the server response trace from that
server request? It will be a short while later with identical local=
remote= and FD values.
If there is anything else on that FD 23 it would be useful to know as well.


2013/03/07 11:40:46.416 kid1| ctx: enter level  0: 'https://www.google.com/'
2013/03/07 11:40:46.416 kid1| http.cc(746) processReplyHeader: HTTP Server 
local=1.2.3.4:50146 remote=173.194.36.18:443 FD 23 flags=1
2013/03/07 11:40:46.416 kid1| http.cc(747) processReplyHeader: HTTP Server 
REPLY:
-
HTTP/1.1 302 Found
Location: https://www.google.co.in/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: X
Set-Cookie: X
P3P: CP="This is not a P3P policy! See 
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more 
info."
Date: Thu, 07 Mar 2013 06:10:46 GMT
Server: gws
Content-Length: 222
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN


302 Moved
302 Moved
The document has moved
https://www.google.co.in/";>here.

--
2013/03/07 11:40:46.416 kid1| ctx: exit level  0
2013/03/07 11:40:46.416 kid1| client_side.cc(1386) sendStartOfMessage: HTTP 
Client local=173.194.36.18:443 remote=1.2.3.4:50145 FD 21 flags=33
2013/03/07 11:40:46.416 kid1| client_side.cc(1387) sendStartOfMessage: HTTP 
Client REPLY:
-
HTTP/1.1 302 Moved Temporarily
Location: https://www.google.co.in/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: X
Set-Cookie: X
P3P: CP="This is not a P3P policy! See 
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more 
info."
Date: Thu, 07 Mar 2013 06:10:46 GMT
Server: gws
Content-Length: 222
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Cache: MISS from a.b.c
X-Cache-Lookup: MISS from a.b.c:8080
Via: 1.1 a.b.c (squid/3.3.2)
Connection: keep-alive




If we assume there is someting terribly broken with where the access.log
data is being generate from.
Can you create a custom log format please which outputs:

   logformat test %>A/%>a:%>p -> %>la:%>lp (%la:%lp)
%
%
[%h{Host}] was giving error, so i changed it to [%{Host}>h]

Here is output:
ABCD.net.in/1.2.3.4:33307 -> 173.194.36.16:443 (-:8081) :::0 -> 
www.google.com/2404:6800:4009:802::1011:443 [www.google.com]

Notice :::0 - somewhere it thinks its IPv6??

If domain has just IPv4 address and no IPv6 address:

ABCD.net.in/1.2.3.4:58347 -> 174.122.92.66:443 (-:8081) 0.0.0.0:0 -> 
www.bigrock.com/174.122.92.65:443 [www.bigrock.com]


If i use dns_v4_first, it logs IPv4 address.

ABCD.mtnl.net.in/1.2.3.4:33559
  -> 74.125.236.147:443 (-:8081) 0.0.0.0:0 ->
www.google.com/74.125.236.146:443 [www.google.com]

Notice the change in IP address though. But may be that is expected as squid 
does its own DNS lookup and picks other IP.


Okay that zero IP:port on the outbound confirmed my suspicion about what 
the code was doing. When using a pinned connection it is not setting the 
real connection details into the log.





If it is still logging the IPv6. I have an experimental patch here:
http

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-07 Thread Amm 
- Original Message -

> From: Amos Jeffries 
> To: squid-users@squid-cache.org
> Cc: 
> Sent: Thursday, 7 March 2013 1:11 PM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 7/03/2013 7:22 p.m., Amm wrote:
>> 
> 


>>  For testing, URL was accessed with curl (curl -k https://www.google.com/)
>> 
>>  Here is debug output: (Google IP has changed but in same subnet, 1.2.3.4 is 
> my public IP replaced)
>> 
>>  2013/03/07 11:40:46.326 kid1| client_side.cc(2325) parseHttpRequest: HTTP 
> Client local=173.194.36.18:443 remote=1.2.3.4:50145 FD 21 flags=33
>>  2013/03/07 11:40:46.326 kid1| client_side.cc(2326) parseHttpRequest: HTTP 
> Client REQUEST:
>>  -
>>  GET / HTTP/1.1
>>  User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 
> NSS/3.13.5.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7
>>  Host: www.google.com
>>  Accept: */*
>> 
>> 
>>  --
>>  2013/03/07 11:40:46.326 kid1| http.cc(2177) sendRequest: HTTP Server 
> local=1.2.3.4:50146 remote=173.194.36.18:443 FD 23 flags=1
>>  2013/03/07 11:40:46.326 kid1| http.cc(2178) sendRequest: HTTP Server 
> REQUEST:
>>  -
>>  GET / HTTP/1.1
>>  User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 
> NSS/3.13.5.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7
>>  Host: www.google.com
>>  Accept: */*
>>  Via: 1.1 a.b.c (squid/3.3.2)
>>  X-Forwarded-For: 1.2.3.4
>>  Cache-Control: max-age=259200
>>  Connection: keep-alive
>> 
>> 
>>  HTTP server REQUEST shows 173.194.36.18, but access.logs show IPv6 address:
>>  1362636646.416     90 1.2.3.4 TCP_MISS/302 1138 GET https://www.google.com/ 
> - PINNED/2404:6800:4009:802::1011 text/html


> This is a really *really* strange outcome. It is indeed looking like a 
> code bug somewhere.
> 
> The cache.log showed the TCP level details being apparently correct. So 
> I think we can ignore everyting up to the point of logging.
> Just to cofirm that can you add the server response trace from that 
> server request? It will be a short while later with identical local= 
> remote= and FD values.
> If there is anything else on that FD 23 it would be useful to know as well.


2013/03/07 11:40:46.416 kid1| ctx: enter level  0: 'https://www.google.com/'
2013/03/07 11:40:46.416 kid1| http.cc(746) processReplyHeader: HTTP Server 
local=1.2.3.4:50146 remote=173.194.36.18:443 FD 23 flags=1
2013/03/07 11:40:46.416 kid1| http.cc(747) processReplyHeader: HTTP Server 
REPLY:
-
HTTP/1.1 302 Found
Location: https://www.google.co.in/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: X
Set-Cookie: X
P3P: CP="This is not a P3P policy! See 
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for 
more info."
Date: Thu, 07 Mar 2013 06:10:46 GMT
Server: gws
Content-Length: 222
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN


302 Moved
302 Moved
The document has moved
https://www.google.co.in/";>here.

--
2013/03/07 11:40:46.416 kid1| ctx: exit level  0
2013/03/07 11:40:46.416 kid1| client_side.cc(1386) sendStartOfMessage: HTTP 
Client local=173.194.36.18:443 remote=1.2.3.4:50145 FD 21 flags=33
2013/03/07 11:40:46.416 kid1| client_side.cc(1387) sendStartOfMessage: HTTP 
Client REPLY:
-
HTTP/1.1 302 Moved Temporarily
Location: https://www.google.co.in/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: X
Set-Cookie: X
P3P: CP="This is not a P3P policy! See 
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for 
more info."
Date: Thu, 07 Mar 2013 06:10:46 GMT
Server: gws
Content-Length: 222
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Cache: MISS from a.b.c
X-Cache-Lookup: MISS from a.b.c:8080
Via: 1.1 a.b.c (squid/3.3.2)
Connection: keep-alive



> If we assume there is someting terribly broken with where the access.log 
> data is being generate from.
> Can you create a custom log format please which outputs:
> 
>   logformat test %>A/%>a:%>p -> %>la:%>lp (%la:%lp) 
> % 
> % 
> and use it for a secondary access_log line. Lets see what gets logged by 
> that.

[%h{Host}] was giving error, so i changed it to [%{Host}>h]

Here is output:
ABCD.net.in/1.2.3.4:33307 -> 173.194.36.16:443 (-:8081) :::0 -> 
www.google.com/2404:6800:4009:802::1011:443 [www.google.com]

Notice :::0 - somewhere it thinks its IPv6??

If domain has just IPv4 address and no IPv6 address:

ABCD.net.in/1.2.3.4:58347 -> 174.122.92.66:443 (-:8081) 0.0.0.0:0 -> 
www.bigrock.com/174.122.92.65:443 [www.bigrock.com]


If i use dns_v4_first, it logs IPv4 address.

ABCD.mtnl.net.in/1.2.3.4:33559
 -> 74.125.236.147:443 (-:808

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amos Jeffries 

On 7/03/2013 7:22 p.m., Amm wrote:


- Original Message -

From: Amos Jeffries

On 7/03/2013 5:30 p.m., Amm wrote:

  - Original Message -

  From: Amos Jeffries

  On 7/03/2013 2:03 a.m., Amm wrote:

I just tried 443 port interception with sslbump and is working

perfectly.

If sslbump none applies for request then it passes requests as

is:

Log shows something like this:

1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT

  23.63.101.48:443 - HIER_DIRECT/23.63.101.48 -

if sslbump server-first applied for request then log shows:
1362574001.569294 192.168.1.1 TCP_MISS/200 515 GET

  https://mail.google.com/mail/images/c.gif? -

PINNED/2404:6800:4009:801::1015

  image/gif

(Note: URL may not be same in both cases, these are just example)

I dont have IPv6, why is it showing IPv6 address, in 2nd case?

  Because you *do* have IPv6, or at least the Squid box does. And Squid

is

  using it successfully to contact the upstream web server.

  Amos


  Nope I do not have IPv6. I have been begging my ISP to give IPv6.


  

I hear what you are saying. Yet your logs are showing successful IPv6 traffic.
Maybe they enabled it on the router without informing you. Or maybe someone else
on the network setup a IPv6 gateway router (PC running 6to4 and emitting RAs?).
I don't know.

Somehow Squid detected that global IPv6 connectivity was available and is doing
full TCP connection setup and HTTP transactions resulting in over 28KB of data
transferred over IPv6 so far.

Try these three tests:
ping6 mail.google.com
netstat -antup
mtr -n6 mail.google.com


Please trust me. I am a network engineer since about 15 years now. (Not trying 
to brag).

I also appreciate your efforts a lot for always replying.

But I do not have IPv6. The squid is running on my standalone laptop.(there is 
no LAN network)

# ping6 mail.google.com
connect: Network is unreachable

# mtr -n6 mail.google.com
gives EMPTY screen i.e shows nothing except headers.

# list IPv6 addresses
# ip -f inet6 addr list
1: lo:  mtu 16436
 inet6 ::1/128 scope host
valid_lft forever preferred_lft forever


Hence, no interface has IPv6 except lo (loopback)


# list IPv4 addresses
# ip -f inet addr list
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
 inet 127.0.0.1/8 scope host lo
7: ppp1:  mtu 1492 qdisc htb state 
UNKNOWN qlen 3
 inet 1.2.3.4 peer 1.2.3.4/32 scope global ppp1

PPP interface is ADSL connection and just has IPv4 address. ADSL router is in 
bridge mode.



Okay.





NAT happening *into* Squid does not require IPv4 outbound. In cases like these
where the HTTP Host: header can be 100% validated as belonging to the
destination IP address Squid will use DNS to locate the upstream server. In this
case it locates the  and uses it.

You can enable debug_options 11,2 to see the client and server HTTP transaction
IP addressing details.

I enabled debug.

For testing, URL was accessed with curl (curl -k https://www.google.com/)

Here is debug output: (Google IP has changed but in same subnet, 1.2.3.4 is my 
public IP replaced)

2013/03/07 11:40:46.326 kid1| client_side.cc(2325) parseHttpRequest: HTTP 
Client local=173.194.36.18:443 remote=1.2.3.4:50145 FD 21 flags=33
2013/03/07 11:40:46.326 kid1| client_side.cc(2326) parseHttpRequest: HTTP 
Client REQUEST:
-
GET / HTTP/1.1
User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 NSS/3.13.5.0 
zlib/1.2.5 libidn/1.22 libssh2/1.2.7
Host: www.google.com
Accept: */*


--
2013/03/07 11:40:46.326 kid1| http.cc(2177) sendRequest: HTTP Server 
local=1.2.3.4:50146 remote=173.194.36.18:443 FD 23 flags=1
2013/03/07 11:40:46.326 kid1| http.cc(2178) sendRequest: HTTP Server REQUEST:
-
GET / HTTP/1.1
User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 NSS/3.13.5.0 
zlib/1.2.5 libidn/1.22 libssh2/1.2.7
Host: www.google.com
Accept: */*
Via: 1.1 a.b.c (squid/3.3.2)
X-Forwarded-For: 1.2.3.4
Cache-Control: max-age=259200
Connection: keep-alive


HTTP server REQUEST shows 173.194.36.18, but access.logs show IPv6 address:
1362636646.416 90 1.2.3.4 TCP_MISS/302 1138 GET https://www.google.com/ - 
PINNED/2404:6800:4009:802::1011 text/html


This is a really *really* strange outcome. It is indeed looking like a 
code bug somewhere.


The cache.log showed the TCP level details being apparently correct. So 
I think we can ignore everyting up to the point of logging.
Just to cofirm that can you add the server response trace from that 
server request? It will be a short while later with identical local= 
remote= and FD values.

If there is anything else on that FD 23 it would be useful to know as well.


If we assume there is someting terribly broken with where the access.log 
data is being generate from.

Can you create a custom log format please which outputs:

 logformat test %>A/%>a:%>p -> %>la:%>lp (%la:%lp) % 
%

and use it for a secondary access_log line. Lets see what gets logged by 
that.


If it i

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amm 


- Original Message -
> From: Amos Jeffries 
> To: squid-users@squid-cache.org
> Cc: 
> Sent: Thursday, 7 March 2013 11:19 AM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 7/03/2013 5:30 p.m., Amm wrote:
>>  - Original Message -
>>>  From: Amos Jeffries
>>> 
>>>  On 7/03/2013 2:03 a.m., Amm wrote:
>>>>    I just tried 443 port interception with sslbump and is working 
> perfectly.
>>>> 
>>>>    If sslbump none applies for request then it passes requests as 
> is:
>>>>    Log shows something like this:
>>>> 
>>>>    1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT
>>>  23.63.101.48:443 - HIER_DIRECT/23.63.101.48 -
>>>> 
>>>>    if sslbump server-first applied for request then log shows:
>>>>    1362574001.569    294 192.168.1.1 TCP_MISS/200 515 GET
>>>  https://mail.google.com/mail/images/c.gif? - 
> PINNED/2404:6800:4009:801::1015
>>>  image/gif
>>>>    (Note: URL may not be same in both cases, these are just example)
>>>> 
>>>>    I dont have IPv6, why is it showing IPv6 address, in 2nd case?
>>>  Because you *do* have IPv6, or at least the Squid box does. And Squid 
> is
>>>  using it successfully to contact the upstream web server.
>>> 
>>>  Amos
>>> 
>>  Nope I do not have IPv6. I have been begging my ISP to give IPv6.


 
> I hear what you are saying. Yet your logs are showing successful IPv6 traffic.
> Maybe they enabled it on the router without informing you. Or maybe someone 
> else 
> on the network setup a IPv6 gateway router (PC running 6to4 and emitting 
> RAs?). 
> I don't know.
> 
> Somehow Squid detected that global IPv6 connectivity was available and is 
> doing 
> full TCP connection setup and HTTP transactions resulting in over 28KB of 
> data 
> transferred over IPv6 so far.
> 
> Try these three tests:
> ping6 mail.google.com
> netstat -antup
> mtr -n6 mail.google.com


Please trust me. I am a network engineer since about 15 years now. (Not trying 
to brag).

I also appreciate your efforts a lot for always replying.

But I do not have IPv6. The squid is running on my standalone laptop.(there is 
no LAN network)

# ping6 mail.google.com
connect: Network is unreachable

# mtr -n6 mail.google.com
gives EMPTY screen i.e shows nothing except headers.

# list IPv6 addresses
# ip -f inet6 addr list
1: lo:  mtu 16436 
    inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever


Hence, no interface has IPv6 except lo (loopback)


# list IPv4 addresses
# ip -f inet addr list
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    inet 127.0.0.1/8 scope host lo
7: ppp1:  mtu 1492 qdisc htb state 
UNKNOWN qlen 3
    inet 1.2.3.4 peer 1.2.3.4/32 scope global ppp1

PPP interface is ADSL connection and just has IPv4 address. ADSL router is in 
bridge mode.


>>  squid is running on the very same machine.
>> 
>>  Rule used is:
>>  iptables -t nat -A OUTPUT -m owner ! --uid-owner squid -p tcp --dport 443 
> -j REDIRECT --to-ports 8081
>> 
>>  URL accessed is https://www.google.com
>> 
>>  nslookup -q=a www.google.com = 173.194.36.48 (one of many IPs in result)
>>  nslookup -q= www.google.com = 2404:6800:4009:803::1014 (only 1 IPv6 in 
> result)
>> 
>>  access.log:
>>  1362629583.956    532 192.168.1.1 TCP_MISS/200 28088 GET 
> https://www.google.com/ - PINNED/2404:6800:4009:803::1014 text/html
>> 
>>  I used wireshark to monitor the traffic. Result is:
>> 
>>  0.00 192.168.1.1 -> 173.194.36.48 TLSv1 775 Application Data
>>  0.017809 173.194.36.48 -> 192.168.1.1 TCP 68 443 > 40400 [ACK] Seq=1 
> Ack=708 Win=1002 Len=0 TSval= TSecr=
> 
> Your log states the client->Squid connection as being IPv4, this trace 
> confirms _that_.

The wireshark output I gave is not client->squid. Wireshark was run on ppp1 
interface i.e. squid->internet.

# command run was
# tshark -plnippp1 port 443.


> 
>>  Clearly its using IPv4 and not IPv6.
>> 
>>  Note: I have replaced my public IP with 192.168.1.1
>> 
>>  I have a feeling that since I am using REDIRECT, squid receives redirect 
> packets on local (loopback) IPv6 address, so it assumes that connection is 
> IPv6 
> and logs IPv6 address instead. (even though it connects to IPv4 address)


> Notice that:
> * the client->Squid connection and the Squid->server connection are 
> independent TCP connections


Agree.


> * IPv6 is on the Squid->Internet connection side of things

But as shown above squid->internet is also IPv4


> * IPv4 is happening

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amos Jeffries 

On 7/03/2013 5:30 p.m., Amm wrote:

- Original Message -

From: Amos Jeffries

On 7/03/2013 2:03 a.m., Amm wrote:

  I just tried 443 port interception with sslbump and is working perfectly.

  If sslbump none applies for request then it passes requests as is:
  Log shows something like this:

  1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT

23.63.101.48:443 - HIER_DIRECT/23.63.101.48 -


  if sslbump server-first applied for request then log shows:
  1362574001.569294 192.168.1.1 TCP_MISS/200 515 GET

https://mail.google.com/mail/images/c.gif? - PINNED/2404:6800:4009:801::1015
image/gif

  (Note: URL may not be same in both cases, these are just example)

  I dont have IPv6, why is it showing IPv6 address, in 2nd case?

Because you *do* have IPv6, or at least the Squid box does. And Squid is
using it successfully to contact the upstream web server.

Amos


Nope I do not have IPv6. I have been begging my ISP to give IPv6.


I hear what you are saying. Yet your logs are showing successful IPv6 
traffic.
Maybe they enabled it on the router without informing you. Or maybe 
someone else on the network setup a IPv6 gateway router (PC running 6to4 
and emitting RAs?). I don't know.


Somehow Squid detected that global IPv6 connectivity was available and 
is doing full TCP connection setup and HTTP transactions resulting in 
over 28KB of data transferred over IPv6 so far.


Try these three tests:
 ping6 mail.google.com
 netstat -antup
 mtr -n6 mail.google.com



squid is running on the very same machine.

Rule used is:
iptables -t nat -A OUTPUT -m owner ! --uid-owner squid -p tcp --dport 443 -j 
REDIRECT --to-ports 8081

URL accessed is https://www.google.com

nslookup -q=a www.google.com = 173.194.36.48 (one of many IPs in result)
nslookup -q= www.google.com = 2404:6800:4009:803::1014 (only 1 IPv6 in 
result)

access.log:
1362629583.956532 192.168.1.1 TCP_MISS/200 28088 GET 
https://www.google.com/ - PINNED/2404:6800:4009:803::1014 text/html

I used wireshark to monitor the traffic. Result is:

0.00 192.168.1.1 -> 173.194.36.48 TLSv1 775 Application Data
0.017809 173.194.36.48 -> 192.168.1.1 TCP 68 443 > 40400 [ACK] Seq=1 Ack=708 
Win=1002 Len=0 TSval= TSecr=


Your log states the client->Squid connection as being IPv4, this trace 
confirms _that_.



Clearly its using IPv4 and not IPv6.

Note: I have replaced my public IP with 192.168.1.1

I have a feeling that since I am using REDIRECT, squid receives redirect 
packets on local (loopback) IPv6 address, so it assumes that connection is IPv6 
and logs IPv6 address instead. (even though it connects to IPv4 address)


Notice that:
 * the client->Squid connection and the Squid->server connection are 
independent TCP connections

 * IPv6 is on the Squid->Internet connection side of things
 * IPv4 is happening on the client->Squid connection
 * REDIRECT is happening on the client->173.194.36.48 packets

NAT happening *into* Squid does not require IPv4 outbound. In cases like 
these where the HTTP Host: header can be 100% validated as belonging to 
the destination IP address Squid will use DNS to locate the upstream 
server. In this case it locates the  and uses it.


You can enable debug_options 11,2 to see the client and server HTTP 
transaction IP addressing details.



So I tried to change iptables rule to:
iptables -t nat -A OUTPUT -m owner ! --uid-owner squid -p tcp --dport 443 -j 
DNAT --to 127.0.0.1:8081

still it logs IPv6 address in access.log. So do not know why it assumes IPv6.


The "iptables" tool only changes IPv4 firewall settings on your machine. 
It does not affect IPv6 traffic in any way.


There is a different tool "ip6tables" which operates very similar, but 
only affects the IPv6 firewall settings on your machine.



So may be somewhere there is a bug. (either logical or coding)


There is definitely a Logical one - clearly your network is not doing 
what you think it is doing.
Coding one(s) still unclear - we need to get you sorted out about what 
is happening on the IPv6 side of the network before that can be known.


Amos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amm 




- Original Message -
> From: Amos Jeffries 
> To: squid-users@squid-cache.org
> Cc: 
> Sent: Thursday, 7 March 2013 4:11 AM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 7/03/2013 2:03 a.m., Amm wrote:
>>> 
>>  I just tried 443 port interception with sslbump and is working perfectly.
>> 
>>  If sslbump none applies for request then it passes requests as is:
>>  Log shows something like this:
>> 
>>  1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT 
> 23.63.101.48:443 - HIER_DIRECT/23.63.101.48 -
>> 
>> 
>>  if sslbump server-first applied for request then log shows:
>>  1362574001.569    294 192.168.1.1 TCP_MISS/200 515 GET 
> https://mail.google.com/mail/images/c.gif? - PINNED/2404:6800:4009:801::1015 
> image/gif
>> 
>>  (Note: URL may not be same in both cases, these are just example)
>> 
>>  I dont have IPv6, why is it showing IPv6 address, in 2nd case?
> 
> Because you *do* have IPv6, or at least the Squid box does. And Squid is 
> using it successfully to contact the upstream web server.
> 
> Amos
>

Nope I do not have IPv6. I have been begging my ISP to give IPv6.

squid is running on the very same machine.

Rule used is:
iptables -t nat -A OUTPUT -m owner ! --uid-owner squid -p tcp --dport 443 -j 
REDIRECT --to-ports 8081

URL accessed is https://www.google.com

nslookup -q=a www.google.com = 173.194.36.48 (one of many IPs in result)
nslookup -q= www.google.com = 2404:6800:4009:803::1014 (only 1 IPv6 in 
result)

access.log:
1362629583.956    532 192.168.1.1 TCP_MISS/200 28088 GET 
https://www.google.com/ - PINNED/2404:6800:4009:803::1014 text/html

I used wireshark to monitor the traffic. Result is:

0.00 192.168.1.1 -> 173.194.36.48 TLSv1 775 Application Data
0.017809 173.194.36.48 -> 192.168.1.1 TCP 68 443 > 40400 [ACK] Seq=1 Ack=708 
Win=1002 Len=0 TSval= TSecr=

Clearly its using IPv4 and not IPv6.

Note: I have replaced my public IP with 192.168.1.1

I have a feeling that since I am using REDIRECT, squid receives redirect 
packets on local (loopback) IPv6 address, so it assumes that connection is IPv6 
and logs IPv6 address instead. (even though it connects to IPv4 address)

So I tried to change iptables rule to:
iptables -t nat -A OUTPUT -m owner ! --uid-owner squid -p tcp --dport 443 -j 
DNAT --to 127.0.0.1:8081

still it logs IPv6 address in access.log. So do not know why it assumes IPv6.

So may be somewhere there is a bug. (either logical or coding)

Regards,

Amm.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amos Jeffries 

On 7/03/2013 2:03 a.m., Amm wrote:

- Original Message -

From: Amos Jeffries

On 6/03/2013 1:40 p.m., Alex Rousskov wrote:

  On 03/05/2013 03:09 AM, Amos Jeffries wrote:



  Squid tunnel functionality requires a CONNECT wrapper to generate
  outgoing connections.
  It is not yet setup to do the raw-TCP type of bypass the intercepted
  traffic would require.

  Are you sure? IIRC, "ssl_bump none" tunneling code works for

intercepted

  connections, and that is what we claim in squid.conf:

Hmm. Yes I see the code now.

Looks like it should work form IPv4 but IPv6 intercepted HTTPS might be
missing the [] around the IP.

Amos


I just tried 443 port interception with sslbump and is working perfectly.

If sslbump none applies for request then it passes requests as is:
Log shows something like this:

1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT 23.63.101.48:443 - 
HIER_DIRECT/23.63.101.48 -


if sslbump server-first applied for request then log shows:
1362574001.569294 192.168.1.1 TCP_MISS/200 515 GET 
https://mail.google.com/mail/images/c.gif? - PINNED/2404:6800:4009:801::1015 
image/gif

(Note: URL may not be same in both cases, these are just example)

I dont have IPv6, why is it showing IPv6 address, in 2nd case?


Because you *do* have IPv6, or at least the Squid box does. And Squid is 
using it successfully to contact the upstream web server.


Amos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Christos Tsantilas 
On 03/06/2013 06:15 AM, Amm wrote:
>> On 03/04/2013 10:11 PM, Amm wrote:
>>
   # Let user specify domains to avoid decrypting, such as internet 
>> banking
   acl bump-bypass dstdomain .commbank.com.au 
   ssl_bump none bump-bypass
   ssl_bump server-first all 
>>
>>
>>>   This will not work for intercepting traffic. Because domain is known
>>>   only after SSL connection is established. So certificate stage etc
>>>   has already passed.
>>
>> It will work but only if the reverse DNS lookup for the intercepted IP
>> address works: ssl_bump supports slow ACLs, and dstdomain is a slow ACL
>> if given an IP address.
> 
> As per http://www.squid-cache.org/Doc/config/acl/  its a fast ACL.
> 
> acl aclname dstdomain   .foo.com ...
> # Destination server from URL [fast]

The documentation should say that it is fast in most cases

If the user has use the ip address and not the host name as part of the
url, then squid has to do a reverse lookup to find the domain name.

In the case of transparent SSL interception, squid will have only the ip
address of the destination server so the reverse lookup required.

The problem with the reverse lookup is that in most cases will not give
you the correct domain name. For example a "host www.paypal.com" return
the ip address "23.55.226.234". But the "host 23.55.226.234" return as
domain name: <-something->.akamaitechnologies.com

Also the paypal example maybe says that it is difficult to find a
correct ip address range for some SSL sites...


Regards,
   Christos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-06 Thread Amm 




- Original Message -
> From: Amos Jeffries 
> To: squid-users@squid-cache.org
> Cc: 
> Sent: Wednesday, 6 March 2013 11:36 AM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 6/03/2013 1:40 p.m., Alex Rousskov wrote:
>>  On 03/05/2013 03:09 AM, Amos Jeffries wrote:
>> 
>> 
>>>  Squid tunnel functionality requires a CONNECT wrapper to generate
>>>  outgoing connections.
>>>  It is not yet setup to do the raw-TCP type of bypass the intercepted
>>>  traffic would require.
>>  Are you sure? IIRC, "ssl_bump none" tunneling code works for 
> intercepted
>>  connections, and that is what we claim in squid.conf:
> 
> Hmm. Yes I see the code now.
> 
> Looks like it should work form IPv4 but IPv6 intercepted HTTPS might be 
> missing the [] around the IP.
> 
> Amos
>

I just tried 443 port interception with sslbump and is working perfectly.

If sslbump none applies for request then it passes requests as is:
Log shows something like this:

1362574305.069  90590 192.168.1.1 TCP_MISS/200 3600 CONNECT 23.63.101.48:443 - 
HIER_DIRECT/23.63.101.48 -


if sslbump server-first applied for request then log shows:
1362574001.569    294 192.168.1.1 TCP_MISS/200 515 GET 
https://mail.google.com/mail/images/c.gif? - PINNED/2404:6800:4009:801::1015 
image/gif

(Note: URL may not be same in both cases, these are just example)

I dont have IPv6, why is it showing IPv6 address, in 2nd case?

Using squid 3.3.2.

Regards

Amm

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Alex Rousskov 
On 03/05/2013 09:15 PM, Amm wrote:
> - Original Message -
>> From: Alex Rousskov 
>> To: "squid-users@squid-cache.org" 
>> Cc: 
>> Sent: Wednesday, 6 March 2013 6:20 AM
>> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
>>
>> On 03/04/2013 10:11 PM, Amm wrote:
>>
>>>>   # Let user specify domains to avoid decrypting, such as internet banking
>>>>   acl bump-bypass dstdomain .commbank.com.au 
>>>>   ssl_bump none bump-bypass
>>>>   ssl_bump server-first all 


>>>   This will not work for intercepting traffic. Because domain is known
>>>   only after SSL connection is established. So certificate stage etc
>>>   has already passed.

>> It will work but only if the reverse DNS lookup for the intercepted IP
>> address works: ssl_bump supports slow ACLs, and dstdomain is a slow ACL
>> if given an IP address.


> As per http://www.squid-cache.org/Doc/config/acl/  its a fast ACL.
> 
> acl aclname dstdomain   .foo.com ...
> # Destination server from URL [fast]

... but could be a slow ACL. Read a few lines lower:

> # For dstdomain and dstdom_regex a reverse lookup is tried if a IP
> # based URL is used and no match is found. The name "none" is used
> # if the reverse lookup fails.


>>>   I am also assuming that squid checks IP based ACLs for ssl_bump
>>>   before establishing connection with client.

>> Squid checks all ssl_bump ACLs before establishing a TCP connection with
>> the server. The TCP connection from the client is already accepted (or
>> intercepted) by the time ssl_bump ACL is checked.


> What I would like to know is, does squid check ssl_bump ACL before starting
> SSL connection with client OR after? (for intercepting on https_port)

Squid does not establish an SSL connection with the TCP client if
"ssl_bump none" matches.


HTH,

Alex.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Amos Jeffries 

On 6/03/2013 1:40 p.m., Alex Rousskov wrote:

On 03/05/2013 03:09 AM, Amos Jeffries wrote:



Squid tunnel functionality requires a CONNECT wrapper to generate
outgoing connections.
It is not yet setup to do the raw-TCP type of bypass the intercepted
traffic would require.

Are you sure? IIRC, "ssl_bump none" tunneling code works for intercepted
connections, and that is what we claim in squid.conf:


Hmm. Yes I see the code now.

Looks like it should work form IPv4 but IPv6 intercepted HTTPS might be 
missing the [] around the IP.


Amos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Amm 



- Original Message -
> From: Alex Rousskov 
> To: "squid-users@squid-cache.org" 
> Cc: 
> Sent: Wednesday, 6 March 2013 6:20 AM
> Subject: Re: [squid-users] Bypassing SSL Bump for dstdomain
> 
> On 03/04/2013 10:11 PM, Amm wrote:
> 
>>>  # Let user specify domains to avoid decrypting, such as internet 
> banking
>>>  acl bump-bypass dstdomain .commbank.com.au 
>>>  ssl_bump none bump-bypass
>>>  ssl_bump server-first all 
> 
> 
>>  This will not work for intercepting traffic. Because domain is known
>>  only after SSL connection is established. So certificate stage etc
>>  has already passed.
> 
> It will work but only if the reverse DNS lookup for the intercepted IP
> address works: ssl_bump supports slow ACLs, and dstdomain is a slow ACL
> if given an IP address.

As per http://www.squid-cache.org/Doc/config/acl/  its a fast ACL.

acl aclname dstdomain   .foo.com ...
    # Destination server from URL [fast]

Also depending on reverse lookup for bypassing ssl_bump is can be
insecure w.r.t. policy. Rare but still somewhat insecure.


>>  I am also assuming that squid checks IP based ACLs for ssl_bump
>>  before establishing connection with client.
> 
> Squid checks all ssl_bump ACLs before establishing a TCP connection with
> the server. The TCP connection from the client is already accepted (or
> intercepted) by the time ssl_bump ACL is checked.

What I would like to know is, does squid check ssl_bump ACL before starting
SSL connection with client OR after? (for intercepting on https_port)

Otherwise ssl_bump server-first OR none feature does not help much.

Regards,

Amm.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Alex Rousskov 
On 03/04/2013 10:11 PM, Amm wrote:

>> # Let user specify domains to avoid decrypting, such as internet banking
>> acl bump-bypass dstdomain .commbank.com.au 
>> ssl_bump none bump-bypass
>> ssl_bump server-first all 


> This will not work for intercepting traffic. Because domain is known
> only after SSL connection is established. So certificate stage etc
> has already passed.

It will work but only if the reverse DNS lookup for the intercepted IP
address works: ssl_bump supports slow ACLs, and dstdomain is a slow ACL
if given an IP address.


> You should try ACL check based on real IP or IP range. Ofcourse this
> assumes that IP will never change for those banks.

Agreed. And one can combine fast IP-based rules with slower reverse DNS
lookups, of course. Each approach has its own flaws.


> I am also assuming that squid checks IP based ACLs for ssl_bump
> before establishing connection with client.

Squid checks all ssl_bump ACLs before establishing a TCP connection with
the server. The TCP connection from the client is already accepted (or
intercepted) by the time ssl_bump ACL is checked.


> Or you need to create rules at firewall level which will *not* divert
> traffic for those sites to squid.

Agreed. That would be a better alternative to IP-based ssl_bump ACLs.


Thank you,

Alex.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Alex Rousskov 
On 03/05/2013 03:09 AM, Amos Jeffries wrote:


> Squid tunnel functionality requires a CONNECT wrapper to generate
> outgoing connections.
> It is not yet setup to do the raw-TCP type of bypass the intercepted
> traffic would require.

Are you sure? IIRC, "ssl_bump none" tunneling code works for intercepted
connections, and that is what we claim in squid.conf:

> none
> Become a TCP tunnel without decoding the connection.
> Works with both CONNECT requests and intercepted SSL
> connections. This is the default behavior when no
> ssl_bump option is given or no ssl_bump ACLs match.

HTH,

Alex.

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Dan Charlesworth 
Cool -- thanks folks. That makes sense.

I guess if the situation is ever called for, IPs will have to suffice.

On 05/03/2013, at 9:09 PM, Amos Jeffries  wrote:

> On 5/03/2013 6:11 p.m., Amm wrote:
>>> 
>>> From: Dan Charlesworth 
>>> To: squid-users@squid-cache.org
>>> Sent: Tuesday, 5 March 2013 10:21 AM
>>> Subject: [squid-users] Bypassing SSL Bump for dstdomain
>>> 
>>> Hi
>>> 
>>> I've recently set up a very simple Squid 3.3.1 deployment to test out 
>>> Server First bumping and Mimicking in a REDIRECT type intercept 
>>> configuration.
>>> 
>>> It's working quite nicely, but I'm trying to accommodate a scenario where 
>>> an admin would like to disable bumping for certain webistes, for example 
>>> internet banking ones.
>>> 
>>> I basically have the exact same "ssl_bump" parameters from the config 
>>> example and yet requests matching the ACL are still being bumped as 
>>> evidenced by:
>>> - The full HTTPS URLs being recorded in the access log.
>>> - My client browser continuing to show that the certificate is signed by 
>>> the squid-signed CA when accessing the dstdomain.
>>> 
>>> I feel like I'm making some obvious mistake here, but can't see the forest 
>>> right now.
>>> 
>>> ...
>>> 
>>> # Let user specify domains to avoid decrypting, such as internet banking
>>> acl bump-bypass dstdomain .commbank.com.au
>>> 
>>> ...
>>> 
>>> ssl_bump none bump-bypass
>>> ssl_bump server-first all
>> 
>> 
>> This will not work for intercepting traffic. Because domain is known only 
>> after SSL connection is established. So certificate stage etc has already 
>> passed.
>> 
>> 
>> You should try ACL check based on real IP or IP range. Ofcourse this assumes 
>> that IP will never change for those banks.
>> 
>> I am also assuming that squid checks IP based ACLs for ssl_bump before 
>> establishing connection with client. (I have personally not tried this setup 
>> so can not tell for sure)
>> 
>> 
>> Or you need to create rules at firewall level which will *not* divert 
>> traffic for those sites to squid.
>> 
>> Amm.
> 
> Also, Squid tunnel functionality requires a CONNECT wrapper to generate 
> outgoing connections.
> It is not yet setup to do the raw-TCP type of bypass the intercepted traffic 
> would require.
> 
> Amos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-05 Thread Amos Jeffries 

On 5/03/2013 6:11 p.m., Amm wrote:


From: Dan Charlesworth 
To: squid-users@squid-cache.org
Sent: Tuesday, 5 March 2013 10:21 AM
Subject: [squid-users] Bypassing SSL Bump for dstdomain

Hi

I've recently set up a very simple Squid 3.3.1 deployment to test out Server 
First bumping and Mimicking in a REDIRECT type intercept configuration.

It's working quite nicely, but I'm trying to accommodate a scenario where an 
admin would like to disable bumping for certain webistes, for example internet 
banking ones.

I basically have the exact same "ssl_bump" parameters from the config example 
and yet requests matching the ACL are still being bumped as evidenced by:
- The full HTTPS URLs being recorded in the access log.
- My client browser continuing to show that the certificate is signed by the 
squid-signed CA when accessing the dstdomain.

I feel like I'm making some obvious mistake here, but can't see the forest 
right now.

...

# Let user specify domains to avoid decrypting, such as internet banking
acl bump-bypass dstdomain .commbank.com.au

...

ssl_bump none bump-bypass
ssl_bump server-first all



This will not work for intercepting traffic. Because domain is known only after 
SSL connection is established. So certificate stage etc has already passed.


You should try ACL check based on real IP or IP range. Ofcourse this assumes 
that IP will never change for those banks.

I am also assuming that squid checks IP based ACLs for ssl_bump before 
establishing connection with client. (I have personally not tried this setup so 
can not tell for sure)


Or you need to create rules at firewall level which will *not* divert traffic 
for those sites to squid.

Amm.


Also, Squid tunnel functionality requires a CONNECT wrapper to generate 
outgoing connections.
It is not yet setup to do the raw-TCP type of bypass the intercepted 
traffic would require.


Amos

Re: [squid-users] Bypassing SSL Bump for dstdomain

2013-03-04 Thread Amm 
>
> From: Dan Charlesworth 
>To: squid-users@squid-cache.org 
>Sent: Tuesday, 5 March 2013 10:21 AM
>Subject: [squid-users] Bypassing SSL Bump for dstdomain
> 
>Hi
>
>I've recently set up a very simple Squid 3.3.1 deployment to test out Server 
>First bumping and Mimicking in a REDIRECT type intercept configuration.
>
>It's working quite nicely, but I'm trying to accommodate a scenario where an 
>admin would like to disable bumping for certain webistes, for example internet 
>banking ones.
>
>I basically have the exact same "ssl_bump" parameters from the config example 
>and yet requests matching the ACL are still being bumped as evidenced by:
>- The full HTTPS URLs being recorded in the access log.
>- My client browser continuing to show that the certificate is signed by the 
>squid-signed CA when accessing the dstdomain.
>
>I feel like I'm making some obvious mistake here, but can't see the forest 
>right now.
>
>...
>
># Let user specify domains to avoid decrypting, such as internet banking
>acl bump-bypass dstdomain .commbank.com.au 
>
> ...
> 

>ssl_bump none bump-bypass
>ssl_bump server-first all



This will not work for intercepting traffic. Because domain is known only after 
SSL connection is established. So certificate stage etc has already passed.


You should try ACL check based on real IP or IP range. Ofcourse this assumes 
that IP will never change for those banks.

I am also assuming that squid checks IP based ACLs for ssl_bump before 
establishing connection with client. (I have personally not tried this setup so 
can not tell for sure)


Or you need to create rules at firewall level which will *not* divert traffic 
for those sites to squid.

Amm.