IPFW with NAT (breakage with vlanhwtag enabled) Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-09 Thread Dr Josef Karthauser 

> On 8 Apr 2016, at 10:03, Dr Josef Karthauser  wrote:
> 
>> On 8 Apr 2016, at 06:51, Ian Smith > > wrote:
>> 
>> On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:
>> 
>>> Looks like the first packet is being retransmitted, which means that 
>>> the nat is probably misconfigured and the TCP connection is broken in
>>> some strange way.
>> 
>>> Does anyone have a clue as to where to look? The ipfw rules are
>>> simple enough - what have I missed?
>> 
>> Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
>> last para.  If not, no idea ..

So, disabling TSO did partially fix the problem; at least the “duplicate data” 
issue.

However, I’ve now added an https service in the jails (an haproxy), and that 
fails a TLS handshake from some hosts.

Bizarrely that problem goes away when I disable hw vlan tag processing 
(-vlanhwtag); that seems weird, and perhaps another bug.

The configuration of my machine is as follows:

  vlan10 (on igb0) [public address] <— [ipfw nat] -> igb1 [private address 
in a jail on the host, also bound to a physical network]

Is there any obvious reason why hardware vlan tagging should get in the way of 
a NAT session? I can’t think why that would be, but disabling it definitely 
fixes the problem.

Joe

— 
Dr Josef Karthauser
Chief Technical Officer
(01225) 300371 / (07703) 596893
www.truespeed.com 
  / theTRUESPEED   
  @theTRUESPEED 
 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-08 Thread Dr Josef Karthauser 
> On 8 Apr 2016, at 06:51, Ian Smith  wrote:
> 
> On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:
> 
> [ AppleMail msgs fail to quote properly in pine, so a partial quote: ]
> 
>> Looks like the first packet is being retransmitted, which means that 
>> the nat is probably misconfigured and the TCP connection is broken in
>> some strange way.
> 
>> Does anyone have a clue as to where to look? The ipfw rules are
>> simple enough - what have I missed?
> 
> Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
> last para.  If not, no idea ..
> 

Thanks Ian,

It was exactly that issue! I wish I had remembered that I’d seen that in the 
man page; would have saved hours of debugging :)

Joe

— 
Dr Josef Karthauser
Chief Technical Officer
(01225) 300371 / (07703) 596893
www.truespeed.com
  / theTRUESPEED 
  @theTRUESPEED
 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-07 Thread Ian Smith 
On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:

[ AppleMail msgs fail to quote properly in pine, so a partial quote: ]

 > Looks like the first packet is being retransmitted, which means that 
 > the nat is probably misconfigured and the TCP connection is broken in
 > some strange way.

 > Does anyone have a clue as to where to look? The ipfw rules are
 > simple enough - what have I missed?

Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
last para.  If not, no idea ..

cheers, Ian
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-07 Thread Dr Josef Karthauser 

> On 8 Apr 2016, at 00:11, Dr Josef Karthauser  wrote:
> 
>> On 7 Apr 2016, at 17:08, Dr Josef Karthauser > > wrote:
>> 
>> Looks like the first packet is being retransmitted, which means that the nat 
>> is probably misconfigured and the TCP connection is broken in some strange 
>> way.
>> 
>> Does anyone have a clue as to where to look? The ipfw rules are simple 
>> enough - what have I missed?
> 
> Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
> capture and taken a look and I get a packet that I’ve included below.
> 
> It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
> response. Looking at this I’d be inclined to think it’s a bug in the 
> webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
> server directly from the host machine on the private IP address (bypassing 
> the NAT), the HTTP response  received is perfectly fine. It’s only when I do 
> an HTTP request to the public IP address and go through the NAT that I 
> experience the problem.
> 
> How could this happen? Is it a buggy packet reassembly in the kernel perhaps?
> 

Adding: "ipfw add reass all from any to any” to the beginning of the ipfw rule 
set doesn’t make any difference to the behaviour. 

Joe

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-07 Thread Dr Josef Karthauser 

> On 7 Apr 2016, at 17:08, Dr Josef Karthauser  wrote:
> 
> Looks like the first packet is being retransmitted, which means that the nat 
> is probably misconfigured and the TCP connection is broken in some strange 
> way.
> 
> Does anyone have a clue as to where to look? The ipfw rules are simple enough 
> - what have I missed?

Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
capture and taken a look and I get a packet that I’ve included below.

It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
response. Looking at this I’d be inclined to think it’s a bug in the 
webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
server directly from the host machine on the private IP address (bypassing the 
NAT), the HTTP response  received is perfectly fine. It’s only when I do an 
HTTP request to the public IP address and go through the NAT that I experience 
the problem.

How could this happen? Is it a buggy packet reassembly in the kernel perhaps?

Joe

p.s here’s the strange packet with an HTTP response injected in the middle of a 
HTML stream:


23:01:07.204016 IP (tos 0x0, ttl 64, id 4190, offset 0, flags [DF], proto TCP 
(6), length 1500)
31.210.26.216.8080 > infiniverse.karthauser.co.uk.62475: Flags [.], cksum 
0xda1c (incorrect -> 0x7ff7), seq 8689:10137, ack 86, win 1040, options 
[nop,nop,TS val 124159447 ecr 1737359970], length 1448
.g.).
.f..g..b   Other Documentation

http://tomcat.apache.org/connectors-doc/;>Tomcat Connectors
http://tomcat.apache.org/connectors-doc/;>mod_jk Documentation
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
Date: Thu, 07 Apr 2016 23:01:05 GMT

2000






Apache Tomcat/7.0.68








http://tomcat.apache.org/;>Home
Documentation
Configuration
Examples
http://wiki.apache.org/tomcat/FrontPage;>Wiki
http://tomcat.apache.org/lists.html;>Mailing Lists
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

2016-04-07 Thread Dr Josef Karthauser 
I’m scratching my head with an IPFW / NAT configuration; could someone please 
throw me a bone?

I’ve got a jail, and I’m NATing using IPFW to connect it to the outside world.

In particular I’m forwarding port 8080 from the host’s public address to the 
jail’s private address.

When I pull an HTTP connection from port publicip:8080 I get the first packet 
of the TCP stream twice, and then the HTTP connection fails.
That ought not to happen :(.

The firewall rule is very simple

nat 1 config if vlan10 reset redirect_port tcp 10.17.0.16:8080 8080 // NAT for 
jails - forward to portal on 8080
nat 1 ip from any to any via vlan10 in
nat 1 ip from any to any via vlan10 out

add allow ip from any to any


If I tcpdump on the host:

# tcpdump -i vlan10 port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan10, link-type EN10MB (Ethernet), capture size 65535 bytes

17:02:02.478760 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [S], seq 3088565770, 
win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 672977930 ecr 
0,sackOK,eol], length 0
17:02:02.478797 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [S.], seq 425576427, 
ack 3088565771, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 
1035319863 ecr 672977930], length 0
17:02:02.480137 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1, win 
4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 0
17:02:02.480393 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 85
17:02:02.714225 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672978161 ecr 1035319863], length 85
17:02:02.975220 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672978421 ecr 1035319863], length 85
17:02:02.975239 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1:1449, ack 
86, win 1040, options [nop,nop,TS val 1035320360 ecr 672977931], length 1448
17:02:03.079324 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1449, win 
4096, options [nop,nop,TS val 672978522 ecr 1035320360], length 0
17:02:03.079336 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1449:4345, 
ack 86, win 1040, options [nop,nop,TS val 1035320464 ecr 672978522], length 2896
17:02:03.080931 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 4345, win 
4050, options [nop,nop,TS val 672978523 ecr 1035320464], length 0
17:02:03.578732 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 4345:5793, 
ack 86, win 1040, options [nop,nop,TS val 1035320963 ecr 672978523], length 1448
17:02:03.725858 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 5793, win 
4096, options [nop,nop,TS val 672979158 ecr 1035320963], length 0
17:02:03.725888 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 5793:8689, 
ack 86, win 1040, options [nop,nop,TS val 1035321110 ecr 672979158], length 2896
17:02:03.727352 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 8689, win 
4050, options [nop,nop,TS val 672979159 ecr 1035321110], length 0
17:02:04.260416 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 8689:10137, 
ack 86, win 1040, options [nop,nop,TS val 1035321645 ecr 672979159], length 1448
17:02:04.340844 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 10137, win 
4096, options [nop,nop,TS val 672979770 ecr 1035321645], length 0
17:02:04.340855 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 
10137:13033, ack 86, win 1040, options [nop,nop,TS val 1035321725 ecr 
672979770], length 2896
17:02:04.342775 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [F.], seq 86, ack 
11585, win 4096, options [nop,nop,TS val 672979771 ecr 1035321725], length 0
17:02:04.342803 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 
13033:15929, ack 87, win 1040, options [nop,nop,TS val 1035321727 ecr 
672979771], length 2896
17:02:04.343154 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565856, 
win 0, length 0
17:02:04.30 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, 
win 0, length 0
17:02:04.344740 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, 
win 0, length 0

And the client doing the http request gets:

phoenix:~ joe$ curl -v http://X.X.X.216:8080/
*   Trying 31.210.26.216...
* Connected to X.X.X.216 port 8080 (#0)
> GET / HTTP/1.1
> Host: x.x.com:8080
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Date: Thu, 07 Apr 2016 16:02:02 GMT
< 






Apache Tomcat/7.0.68








http://tomcat.apache.org/;>Home
Documentation
Configuration
Examples
http://wiki.apache.org/tomcat/FrontPage;>Wiki
[CUT]


Other Documentation