IPFW with NAT (breakage with vlanhwtag enabled) Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 8 Apr 2016, at 10:03, Dr Josef Karthauser  wrote:
> 
>> On 8 Apr 2016, at 06:51, Ian Smith > > wrote:
>> 
>> On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:
>> 
>>> Looks like the first packet is being retransmitted, which means that 
>>> the nat is probably misconfigured and the TCP connection is broken in
>>> some strange way.
>> 
>>> Does anyone have a clue as to where to look? The ipfw rules are
>>> simple enough - what have I missed?
>> 
>> Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
>> last para.  If not, no idea ..

So, disabling TSO did partially fix the problem; at least the “duplicate data” 
issue.

However, I’ve now added an https service in the jails (an haproxy), and that 
fails a TLS handshake from some hosts.

Bizarrely that problem goes away when I disable hw vlan tag processing 
(-vlanhwtag); that seems weird, and perhaps another bug.

The configuration of my machine is as follows:

  vlan10 (on igb0) [public address] <— [ipfw nat] -> igb1 [private address 
in a jail on the host, also bound to a physical network]

Is there any obvious reason why hardware vlan tagging should get in the way of 
a NAT session? I can’t think why that would be, but disabling it definitely 
fixes the problem.

Joe

— 
Dr Josef Karthauser
Chief Technical Officer
(01225) 300371 / (07703) 596893
www.truespeed.com 
  / theTRUESPEED   
  @theTRUESPEED 
 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 8 Apr 2016, at 06:51, Ian Smith  wrote:
> 
> On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:
> 
> [ AppleMail msgs fail to quote properly in pine, so a partial quote: ]
> 
>> Looks like the first packet is being retransmitted, which means that 
>> the nat is probably misconfigured and the TCP connection is broken in
>> some strange way.
> 
>> Does anyone have a clue as to where to look? The ipfw rules are
>> simple enough - what have I missed?
> 
> Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
> last para.  If not, no idea ..
> 

Thanks Ian,

It was exactly that issue! I wish I had remembered that I’d seen that in the 
man page; would have saved hours of debugging :)

Joe

— 
Dr Josef Karthauser
Chief Technical Officer
(01225) 300371 / (07703) 596893
www.truespeed.com
  / theTRUESPEED 
  @theTRUESPEED
 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Ian Smith]

On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:

[ AppleMail msgs fail to quote properly in pine, so a partial quote: ]

 > Looks like the first packet is being retransmitted, which means that 
 > the nat is probably misconfigured and the TCP connection is broken in
 > some strange way.

 > Does anyone have a clue as to where to look? The ipfw rules are
 > simple enough - what have I missed?

Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
last para.  If not, no idea ..

cheers, Ian
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 8 Apr 2016, at 00:11, Dr Josef Karthauser  wrote:
> 
>> On 7 Apr 2016, at 17:08, Dr Josef Karthauser > > wrote:
>> 
>> Looks like the first packet is being retransmitted, which means that the nat 
>> is probably misconfigured and the TCP connection is broken in some strange 
>> way.
>> 
>> Does anyone have a clue as to where to look? The ipfw rules are simple 
>> enough - what have I missed?
> 
> Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
> capture and taken a look and I get a packet that I’ve included below.
> 
> It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
> response. Looking at this I’d be inclined to think it’s a bug in the 
> webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
> server directly from the host machine on the private IP address (bypassing 
> the NAT), the HTTP response  received is perfectly fine. It’s only when I do 
> an HTTP request to the public IP address and go through the NAT that I 
> experience the problem.
> 
> How could this happen? Is it a buggy packet reassembly in the kernel perhaps?
> 

Adding: "ipfw add reass all from any to any” to the beginning of the ipfw rule 
set doesn’t make any difference to the behaviour. 

Joe

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 7 Apr 2016, at 17:08, Dr Josef Karthauser  wrote:
> 
> Looks like the first packet is being retransmitted, which means that the nat 
> is probably misconfigured and the TCP connection is broken in some strange 
> way.
> 
> Does anyone have a clue as to where to look? The ipfw rules are simple enough 
> - what have I missed?

Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
capture and taken a look and I get a packet that I’ve included below.

It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
response. Looking at this I’d be inclined to think it’s a bug in the 
webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
server directly from the host machine on the private IP address (bypassing the 
NAT), the HTTP response  received is perfectly fine. It’s only when I do an 
HTTP request to the public IP address and go through the NAT that I experience 
the problem.

How could this happen? Is it a buggy packet reassembly in the kernel perhaps?

Joe

p.s here’s the strange packet with an HTTP response injected in the middle of a 
HTML stream:


23:01:07.204016 IP (tos 0x0, ttl 64, id 4190, offset 0, flags [DF], proto TCP 
(6), length 1500)
31.210.26.216.8080 > infiniverse.karthauser.co.uk.62475: Flags [.], cksum 
0xda1c (incorrect -> 0x7ff7), seq 8689:10137, ack 86, win 1040, options 
[nop,nop,TS val 124159447 ecr 1737359970], length 1448
.g.).
.f..g..b   Other Documentation

http://tomcat.apache.org/connectors-doc/;>Tomcat Connectors
http://tomcat.apache.org/connectors-doc/;>mod_jk Documentation
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
Date: Thu, 07 Apr 2016 23:01:05 GMT

2000






Apache Tomcat/7.0.68








http://tomcat.apache.org/;>Home
Documentation
Configuration
Examples
http://wiki.apache.org/tomcat/FrontPage;>Wiki
http://tomcat.apache.org/lists.html;>Mailing Lists
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"