Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Ian Smith]

On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote:

[ AppleMail msgs fail to quote properly in pine, so a partial quote: ]

 > Looks like the first packet is being retransmitted, which means that 
 > the nat is probably misconfigured and the TCP connection is broken in
 > some strange way.

 > Does anyone have a clue as to where to look? The ipfw rules are
 > simple enough - what have I missed?

Do you have TSO enabled on that NIC?  If so, see ipfw(8) BUGS, third 
last para.  If not, no idea ..

cheers, Ian
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

FreeBSD and Intel XL710 Ethernet drivers and malicious driver detection

[Alvin Wong]

We have recently come across what appears to be a driver/firmware problem
with our Intel Ethernet Converged Network Adapter X710-DA2.  Wanted to see
if anyone had insight into whether it was a known bug and if any
fixes/workarounds exist.

Symptoms:  On two FreeBSD 10.2-STABLE r292570 amd64 instances with these
cards -- we experienced packet loss (incrementing packet error counts) at
exactly the same time we observed /var/log/messages such as those below.
Here are the details of our hardware/driver/firmware:

Hardware:

Intel Ethernet Card:
X710DA2
Dual Port, SFP 10GbE,
PCIe 3.0, x8
VendorID:8086,
DvID:1572

FreeBSD Drivers we are running:

# sysctl -a | grep dev.ixl.0.%desc
dev.ixl.0.%desc: Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.3

Intel Firmware we were running:

# sysctl -a | grep dev.ixl.0.fw_version
dev.ixl.0.fw_version: f4.40 a1.4 n04.53 e80001dc0

=== /var/log/messages logs 

Apr  2 00:45:40 server3 kernel: ixl0: Malicious Driver Detection event 0x02
on TX queue 15 pf number 0x00
Apr  2 00:45:40 server3 kernel: ixl0: MDD TX event is for this function
0x0001ixl0: Malicious Driver Detection event 0x02 on TX queue 3 pf
number 0x00
Apr  2 00:45:40 server3 kernel: ixl0: MDD TX event is for this function
0x0001
Apr  2 00:45:41 server3 kernel: ixl0: Malicious Driver Detection event 0x02
on TX queue 12 pf number 0x00
Apr  2 00:45:41 server3 kernel: ixl0: MDD TX event is for this function
0x0001
Apr  2 00:45:43 server3 kernel: ixl0: Malicious Driver Detection event 0x02
on TX queue 1 pf number 0x00
Apr  2 00:45:43 server3 kernel: ixl0: MDD TX event is for this function
0x0001
Apr  2 01:12:03 server3 kernel: ixl0: Interface stopped DISTRIBUTING,
possible flapping


And on another box.

Apr  2 02:18:19 server4 kernel: ixl1: Malicious Driver Detection event 0x02
on TX queue 12 pf number 0x01
Apr  2 02:18:19 server4 kernel: ixl1: MDD TX event is for this function
0x0001
Apr  2 02:18:20 server4 kernel: ixl1: Malicious Driver Detection event 0x02
on TX queue 1 pf number 0x01
Apr  2 02:18:20 server4 kernel: ixl1: MDD TX event is for this function
0x0001

===

We noticed that the FreeBSD 10.3 RELEASE & even STABLE repositories Intel
IXL drivers are still version 1.4.3 but Intel's download site had 1.4.27
for quite a while now.

Anyone know if there was a reason for discrepancy?

For Intel reps here, should we also upgrade firmware to the latest
dev.ixl.0.fw_version: f5.0 a1.5 n05.02 e80002282 ?

Thanks,
Alvin Wong
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 8 Apr 2016, at 00:11, Dr Josef Karthauser  wrote:
> 
>> On 7 Apr 2016, at 17:08, Dr Josef Karthauser > > wrote:
>> 
>> Looks like the first packet is being retransmitted, which means that the nat 
>> is probably misconfigured and the TCP connection is broken in some strange 
>> way.
>> 
>> Does anyone have a clue as to where to look? The ipfw rules are simple 
>> enough - what have I missed?
> 
> Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
> capture and taken a look and I get a packet that I’ve included below.
> 
> It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
> response. Looking at this I’d be inclined to think it’s a bug in the 
> webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
> server directly from the host machine on the private IP address (bypassing 
> the NAT), the HTTP response  received is perfectly fine. It’s only when I do 
> an HTTP request to the public IP address and go through the NAT that I 
> experience the problem.
> 
> How could this happen? Is it a buggy packet reassembly in the kernel perhaps?
> 

Adding: "ipfw add reass all from any to any” to the beginning of the ipfw rule 
set doesn’t make any difference to the behaviour. 

Joe

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

> On 7 Apr 2016, at 17:08, Dr Josef Karthauser  wrote:
> 
> Looks like the first packet is being retransmitted, which means that the nat 
> is probably misconfigured and the TCP connection is broken in some strange 
> way.
> 
> Does anyone have a clue as to where to look? The ipfw rules are simple enough 
> - what have I missed?

Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap 
capture and taken a look and I get a packet that I’ve included below.

It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP 
response. Looking at this I’d be inclined to think it’s a bug in the 
webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web 
server directly from the host machine on the private IP address (bypassing the 
NAT), the HTTP response  received is perfectly fine. It’s only when I do an 
HTTP request to the public IP address and go through the NAT that I experience 
the problem.

How could this happen? Is it a buggy packet reassembly in the kernel perhaps?

Joe

p.s here’s the strange packet with an HTTP response injected in the middle of a 
HTML stream:


23:01:07.204016 IP (tos 0x0, ttl 64, id 4190, offset 0, flags [DF], proto TCP 
(6), length 1500)
31.210.26.216.8080 > infiniverse.karthauser.co.uk.62475: Flags [.], cksum 
0xda1c (incorrect -> 0x7ff7), seq 8689:10137, ack 86, win 1040, options 
[nop,nop,TS val 124159447 ecr 1737359970], length 1448
.g.).
.f..g..b   Other Documentation

http://tomcat.apache.org/connectors-doc/;>Tomcat Connectors
http://tomcat.apache.org/connectors-doc/;>mod_jk Documentation
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
Date: Thu, 07 Apr 2016 23:01:05 GMT

2000






Apache Tomcat/7.0.68








http://tomcat.apache.org/;>Home
Documentation
Configuration
Examples
http://wiki.apache.org/tomcat/FrontPage;>Wiki
http://tomcat.apache.org/lists.html;>Mailing Lists
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Jenkins build is still unstable: FreeBSD_stable_10 #196

[jenkins-admin]

See 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

[Dr Josef Karthauser]

I’m scratching my head with an IPFW / NAT configuration; could someone please 
throw me a bone?

I’ve got a jail, and I’m NATing using IPFW to connect it to the outside world.

In particular I’m forwarding port 8080 from the host’s public address to the 
jail’s private address.

When I pull an HTTP connection from port publicip:8080 I get the first packet 
of the TCP stream twice, and then the HTTP connection fails.
That ought not to happen :(.

The firewall rule is very simple

nat 1 config if vlan10 reset redirect_port tcp 10.17.0.16:8080 8080 // NAT for 
jails - forward to portal on 8080
nat 1 ip from any to any via vlan10 in
nat 1 ip from any to any via vlan10 out

add allow ip from any to any


If I tcpdump on the host:

# tcpdump -i vlan10 port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan10, link-type EN10MB (Ethernet), capture size 65535 bytes

17:02:02.478760 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [S], seq 3088565770, 
win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 672977930 ecr 
0,sackOK,eol], length 0
17:02:02.478797 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [S.], seq 425576427, 
ack 3088565771, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 
1035319863 ecr 672977930], length 0
17:02:02.480137 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1, win 
4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 0
17:02:02.480393 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 85
17:02:02.714225 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672978161 ecr 1035319863], length 85
17:02:02.975220 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 
1, win 4117, options [nop,nop,TS val 672978421 ecr 1035319863], length 85
17:02:02.975239 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1:1449, ack 
86, win 1040, options [nop,nop,TS val 1035320360 ecr 672977931], length 1448
17:02:03.079324 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1449, win 
4096, options [nop,nop,TS val 672978522 ecr 1035320360], length 0
17:02:03.079336 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1449:4345, 
ack 86, win 1040, options [nop,nop,TS val 1035320464 ecr 672978522], length 2896
17:02:03.080931 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 4345, win 
4050, options [nop,nop,TS val 672978523 ecr 1035320464], length 0
17:02:03.578732 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 4345:5793, 
ack 86, win 1040, options [nop,nop,TS val 1035320963 ecr 672978523], length 1448
17:02:03.725858 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 5793, win 
4096, options [nop,nop,TS val 672979158 ecr 1035320963], length 0
17:02:03.725888 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 5793:8689, 
ack 86, win 1040, options [nop,nop,TS val 1035321110 ecr 672979158], length 2896
17:02:03.727352 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 8689, win 
4050, options [nop,nop,TS val 672979159 ecr 1035321110], length 0
17:02:04.260416 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 8689:10137, 
ack 86, win 1040, options [nop,nop,TS val 1035321645 ecr 672979159], length 1448
17:02:04.340844 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 10137, win 
4096, options [nop,nop,TS val 672979770 ecr 1035321645], length 0
17:02:04.340855 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 
10137:13033, ack 86, win 1040, options [nop,nop,TS val 1035321725 ecr 
672979770], length 2896
17:02:04.342775 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [F.], seq 86, ack 
11585, win 4096, options [nop,nop,TS val 672979771 ecr 1035321725], length 0
17:02:04.342803 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 
13033:15929, ack 87, win 1040, options [nop,nop,TS val 1035321727 ecr 
672979771], length 2896
17:02:04.343154 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565856, 
win 0, length 0
17:02:04.30 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, 
win 0, length 0
17:02:04.344740 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, 
win 0, length 0

And the client doing the http request gets:

phoenix:~ joe$ curl -v http://X.X.X.216:8080/
*   Trying 31.210.26.216...
* Connected to X.X.X.216 port 8080 (#0)
> GET / HTTP/1.1
> Host: x.x.com:8080
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Date: Thu, 07 Apr 2016 16:02:02 GMT
< 






Apache Tomcat/7.0.68








http://tomcat.apache.org/;>Home
Documentation
Configuration
Examples
http://wiki.apache.org/tomcat/FrontPage;>Wiki
[CUT]


Other Documentation

Jenkins build is still unstable: FreeBSD_stable_10 #195

[jenkins-admin]

See 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"