Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable [SOLVED]

[Graham Menhennitt]

On 31/08/2017 22:27, Andrey V. Elsukov wrote:

On 31.08.2017 15:10, Graham Menhennitt wrote:

On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep
options' is:
options=8209b

 nd6 options=29

On 11-Stable (the one with the problems), it's igb1 and the output of
'ifconfig igb1 | grep options' is:
options=6403bb

 nd6 options=29


You need to disable TSO on your interface, ipfw nat is not compatible
with TCP segmentation offloading (this is noted in ipfw(8) BUGS section).

Try to use:
ifconfig igb1 -vlanhwtso -tso4

You can add these option to "ifconfig_igb1" variable in rc.conf.

Thanks very much for that Andrey (and Ian). It fixes the performance 
problem. I did look an the man page for both igb and ipfw but must have 
missed this. I agree, Ian, it would be good if there was some kind of 
warning at runtime.


So, that fixes the performance problems. I have another problem that 
I'll send a separate email about.


Thanks again,

Graham

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Ian Smith]

On Thu, 31 Aug 2017 15:27:47 +0300, Andrey V. Elsukov wrote:
 > On 31.08.2017 15:10, Graham Menhennitt wrote:
 > > On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep
 > > options' is:
 > > options=8209b
 > > 
 > > nd6 options=29
 > > 
 > > On 11-Stable (the one with the problems), it's igb1 and the output of
 > > 'ifconfig igb1 | grep options' is:
 > > options=6403bb
 > > 
 > > nd6 options=29
 > > 
 > 
 > You need to disable TSO on your interface, ipfw nat is not compatible
 > with TCP segmentation offloading (this is noted in ipfw(8) BUGS section).
 > 
 > Try to use:
 > ifconfig igb1 -vlanhwtso -tso4
 > 
 > You can add these option to "ifconfig_igb1" variable in rc.conf.

Specifically:

 Due to the architecture of libalias(3), ipfw nat is not compatible with
 the TCP segmentation offloading (TSO).  Thus, to reliably nat your net-
 work traffic, please disable TSO on your NICs using ifconfig(8).

Since natd also uses libalias, does not that also apply when using natd?  
I forget, and neither libalias(3) nor natd(8) mentions 'tso|TSO'.

Since this comes up so often, including on questions@, I'm wondering if 
an extra test in /etc/rc.d/ipfw at ipfw_prestart() for enablement of 
either $natd_enable (if applicable) or $firewall_nat_enable could then 
and there check ifconfig $natd_interface and/or $firewall_nat_interface 
for the presence of TSO4 and/or VLAN_HWTSO options, and so could warn 
the user - or just run "ifconfig $iface -vlanhwtso -tso4" directly?

While some interfaces such as ngX or pppX need not be up or even exist 
when starting ipfw, such interfaces should never use TSO anyway?  But 
I'm probably missing something obvious ..

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Andrey V. Elsukov]

On 31.08.2017 15:10, Graham Menhennitt wrote:
> On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep
> options' is:
> options=8209b
> 
> nd6 options=29
> 
> On 11-Stable (the one with the problems), it's igb1 and the output of
> 'ifconfig igb1 | grep options' is:
> options=6403bb
> 
> nd6 options=29
> 

You need to disable TSO on your interface, ipfw nat is not compatible
with TCP segmentation offloading (this is noted in ipfw(8) BUGS section).

Try to use:
ifconfig igb1 -vlanhwtso -tso4

You can add these option to "ifconfig_igb1" variable in rc.conf.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Graham Menhennitt]

On 31/08/2017 20:03, Andrey V. Elsukov wrote:

On 31.08.2017 13:01, Andrey V. Elsukov wrote:
Does anybody please have any ideas on this, please? 
Can you show the output of `ifconfig igb1 | grep flags` on stable/10 
and stable/11? 

Sorry, I wanted to write `ifconfig igb1 | grep options`.


Thanks for replying Andrey.

On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep 
options' is:

options=8209b
nd6 options=29

On 11-Stable (the one with the problems), it's igb1 and the output of 
'ifconfig igb1 | grep options' is:

options=6403bb
nd6 options=29

Thanks again for your help,
Graham

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Andrey V. Elsukov]

On 31.08.2017 13:01, Andrey V. Elsukov wrote:
>> Does anybody please have any ideas on this, please?
> 
> Can you show the output of `ifconfig igb1 | grep flags` on stable/10 and
> stable/11?

Sorry, I wanted to write `ifconfig igb1 | grep options`.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Andrey V. Elsukov]

On 29.08.2017 12:33, Graham Menhennitt wrote:
> However, the performance on the 11-Stable box is much worse. For file
> transfers I get about 1/10th the speed. Incoming TLS connections often
> fail to establish. Looking (from outside the box) at the interface in
> Wireshark shows lots of packets being retransmitted.
> 
> This appears to be due to the NAT rule. If I remove that, the
> performance jumps up to be approximately the same as the 10-Stable box.
> The rules are pretty simple:
>   nat 1 config if igb1 deny_in same_ports redirect_port udp
> XXX.XXX.XXX.XXX: 
>   nat 1 ip4 from any to any via igb1
> 
> I can provide the full set of rules if needed, but I think only those
> two lines are relevant.
> 
> Does anybody please have any ideas on this, please?

Can you show the output of `ifconfig igb1 | grep flags` on stable/10 and
stable/11?

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

ipfw kernel NAT performance much worse in 11-Stable than 10-Stable

[Graham Menhennitt]

I have two machines of similar CPU power that I use as routers. One is 
running 11-Stable as of a week ago and the other is 10-Stable from 
around the same time. They both run roughly the same IPFW rules (the 
syntax has changed slightly to run on the newer version). I've been 
using the 10-Stable box for a number of years without problems.


However, the performance on the 11-Stable box is much worse. For file 
transfers I get about 1/10th the speed. Incoming TLS connections often 
fail to establish. Looking (from outside the box) at the interface in 
Wireshark shows lots of packets being retransmitted.


This appears to be due to the NAT rule. If I remove that, the 
performance jumps up to be approximately the same as the 10-Stable box. 
The rules are pretty simple:
  nat 1 config if igb1 deny_in same_ports redirect_port udp 
XXX.XXX.XXX.XXX: 

  nat 1 ip4 from any to any via igb1

I can provide the full set of rules if needed, but I think only those 
two lines are relevant.


Does anybody please have any ideas on this, please?

Thanks for any help,

Graham

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"