Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
hi, just wanted to say it's not just on bce. i had the same problem with my msk interfaces. i just had 1 pipe and 1 divert rule in my ipfw rules. and i couldn't even open a simple 10K html page from apache, even apache didn't respond. eventually i disabled tso on my interfaces and everything went back to normal. From: Garrett Cooper yanef...@gmail.com To: n...@freebsd.org Sent: Thu, July 1, 2010 6:30:53 AM Subject: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img100% 1440KB 13.7KB/s 01:45 real1m59.400s user0m0.031s sys0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img100% 1440KB 1.4MB/s 00:00 real0m0.712s user0m0.018s sys0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). Here are some other details: # ipfw list 00050 divert 8668 ip4 from any to any via bce1 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any ip6 icmp6types 1 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136 65000 allow ip from any to any 65535 deny ip from any to any # ls /etc/natd* ls: /etc/natd*: No such file or directory # uname -a FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0 r209169: Mon Jun 14 12:41:49 PDT 2010 root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE amd64 # pciconf -lv | grep -A 4 bce b...@pci0:7:0:0:class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet -- b...@pci0:3:0:0:class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet Let me know what other info is required. Thanks, -Garrett PS Please keep me CCed in all emails. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Thu, Jul 1, 2010 at 10:32 PM, Garrett Cooper yanef...@gmail.com wrote: On Thu, Jul 1, 2010 at 7:20 PM, Aleksandr A Babaylov @babolo.ru wrote: On Thu, Jul 01, 2010 at 04:54:47PM -0700, Pyun YongHyeon wrote: On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img 100% 1440KB 13.7KB/s 01:45 real 1m59.400s user 0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img 100% 1440KB 1.4MB/s 00:00 real 0m0.712s user 0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. Or ipfw nat. man ipfw | grep nat That uses the kernel module though, and that's horribly broken on my machine with 8-STABLE/9-CURRENT (see: http://www.mail-archive.com/freebsd-net@freebsd.org/msg33518.html ). I wonder if that's related to the TSO issue. As Vitezslav pointed out to me off-list, this is a known issue with libalias and IPFW: Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8). Both the ipfw kernel based nat'ing and natd use libalias, so both will be affected by this issue. This needs to be fixed (I'll see what I can do to help expedite this). Thanks, -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img100% 1440KB 13.7KB/s 01:45 real 1m59.400s user 0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img100% 1440KB 1.4MB/s 00:00 real 0m0.712s user 0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. Here are some other details: # ipfw list 00050 divert 8668 ip4 from any to any via bce1 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any ip6 icmp6types 1 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136 65000 allow ip from any to any 65535 deny ip from any to any # ls /etc/natd* ls: /etc/natd*: No such file or directory # uname -a FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0 r209169: Mon Jun 14 12:41:49 PDT 2010 root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE amd64 # pciconf -lv | grep -A 4 bce b...@pci0:7:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet -- b...@pci0:3:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet Let me know what other info is required. Can you reproduce this issue on other TSO capable drivers? I'm not aware of any TSO issues on bce(4). ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Thu, Jul 01, 2010 at 04:54:47PM -0700, Pyun YongHyeon wrote: On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img100% 1440KB 13.7KB/s 01:45 real1m59.400s user0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img100% 1440KB 1.4MB/s 00:00 real0m0.712s user0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. Or ipfw nat. man ipfw | grep nat ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Thu, 1 Jul 2010, Garrett Cooper wrote: On Thu, Jul 1, 2010 at 4:54 PM, Pyun YongHyeon pyu...@gmail.com wrote: On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img 100% 1440KB 13.7KB/s 01:45 real 1m59.400s user 0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img 100% 1440KB 1.4MB/s 00:00 real 0m0.712s user 0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. There's only one NAT rule here, not clear how many active NAT sessions are involved. I'm tending to doubt this is really a natd issue; natd has no interaction with interface issues like TSO, that I know of, hopefully someone will correct me if I'm wrong about that. Here are some other details: # ipfw list 00050 divert 8668 ip4 from any to any via bce1 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any ip6 icmp6types 1 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136 65000 allow ip from any to any 65535 deny ip from any to any # ls /etc/natd* ls: /etc/natd*: No such file or directory I assume that's the 'open' rc.firewall ruleset? So you have no natd.conf, and are taking all defaults? Just to check the config: # grep natd_ /etc/rc.conf # ps axw | grep [n]atd Do you have options IPFIREWALL and IPDIVERT in kernel, or are you loading these as modules? # uname -a FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0 r209169: Mon Jun 14 12:41:49 PDT 2010 root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE amd64 # pciconf -lv | grep -A 4 bce b...@pci0:7:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet -- b...@pci0:3:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet Let me know what other info is required. Can you reproduce this issue on other TSO capable drivers? I'm not aware of any TSO issues on bce(4). Hi Pyun! I'll have to pop in a Copper Intel card that we have laying around in the lab. I think it's em(4) compatible.. I forget... I have a few things to test network wise this weekend, so I'll try and repro a few things this weekend (say, Sunday?). I also have my msk(4) enabled machine in the lab I can test with, but I'll have to install the machine to spec with the Poweredge 2950 I have in the lab. I'm using ipfw because it was easy to setup according to the handbook, but in reality if ipfw is this bad dealing with nat rules, then I need to work with someone to improve how it scales. Unless there's something weird with tagging or something going on with divert sockets, this looks like something else; natd usually works fine at much higher rates, but I can't talk about gigabit. Though in-kernel NAT should be better at the higher throughput end, your 'ISDN' rate and the high CPU usage for natd is certainly not typical. Does this box have a public IP address on bce1? It's not clear whether you're doing this transfer from this box, or from another, through it, ie what address translation is expected? Where is 'somehost'? Hence, knowing natd's config options and net topology might be helpful. cheers, Ian___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Thu, Jul 1, 2010 at 9:19 PM, Ian Smith smi...@nimnet.asn.au wrote: On Thu, 1 Jul 2010, Garrett Cooper wrote: On Thu, Jul 1, 2010 at 4:54 PM, Pyun YongHyeon pyu...@gmail.com wrote: On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img 100% 1440KB 13.7KB/s 01:45 real 1m59.400s user 0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img 100% 1440KB 1.4MB/s 00:00 real 0m0.712s user 0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. There's only one NAT rule here, not clear how many active NAT sessions are involved. I'm tending to doubt this is really a natd issue; natd has no interaction with interface issues like TSO, that I know of, hopefully someone will correct me if I'm wrong about that. Here are some other details: # ipfw list 00050 divert 8668 ip4 from any to any via bce1 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any ip6 icmp6types 1 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136 65000 allow ip from any to any 65535 deny ip from any to any # ls /etc/natd* ls: /etc/natd*: No such file or directory I assume that's the 'open' rc.firewall ruleset? Yes. $ grep ^firewall /etc/rc.conf firewall_type=open So you have no natd.conf, and are taking all defaults? Just to check the config: Correct. $ ls /etc/natd.conf ls: /etc/natd.conf: No such file or directory # grep natd_ /etc/rc.conf $ grep ^natd_ /etc/rc.conf natd_enable=YES natd_interface=bce1 # ps axw | grep [n]atd Do you have options IPFIREWALL and IPDIVERT in kernel, or are you loading these as modules? Modules. $ egrep 'IPDIVERT|IPFIREWALL' /root/TAMESHI_STABLE $ make -VMODULES_OVERRIDE -f /etc/src.conf foo bce bge em bridgestp if_bridge ipdivert ipfw ipfw_nat libalias i2c/smbus ipmi ipmi/ipmi_linux linprocfs linsysfs linux # uname -a FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0 r209169: Mon Jun 14 12:41:49 PDT 2010 root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE amd64 # pciconf -lv | grep -A 4 bce b...@pci0:7:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet -- b...@pci0:3:0:0: class=0x02 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)' class = network subclass = ethernet Let me know what other info is required. Can you reproduce this issue on other TSO capable drivers? I'm not aware of any TSO issues on bce(4). Hi Pyun! I'll have to pop in a Copper Intel card that we have laying around in the lab. I think it's em(4) compatible.. I forget... I have a few things to test network wise this weekend, so I'll try and repro a few things this weekend (say, Sunday?). I also have my msk(4) enabled machine in the lab I can test with, but I'll have to install the machine to spec with the Poweredge 2950 I have in the lab. I'm using ipfw because it was easy to setup according to the handbook, but in reality if ipfw is this bad dealing with nat rules, then I need to work with someone to improve how it scales. Unless there's something weird with tagging or something going on with divert sockets, this looks like something else; Ok. natd usually works fine at much higher rates, but I can't talk about gigabit. Though in-kernel NAT should be better at the higher throughput end, But this panics deterministically as I've shown in another thread on 8-STABLE, so unfortunately I can't use this. your 'ISDN' rate and the high CPU usage for natd is certainly not typical. That I wouldn't doubt. Does this box have a public IP address on bce1? Nope. It's not clear whether you're doing this transfer from this box, or from
Re: Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE
On Thu, Jul 1, 2010 at 7:20 PM, Aleksandr A Babaylov @babolo.ru wrote: On Thu, Jul 01, 2010 at 04:54:47PM -0700, Pyun YongHyeon wrote: On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote: Hi, Just an observation I made while transferring a file: # time scp floppy.img somehost: Password: floppy.img 100% 1440KB 13.7KB/s 01:45 real 1m59.400s user 0m0.031s sys 0m0.028s # sysctl net.inet.tcp.tso=0 net.inet.tcp.tso: 1 - 0 # time scp floppy.img somehost: floppy.img 100% 1440KB 1.4MB/s 00:00 real 0m0.712s user 0m0.018s sys 0m0.018s Going ISDN speeds transferring a 1.44MB file is sad when you have a gigabit uplink :(... natd seems to be doing a LOT of spinning when TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core machine). I would use pf(4) if I have to handle lots of NAT rules. Or ipfw nat. man ipfw | grep nat That uses the kernel module though, and that's horribly broken on my machine with 8-STABLE/9-CURRENT (see: http://www.mail-archive.com/freebsd-net@freebsd.org/msg33518.html ). I wonder if that's related to the TSO issue. Thanks! -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
