Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On 12/16/11, Eugene Grosbein wrote: (Sorry Eugene, I didn't get your message until I searched the web). Do you use NAT? man ipfw clearly states: ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8) Yes, I'm using ipfw and NAT, so this is the problem. Thanks for solving me this. I'm just wondering if it was a good idea to enable TSO by default without at least a warning in the release notes or in UPDATING. This has caused me some headaches, which is ok, I'm just sorry I've also wasted some of YongHyeon's time; what's worse, I believe I won't be the only one hit by this. bye Thanks av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Sun, Dec 18, 2011 at 08:10:20PM +0100, Andrea Venturoli wrote: On 12/16/11, Eugene Grosbein wrote: (Sorry Eugene, I didn't get your message until I searched the web). Do you use NAT? man ipfw clearly states: ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8) Yes, I'm using ipfw and NAT, so this is the problem. Thanks for solving me this. I'm just wondering if it was a good idea to enable TSO by default without at least a warning in the release notes or in UPDATING. This has caused me some headaches, which is ok, I'm just sorry I've also wasted some of YongHyeon's time; what's worse, I believe I won't be the That's fine. This made me re-read fxp(4) TSO path and I was able to identify possible flaw in the implementation. Will commit the fix. only one hit by this. bye Thanks av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On 12/15/11 23:13, YongHyeon PYUN wrote: I tried netstat -ind, but it shows no Ierrs/Idrop/Oerrs/Odrop. Use -s option which will show statistics for each network protocols. Search 'discarded for bad checksums' from the output. Still all bad counters at zero. You'll see tso.dump and notso.dump: they are both from the same client downloading the same (random) file (the file name was changed only to prevent possible caching). See notso.dump is perfect, while tso.dump shows a lot of potential problems. Thanks. Thanks go to you! :-) Thanks for testing. Based on dump file, I tried various MTU configuration and I was not able to reproduce it. By chance, are you using firewall(pf/ipfw/ipf) or bridge(4)? If I remember correctly some firewall rules are not compatible with TSO. For bridge, if one member of bridge does not support TSO, TSO should be disabled. Very interesting: I'm not using bridge on this host, but I'm using ipfw! Which rules are incompatible? Any pointer on this? I'm also using CARP, in case it could matter, but not on this interface. bye Thanks av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
15.12.2011 02:33, Andrea Venturoli пишет: Hello. I recently installed 8.2 with the following card: dmesg ... fxp0: Intel 82550 Pro/100 Ethernet port 0xdc00-0xdc3f mem 0xfebfb000-0xfebfbfff,0xfebc-0xfebd irq 20 at device 5.0 on pci4 ... pciconv -lv ... fxp0@pci0:4:5:0:class=0x02 card=0x00408086 chip=0x12298086 rev=0x0c hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet ... I had weird problems, especially due to ICMP/fragmentation/MTU/... issues. After hours spent in debugging, I noticed tcpdump showing ICMP packets with wrong checksums and that TSO4 was enabled. ifconfig fxp0 -tso seemed to solve everything. Don't know if it's a bug in the card, in the driver or what... Maybe it shouldn't be enabled by default? Do you use NAT? man ipfw clearly states: ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8) Eugene Grosbein ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On 12/14/11 22:32, YongHyeon PYUN wrote: Wireshark showed some wrong checksums (I believe on the ICMP packet, but I might remember wrong). You can check whether you received bad checksummed frames with netstat(1). I tried netstat -ind, but it shows no Ierrs/Idrop/Oerrs/Odrop. Is simple downloading from client to server is enough to trigger the issue? Yes and no. Depending on where the client is located (on the Internet) and/or the protocol used, I get either failures or ridicuolous performance (i.e. 58-60kB/s without TSO vs. 1-2kB/s with TSO). Packet capture that shows the problem would be great to know what's going on here. I'll send them to you privately. You'll see tso.dump and notso.dump: they are both from the same client downloading the same (random) file (the file name was changed only to prevent possible caching). See notso.dump is perfect, while tso.dump shows a lot of potential problems. Would you try attached patch and let me know it goes? Sorry, it seems extra pull up for TCP payload may not be required here. Try this instead. I see a little increase in performance (2-3kB/s instead of 1-2kB/s); this might however well depend on external factors. Still it is very different from what I'm get without TSO. bye Thanks av. P.S. I can live well without TSO; I'm just doing this to let the software improve. Go ahead only if *you* are interested. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Thu, Dec 15, 2011 at 03:16:51PM +0100, Andrea Venturoli wrote: On 12/14/11 22:32, YongHyeon PYUN wrote: Wireshark showed some wrong checksums (I believe on the ICMP packet, but I might remember wrong). You can check whether you received bad checksummed frames with netstat(1). I tried netstat -ind, but it shows no Ierrs/Idrop/Oerrs/Odrop. Use -s option which will show statistics for each network protocols. Search 'discarded for bad checksums' from the output. Is simple downloading from client to server is enough to trigger the issue? Yes and no. Depending on where the client is located (on the Internet) and/or the protocol used, I get either failures or ridicuolous performance (i.e. 58-60kB/s without TSO vs. 1-2kB/s with TSO). Packet capture that shows the problem would be great to know what's going on here. I'll send them to you privately. You'll see tso.dump and notso.dump: they are both from the same client downloading the same (random) file (the file name was changed only to prevent possible caching). See notso.dump is perfect, while tso.dump shows a lot of potential problems. Thanks. Would you try attached patch and let me know it goes? Sorry, it seems extra pull up for TCP payload may not be required here. Try this instead. I see a little increase in performance (2-3kB/s instead of 1-2kB/s); this might however well depend on external factors. Still it is very different from what I'm get without TSO. Thanks for testing. Based on dump file, I tried various MTU configuration and I was not able to reproduce it. By chance, are you using firewall(pf/ipfw/ipf) or bridge(4)? If I remember correctly some firewall rules are not compatible with TSO. For bridge, if one member of bridge does not support TSO, TSO should be disabled. bye Thanks av. P.S. I can live well without TSO; I'm just doing this to let the software improve. Go ahead only if *you* are interested. I do care driver stability so it would be great if I manage to address the issue. :-) ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Intel 82550 Pro/100 Ethernet and TSO troubles
Hello. I recently installed 8.2 with the following card: dmesg ... fxp0: Intel 82550 Pro/100 Ethernet port 0xdc00-0xdc3f mem 0xfebfb000-0xfebfbfff,0xfebc-0xfebd irq 20 at device 5.0 on pci4 ... pciconv -lv ... fxp0@pci0:4:5:0:class=0x02 card=0x00408086 chip=0x12298086 rev=0x0c hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet ... I had weird problems, especially due to ICMP/fragmentation/MTU/... issues. After hours spent in debugging, I noticed tcpdump showing ICMP packets with wrong checksums and that TSO4 was enabled. ifconfig fxp0 -tso seemed to solve everything. Don't know if it's a bug in the card, in the driver or what... Maybe it shouldn't be enabled by default? I also have fxp1: dmesg ... fxp1: Intel 82559 Pro/100 Ethernet port 0xd880-0xd8bf mem 0xfebfa 000-0xfebfafff,0xfea0-0xfeaf irq 21 at device 6.0 on pci4 ... pciconv -lv ... fxp1@pci0:4:6:0:class=0x02 card=0x000c8086 chip=0x12298086 rev=0x08 hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet ... TSO was not enabled on this by default. Feel free to ask, in case you need any further info. bye av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Wed, Dec 14, 2011 at 08:33:36PM +0100, Andrea Venturoli wrote: Hello. I recently installed 8.2 with the following card: dmesg ... fxp0: Intel 82550 Pro/100 Ethernet port 0xdc00-0xdc3f mem 0xfebfb000-0xfebfbfff,0xfebc-0xfebd irq 20 at device 5.0 on pci4 ... pciconv -lv ... fxp0@pci0:4:5:0:class=0x02 card=0x00408086 chip=0x12298086 rev=0x0c hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet ... I had weird problems, especially due to ICMP/fragmentation/MTU/... issues. After hours spent in debugging, I noticed tcpdump showing ICMP packets with wrong checksums and that TSO4 was enabled. ifconfig fxp0 -tso seemed to solve everything. Don't know if it's a bug in the card, in the driver or what... Maybe it shouldn't be enabled by default? AFAIK the firmware of controller has no known TSO issue so it indicates a bug in driver. What makes me wonder is ICMP ECHO packet should not be affected by TSO and I have no clue at this moment. (Here, I assume you've captured packets on receiver side since bpf sees packets before hardware computes checksum.) If you have a reliable way that reproduces the issue, let me know. I also have fxp1: dmesg ... fxp1: Intel 82559 Pro/100 Ethernet port 0xd880-0xd8bf mem 0xfebfa 000-0xfebfafff,0xfea0-0xfeaf irq 21 at device 6.0 on pci4 ... pciconv -lv ... fxp1@pci0:4:6:0:class=0x02 card=0x000c8086 chip=0x12298086 rev=0x08 hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet ... TSO was not enabled on this by default. TSO is available only on i82550/i82551. Feel free to ask, in case you need any further info. bye av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On 12/14/11 20:59, YongHyeon PYUN wrote: AFAIK the firmware of controller has no known TSO issue so it indicates a bug in driver. What makes me wonder is ICMP ECHO packet should not be affected by TSO and I have no clue at this moment. I wasn't talking about ICMP ECHO. What happened was: a) the client connected to my server, advertising a TCP MSS of X; b) my server started sending with packets larger than X (possibly it misinterpreted MSS size???); c) an ICMP packet arrived, asking my server to send packets no larger than Y (I guess it was ignored); d) my server kept resending the same (too big) packets; e) it eventually reduced packet size, but it was still larger than Y; ... Wireshark showed some wrong checksums (I believe on the ICMP packet, but I might remember wrong). Of course this made a bell ring and removing TSO solved everything. (Here, I assume you've captured packets on receiver side since bpf sees packets before hardware computes checksum.) Yes, although I don't have them anymore. If you have a reliable way that reproduces the issue, let me know. I can turn TSO on again, save the packets and send them to you. I just hope nothing changes on the Internet in between the server and the client. Let me know if you need/want this and I'll arrange in the next few days. bye Thanks av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Wed, Dec 14, 2011 at 10:17:41PM +0100, Andrea Venturoli wrote: On 12/14/11 20:59, YongHyeon PYUN wrote: AFAIK the firmware of controller has no known TSO issue so it indicates a bug in driver. What makes me wonder is ICMP ECHO packet should not be affected by TSO and I have no clue at this moment. I wasn't talking about ICMP ECHO. What happened was: a) the client connected to my server, advertising a TCP MSS of X; b) my server started sending with packets larger than X (possibly it misinterpreted MSS size???); c) an ICMP packet arrived, asking my server to send packets no larger than Y (I guess it was ignored); d) my server kept resending the same (too big) packets; e) it eventually reduced packet size, but it was still larger than Y; ... Wireshark showed some wrong checksums (I believe on the ICMP packet, but I might remember wrong). You can check whether you received bad checksummed frames with netstat(1). Of course this made a bell ring and removing TSO solved everything. (Here, I assume you've captured packets on receiver side since bpf sees packets before hardware computes checksum.) Yes, although I don't have them anymore. If you have a reliable way that reproduces the issue, let me know. I can turn TSO on again, save the packets and send them to you. I just hope nothing changes on the Internet in between the server and the client. Let me know if you need/want this and I'll arrange in the next few days. Let me know MSS of both client and server. Is simple downloading from client to server is enough to trigger the issue? Packet capture that shows the problem would be great to know what's going on here. bye Thanks av. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Wed, Dec 14, 2011 at 01:32:42PM -0800, YongHyeon PYUN wrote:
On Wed, Dec 14, 2011 at 10:17:41PM +0100, Andrea Venturoli wrote:
On 12/14/11 20:59, YongHyeon PYUN wrote:
AFAIK the firmware of controller has no known TSO issue so it
indicates a bug in driver.
What makes me wonder is ICMP ECHO packet should not be affected by
TSO and I have no clue at this moment.
I wasn't talking about ICMP ECHO.
What happened was:
a) the client connected to my server, advertising a TCP MSS of X;
b) my server started sending with packets larger than X (possibly it
misinterpreted MSS size???);
c) an ICMP packet arrived, asking my server to send packets no larger
than Y (I guess it was ignored);
d) my server kept resending the same (too big) packets;
e) it eventually reduced packet size, but it was still larger than Y;
...
Wireshark showed some wrong checksums (I believe on the ICMP packet, but
I might remember wrong).
You can check whether you received bad checksummed frames with
netstat(1).
Of course this made a bell ring and removing TSO solved everything.
(Here, I assume you've
captured packets on receiver side since bpf sees packets before
hardware computes checksum.)
Yes, although I don't have them anymore.
If you have a reliable way that reproduces the issue, let me know.
I can turn TSO on again, save the packets and send them to you.
I just hope nothing changes on the Internet in between the server and
the client.
Let me know if you need/want this and I'll arrange in the next few days.
Let me know MSS of both client and server.
Is simple downloading from client to server is enough to trigger
the issue? Packet capture that shows the problem would be great to
know what's going on here.
Would you try attached patch and let me know it goes?
Index: sys/dev/fxp/if_fxp.c
===
--- sys/dev/fxp/if_fxp.c (revision 228504)
+++ sys/dev/fxp/if_fxp.c (working copy)
@@ -1454,7 +1454,8 @@
return (ENOBUFS);
}
tcp = (struct tcphdr *)(mtod(m, char *) + poff);
- m = m_pullup(m, poff + sizeof(struct tcphdr) + tcp-th_off);
+ m = m_pullup(m, poff + sizeof(struct tcphdr) +
+ (tcp-th_off 2));
if (m == NULL) {
*m_head = NULL;
return (ENOBUFS);
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: Intel 82550 Pro/100 Ethernet and TSO troubles
On Wed, Dec 14, 2011 at 03:22:06PM -0800, YongHyeon PYUN wrote:
On Wed, Dec 14, 2011 at 01:32:42PM -0800, YongHyeon PYUN wrote:
On Wed, Dec 14, 2011 at 10:17:41PM +0100, Andrea Venturoli wrote:
On 12/14/11 20:59, YongHyeon PYUN wrote:
AFAIK the firmware of controller has no known TSO issue so it
indicates a bug in driver.
What makes me wonder is ICMP ECHO packet should not be affected by
TSO and I have no clue at this moment.
I wasn't talking about ICMP ECHO.
What happened was:
a) the client connected to my server, advertising a TCP MSS of X;
b) my server started sending with packets larger than X (possibly it
misinterpreted MSS size???);
c) an ICMP packet arrived, asking my server to send packets no larger
than Y (I guess it was ignored);
d) my server kept resending the same (too big) packets;
e) it eventually reduced packet size, but it was still larger than Y;
...
Wireshark showed some wrong checksums (I believe on the ICMP packet, but
I might remember wrong).
You can check whether you received bad checksummed frames with
netstat(1).
Of course this made a bell ring and removing TSO solved everything.
(Here, I assume you've
captured packets on receiver side since bpf sees packets before
hardware computes checksum.)
Yes, although I don't have them anymore.
If you have a reliable way that reproduces the issue, let me know.
I can turn TSO on again, save the packets and send them to you.
I just hope nothing changes on the Internet in between the server and
the client.
Let me know if you need/want this and I'll arrange in the next few days.
Let me know MSS of both client and server.
Is simple downloading from client to server is enough to trigger
the issue? Packet capture that shows the problem would be great to
know what's going on here.
Would you try attached patch and let me know it goes?
Sorry, it seems extra pull up for TCP payload may not be required
here. Try this instead.
Index: sys/dev/fxp/if_fxp.c
===
--- sys/dev/fxp/if_fxp.c(revision 228504)
+++ sys/dev/fxp/if_fxp.c(working copy)
@@ -1454,7 +1454,7 @@
return (ENOBUFS);
}
tcp = (struct tcphdr *)(mtod(m, char *) + poff);
- m = m_pullup(m, poff + sizeof(struct tcphdr) + tcp-th_off);
+ m = m_pullup(m, poff + (tcp-th_off 2));
if (m == NULL) {
*m_head = NULL;
return (ENOBUFS);
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
