vmxnet3 LROv6 performance issues
Hi, this started as a Debian bug (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816377), but since it's affecting SLES as well I'm hoping to get some help here. Back in 2014 we migrated our VMware farm from old HP blade servers to new ones Old: HP BL490c Gen6, Flex10 something, BCM57711E New: HP BL460c Gen8, HP FlexFabric 630FLB, BCM57840 The new network chipset apparently sports IPV6 LRO support in hardware. Unfortunately that support was broken with in-tree vmxnet3 kernel modules back then, TCPv6 connections were stalling all over the place. Disabling LRO or using the vmxnet3 module provided by VMware fixed the issue. We are now seeing the issue again. Not as prominent as before, but still affecting some workloads pretty badly (for example a simple iperf3 benchmark towards an affected VM). client% iperf3 -c ping.lrz.de -t 30 Connecting to host ping.lrz.de, port 5201 [ 4] local 2001:4ca0:0:f000:bf47:886b:a813:df4f port 60174 connected to 2001:4ca0:0:101::81bb:a11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 305 KBytes 2.50 Mbits/sec 34 8.37 KBytes [ 4] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 24 2.79 KBytes [ 4] 2.00-3.00 sec 251 KBytes 2.06 Mbits/sec 75 2.79 KBytes [ 4] 3.00-4.00 sec 251 KBytes 2.06 Mbits/sec 124 2.79 KBytes [ 4] 4.00-5.00 sec 88.7 MBytes 744 Mbits/sec 24332 KBytes [ 4] 5.00-6.00 sec 111 MBytes 931 Mbits/sec0476 KBytes [ 4] 6.00-7.00 sec 110 MBytes 921 Mbits/sec0478 KBytes [ 4] 7.00-8.00 sec 110 MBytes 923 Mbits/sec0481 KBytes [ 4] 8.00-9.00 sec 110 MBytes 921 Mbits/sec0502 KBytes [ 4] 9.00-10.00 sec 19.8 MBytes 166 Mbits/sec 27 2.79 KBytes [ 4] 10.00-11.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 11.00-12.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 12.00-13.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 13.00-14.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 14.00-15.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 15.00-16.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 16.00-17.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 17.00-18.00 sec 0.00 Bytes 0.00 bits/sec 16 2.79 KBytes [ 4] 18.00-19.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 19.00-20.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 20.00-21.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 21.00-22.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 22.00-23.00 sec 0.00 Bytes 0.00 bits/sec 22 2.79 KBytes [ 4] 23.00-24.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 24.00-25.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 25.00-26.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 26.00-27.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 27.00-28.00 sec 0.00 Bytes 0.00 bits/sec 16 2.79 KBytes [ 4] 28.00-29.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes [ 4] 29.00-30.00 sec 0.00 Bytes 0.00 bits/sec 20 2.79 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-30.00 sec 550 MBytes 154 Mbits/sec 702 sender [ 4] 0.00-30.00 sec 547 MBytes 153 Mbits/sec receiver iperf Done. server# ethtool -K eth0 lro off client% iperf3 -c ping.lrz.de -t 30 Connecting to host ping.lrz.de, port 5201 [ 4] local 2001:4ca0:0:f000:bf47:886b:a813:df4f port 60228 connected to 2001:4ca0:0:101::81bb:a11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 112 MBytes 942 Mbits/sec0477 KBytes [ 4] 1.00-2.00 sec 110 MBytes 921 Mbits/sec0499 KBytes [ 4] 2.00-3.00 sec 110 MBytes 924 Mbits/sec0499 KBytes [ 4] 3.00-4.00 sec 109 MBytes 918 Mbits/sec0499 KBytes [ 4] 4.00-5.00 sec 110 MBytes 919 Mbits/sec0523 KBytes [ 4] 5.00-6.00 sec 110 MBytes 926 Mbits/sec0523 KBytes [ 4] 6.00-7.00 sec 110 MBytes 919 Mbits/sec0547 KBytes [ 4] 7.00-8.00 sec 110 MBytes 927 Mbits/sec0547 KBytes [ 4] 8.00-9.00 sec 110 MBytes 924 Mbits/sec0629 KBytes [ 4] 9.00-10.00 sec 110 MBytes 923 Mbits/sec0629 KBytes [ 4] 10.00-11.00 sec 110 MBytes 923 Mbits/sec0629 KBytes [ 4] 11.00-12.00 sec 110 MBytes 923 Mbits/sec0629 KBytes [ 4] 12.00-13.00 sec 109 MBytes 912 Mbits/sec0629 KBytes [ 4]
Re: [IPv6] BUG: NULL pointer dereference in(?) ip6_flush_pending_frames
YOSHIFUJI Hideaki / 吉藤英明: Hi, BUG: unable to handle kernel NULL pointer dereference at virtual address 008c : EIP is at ip6_flush_pending_frames+0x97/0x121 I think I've found a bug. [...] Anyway, please try this. FTR, I tried 2.6.22.6 without the patch and it failed as well. The patched kernel is running since yesterday evening (about 8h now) and seems to be stable so far. Too early to tell for sure, but I guess we have a fix. Thanks Yoshifuji! Regards, Bernhard - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[IPv6] BUG: NULL pointer dereference in(?) ip6_flush_pending_frames
Hi,
I'm running a public Teredo relay (IPv4-to-IPv6 migration protocol)
using Miredo. Every once in a while (a few minutes to days after
daemon restart) it becomes unusable and I see the following kernel
message:
BUG: unable to handle kernel NULL pointer dereference at virtual address
008c
printing eip:
c02640e6
*pde =
Oops: [#17]
SMP
Modules linked in: ip6table_filter ip6_tables af_packet tun bitrev crc32
ipt_LOG xt_tcpudp iptable_filter iptable_mangle ip_tables x_tables
dm_mod capability commoncap iTCO_wdt floppy e1000 rtc unix
CPU:0
EIP:0060:[c02640e6]Not tainted VLI
EFLAGS: 00210246 (2.6.21.3-iabg-pe750 #1)
EIP is at ip6_flush_pending_frames+0x97/0x121
eax: ebx: d3e3ca80 ecx: db590380 edx: d3e3caf0
esi: d3e3cc80 edi: db590380 ebp: 0002 esp: d4af7cd4
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process miredo (pid: 17615, ti=d4af6000 task=cfd60030 task.ti=d4af6000)
Stack: 05d0 d4af7d44 d4af7d54 d4af7d54 db590380
c0275ab5
0040 d4af7d48 df4c6780
0040
d4af7f44 d3e3ca80 3a00 001c 003a
Call Trace:
[c0275ab5] rawv6_sendmsg+0x840/0xa63
[c0258a09] inet_sendmsg+0x3b/0x45
[c021df73] sock_sendmsg+0xbc/0xd4
[c0123f99] autoremove_wake_function+0x0/0x35
[e087c911] tun_chr_aio_read+0x29e/0x2a8 [tun]
[c011025a] default_wake_function+0x0/0xc
[c021e29c] sys_sendto+0x118/0x138
[c014d03c] do_readv_writev+0x17d/0x187
[e087c673] tun_chr_aio_read+0x0/0x2a8 [tun]
[c021ef2e] sys_socketcall+0x15e/0x242
[c0102560] syscall_call+0x7/0xb
===
Code: 8d 43 70 8b 48 04 39 c1 74 31 85 c9 74 2d ff 48 08 8b 11 8b 41 04
c7 41 04 00 00 00 00 c7 01 00 00 00 00 89 42 04 89 10 8b 41 28 8b b8
8c 00 00 00 85 ff 0f 85 6b ff ff ff eb 94 83 a3 84 01 00
EIP: [c02640e6] ip6_flush_pending_frames+0x97/0x121 SS:ESP
0068:d4af7cd4
I have not found anything related on netdev, I'll try a new kernel to be
sure. Do you need any more information to debug this issue?
Hardware is a Dell PowerEdge 750 (i386 P4 HT), vanilla kernel 2.6.21.3
running Debian testing.
Thanks,
Bernhard
-
To unsubscribe from this list: send the line unsubscribe netdev in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [IPv6] PROBLEM? Network unreachable despite correct route
Jarek Poplawski wrote: ip -6 route: 2001:4ca0:0:f000::/64 dev eth0 proto kernel metric 256 expires 86322sec mtu 1500 advmss 1440 fragtimeout 4294967295 fe80::/64 dev eth0 metric 256 expires 21225804sec mtu 1500 advmss 1440 fragtimeout 4294967295 ff00::/8 dev eth0 metric 256 expires 21225804sec mtu 1500 advmss 1440 fragtimeout 4294967295 default via fe80::2d0:4ff:fe12:2400 dev eth0 proto kernel metric 1024 expires 1717sec mtu 1500 advmss 1440 fragtimeout 64 unreachable default dev lo proto none metric -1 error -101 fragtimeout 255 Did you analyze this dev lo warning? That one is default. Recent kernels (since 2.6.12 or so, I think when the default on-link assumption was killed) have a default route pointing to unreachable default lo on bootup. Routes learned from RA or added statically are installed with a better metric and are preferred that way. I think the use is to have a Network unreachable returned immediately if no IPv6 router is present. Regards, Bernhard - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[IPv6] PROBLEM? Network unreachable despite correct route
Hi, I'm having a really ugly problem I'm trying to pinpoint, but failed so far. I'm neither completely convinced it is not related to my local setup(s), nor do I have any clue how this might be caused. I have several boxes with native IPv6 connectivity at various places. Some of them show symptoms of a lost default route for small periods of time (10-15 seconds several times a day). By symptoms I mean - traceroute6 from the affected box to any other host dies immediately (the network unreachable does not come from the first hop (the upstream router), but from the local stack itself) - a local running OpenVPN 2.1_rc1b with UDPv6 transport patched in shows the following output in the syslog file Tue Jan 9 16:48:28 2007 write UDPv6 []: Network is unreachable (code=101) - mtr from the outside to the machine shows that the affected box does not respond anymore, while the hop before (the router) is clean. - new connects to the box (e.g. ssh) from the outside are stuck (packets get lost, since I'm running my client with tcp_retries=1 I get a timeout At the same time, established ssh connections to the box work fine. I can do ip -6 route and it shows the default route, both preferred and valid lifetime not exceeded (far from that). The systems I'm observing this are: - Dell PowerEdge 750 (P4 with HT), Debian Etch, self compiled kernel 2.6.17.11, connected (e1000) to two upstream Cisco 7200, default route is learned from RIPng (Quagga), static addresses - Dell OptiPlex GXsomething (P4 with HT, Single Core), SuSE 10.2, distribution kernel 2.6.18.5-3-default, connected (tg3) to one upstream Cisco 6500/Sup720, default route learned through stateless autoconfiguration (RA) - self built AMD Athlon64 (x86_64), Ubuntu Edgy, Distribution kernel 2.6.17-10-generic, connected (forcedeth) to an upstream Linux box (2.6.20-rc3), default route learned through stateless autoconfiguration (RA) as well. My current believe is that this is an regression introduced in 2.6.17. I have searched for several weeks now why box #1 (the PowerEdge) shows signs of unreachability in the monitoring, but could not find any clue (or verify any reachability problems when I got the monitoring alert). At the same time, a sibling (same hardware, same switch, same network segment, route also learned through Quagga, but different kernel (2.6.16)) of this box did not show any symptoms, so I ruled out the local network. Also, I upgraded box #2 from SuSE 10.1 (distribution kernel 2.6.16-something) to SuSE 10.2 yesterday. While it was running the OpenVPN/UDPv6 daemon the whole time, there has been exactly _one_ occurence of the Network is unreachable message in the past two weeks before the upgrade (and I can correlate this message with network maintainance where the VPN endpoint was indeed unreachable). Since the upgrade, I have at least 50 lines of that sort in syslog (in about a day). It is pretty hard to trace this. It seems to appear very seldom, it is not long and I cannot predict the time where it happens by doing more network load or anything else on that machine. IPv4 is fine and without loss in all cases. All network components are dual-stacked, so if there was an L2 issue between the router and the host it would affect IPv4 as well. Is anyone aware of any issue which might cause this? I've upgraded the PowerEdge to 2.6.19.1 now, but it is too early to tell whether this problem still exists. Does anyone recall a bugreport and maybe a fix for it? A patch or a link to a changeset would be even better, so I could report that to SuSE and Ubuntu to have it included in future kernels. Thanks, Bernhard - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [IPv6] PROBLEM? Network unreachable despite correct route
On Tue, Jan 09, 2007 at 08:36:24PM +0100, Bernhard Schmidt wrote: Hi, I did some additional testing I'm having a really ugly problem I'm trying to pinpoint, but failed so far. I'm neither completely convinced it is not related to my local setup(s), nor do I have any clue how this might be caused. [...] - Dell OptiPlex GXsomething (P4 with HT, Single Core), SuSE 10.2, distribution kernel 2.6.18.5-3-default, connected (tg3) to one upstream Cisco 6500/Sup720, default route learned through stateless autoconfiguration (RA) Running tcpdump on this (target) box shows that ICMPv6 echo requests (which is what mtr sends to the target box) are received by the box, but not replied to 01:02:09.884692 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 54173, length 64 01:02:09.884706 IP6 2001:4ca0:0:f000:211:43ff:fe7e: 2001:a60:f001:1:218:f3ff:fe66:: ICMP6, echo reply, seq 54173, length 64 01:02:10.428063 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 55453, length 64 01:02:11.056871 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 56733, length 64 01:02:11.700772 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 58013, length 64 [...] 01:02:17.301169 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 3998, length 64 01:02:17.941020 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 5278, length 64 01:02:18.581037 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 6558, length 64 01:02:18.581050 IP6 2001:4ca0:0:f000:211:43ff:fe7e: 2001:a60:f001:1:218:f3ff:fe66:: ICMP6, echo reply, seq 6558, length 64 while this is happening, the SSH session (between the very same hosts) is perfectly fine. ip6_tables.ko is not loaded, there is no other ICMPv6 packet (e.g. neighbor solicitation or router advertisement) anywhere near the beginning of my problem. Incoming TCP SYN (an additional SSH session I tried to establish when I saw the box was not responding) are also visible on the interface, but not answered. 01:18:35.638744 IP6 2001:a60:f001:1:218:f3ff:fe66:.57045 2001:4ca0:0:f000:211:43ff:fe7e:.22: SWE 1448406153:1448406153(0) win 5760 mss 1440,sackOK,timestamp 13958554 0,nop,wscale 2 01:18:35.701523 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 41148, length 64 01:18:36.328728 IP6 2001:a60:f001:1:218:f3ff:fe66: 2001:4ca0:0:f000:211:43ff:fe7e:: ICMP6, echo request, seq 42428, length 64 I managed to pull ip -6 route, ip -6 neigh and ip -6 addr while the box was not responding: ip -6 route: 2001:4ca0:0:f000::/64 dev eth0 proto kernel metric 256 expires 86322sec mtu 1500 advmss 1440 fragtimeout 4294967295 fe80::/64 dev eth0 metric 256 expires 21225804sec mtu 1500 advmss 1440 fragtimeout 4294967295 ff00::/8 dev eth0 metric 256 expires 21225804sec mtu 1500 advmss 1440 fragtimeout 4294967295 default via fe80::2d0:4ff:fe12:2400 dev eth0 proto kernel metric 1024 expires 1717sec mtu 1500 advmss 1440 fragtimeout 64 unreachable default dev lo proto none metric -1 error -101 fragtimeout 255 ip -6 neigh: fe80::2d0:4ff:fe12:2400 dev eth0 lladdr 00:d0:04:12:24:00 router REACHABLE ip -6 addr: 2: eth0: BROADCAST,MULTICAST,NOTRAILERS,UP,1 mtu 1500 qlen 1000 inet6 2001:4ca0:0:f000:211:43ff:fe7e:/64 scope global dynamic valid_lft 86318sec preferred_lft 14318sec inet6 fe80::211:43ff:fe7e:/64 scope link valid_lft forever preferred_lft forever Nothing in dmesg or any file in /var/log (except the notorious Network is unreachable messages from OpenVPN). I was wrong before by the way, some outgoing connections from the affected machine still work, I was able to ping6, traceroute6 and telnet. At least on this particular machine, I am very sure I have seen Network unreachable on outgoing connects at some point. I'll try to downgrade this machine to 2.6.16 (and eventually upgrade to 2.6.19.1) and have a look whether the problem is gone. - Dell PowerEdge 750 (P4 with HT), Debian Etch, self compiled kernel 2.6.17.11, connected (e1000) to two upstream Cisco 7200, default route is learned from RIPng (Quagga), static addresses Still too soon to be absolutely sure, but I think the problem is gone since the upgrade to 2.6.19.1. Regards, Bernhard - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
