Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
--On 04 July 2019 10:59 +0200 Christian M wrote: From my tests I found that 10.4-RELEASE was as fast as you could expect (>10Gbit/s), then something changed in 11.0-RELEASE and carried on and got even worse in 12.0-RELEASE. Would it not be a good idea to begin there (10.4 -> 11.0), and try to identify what changes could impact the performance drop so significantly? This is perhaps a much harder task than it sounds for someone that knows nothing about what changes was made, and how difficult it would be to identify what changes actually could be relevant. Just a thought. Looking at FreeBSD source - there doesn't appear to be a lot that has changed (at least in sys/xen) even from 10.4 through to 12 - certainly nothing networking I can see (unless I'm using svn log 'wrong', or there's other xen stuff within FreeBSD I should be looking at). So this could be caused by other changes that have happened with the OS over that time. If I get time - I'll see if I can follow the thread and setup a similar test system here, to be fair we've not had a performance issue with Xen Networking (as nothing we do needs the performance - yet). We've been much more plagued by packet weirdness which virtio has fixed. Even if I can re-create things here - it may only be an "oh yeah, so it is" moment. -Kp ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
--On 04 July 2019 09:23 +0200 Roger Pau Monné wrote: As a workaround you can switch to the emulated network card by setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will give you worse performance than a fully working PV network card, but at least should be consistent. There are others that have switched to virtio-net, but I have no idea how to do that with XCP. Just to add my $0.02's to the conversation... As Roger knows we've had numerous issues with Xen xn based networking and FreeBSD guests over the years. We currently run VirtIO (vtnet) on XenServer 7.1 and it solves all these issues (it's probably not quite as 'performant' as xn - but it's close for what we're using it with - it is miles better than e1000 performance). Unfortunately in XenServer 7.6 virtio is no longer compiled into Qemu from what I can see. XCP-ng has the same issue (as it's based on XenServer) - so virtio is currently not an option with XCP-ng either. e1000/rtl8139 will solve 'weirdness' (think of VM's routing traffic, doing DHCP or VPN duties) - but virtio solves the same issues, with much better performance. I've posted to the XCP-ng forums asking if virtio can be enabled in XCP-ng builds (as this is probably more likely that getting XenServer to release with it enabled). Aside from getting Xen xn 'fixed' for the cases it currently fails (no small undertaking from what I understand) - it leaves FreeBSD pretty much stuck, at least for some usage cases. Regards, -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Thanks Roger. I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got 1000baseT full duplex auto selected, so I expected throughput close to that, but to my surprise this are the results: $ ifconfig em0: flags=8843 metric 0 mtu 1500 options=812099 ether 56:65:6f:f3:02:fb inet 172.31.16.125 netmask 0xff00 broadcast 172.31.16.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff00 groups: lo nd6 options=21 $ iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec0368 KBytes [ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec0368 KBytes [ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec0368 KBytes [ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec0368 KBytes [ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec0368 KBytes [ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec0385 KBytes [ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec0385 KBytes [ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec0385 KBytes [ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec0385 KBytes [ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec0385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec0 sender [ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec receiver iperf Done. $ iperf3 -c 172.31.16.126 -R Connecting to host 172.31.16.126, port 5201 Reverse mode, remote host 172.31.16.126 is sending [ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec [ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec [ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec [ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec [ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec [ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec [ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec [ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec [ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec [ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec0 sender [ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec receiver iperf Done. Good luck with your talk. Regards, Christian Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monné : > On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné < > roger@citrix.com>: > > > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and > .128. > > > VIF > > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > > > I feel the throughput is more consistent now, not all over the place > as > > > > before, even between runs. But the Retr column (tcp retries) in > iperf3 > > > has > > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > > issue > > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it > to > > > be, > > > > and 0 retries. > > > > > > > > 12.0-STABLE: > > > > > > > > Connecting to host 172.31.16.128, port 5201 > > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port > 5201 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 > KBytes > > > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 > KBytes > > > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 > KBytes > > > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 > KBytes > > > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 > KBytes > > > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 > KBytes > > > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 > KBytes > > > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 > KBytes > > > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 > KBytes > > > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 > KBytes > > > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné : > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > > VIF > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > I feel the throughput is more consistent now, not all over the place as > > > before, even between runs. But the Retr column (tcp retries) in iperf3 > > has > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > issue > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > > be, > > > and 0 retries. > > > > > > 12.0-STABLE: > > > > > > Connecting to host 172.31.16.128, port 5201 > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > Can you paste the output of ifconfig for both the interfaces used in > > the test? > > > > Are you sure all hardware offloading capabilities are turned off on > > both interfaces? > > > > Can you check what's causing those retries? > > > > Either using tcpdump, whireshark or some other tool to analyze the > > network traffic and detect the errors that cause such retries? > > > > Thanks, Roger. > > > > 172.31.16.127 (12.0-STABLE): > > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff00 > groups: lo > nd6 options=21 > xn0: flags=8843 metric 0 mtu 1500 > options=501 I would try to disable rxcsum, tso4 and lro also. > ether 6e:83:99:ed:ce:f7 > inet 172.31.16.127 netmask 0xff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29 > > ethtool -k vif68.0 > Features for vif68.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] >
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné : > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > VIF > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > I feel the throughput is more consistent now, not all over the place as > > before, even between runs. But the Retr column (tcp retries) in iperf3 > has > > jumped up considerably from hundreds/s to thousands/s. > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > issue > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > be, > > and 0 retries. > > > > 12.0-STABLE: > > > > Connecting to host 172.31.16.128, port 5201 > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > Can you paste the output of ifconfig for both the interfaces used in > the test? > > Are you sure all hardware offloading capabilities are turned off on > both interfaces? > > Can you check what's causing those retries? > > Either using tcpdump, whireshark or some other tool to analyze the > network traffic and detect the errors that cause such retries? > > Thanks, Roger. > 172.31.16.127 (12.0-STABLE): lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff00 groups: lo nd6 options=21 xn0: flags=8843 metric 0 mtu 1500 options=501 ether 6e:83:99:ed:ce:f7 inet 172.31.16.127 netmask 0xff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29 ethtool -k vif68.0 Features for vif68.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] 172.31.16.128 (12.0-STABLE): lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff00 groups: lo nd6 options=21 xn0: flags=8843 metric 0 mtu 1500 options=501
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > I feel the throughput is more consistent now, not all over the place as > before, even between runs. But the Retr column (tcp retries) in iperf3 has > jumped up considerably from hundreds/s to thousands/s. > > Just a reminder, I have tested this with 11.0-RELEASE also, where the issue > appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, > and 0 retries. > > 12.0-STABLE: > > Connecting to host 172.31.16.128, port 5201 > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 Can you paste the output of ifconfig for both the interfaces used in the test? Are you sure all hardware offloading capabilities are turned off on both interfaces? Can you check what's causing those retries? Either using tcpdump, whireshark or some other tool to analyze the network traffic and detect the errors that cause such retries? Thanks, Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF cheksum offloading is turned off, and -txcsum for xn0 for both VM's. I feel the throughput is more consistent now, not all over the place as before, even between runs. But the Retr column (tcp retries) in iperf3 has jumped up considerably from hundreds/s to thousands/s. Just a reminder, I have tested this with 11.0-RELEASE also, where the issue appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, and 0 retries. 12.0-STABLE: Connecting to host 172.31.16.128, port 5201 [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 sender [ 5] 0.00-20.00 sec 2.05 GBytes 881 Mbits/sec receiver 10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for xn0): Connecting to host 172.31.16.122, port 5201 [ 5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 832 MBytes 6.98 Gbits/sec0580 KBytes [ 5] 1.00-2.00 sec 1.24 GBytes 10.6 Gbits/sec0 1.04 MBytes [ 5] 2.00-3.00 sec 1.34 GBytes 11.5 Gbits/sec0 1.57 MBytes [ 5] 3.00-4.00 sec 1.41 GBytes 12.1 Gbits/sec0 1.76 MBytes [ 5] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec0 1.77 MBytes [ 5] 5.00-6.00 sec 1.34 GBytes 11.5 Gbits/sec0 1.77 MBytes [ 5] 6.00-7.00 sec 1.37 GBytes 11.8 Gbits/sec0 1.77 MBytes [ 5] 7.00-8.00 sec 1.39 GBytes 12.0 Gbits/sec0 1.77 MBytes [ 5] 8.00-9.00 sec 1.35 GBytes 11.6 Gbits/sec0 1.77 MBytes [ 5] 9.00-10.00 sec 1.42 GBytes 12.2 Gbits/sec0 1.77 MBytes [ 5] 10.00-11.00 sec 1.34 GBytes 11.5 Gbits/sec0 1.77 MBytes [ 5] 11.00-12.00 sec 1.32 GBytes 11.4 Gbits/sec0 1.77 MBytes [ 5] 12.00-13.00 sec 1.34 GBytes 11.6 Gbits/sec0 1.77 MBytes [ 5] 13.00-14.00 sec 1.49 GBytes 12.8 Gbits/sec0 1.77 MBytes [ 5] 14.00-15.00 sec 1.36 GBytes 11.7 Gbits/sec0 1.77 MBytes [ 5] 15.00-16.00 sec 1.37 GBytes 11.7 Gbits/sec0 1.77 MBytes [ 5] 16.00-17.00 sec 1.48 GBytes 12.7 Gbits/sec0 1.77 MBytes [ 5] 17.00-18.00 sec 1.36 GBytes 11.7 Gbits/sec0 1.77 MBytes [ 5] 18.00-19.00 sec 1.47 GBytes 12.6 Gbits/sec0 1.77 MBytes [ 5] 19.00-20.00 sec 1.36 GBytes 11.7 Gbits/sec0 1.77 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 26.9 GBytes 11.6 Gbits/sec0 sender [ 5] 0.00-20.01 sec 26.9 GBytes 11.6 Gbits/sec receiver ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
26.06.2019 20:17, Christian M wrote: > Started ntpd in the two main 12-0-RELEASE vm's I've been testing with. > drift files show -0.262 and -1.144 after about two hours. This is very good, so it should not be timekeeping problem. There were many fixes to FreeBSD12 after the release. Can you update to 12.0-STABLE? ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Started ntpd in the two main 12-0-RELEASE vm's I've been testing with. drift files show -0.262 and -1.144 after about two hours. Den ons 26 juni 2019 kl 12:50 skrev Eugene Grosbein : > 26.06.2019 17:44, Christian M wrote: > > > Sorry to say, but no. Nothing changed :( > > > > The iperf issue concerns me a bit also: > > Try starting ntpd inside VM guests, wait an hour then post contents of > /var/db/ntpd.drift > This is to verify quality of time source provided to VMs by hypervisor. > > ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
26.06.2019 17:26, Christian M wrote: > I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum > disappeared. [skip] Data looks good. Did it run better this time? ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum disappeared. netstat -sp ip for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): ip: 3664084 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 3664080 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 4 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 7003245 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header ip: 8217827 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 8217827 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4300027 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header netstat -sp tcp for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): tcp: 7002523 packets sent 6866670 data packets (9939548357 bytes) 109393 data packets (101233068 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 27039 ack-only packets (90 delayed) 0 URG only packets 0 window probe packets 0 window update packets 125 control packets 3664099 packets received 3387212 acks (for 9939333715 bytes) 270936 duplicate acks 0 acks for unsent data 915 packets (36547 bytes) received in-sequence 4887 completely duplicate packets (208 bytes) 4853 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 5694 window update packets 6 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 88 connection requests 2 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 89 connections established (including accepts) 85 times used RTT from hostcache 85 times used RTT variance from hostcache 85 times used slow-start threshold from hostcache 90 connections closed (including 57 drops) 51 connections updated cached RTT on close 51 connections updated cached RTT variance on close 23 connections updated cached ssthresh on close 1 embryonic connection dropped 3387212 segments updated rtt (of 3313837 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 759139 correct ACK header predictions 70 correct data packet header predictions 2 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 2 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 2 cookies sent 0 cookies received 1 hostcache entry added 0 bucket overflow 5602 SACK recovery episodes 109329 segment rexmits in SACK recovery episodes 101663360 byte rexmits in SACK recovery episodes 249723 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
26.06.2019 15:11, Christian M wrote: > Running tcpdump on the host while running iperf3 between the 12.0 VM's > results in a lot of incorrect cksum like this. > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 Incorrect checksum shown by tcpdump is normal and expected for outgoing packets if txcsum offload is not disabled because networking stack skips checksum calculation offloading it to "hardware". It still indicates a bug at sending side if seen for outgoing packets when txcsum is disabled. Incorrect checksum shown by tcpdump for incoming packets always indicates a problem at sending side or intermediate network (if any). You should check output of "netstat -sp ip" and "netstat -sp tcp" for non-zero and growing checksum error counters at receiving side. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
checksum offloading doe not seem to have any affect when done on the VM VIF's. Both 12.0-RELEASE VIF's now have: other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off ethtool -k vif54.0 Features for vif54.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Both vif's look the same as above. Running tcpdump on the host while running iperf3 between the 12.0 VM's results in a lot of incorrect cksum like this. tcpdump -i vif54.0 -v -nn| grep -i incorrect 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 ... Also noticed that iperf3 does not work to run every time, more than 50% of the tries with iperf3 -c I get: Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected Or nothing happens and I need to abort, then restart iperf3 -s on the other VM, and retry. The results from iperf3 are really inconsistent still, and I cant get a good feel of any pattern. Currently as I type this, a result looks like this: Host: tcpdump running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 37.8 MBytes 316 Mbits/sec 176 1.43 KBytes [ 5] 1.00-2.00 sec 133 MBytes 1.11 Gbits/sec 1022 49.8 KBytes [ 5] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec 784191 KBytes [ 5] 3.00-4.00 sec 132 MBytes 1.11 Gbits/sec 201345 KBytes [ 5] 4.00-5.00 sec 95.3 MBytes 799 Mbits/sec 438180 KBytes [ 5] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 1256353 KBytes [ 5] 6.00-7.00 sec 144 MBytes 1.20 Gbits/sec 1242275 KBytes [ 5] 7.00-8.00 sec 94.5 MBytes 792 Mbits/sec 593167 KBytes [ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 326169 KBytes [ 5] 9.00-10.00 sec 57.0 MBytes 478 Mbits/sec 1013 1.43 KBytes [ 5] 10.00-11.00 sec 31.3 MBytes 262 Mbits/sec 463 1.43 KBytes [ 5] 11.00-12.00 sec 42.3 MBytes 355 Mbits/sec 544 78.3 KBytes [ 5] 12.00-13.00 sec 41.3 MBytes 346 Mbits/sec 624 1.43 KBytes [ 5] 13.00-14.00 sec 52.7 MBytes 442 Mbits/sec 739 1.43 KBytes [ 5] 14.00-15.00 sec 20.4 MBytes 171 Mbits/sec 384 1.43 KBytes [ 5] 15.00-16.00 sec 74.5 MBytes 625 Mbits/sec 489299 KBytes [ 5] 16.00-17.00 sec 40.1 MBytes 337 Mbits/sec 835 1.43 KBytes [ 5] 17.00-18.00 sec 43.2 MBytes 363 Mbits/sec 372 1.43 KBytes [ 5] 18.00-19.00 sec 58.9 MBytes 494 Mbits/sec 723 1.43 KBytes [ 5] 19.00-20.00 sec 21.2 MBytes 178 Mbits/sec 300 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec 12524 sender [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec receiver But please be sure, this can change dramatically without making any changes what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as stated before), and tcpdump is running on the host. xn0: flags=8843 metric 0 mtu 1500 options=503 ether 12:ec:6a:5c:af:df inet 172.31.16.125 netmask 0xff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29 xn0: flags=8843 metric 0 mtu 1500 options=503 ether aa:4f:f7:4c:a0:e6 inet 172.31.16.126 netmask 0xff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29 Absolutely nothing changed this time (restarted iperf3 -s though since I ran ifconfig do get the info above), tcpdump still running on host (never even
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
25.06.2019 19:40, Christian M wrote: > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0 > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF > via XCP-ng earlier. It made no difference unfortunately. Then you should try to bring the interface to promiscuous mode manually with "ifconfig xn0 promisc" any retry. If this does not make a difference, then you may be facing some timekeeping problem as tcpdump makes additional CPU load on the system that can force scheduler behaviour to change. As next step, you should check your kern.timecounter.* and kern.eventtimer.* sysctls. If default value of kern.eventtimer.periodic is 0 (it depends on detected "hardware"), you should change it kern.eventtimer.periodic=1 and retry the test. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
oh yes, sorry I am using: 0:~ # xl info host : x0 release : 4.12.14-lp150.12.64-default version : #1 SMP Mon Jun 17 16:53:50 UTC 2019 (3edfd41) machine : x86_64 nr_cpus : 4 max_cpu_id : 3 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3192 hw_caps : bfebfbff:77faf3bf:2c100800:0021:0001:27ab::0100 virt_caps : hvm hvm_directio total_memory : 32461 free_memory : 5319 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 10 xen_extra : .3_04-lp150.2.1 xen_version : 4.10.3_04-lp150.2.1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0x8000 xen_changeset : xen_commandline : dom0_mem=3072M,max:3072M cc_compiler : gcc (SUSE Linux) 7.4.0 cc_compile_by : abuild cc_compile_domain : suse.de cc_compile_date : Thu May 9 16:10:08 UTC 2019 build_id : bf77d035ff05eb55d9616e418f73cbcbd96f747b xend_config_format : 4 Since you peaked my interest, I tryed playing around a little. Initially to day, the retransmits where between 4-15 retransmits. I then changed the network interface from hypervisor default to E1000 (still xn0 in freebsd) and the retransmits jumped up to 700 to 800! After changing the network interface back to hypervisor default, the number of retranmits remains high in the 3-digit area. Fooling around with mtu size did not change anything. Something is not quite right. On 6/24/19 10:24 PM, Christian M wrote: Thanks for your input Bjarne. Previously I tested with iperf2, but I have made som tests with iperf3 now also and noticed there are a lot of "Retr" (TCP retries) in some cases. Went back here to your post and saw that your results also showed a lot of retries in some cases. My new tests showed similar results to yours (although not nearly as hi throughput as you have). When Linux is client the retries are 0, with FreeBSD as client the number is > 0. I'm not sure if our numbers are considered high though, and something that actually is a problem? Although, on a internal network with no external factors like interference, I feel this should always be 0 no matter what? What hypervisor were you running in your tests? Version? Regards, Christian Den mån 24 juni 2019 kl 17:56 skrev Bjarne : Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3 -s / iperf3 -c I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD 3,5Gbit/sec FBSD -> Centos: 3GBit/Sec Centos -> FBSD: 12 Gbit/sec :snip -- mvh, Bjarne ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
There's a mistake in my reply below. On Tue, Jun 25, 2019 at 04:56:43PM +0200, Roger Pau Monné wrote: > On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote: > > I've made two tests while running tcpdump on the xcp-ng host. I'm not at > > all qualified to interpret the .pcap files from tcpdump, but I've put them > > on Google Drive and linked them below the two tests. Perhaps someone more > > qualified could have a look for anything useful in there. Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > > > Host: XCP-ng 7.6.0 > > Network: Private Network on host, not connected to any PIF. > > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) > > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) > > > > On the host I listen with tcpdump on the VIF for VM1 in both tests. > > > > VM1 as client: > > > > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap > > Can you check the capabilities of vif42.0? (ie: whether csum > offloading is actually disabled on the host?) > > > xcp-ng-vm1-client.pcap (80M): > > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU > > On VM1: iperf3 -c 172.31.16.126 > > On VM2: iperf3 -s > > I've taken a look at the dump and the checksum is wrong (or maybe > missing) for all? packets. > > Packets with source 172.31.16.125 all have the TCP checksum set to > 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum ^ 172.31.16.126 ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0 -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF via XCP-ng earlier. It made no difference unfortunately. Den tis 25 juni 2019 kl 14:08 skrev Eugene Grosbein : > 25.06.2019 18:55, Christian M wrote: > > > Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > Generally this indicates broken "hardware" offload for some features > like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. > > Some of them may be automatically disabled while interface is put in > promiscuous mode > by running tcpdump and enabled back after tcpdump terminated. > > You should look at "options" line in the output of ifconfig for the > interface > and manually disable offloads (look for ifconfig(8) manual for details). > They retry tests without tcpdump running. > > ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
25.06.2019 18:55, Christian M wrote: > Please note the > extremely uneven throughput for test 2 below. It's almost like the > throughput increased when running tcpdump simultaneously. Generally this indicates broken "hardware" offload for some features like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. Some of them may be automatically disabled while interface is put in promiscuous mode by running tcpdump and enabled back after tcpdump terminated. You should look at "options" line in the output of ifconfig for the interface and manually disable offloads (look for ifconfig(8) manual for details). They retry tests without tcpdump running. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3 -s / iperf3 -c I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD 3,5Gbit/sec FBSD -> Centos: 3GBit/Sec Centos -> FBSD: 12 Gbit/sec Centos <-> Centos: 12 Gbit/sec OpenSuse <-> OpenSuse 13 GBit/sec So it ssems there is some fishy when Freebsd is used as client xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm has been optimized in any way, everything is out of the box. === Test 1, Freebsd 12 -> freebsd 12 === Server Freebsd 12: ccepted connection from 192.168.1.100, port 39518 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver Client, freebsd 12: Connecting to host 10.192.168.99, port 5201 [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 sender [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver == test 2, centos 7.6 -> Freebsd 12 == Server Freebsd 12: Accepted connection from 192.168.1.162, port 49180 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver Client, centos 7.6: Connecting to host 192.168.1.99, port 5201 [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver
Re: Very slow and inconsistent internal network speed (between VM's on the same host) for FreeBSD 11.0+ as guest on XCP-ng/XenServer
Den tors 20 juni 2019 kl 16:39 skrev Roger Pau Monné : > > Do you see the same issues with external connections? Have you tested > throughput between two FreeBSD 12.0 VM running on different hosts? > > I've tested 12.0-RELEASE between two hosts (XCP-ng 7.6.0 and Citrix CXenserver 7.2.0) over Gbit link, and iperf results is about 600Mbit/s back and forth. I also tested 12.0-RELEASE on same XCP-ng host, but changed the network from Private Network to the Internal Network (that's connected to a PIF). That increased speed slightly, but It's still below 100mbit back and forth. > > > Hm, OK that's weird, I don't think however it's related to Xen. Have > you tried if the same happens on a bare-metal install of FreeBSD? > > Or when running on a different hypervisor? > > I have not. I only have XCP-ng 7.6.0 and Citrix XenServer 7.x to play with currently. I have no server at this moment to test a bare metal install on, but if you believe this is an important test I could perhaps find a machine to try it on. Please let me know if I should proceed with this. > > > > > Any ideas of how to proceed now to find a solution for this? > > Maybe you can try to run wireshark/tcpdump or some other similar > software in order to try to detect if there are errors on the > transmitted packets? > > You could run the sniffer on the host and attach it to the backend > interfaces (vifX.X) or the bridge if you are using bridged > networking. > > The 12.0 <-> 12.0 case seems quite bad, so I would start with that > one. > > Roger. > I have now run tshark (wireshark) to capture traffic between 12.0 <-> 12.0 on the private network (running tshark on freebsd). I'm really not qualified to interpret the output, so I'll link to the capture-output here: capture-output.pcap: https://drive.google.com/open?id=1zU6qhZO7OOKfLRugXJovYIQuFCLrouH6 (218MB) output.txt (tshark -r capture-output.pcap > output.txt): https://drive.google.com/open?id=1ORJy_jhDyAIgmw1WnXBsPQSlIxqEzVrF (34MB) I'm not really sure how to run it on XCP-ng, or is there perhaps another tool I can use for that other than tshark? Thanks, Christian ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
