Re: Networking under Xen
On 2015-07-15 00:54:25 (-0700), Colin Percival cperc...@freebsd.org wrote:
In my tests, deleting these lines from pf_ioctl.c
3570 /* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
3571 if ((*m)-m_pkthdr.csum_flags CSUM_DELAY_DATA) {
3572 in_delayed_cksum(*m);
3573 (*m)-m_pkthdr.csum_flags = ~CSUM_DELAY_DATA;
3574 }
unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these lines
are there in the first place, which is why I didn't want to simply go in
and remove them -- but it may be that wrapping those lines in something
like if ((csum_flags CSUM_TSO) == 0) would solve the problem without
breaking anything else.
I think the reason for this checksum calculation is that pf sometimes
modifies the packet, so it also updates the checksum.
In case of TSO (or CSUM_TCP) that doesn't actually work because the TCP
checksum is the pseudo IP(6) header checksum, not the final checksum.
Starting from a full checksum the update is correct however, so that's
why pf works on non-TSO interfaces.
It doesn't work on Xen TSO interfaces because (I assume) it expects to
get the pseudo header checksum, not the full checksum.
It's not entirely clear to my why it's not broken on my hardware (which
claims TSO support), but perhaps Xen is more picky than actual hardware.
Adding the if (CSUM_TSO) check will improve matters, but it will still
break on packets which get changed by pf (rdr/nat/...).
I think the real solution is to make pf_cksum_fixup() a bit more
intelligent. It should look at m-m_pkthdr.csum_flags to determine if it
should touch the checksum or not.
If CSUM_TCP is set we know we've got a pseudo header checksum and we
shouldn't touch it unless we've modified the source or destination IP
address (or length or protocol, but that's less likely).
If CSUM_TCP is not set we should have a full checksum and should do the
fixup as before.
Actually, perhaps CSUM_DELAY_DATA is better than CSUM_TCP, because we'll
have to do the same for UDP.
Regards,
Kristof
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
On 07/26/15 06:16, Kristof Provost wrote: On 2015-07-15 00:54:25 (-0700), Colin Percival cperc...@freebsd.org wrote: In my tests, deleting these lines from pf_ioctl.c 3570 /* We need a proper CSUM befor we start (s. OpenBSD ip_output) */ [...] unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these lines are there in the first place, which is why I didn't want to simply go in and remove them -- but it may be that wrapping those lines in something like if ((csum_flags CSUM_TSO) == 0) would solve the problem without breaking anything else. I think the reason for this checksum calculation is that pf sometimes modifies the packet, so it also updates the checksum. Aha, this is exactly the sort of thing I was worried about. I'm glad you understand this stuff better than me. It doesn't work on Xen TSO interfaces because (I assume) it expects to get the pseudo header checksum, not the full checksum. It's not entirely clear to my why it's not broken on my hardware (which claims TSO support), but perhaps Xen is more picky than actual hardware. I'm not 100% certain about this, but I don't think Xen is doing anything with the checksum; rather, everything is being passed through to the underlying hardware, and some NICs are pickier than others. I'll refrain from commenting on your plans for fixing this since, as I mentioned above, you understand how pf and the network stack work far better than I do. :-) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
On 19 Jul 2015, at 12:14, Kristof Provost wrote: On 19 Jul 2015, at 17:58, George Neville-Neil g...@freebsd.org wrote: Since Kristof is already working on this, I'll let him address it. If necessary open a PR on this specifically please. It’s on my todo list, but I’d like to take another run at PR 172648 first. Works for me. Thanks, George ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
On 19 Jul 2015, at 17:58, George Neville-Neil g...@freebsd.org wrote: Since Kristof is already working on this, I'll let him address it. If necessary open a PR on this specifically please. It’s on my todo list, but I’d like to take another run at PR 172648 first. Regards, Kristof ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
Also i have noticed, when TSO is disabled IPv4 TCP Performance also drops from ~10 Gb/s to ~3 Gb/s, the same as with IPv6. I have read threads from around 2010 about the networking stack not having support for TSO on IPv6 Packets. May this be the problem in this case? On 14 Jul 2015, at 23:44, Sydney Meyer syd.me...@gmail.com wrote: On 14 Jul 2015, at 18:15, Mark Felder f...@freebsd.org wrote: On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: Hello everybody, i have noticed some odd behaviour with networking under Xen with FreeBSD 10 as a DomU. - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. (measured with iperf) What is the before and after here? When is FreeBSD successfully doing 10Gbit/s and when isn't it? Is pf enabled? Are you scrubbing? With two clean 10.1 AMD64 DomU installations both with a single, pinned cpu, without pf enabled the TCP performance between the two hosts, measured with iperf, differs between ~10 Gb/s on IPv4 and ~3 Gb/s on IPv6. With pf enabled and scrub in all the difference is almost the same. - Dropped/Stalled Connections with TCP Segmentation Offload and pf enabled. TSO is a known issue. I've been turning it off for years to get FreeBSD to play nice on Xen. This one i am still investigating, because it happens only in certain situations (which are not clear to me, atm), but the host seems to drop ACK Packets in some situations like when connected to via IPSEC or via double NAT. This happens only when pf it actually enabled. Disabling TSO on the xn-interface seems to help. - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 Mbit/s (iperf). Are you saying FreeBSD non-IPSEC kernel can do 10Gbit/s TCP performance, but IPSEC kernel immediately drops it to 200Mbit/s? As for the apparent performance drop with IPSEC enabled Kernels without security associations installed, i am unable to reproduce this now, not on 10.0 or 10.1 nor 10 STABLE. Only when actually _using_ IPSec the performance drops from ~10Gb/s to around ~200Mb/s whether actually encrypting esp traffic or not. This clearly must have been a mistake on my side, although i could have sworn that i checked this two times before asking on the forums and the -net mailing list a few weeks ago. Well then, i am sincerely sorry about this one. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Networking under Xen
Hello everybody, i have noticed some odd behaviour with networking under Xen with FreeBSD 10 as a DomU. - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. (measured with iperf) - Dropped/Stalled Connections with TCP Segmentation Offload and pf enabled. - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 Mbit/s (iperf). I have observed this behaviour under Freebsd 10, 10.1 and 10 Stable. The Hypervisor is running Xen 4.4 with a Linux 3.16 Dom0. Any Suggestions? ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: Hello everybody, i have noticed some odd behaviour with networking under Xen with FreeBSD 10 as a DomU. - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. (measured with iperf) What is the before and after here? When is FreeBSD successfully doing 10Gbit/s and when isn't it? Is pf enabled? Are you scrubbing? - Dropped/Stalled Connections with TCP Segmentation Offload and pf enabled. TSO is a known issue. I've been turning it off for years to get FreeBSD to play nice on Xen. - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 Mbit/s (iperf). Are you saying FreeBSD non-IPSEC kernel can do 10Gbit/s TCP performance, but IPSEC kernel immediately drops it to 200Mbit/s? ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: Networking under Xen
On 07/14/15 09:15, Mark Felder wrote: On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: - Dropped/Stalled Connections with TCP Segmentation Offload and pf enabled. TSO is a known issue. I've been turning it off for years to get FreeBSD to play nice on Xen. More precisely *pf* is a known issue. It's inserting TCP checksums into TSO packets. I reported this a long time ago and I don't know why pf still hasn't been fixed. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
