Re: Networking under Xen
On 07/26/15 06:16, Kristof Provost wrote: > On 2015-07-15 00:54:25 (-0700), Colin Percival wrote: >> In my tests, deleting these lines from pf_ioctl.c >> 3570 /* We need a proper CSUM befor we start (s. OpenBSD ip_output) */ >> [...] >> unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these lines >> are there in the first place, which is why I didn't want to simply go in >> and remove them -- but it may be that wrapping those lines in something >> like "if ((csum_flags & CSUM_TSO) == 0)" would solve the problem without >> breaking anything else. > > I think the reason for this checksum calculation is that pf sometimes > modifies the packet, so it also updates the checksum. Aha, this is exactly the sort of thing I was worried about. I'm glad you understand this stuff better than me. > It doesn't work on Xen TSO interfaces because (I assume) it expects to > get the pseudo header checksum, not the full checksum. > It's not entirely clear to my why it's not broken on my hardware (which > claims TSO support), but perhaps Xen is more picky than actual hardware. I'm not 100% certain about this, but I don't think Xen is doing anything with the checksum; rather, everything is being passed through to the underlying hardware, and some NICs are pickier than others. I'll refrain from commenting on your plans for fixing this since, as I mentioned above, you understand how pf and the network stack work far better than I do. :-) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On 2015-07-15 00:54:25 (-0700), Colin Percival wrote:
> In my tests, deleting these lines from pf_ioctl.c
>
> 3570 /* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
> 3571 if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) {
> 3572 in_delayed_cksum(*m);
> 3573 (*m)->m_pkthdr.csum_flags &= ~CSUM_DELAY_DATA;
> 3574 }
>
> unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these lines
> are there in the first place, which is why I didn't want to simply go in
> and remove them -- but it may be that wrapping those lines in something
> like "if ((csum_flags & CSUM_TSO) == 0)" would solve the problem without
> breaking anything else.
>
I think the reason for this checksum calculation is that pf sometimes
modifies the packet, so it also updates the checksum.
In case of TSO (or CSUM_TCP) that doesn't actually work because the TCP
checksum is the pseudo IP(6) header checksum, not the final checksum.
Starting from a full checksum the update is correct however, so that's
why pf works on non-TSO interfaces.
It doesn't work on Xen TSO interfaces because (I assume) it expects to
get the pseudo header checksum, not the full checksum.
It's not entirely clear to my why it's not broken on my hardware (which
claims TSO support), but perhaps Xen is more picky than actual hardware.
Adding the if (CSUM_TSO) check will improve matters, but it will still
break on packets which get changed by pf (rdr/nat/...).
I think the real solution is to make pf_cksum_fixup() a bit more
intelligent. It should look at m->m_pkthdr.csum_flags to determine if it
should touch the checksum or not.
If CSUM_TCP is set we know we've got a pseudo header checksum and we
shouldn't touch it unless we've modified the source or destination IP
address (or length or protocol, but that's less likely).
If CSUM_TCP is not set we should have a full checksum and should do the
fixup as before.
Actually, perhaps CSUM_DELAY_DATA is better than CSUM_TCP, because we'll
have to do the same for UDP.
Regards,
Kristof
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
Also i have noticed, when TSO is disabled IPv4 TCP Performance also drops from ~10 Gb/s to ~3 Gb/s, the same as with IPv6. I have read threads from around 2010 about the networking stack not having support for TSO on IPv6 Packets. May this be the problem in this case? > On 14 Jul 2015, at 23:44, Sydney Meyer wrote: > > >> On 14 Jul 2015, at 18:15, Mark Felder wrote: >> >> >> >> On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: >>> Hello everybody, >>> >>> i have noticed some odd behaviour with networking under Xen with FreeBSD >>> 10 as a DomU. >>> >>> - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. >>> (measured with iperf) >>> >> >> What is the "before" and "after" here? When is FreeBSD successfully >> doing 10Gbit/s and when isn't it? Is pf enabled? Are you scrubbing? > > With two clean 10.1 AMD64 DomU installations both with a single, pinned cpu, > without pf enabled the TCP performance between the two hosts, measured with > iperf, differs between ~10 Gb/s on IPv4 and ~3 Gb/s on IPv6. With pf enabled > and "scrub in all" the difference is almost the same. >> >>> - Dropped/Stalled Connections with TCP Segmentation Offload and pf >>> enabled. >>> >> >> TSO is a known issue. I've been turning it off for years to get FreeBSD >> to play nice on Xen. > > This one i am still investigating, because it happens only in "certain" > situations (which are not clear to me, atm), but the host seems to drop ACK > Packets in some situations like when connected to via IPSEC or via double > NAT. This happens only when pf it actually enabled. Disabling TSO on the > xn-interface seems to help. >> >>> - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 >>> Mbit/s (iperf). >>> >> >> Are you saying FreeBSD non-IPSEC kernel can do 10Gbit/s TCP performance, >> but IPSEC kernel immediately drops it to 200Mbit/s? > > As for the apparent performance drop with IPSEC enabled Kernels without > security associations installed, i am unable to reproduce this now, not on > 10.0 or 10.1 nor 10 STABLE. Only when actually _using_ IPSec the performance > drops from ~10Gb/s to around ~200Mb/s whether actually encrypting esp traffic > or not. > This clearly must have been a mistake on my side, although i could have sworn > that i checked this two times before asking on the forums and the -net > mailing list a few weeks ago. Well then, i am sincerely sorry about this one. >> ___ >> freebsd-xen@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-xen >> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org" > ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On 19 Jul 2015, at 12:14, Kristof Provost wrote: On 19 Jul 2015, at 17:58, George Neville-Neil wrote: Since Kristof is already working on this, I'll let him address it. If necessary open a PR on this specifically please. It’s on my todo list, but I’d like to take another run at PR 172648 first. Works for me. Thanks, George ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
> On 19 Jul 2015, at 17:58, George Neville-Neil wrote: > Since Kristof is already working on this, I'll let him address it. If > necessary > open a PR on this specifically please. It’s on my todo list, but I’d like to take another run at PR 172648 first. Regards, Kristof ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On 15 Jul 2015, at 3:54, Colin Percival wrote:
On 07/15/15 00:44, Kristof Provost wrote:
On 14 Jul 2015, at 21:23, Mark Felder wrote:
On Tue, Jul 14, 2015, at 13:54, Colin Percival wrote:
More precisely *pf* is a known issue. It's inserting TCP checksums
into TSO packets. I reported this a long time ago and I don't know
why pf still hasn't been fixed.
I think Kristof may have solved this in HEAD with all of the care
and
feeding he has been doing to pf, but I haven't verified.
I wouldn’t expect so. As far as I know I’ve not fixed anything
TSO or
checksum related.
In my tests, deleting these lines from pf_ioctl.c
3570/* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
3571if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) {
3572in_delayed_cksum(*m);
3573(*m)->m_pkthdr.csum_flags &= ~CSUM_DELAY_DATA;
3574}
unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these
lines
are there in the first place, which is why I didn't want to simply go
in
and remove them -- but it may be that wrapping those lines in
something
like "if ((csum_flags & CSUM_TSO) == 0)" would solve the problem
without
breaking anything else.
Since Kristof is already working on this, I'll let him address it. If
necessary
open a PR on this specifically please.
Best,
George
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On 07/15/15 00:44, Kristof Provost wrote:
>> On 14 Jul 2015, at 21:23, Mark Felder wrote:
>> On Tue, Jul 14, 2015, at 13:54, Colin Percival wrote:
>>> More precisely *pf* is a known issue. It's inserting TCP checksums
>>> into TSO packets. I reported this a long time ago and I don't know
>>> why pf still hasn't been fixed.
>>
>> I think Kristof may have solved this in HEAD with all of the care and
>> feeding he has been doing to pf, but I haven't verified.
>
> I wouldn’t expect so. As far as I know I’ve not fixed anything TSO or
> checksum related.
In my tests, deleting these lines from pf_ioctl.c
3570/* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
3571if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) {
3572in_delayed_cksum(*m);
3573(*m)->m_pkthdr.csum_flags &= ~CSUM_DELAY_DATA;
3574}
unbreaks pf+TSO on EC2 instances. I'm not entirely sure why these lines
are there in the first place, which is why I didn't want to simply go in
and remove them -- but it may be that wrapping those lines in something
like "if ((csum_flags & CSUM_TSO) == 0)" would solve the problem without
breaking anything else.
--
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
> On 14 Jul 2015, at 21:23, Mark Felder wrote: > On Tue, Jul 14, 2015, at 13:54, Colin Percival wrote: >> On 07/14/15 09:15, Mark Felder wrote: >> More precisely *pf* is a known issue. It's inserting TCP checksums >> into TSO packets. I reported this a long time ago and I don't know >> why pf still hasn't been fixed. >> > > I think Kristof may have solved this in HEAD with all of the care and > feeding he has been doing to pf, but I haven't verified. > I wouldn’t expect so. As far as I know I’ve not fixed anything TSO or checksum related. In fact, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=172648 is at the top of my TODO list right now. Regards, Kristof ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
> On 14 Jul 2015, at 18:15, Mark Felder wrote: > > > > On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: >> Hello everybody, >> >> i have noticed some odd behaviour with networking under Xen with FreeBSD >> 10 as a DomU. >> >> - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. >> (measured with iperf) >> > > What is the "before" and "after" here? When is FreeBSD successfully > doing 10Gbit/s and when isn't it? Is pf enabled? Are you scrubbing? With two clean 10.1 AMD64 DomU installations both with a single, pinned cpu, without pf enabled the TCP performance between the two hosts, measured with iperf, differs between ~10 Gb/s on IPv4 and ~3 Gb/s on IPv6. With pf enabled and "scrub in all" the difference is almost the same. > >> - Dropped/Stalled Connections with TCP Segmentation Offload and pf >> enabled. >> > > TSO is a known issue. I've been turning it off for years to get FreeBSD > to play nice on Xen. This one i am still investigating, because it happens only in "certain" situations (which are not clear to me, atm), but the host seems to drop ACK Packets in some situations like when connected to via IPSEC or via double NAT. This happens only when pf it actually enabled. Disabling TSO on the xn-interface seems to help. > >> - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 >> Mbit/s (iperf). >> > > Are you saying FreeBSD non-IPSEC kernel can do 10Gbit/s TCP performance, > but IPSEC kernel immediately drops it to 200Mbit/s? As for the apparent performance drop with IPSEC enabled Kernels without security associations installed, i am unable to reproduce this now, not on 10.0 or 10.1 nor 10 STABLE. Only when actually _using_ IPSec the performance drops from ~10Gb/s to around ~200Mb/s whether actually encrypting esp traffic or not. This clearly must have been a mistake on my side, although i could have sworn that i checked this two times before asking on the forums and the -net mailing list a few weeks ago. Well then, i am sincerely sorry about this one. > ___ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org" ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On Tue, Jul 14, 2015, at 13:54, Colin Percival wrote: > On 07/14/15 09:15, Mark Felder wrote: > > On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: > >> - Dropped/Stalled Connections with TCP Segmentation Offload and pf > >> enabled. > > > > TSO is a known issue. I've been turning it off for years to get FreeBSD > > to play nice on Xen. > > More precisely *pf* is a known issue. It's inserting TCP checksums > into TSO packets. I reported this a long time ago and I don't know > why pf still hasn't been fixed. > I think Kristof may have solved this in HEAD with all of the care and feeding he has been doing to pf, but I haven't verified. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On 07/14/15 09:15, Mark Felder wrote: > On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: >> - Dropped/Stalled Connections with TCP Segmentation Offload and pf >> enabled. > > TSO is a known issue. I've been turning it off for years to get FreeBSD > to play nice on Xen. More precisely *pf* is a known issue. It's inserting TCP checksums into TSO packets. I reported this a long time ago and I don't know why pf still hasn't been fixed. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: Networking under Xen
On Tue, Jul 14, 2015, at 07:36, Sydney Meyer wrote: > Hello everybody, > > i have noticed some odd behaviour with networking under Xen with FreeBSD > 10 as a DomU. > > - IPv6 (TCP) bandwith drops from ~10 Gbit/s IPv4 to around 3 Gbit/s IPv6. > (measured with iperf) > What is the "before" and "after" here? When is FreeBSD successfully doing 10Gbit/s and when isn't it? Is pf enabled? Are you scrubbing? > - Dropped/Stalled Connections with TCP Segmentation Offload and pf > enabled. > TSO is a known issue. I've been turning it off for years to get FreeBSD to play nice on Xen. > - IPSEC-enabled Kernel TCP Performance drops from ~10 Gbit/s to ~200 > Mbit/s (iperf). > Are you saying FreeBSD non-IPSEC kernel can do 10Gbit/s TCP performance, but IPSEC kernel immediately drops it to 200Mbit/s? ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
