Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On Fri, Aug 17, 2018 at 4:19 PM Shannon Nelson wrote: > > > On 8/16/2018 2:36 PM, Shannon Nelson wrote: > > On 8/16/2018 2:15 PM, Alexander Duyck wrote: > >> On Tue, Aug 14, 2018 at 10:10 AM Shannon Nelson > >> wrote: > >>> > >>> On 8/14/2018 8:30 AM, Alexander Duyck wrote: > On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson > wrote: > > > > This set of patches implements IPsec hardware offload for VF > > devices in > > Intel's 10Gbe x540 family of Ethernet devices. > >>> > >>> [...] > >>> > > So the one question I would have about this patch set is what happens > if you are setting up a ipsec connection between the PF and one of the > VFs on the same port/function? Do the ipsec offloads get translated > across the Tx loopback or do they end up causing issues? Specifically > I would be interested in seeing the results of a test either between > two VFs, or the PF and one of the VFs on the same port. > > - Alex > > >>> > >>> There is definitely something funky in the internal switch connection, > >>> as messages going from PF to VF with an offloaded encryption don't seem > >>> to get received by the VF, at least when in a VEB setup. If I only set > >>> up offloads on the Rx on both PF and VF, and don't offload the Tx, then > >>> things work. > >>> > >>> I don't have a setup to test this, but I suspect that in a VEPA > >>> configuration, with packets going out to the switch and turned around > >>> back in, the Tx encryption offload would happen as expected. > >>> > >>> sln > >> > >> We should probably look at adding at least one patch to the set then > >> that disables IPsec Tx offload if SR-IOV is enabled with VEB so that > >> we don't end up breaking connections should a VF be migrated from a > >> remote system to a local one that it is connected to. > >> > >> - Alex > >> > > > > The problem with this is that someone could set up an IPsec connection > > on the PF for Tx and Rx use, then set num_vfs, start some VFs, and we > > still can end up in the same place. I don't think we want to disallow > > all Tx IPsec offload. > > > > Maybe we can catch it in ixgbe_ipsec_offload_ok()? If it can find that > > the dest mac is on the internal switch, perhaps it can NAK the Tx > > offload? That would force the XFRM xmit code to do a regular SW encrypt > > before sending the packet. I'll look into this. > > > > sln > > This would be a great idea, but the xdo_state_offload_ok() callback > happens in the network stack before routing has happened, so there is no > mac address yet in the skb. We may be stuck with NAKing *all* Tx > offloads when num_vfs != 0. It works, and it is better than no offload > at all, but it sure harshes the vibe. Blech. > > sln You can probably just think of the Tx offload as being lumped in with all the other offloads that don't work when SR-IOV is enabled such as ATR and RSC. - Alex
Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On 8/16/2018 2:36 PM, Shannon Nelson wrote: On 8/16/2018 2:15 PM, Alexander Duyck wrote: On Tue, Aug 14, 2018 at 10:10 AM Shannon Nelson wrote: On 8/14/2018 8:30 AM, Alexander Duyck wrote: On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson wrote: This set of patches implements IPsec hardware offload for VF devices in Intel's 10Gbe x540 family of Ethernet devices. [...] So the one question I would have about this patch set is what happens if you are setting up a ipsec connection between the PF and one of the VFs on the same port/function? Do the ipsec offloads get translated across the Tx loopback or do they end up causing issues? Specifically I would be interested in seeing the results of a test either between two VFs, or the PF and one of the VFs on the same port. - Alex There is definitely something funky in the internal switch connection, as messages going from PF to VF with an offloaded encryption don't seem to get received by the VF, at least when in a VEB setup. If I only set up offloads on the Rx on both PF and VF, and don't offload the Tx, then things work. I don't have a setup to test this, but I suspect that in a VEPA configuration, with packets going out to the switch and turned around back in, the Tx encryption offload would happen as expected. sln We should probably look at adding at least one patch to the set then that disables IPsec Tx offload if SR-IOV is enabled with VEB so that we don't end up breaking connections should a VF be migrated from a remote system to a local one that it is connected to. - Alex The problem with this is that someone could set up an IPsec connection on the PF for Tx and Rx use, then set num_vfs, start some VFs, and we still can end up in the same place. I don't think we want to disallow all Tx IPsec offload. Maybe we can catch it in ixgbe_ipsec_offload_ok()? If it can find that the dest mac is on the internal switch, perhaps it can NAK the Tx offload? That would force the XFRM xmit code to do a regular SW encrypt before sending the packet. I'll look into this. sln This would be a great idea, but the xdo_state_offload_ok() callback happens in the network stack before routing has happened, so there is no mac address yet in the skb. We may be stuck with NAKing *all* Tx offloads when num_vfs != 0. It works, and it is better than no offload at all, but it sure harshes the vibe. Blech. sln
Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On 8/16/2018 2:15 PM, Alexander Duyck wrote: On Tue, Aug 14, 2018 at 10:10 AM Shannon Nelson wrote: On 8/14/2018 8:30 AM, Alexander Duyck wrote: On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson wrote: This set of patches implements IPsec hardware offload for VF devices in Intel's 10Gbe x540 family of Ethernet devices. [...] So the one question I would have about this patch set is what happens if you are setting up a ipsec connection between the PF and one of the VFs on the same port/function? Do the ipsec offloads get translated across the Tx loopback or do they end up causing issues? Specifically I would be interested in seeing the results of a test either between two VFs, or the PF and one of the VFs on the same port. - Alex There is definitely something funky in the internal switch connection, as messages going from PF to VF with an offloaded encryption don't seem to get received by the VF, at least when in a VEB setup. If I only set up offloads on the Rx on both PF and VF, and don't offload the Tx, then things work. I don't have a setup to test this, but I suspect that in a VEPA configuration, with packets going out to the switch and turned around back in, the Tx encryption offload would happen as expected. sln We should probably look at adding at least one patch to the set then that disables IPsec Tx offload if SR-IOV is enabled with VEB so that we don't end up breaking connections should a VF be migrated from a remote system to a local one that it is connected to. - Alex The problem with this is that someone could set up an IPsec connection on the PF for Tx and Rx use, then set num_vfs, start some VFs, and we still can end up in the same place. I don't think we want to disallow all Tx IPsec offload. Maybe we can catch it in ixgbe_ipsec_offload_ok()? If it can find that the dest mac is on the internal switch, perhaps it can NAK the Tx offload? That would force the XFRM xmit code to do a regular SW encrypt before sending the packet. I'll look into this. sln
Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On Tue, Aug 14, 2018 at 10:10 AM Shannon Nelson wrote: > > On 8/14/2018 8:30 AM, Alexander Duyck wrote: > > On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson > > wrote: > >> > >> This set of patches implements IPsec hardware offload for VF devices in > >> Intel's 10Gbe x540 family of Ethernet devices. > > [...] > > > > > So the one question I would have about this patch set is what happens > > if you are setting up a ipsec connection between the PF and one of the > > VFs on the same port/function? Do the ipsec offloads get translated > > across the Tx loopback or do they end up causing issues? Specifically > > I would be interested in seeing the results of a test either between > > two VFs, or the PF and one of the VFs on the same port. > > > > - Alex > > > > There is definitely something funky in the internal switch connection, > as messages going from PF to VF with an offloaded encryption don't seem > to get received by the VF, at least when in a VEB setup. If I only set > up offloads on the Rx on both PF and VF, and don't offload the Tx, then > things work. > > I don't have a setup to test this, but I suspect that in a VEPA > configuration, with packets going out to the switch and turned around > back in, the Tx encryption offload would happen as expected. > > sln We should probably look at adding at least one patch to the set then that disables IPsec Tx offload if SR-IOV is enabled with VEB so that we don't end up breaking connections should a VF be migrated from a remote system to a local one that it is connected to. - Alex
Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On 8/14/2018 8:30 AM, Alexander Duyck wrote: On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson wrote: This set of patches implements IPsec hardware offload for VF devices in Intel's 10Gbe x540 family of Ethernet devices. [...] So the one question I would have about this patch set is what happens if you are setting up a ipsec connection between the PF and one of the VFs on the same port/function? Do the ipsec offloads get translated across the Tx loopback or do they end up causing issues? Specifically I would be interested in seeing the results of a test either between two VFs, or the PF and one of the VFs on the same port. - Alex There is definitely something funky in the internal switch connection, as messages going from PF to VF with an offloaded encryption don't seem to get received by the VF, at least when in a VEB setup. If I only set up offloads on the Rx on both PF and VF, and don't offload the Tx, then things work. I don't have a setup to test this, but I suspect that in a VEPA configuration, with packets going out to the switch and turned around back in, the Tx encryption offload would happen as expected. sln
Re: [Intel-wired-lan] [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
On Mon, Aug 13, 2018 at 11:43 AM Shannon Nelson wrote: > > This set of patches implements IPsec hardware offload for VF devices in > Intel's 10Gbe x540 family of Ethernet devices. > > The IPsec HW offload feature has been in the x540/Niantic family of > network devices since their release in 2009, but there was no Linux > kernel support for the offload until 2017. After the XFRM code added > support for the offload last year, the hw offload was added to the ixgbe > PF driver. > > Since the related x540 VF device uses same setup as the PF for implementing > the offload, adding the feature to the ixgbevf seemed like a good idea. > In this case, the PF owns the device registers, so the VF simply packages > up the request information into a VF<->PF message and the PF does the > device configuration. The resulting IPsec throughput is roughly equivalent > to what we see in the PF - nearly line-rate, with the expected drop in CPU > cycles burned. (I'm not great at performance statistics, I'll let better > folks do the actual measurements as they pertain to their own usage) > > To make use of the capability, first two things are needed: the PF must > be told to enable the offload for VFs (it is off by default) and the VF > must be trusted. A new ethtool priv-flag for ixgbe is added to control > VF offload support. For example: > > ethtool --set-priv-flags eth0 vf-ipsec on > ip link set eth0 vf 1 trust on > > Once those are set up and the VF device is UP, the user can add SAs the > same as for PFs, whether the VF is in the host or has been assigned to > a VM. > > Note that the x540 chip supports a total of 1024 Rx plus 1024 Tx Security > Associations (SAs), shared among the PF and VFs that might request them. > It is entirely possible for a single VF to soak up all the offload > capability, which would likely annoy some people. It seems rather > arbitrary to try to set a limit for how many a VF could be allowed, > but this is mitigated somewhat by the need for "trust" and "vf-ipsec" > to be enabled. I suppose we could come up with a way to make a limit > configurable, but there is no existing method for adding that kind > configuration so I'll leave that to a future discussion. > > Currently this doesn't support Tx offload as the hardware encryption > engine doesn't seem to engage on the Tx packets. This may be a lingering > driver bug, more investigation is needed. Until then, requests for a Tx > offload are failed and the userland requester will need to add Tx SAs > without the offload attribute. > > Given that we don't have Tx offload support, the benefit here is less > than it could be, but is definitely still noticeable. For example, with > informal iperf testing over a 10Gbps link, with full offload in a PF on > one side and a VF in a VM on the other side on a CPU with AES instructions: > > Reference: > No IPsec: 9.4 Gbps > IPsec offload btwn two PFs: 9.2 Gbps > VF as the iperf receiver: > IPsec offload on PF, none on VF: 6.8 Gbps > IPsec offload on PF and VF: 9.2 Gbps << biggest benefit > VF as the iperf sender: > IPsec offload on PF, none on VF: 4.8 Gbps > IPsec offload on PF and VF: 4.8 Gbps > > The iperf traffic is primarily uni-directional, and we can see the most > benefit when VF is the iperf server and is receiving the test traffic. > Watching output from sar also shows a nice decrease in CPU utilization. > So the one question I would have about this patch set is what happens if you are setting up a ipsec connection between the PF and one of the VFs on the same port/function? Do the ipsec offloads get translated across the Tx loopback or do they end up causing issues? Specifically I would be interested in seeing the results of a test either between two VFs, or the PF and one of the VFs on the same port. - Alex
[PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs
This set of patches implements IPsec hardware offload for VF devices in Intel's 10Gbe x540 family of Ethernet devices. The IPsec HW offload feature has been in the x540/Niantic family of network devices since their release in 2009, but there was no Linux kernel support for the offload until 2017. After the XFRM code added support for the offload last year, the hw offload was added to the ixgbe PF driver. Since the related x540 VF device uses same setup as the PF for implementing the offload, adding the feature to the ixgbevf seemed like a good idea. In this case, the PF owns the device registers, so the VF simply packages up the request information into a VF<->PF message and the PF does the device configuration. The resulting IPsec throughput is roughly equivalent to what we see in the PF - nearly line-rate, with the expected drop in CPU cycles burned. (I'm not great at performance statistics, I'll let better folks do the actual measurements as they pertain to their own usage) To make use of the capability, first two things are needed: the PF must be told to enable the offload for VFs (it is off by default) and the VF must be trusted. A new ethtool priv-flag for ixgbe is added to control VF offload support. For example: ethtool --set-priv-flags eth0 vf-ipsec on ip link set eth0 vf 1 trust on Once those are set up and the VF device is UP, the user can add SAs the same as for PFs, whether the VF is in the host or has been assigned to a VM. Note that the x540 chip supports a total of 1024 Rx plus 1024 Tx Security Associations (SAs), shared among the PF and VFs that might request them. It is entirely possible for a single VF to soak up all the offload capability, which would likely annoy some people. It seems rather arbitrary to try to set a limit for how many a VF could be allowed, but this is mitigated somewhat by the need for "trust" and "vf-ipsec" to be enabled. I suppose we could come up with a way to make a limit configurable, but there is no existing method for adding that kind configuration so I'll leave that to a future discussion. Currently this doesn't support Tx offload as the hardware encryption engine doesn't seem to engage on the Tx packets. This may be a lingering driver bug, more investigation is needed. Until then, requests for a Tx offload are failed and the userland requester will need to add Tx SAs without the offload attribute. Given that we don't have Tx offload support, the benefit here is less than it could be, but is definitely still noticeable. For example, with informal iperf testing over a 10Gbps link, with full offload in a PF on one side and a VF in a VM on the other side on a CPU with AES instructions: Reference: No IPsec: 9.4 Gbps IPsec offload btwn two PFs: 9.2 Gbps VF as the iperf receiver: IPsec offload on PF, none on VF: 6.8 Gbps IPsec offload on PF and VF: 9.2 Gbps << biggest benefit VF as the iperf sender: IPsec offload on PF, none on VF: 4.8 Gbps IPsec offload on PF and VF: 4.8 Gbps The iperf traffic is primarily uni-directional, and we can see the most benefit when VF is the iperf server and is receiving the test traffic. Watching output from sar also shows a nice decrease in CPU utilization. Shannon Nelson (8): ixgbe: reload ipsec ip table after sa tables ixgbe: prep ipsec constants for later use ixgbe: add VF ipsec management ixgbe: add VF IPsec offload enable flag ixgbe: add VF IPsec offload request message handling ixgbevf: add defines for IPsec offload request ixgbevf: add VF ipsec offload code ixgbevf: enable VF ipsec offload operations drivers/net/ethernet/intel/ixgbe/ixgbe.h | 20 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 9 + drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c| 275 - drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.h| 13 + drivers/net/ethernet/intel/ixgbe/ixgbe_mbx.h | 5 + drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c| 17 +- drivers/net/ethernet/intel/ixgbevf/Makefile | 1 + drivers/net/ethernet/intel/ixgbevf/defines.h | 10 +- drivers/net/ethernet/intel/ixgbevf/ethtool.c | 2 + drivers/net/ethernet/intel/ixgbevf/ipsec.c| 673 ++ drivers/net/ethernet/intel/ixgbevf/ipsec.h| 66 +++ drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 33 ++ drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 74 ++- drivers/net/ethernet/intel/ixgbevf/mbx.h | 5 + drivers/net/ethernet/intel/ixgbevf/vf.c | 4 + 15 files changed, 1163 insertions(+), 44 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbevf/ipsec.c create mode 100644 drivers/net/ethernet/intel/ixgbevf/ipsec.h -- 2.7.4
