> On 29.09.2020., at 19:29, Joshua Moore <j...@jcm.me> wrote: > > The driver here is performance. I like the idea of assigning CPUs dedicated > to the DPDK namespace and avoiding kernel interrupts. My understanding was > that IOMMU was needed for this but I may be completely off with that thinking.
no, IOMMU is not needed for that. it wil lalso work in no-iommu mode with slightly lower performance (as software needs to do translation between PA and VA). > > It doesn’t seem in retrospect of this discussion that IOMMU offers any > performance increase and that I should simply stick with SR-IOV + > enable_unsafe_noiommu_mode. There is very minimal performance difference between using and not using iommu. > > Should I still expect 100% CPU with IOMMU disabled? CPU utilisation doesn't have anything with IOMMU. If interrupts are disabled only way for software to know that packet is received is to constantly poll. And polling means 100% cpu. > Will I still be able to allocate CPU cores to namespaces without IOMMU? yes. > > > Thanks for working with me on this. > > > —Josh > >> On Sep 29, 2020, at 9:35 AM, Yichen Wang (yicwang) <yicw...@cisco.com >> <mailto:yicw...@cisco.com>> wrote: >> >> >> I am not sure if it worth to try to add “iommu=pt intel_iommu=on” in the >> GRUB? We normally do it in both Linux host/guest, but in the case of ESXi >> not sure if that will help with the vfio-pci IOMMU issue… >> >> Regards, >> Yichen >> >> From: <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> on behalf of >> "Damjan Marion via lists.fd.io <http://lists.fd.io/>" >> <dmarion=me....@lists.fd.io <mailto:dmarion=me....@lists.fd.io>> >> Reply-To: "dmar...@me.com <mailto:dmar...@me.com>" <dmar...@me.com >> <mailto:dmar...@me.com>> >> Date: Tuesday, September 29, 2020 at 7:48 AM >> To: Joshua Moore <j...@jcm.me <mailto:j...@jcm.me>> >> Cc: "Benoit Ganne (bganne)" <bga...@cisco.com <mailto:bga...@cisco.com>>, >> "vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>" <vpp-dev@lists.fd.io >> <mailto:vpp-dev@lists.fd.io>> >> Subject: Re: [vpp-dev] VPP on ESXI with i40evf (SR-IOV Passthrough) Driver >> >> Joshua, >> >> What is your motivation for using IOMMU inside VM? >> It is typically used by hypervisor to protect from misbehaving VMs. >> So unless you want to do nested virtualisation, IOMMU inside your VM doesn’t >> bring lot of value. >> Can you simply try to turn off iommu in the VM and turn >> “enable_unsafe_noiommu_mode”. >> >> Another possible problem with AVF may be version of ESXi PF driver. AVF >> communicates >> with PF over the virtua lchannel and PF driver needs to support it. In >> linux, that works well with recent kernels >> but I’m not sure what is the state of ESXi i40e PF driver…. >> >> — >> Damjan >> >> On 29.09.2020., at 14:04, Joshua Moore <j...@jcm.me <mailto:j...@jcm.me>> >> wrote: >> >> Ben, >> >> Of course there's a PCI switch. My main purpose of leveraging SR-IOV with VF >> allocation is to allow the internal eswitch on the Intel NIC to handle >> switching in hardware instead of the vswitch on the ESXI hypervisor. I don't >> really care so much about isolation of the PCI devices nor the risk of bad >> firmware on the NIC. I will control/trust all of the VMs with access to the >> VFs as well as the device attached to the PF. >> >> So just to confirm, I need to expect 100% CPU utilization with VPP/DPDK + >> IOMMU? If so, what's the best way to monitor CPU-related performance impact >> if I always see 100%? Also I want to confirm that enable_unsafe_noiommu_mode >> still enables the performance benefits of SR-IOV and the only tradeoff is >> the aforementioned isolation/security concern? >> >> >> Thanks for your help, >> >> --Josh >> >> On Tue, Sep 29, 2020 at 5:23 AM Benoit Ganne (bganne) <bga...@cisco.com >> <mailto:bga...@cisco.com>> wrote: >> Hi Joshua, >> >> Glad it solves the vfio issue. Looking at the dmesg output, I suspect the >> issue is that the PCIe topology advertised is not fully supported by vfio + >> IOMMU: it looks like your VF is behind a PCIe switch, so the CPU PCIe IOMMU >> root-complex port cannot guarantee full isolation: all devices behind the >> PCIe switch can talk peer-to-peer directly w/o going through the CPU PCIe >> root-complex port. >> As the CPU IOMMU cannot fully isolate your device, vfio refuses to bind >> unless you allow for unsafe IOMMU config - rightfully, as it seems to be >> your case. >> Anyway, it still means you should benefit from the IOMMU to prevent the >> device to read/write everywhere in host memory. You might not however >> prevent a malign firmware running on the NIC to harm other devices behind >> the same PCIe switch. >> >> Regarding the VM crash, note that VPP is polling the interfaces so it will >> always uses 100% CPU. >> Does the VM also crashes if you run stress-test the CPU eg. >> ~# stress-ng --matrix 0 -t 1m >> >> Best >> ben >> >> > -----Original Message----- >> > From: Joshua Moore <j...@jcm.me <mailto:j...@jcm.me>> >> > Sent: mardi 29 septembre 2020 12:07 >> > To: Benoit Ganne (bganne) <bga...@cisco.com <mailto:bga...@cisco.com>> >> > Cc: Damjan Marion <dmar...@me.com <mailto:dmar...@me.com>>; >> > vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> >> > Subject: Re: [vpp-dev] VPP on ESXI with i40evf (SR-IOV Passthrough) Driver >> > >> > Hello Ben, >> > >> > echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode >> > sudo dpdk-devbind --bind=vfio-pci 0000:13:00.0 >> > >> > >> > The above commands successfully resulted in vfio-pci driver binding to the >> > NIC. However, as soon as I assigned the NIC to VPP and restarted the >> > service, my VM CPU shot up and the VM crashes. >> > >> > >> > Regarding IOMMU I do have it enabled in the host's BIOS, ESXI "Expose >> > IOMMU to the guest OS" option, and I have set the GRUB_CMDLINE_LINUX per >> > the below wiki: >> > >> > https://wiki.fd.io/view/VPP/How_To_Optimize_Performance_(System_Tuning) >> > <https://wiki.fd.io/view/VPP/How_To_Optimize_Performance_(System_Tuning)> >> > >> > root@test:~# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX >> > GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity" >> > GRUB_CMDLINE_LINUX="intel_iommu=on isolcpus=1-7 nohz_full=1-7 >> > hugepagesz=1GB hugepages=16 default_hugepagesz=1GB" >> > >> > Full dmesg output can be found at: http://jcm.me/dmesg.txt >> > <http://jcm.me/dmesg.txt> >> >> >> > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17591): https://lists.fd.io/g/vpp-dev/message/17591 Mute This Topic: https://lists.fd.io/mt/77164974/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
