Re: help
Sure, I will submit a patch (it would be a first). Thanks On Fri, Aug 11, 2023 at 3:30 PM Thomas Monjalon wrote: > Thanks for the info. > Do you think it should be written in the vmxnet3 page of the DPDK > documentation? > If yes, would you like to initiate a patch for review? > > > 11/08/2023 10:43, Igor de Paula: > > Hi again, > > I got this resolved with VMWARE support so I thought to share it here. > > What I originally wanted was to use IOVA-VA on an AMD host. Which didn't > > work. I have learned that the ESXI version that supports > > virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI > 7.0 > > U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already > > as far as I know. > > > > > > On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin > > wrote: > > > > > [AMD Official Use Only - General] > > > > > > Like I said earlier, trying with the Intel host I have on VMWARE, > > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > > > > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could > > > it be possible the specific version EXSI hypervisor supports the HW > IOMMU > > > specific to the platform. My suspicion will be in case on AMD platform > > > changes for required to enable HW iommu might not be available to > specific > > > EXSI (hypervior OS) used. > > > > > > > > > > > > I am not an expert on virtio_user PMD, but I can check if it will work > > > with PA with deferred setting for vmx_net3 PMD are ok? > > > > > > > > > > > > *From:* Igor de Paula > > > *Sent:* Tuesday, July 25, 2023 8:42 PM > > > *To:* Varghese, Vipin > > > *Cc:* Yigit, Ferruh ; Jochen Behrens < > > > jbehr...@vmware.com>; Thomas Monjalon ; > > > users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil < > > > nikhil.agar...@amd.com>; Ronak Doshi ; Immanni, > Venkat > > > ; Chenbo Xia > > > *Subject:* Re: help > > > > > > > > > > > > *Caution:* This message originated from an External Source. Use proper > > > caution when opening attachments, clicking links, or responding. > > > > > > > > > > > > Well, > > > Like I said earlier, trying with the Intel host I have on VMWARE, > > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > > Meaning, PA works regardless if IOMMU is enabled or not. From my > > > experience anyway. > > > That's why I thought that: > > > virtio_user needs VA to work. > > > For some reason VMXNET3 does not work with VA (only on AMD host). > > > > > > > > > > > > > > > > > > On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin < > vipin.vargh...@amd.com> > > > wrote: > > > > > > [AMD Official Use Only - General] > > > > > > > > > > > > Thanks Igor, > > > > > > > > > > > > As suspected the vmx_net3 works with > > > > > > > > > > > >1. Iommu: disabled > > >2. enable_unsafe_iommu: enabled > > >3. dpdk eal iova mode: PA > > > > > > > > > > > > as pointed by you in logs, the virtio_user fails as it expects VA too. > > > > > > > > > > > > Will check and get back. > > > > > > > > > > > > *From:* Igor de Paula > > > *Sent:* Tuesday, July 25, 2023 8:16 PM > > > *To:* Yigit, Ferruh > > > *Cc:* Jochen Behrens ; Thomas Monjalon < > > > tho...@monjalon.net>; users@dpdk.org; Gupta, Nipun < > nipun.gu...@amd.com>; > > > Agarwal, Nikhil ; Ronak Doshi < > dos...@vmware.com>; > > > Immanni, Venkat ; Varghese, Vipin < > > > vipin.vargh...@amd.com>; Chenbo Xia > > > *Subject:* Re: help > > > > > > > > > > > > *Caution:* This message originated from an External Source. Use proper > > > caution when opening attachments, clicking links, or responding. > > > > > > > > > > > > Hi, > > > Attaching the logs of EAL when trying to run a configuration with > > > virtio_user port when IOMMU is > > > disabled and enable_unsafe_iommu is enabled. As you can see it forces > IOVA > > > as PA but t
Re: help
Thanks for the info. Do you think it should be written in the vmxnet3 page of the DPDK documentation? If yes, would you like to initiate a patch for review? 11/08/2023 10:43, Igor de Paula: > Hi again, > I got this resolved with VMWARE support so I thought to share it here. > What I originally wanted was to use IOVA-VA on an AMD host. Which didn't > work. I have learned that the ESXI version that supports > virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7.0 > U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already > as far as I know. > > > On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin > wrote: > > > [AMD Official Use Only - General] > > > > Like I said earlier, trying with the Intel host I have on VMWARE, > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could > > it be possible the specific version EXSI hypervisor supports the HW IOMMU > > specific to the platform. My suspicion will be in case on AMD platform > > changes for required to enable HW iommu might not be available to specific > > EXSI (hypervior OS) used. > > > > > > > > I am not an expert on virtio_user PMD, but I can check if it will work > > with PA with deferred setting for vmx_net3 PMD are ok? > > > > > > > > *From:* Igor de Paula > > *Sent:* Tuesday, July 25, 2023 8:42 PM > > *To:* Varghese, Vipin > > *Cc:* Yigit, Ferruh ; Jochen Behrens < > > jbehr...@vmware.com>; Thomas Monjalon ; > > users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil < > > nikhil.agar...@amd.com>; Ronak Doshi ; Immanni, Venkat > > ; Chenbo Xia > > *Subject:* Re: help > > > > > > > > *Caution:* This message originated from an External Source. Use proper > > caution when opening attachments, clicking links, or responding. > > > > > > > > Well, > > Like I said earlier, trying with the Intel host I have on VMWARE, > > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > Meaning, PA works regardless if IOMMU is enabled or not. From my > > experience anyway. > > That's why I thought that: > > virtio_user needs VA to work. > > For some reason VMXNET3 does not work with VA (only on AMD host). > > > > > > > > > > > > On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin > > wrote: > > > > [AMD Official Use Only - General] > > > > > > > > Thanks Igor, > > > > > > > > As suspected the vmx_net3 works with > > > > > > > >1. Iommu: disabled > >2. enable_unsafe_iommu: enabled > >3. dpdk eal iova mode: PA > > > > > > > > as pointed by you in logs, the virtio_user fails as it expects VA too. > > > > > > > > Will check and get back. > > > > > > > > *From:* Igor de Paula > > *Sent:* Tuesday, July 25, 2023 8:16 PM > > *To:* Yigit, Ferruh > > *Cc:* Jochen Behrens ; Thomas Monjalon < > > tho...@monjalon.net>; users@dpdk.org; Gupta, Nipun ; > > Agarwal, Nikhil ; Ronak Doshi ; > > Immanni, Venkat ; Varghese, Vipin < > > vipin.vargh...@amd.com>; Chenbo Xia > > *Subject:* Re: help > > > > > > > > *Caution:* This message originated from an External Source. Use proper > > caution when opening attachments, clicking links, or responding. > > > > > > > > Hi, > > Attaching the logs of EAL when trying to run a configuration with > > virtio_user port when IOMMU is > > disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA > > as PA but the viritui_user needs IOVA as VA. > > I am also attaching the output of dmesg. I am not sure which kernel logs > > you wanted... if there is anything else please let me know.. > > Regarding the ESXI logs, they are HUGE so I will send to you on a separate > > email. > > > > > > > > On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit wrote: > > > > On 7/21/2023 12:39 PM, Igor de Paula wrote: > > > I am trying to use virtio_user for an interface with the > > > kernel: > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html < > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > > > I think this requires IOVA as va. > > > > > > > I am not sure if virtio-user has IOVA as VA
Re: help
Hi again, I got this resolved with VMWARE support so I thought to share it here. What I originally wanted was to use IOVA-VA on an AMD host. Which didn't work. I have learned that the ESXI version that supports virtual IOMMU in AMD hosts (which is a prerequisite to IOVA-VA) is ESXI 7.0 U1. After updating it worked. On Intel hosts ESXI 6.7 supports it already as far as I know. On Tue, Jul 25, 2023 at 6:19 PM Varghese, Vipin wrote: > [AMD Official Use Only - General] > > Like I said earlier, trying with the Intel host I have on VMWARE, > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > With IOMMU enabled, VMXNET3 works with VA as well as PA. > > [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could > it be possible the specific version EXSI hypervisor supports the HW IOMMU > specific to the platform. My suspicion will be in case on AMD platform > changes for required to enable HW iommu might not be available to specific > EXSI (hypervior OS) used. > > > > I am not an expert on virtio_user PMD, but I can check if it will work > with PA with deferred setting for vmx_net3 PMD are ok? > > > > *From:* Igor de Paula > *Sent:* Tuesday, July 25, 2023 8:42 PM > *To:* Varghese, Vipin > *Cc:* Yigit, Ferruh ; Jochen Behrens < > jbehr...@vmware.com>; Thomas Monjalon ; > users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil < > nikhil.agar...@amd.com>; Ronak Doshi ; Immanni, Venkat > ; Chenbo Xia > *Subject:* Re: help > > > > *Caution:* This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > > Well, > Like I said earlier, trying with the Intel host I have on VMWARE, > specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > With IOMMU enabled, VMXNET3 works with VA as well as PA. > Meaning, PA works regardless if IOMMU is enabled or not. From my > experience anyway. > That's why I thought that: > virtio_user needs VA to work. > For some reason VMXNET3 does not work with VA (only on AMD host). > > > > > > On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin > wrote: > > [AMD Official Use Only - General] > > > > Thanks Igor, > > > > As suspected the vmx_net3 works with > > > >1. Iommu: disabled >2. enable_unsafe_iommu: enabled >3. dpdk eal iova mode: PA > > > > as pointed by you in logs, the virtio_user fails as it expects VA too. > > > > Will check and get back. > > > > *From:* Igor de Paula > *Sent:* Tuesday, July 25, 2023 8:16 PM > *To:* Yigit, Ferruh > *Cc:* Jochen Behrens ; Thomas Monjalon < > tho...@monjalon.net>; users@dpdk.org; Gupta, Nipun ; > Agarwal, Nikhil ; Ronak Doshi ; > Immanni, Venkat ; Varghese, Vipin < > vipin.vargh...@amd.com>; Chenbo Xia > *Subject:* Re: help > > > > *Caution:* This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > > Hi, > Attaching the logs of EAL when trying to run a configuration with > virtio_user port when IOMMU is > disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA > as PA but the viritui_user needs IOVA as VA. > I am also attaching the output of dmesg. I am not sure which kernel logs > you wanted... if there is anything else please let me know.. > Regarding the ESXI logs, they are HUGE so I will send to you on a separate > email. > > > > On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit wrote: > > On 7/21/2023 12:39 PM, Igor de Paula wrote: > > I am trying to use virtio_user for an interface with the > > kernel: > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > > I think this requires IOVA as va. > > > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, > he may know better. > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and > '--iova-mode=pa'? > > > > It does work with Intel host and IOMMU > > enabled. Part of the negotiation when setting it up is getting the IOMMU > > number so I thought it has to have IOMMU. > > > > Yes, issue looks like related to the IOMMU, and it may be either related > to HW support, or ESXi iommu driver support, we will check using below > information you provided. > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > > again, that didn't work. > > ESXI version - VMware ESXi, 7.0.0, 16324942 > > AMD: AMD EPYC 7452 32-Core Processo > > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4
Re: Help Running Example
I'm going to get a dedicated NIC so that I can continue on the other machine, but in the meantime I discovered that one of my old laptops places the ethernet NIC in a unique IOMMU group: 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetLink BCM57780 Gigabit Ethernet PCIe (rev 01) 1 [1.726704] pci :00:01.0: Adding to iommu group 1 1 [1.726892] pci :01:00.0: Adding to iommu group 1 2 [1.726720] pci :00:02.0: Adding to iommu group 2 3 [1.726734] pci :00:14.0: Adding to iommu group 3 4 [1.726751] pci :00:16.0: Adding to iommu group 4 5 [1.726764] pci :00:1a.0: Adding to iommu group 5 6 [1.726778] pci :00:1b.0: Adding to iommu group 6 7 [1.726792] pci :00:1c.0: Adding to iommu group 7 8 [1.726805] pci :00:1c.1: Adding to iommu group 8 9 [1.726818] pci :00:1c.2: Adding to iommu group 9 10 [1.726833] pci :00:1d.0: Adding to iommu group 10 11 [1.726859] pci :00:1f.0: Adding to iommu group 11 11 [1.726872] pci :00:1f.2: Adding to iommu group 11 11 [1.726886] pci :00:1f.3: Adding to iommu group 11 12 [1.726905] pci :02:00.0: Adding to iommu group 12 13 [1.726920] pci :03:00.0: Adding to iommu group 13 14 [1.726933] pci :04:00.0: Adding to iommu group 14 After following the same steps on this machine, and commenting out the port counting as suggested, I have a new error: EAL: Detected CPU lcores: 4 EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized TELEMETRY: No legacy callbacks, legacy socket not created EAL: Error - exiting with code: 1 Cause: Cannot create mbuf pool Based on my research, it looks like this can be caused by a lack of hugepages. Here is an excerpt from /proc/meminfo: HugePages_Total:1024 HugePages_Free: 1024 HugePages_Rsvd:0 HugePages_Surp:0 Which seems fine to me. Any idea what might be causing this problem? Thanks again. On Mon, Aug 7, 2023 at 11:25 PM Fuji Nafiul wrote: > > In the case of the " number of ports must be even " error, it's a simple one. > if you look into the code, you will see that it is checking for the number of > ports and it is giving an error if the port number is not even as the > skeleton app mainly receives packet on 1 port and forward them to a different > port (as far as I remember). So, for the sake of simplicity, if you have odd > numbers of ports or just a single port, don't worry, just comment out the > port number checking and you can even comment out the tx part ( > rte_eth_tx_burst() ) and rather free all received packets. Then run and > monitor the packet counting which will give you a kick start and later you > can try replying to arp requests to expose IP by which you will be able to > establish udp/tcp connection.. I suggest try udp first (I also only worked > with UDP in dpdk) and you will get a lot of help about this from "bond" app > in the official examples. > > On Sun, Aug 6, 2023 at 9:34 PM Alan Beadle wrote: >> >> Hi, >> >> I need some help getting DPDK working. I am running Ubuntu 20.04 with >> a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu >> 5.15 kernel with the same results. >> >> Here is my NIC info from lspci: >> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) >> I219-LM (rev 09) >> >> I built and installed DPDK from source, and applied the following boot >> flags: "intel_iommu=on iommu=pt" >> >> After booting I did the following as root: >> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages >> ifconfig enp0s31f6 down >> dpdk-devbind.py --bind=vfio-pci :00:1f.6 >> >> All of this appeared to work. >> >> I tried running the "skeleton" example program and got the following output: >> sudo ./build/basicfwd >> EAL: Detected CPU lcores: 16 >> EAL: Detected NUMA nodes: 1 >> EAL: Detected shared linkage of DPDK >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >> EAL: Selected IOVA mode 'VA' >> EAL: VFIO support initialized >> EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU >> group bound to VFIO or unbound >> EAL: Requested device :00:1f.6 cannot be used >> TELEMETRY: No legacy callbacks, legacy socket not created >> EAL: Error - exiting with code: 1 >> Cause: Error: number of ports must be even >> >> I'm not at all familiar with DPDK or VFIO. What might the problem be? >> >> -Alan
Re: Help Running Example
On Tue, 8 Aug 2023 11:31:52 -0400
Alan Beadle wrote:
> Here is how I checked what other devices are in the same group as the NIC:
>
> I ran this command as root:
> dmesg|egrep group|awk '{print $NF" "$0}'|sort -n
>
> Here is an excerpt of the output showing the group that the NIC is in:
>
> 10 [ 17.029705] pci :00:1f.0: Adding to iommu group 10
> 10 [ 17.029732] pci :00:1f.2: Adding to iommu group 10
> 10 [ 17.029761] pci :00:1f.3: Adding to iommu group 10
> 10 [ 17.029788] pci :00:1f.4: Adding to iommu group 10
> 10 [ 17.029815] pci :00:1f.5: Adding to iommu group 10
> 10 [ 17.029842] pci :00:1f.6: Adding to iommu group 10
>
>
> And here is an excerpt of the lspci output showing what each of those
> devices is:
>
> 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI
> Controller (rev 09)
> 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset
> Family Power Management Controller (rev 09)
> 00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09)
> 00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
> 00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series
> Chipset Family SPI Controller (rev 09)
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
>
> Based on this grouping, it seems like I can't feasibly unbind all of
> these, unless I misunderstand something.
>
> -Alan
>
> On Tue, Aug 8, 2023 at 11:25 AM Alan Beadle wrote:
> >
> > Thanks Stephen. It looks like my memory controller is in the same
> > IOMMU group. I assume this means I won't be able to do this with this
> > NIC?
> >
> > -Alan
> >
> > On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
> > wrote:
> > >
> > > On Mon, 7 Aug 2023 12:40:21 -0700
> > > Stephen Hemminger wrote:
> > >
> > > > On Sun, 6 Aug 2023 11:33:43 -0400
> > > > Alan Beadle wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > > > 5.15 kernel with the same results.
> > > > >
> > > > > Here is my NIC info from lspci:
> > > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > > > I219-LM (rev 09)
> > > > >
> > > > > I built and installed DPDK from source, and applied the following boot
> > > > > flags: "intel_iommu=on iommu=pt"
> > > > >
> > > > > After booting I did the following as root:
> > > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > > > ifconfig enp0s31f6 down
> > > > > dpdk-devbind.py --bind=vfio-pci :00:1f.6
> > > > >
> > > > > All of this appeared to work.
> > > > >
> > > > > I tried running the "skeleton" example program and got the following
> > > > > output:
> > > > > sudo ./build/basicfwd
> > > > > EAL: Detected CPU lcores: 16
> > > > > EAL: Detected NUMA nodes: 1
> > > > > EAL: Detected shared linkage of DPDK
> > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > EAL: Selected IOVA mode 'VA'
> > > > > EAL: VFIO support initialized
> > > > > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > > > group bound to VFIO or unbound
> > > > > EAL: Requested device :00:1f.6 cannot be used
> > > > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > > > EAL: Error - exiting with code: 1
> > > > > Cause: Error: number of ports must be even
> > > > >
> > > > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > > > >
> > > > > -Alan
> > > >
> > > > IOMMU groups are when multiple PCI devices share the same channel
> > > > in the IOMMU. The group is used to determine what mapping to use when
> > > > device does DMA. Since this is a security thing, devices in same IOMMU
> > > > group can not be shared between kernel and non-kernel usage.
> > > >
> > > > The IOMMU group is determined by wiring on the motherboard.
> > > > Usually it is things like multiple Ethernet ports sharing the same
> > > > group.
> > > > But can be much more confused.
> > > >
> > > > The only option is to unbind all devices in the group before using
> > > > one with DPDK.
> > >
> > > More info on IOMMU groups is in kernel documentation:
> > > https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> > >
> > > and in this article
> > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive
> > >
Right you need to find a different system, use a VM or add an external NIC to
use DPDK.
Re: Help Running Example
Here is how I checked what other devices are in the same group as the NIC:
I ran this command as root:
dmesg|egrep group|awk '{print $NF" "$0}'|sort -n
Here is an excerpt of the output showing the group that the NIC is in:
10 [ 17.029705] pci :00:1f.0: Adding to iommu group 10
10 [ 17.029732] pci :00:1f.2: Adding to iommu group 10
10 [ 17.029761] pci :00:1f.3: Adding to iommu group 10
10 [ 17.029788] pci :00:1f.4: Adding to iommu group 10
10 [ 17.029815] pci :00:1f.5: Adding to iommu group 10
10 [ 17.029842] pci :00:1f.6: Adding to iommu group 10
And here is an excerpt of the lspci output showing what each of those
devices is:
00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI
Controller (rev 09)
00:1f.2 Memory controller: Intel Corporation C620 Series Chipset
Family Power Management Controller (rev 09)
00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09)
00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series
Chipset Family SPI Controller (rev 09)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
I219-LM (rev 09)
Based on this grouping, it seems like I can't feasibly unbind all of
these, unless I misunderstand something.
-Alan
On Tue, Aug 8, 2023 at 11:25 AM Alan Beadle wrote:
>
> Thanks Stephen. It looks like my memory controller is in the same
> IOMMU group. I assume this means I won't be able to do this with this
> NIC?
>
> -Alan
>
> On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
> wrote:
> >
> > On Mon, 7 Aug 2023 12:40:21 -0700
> > Stephen Hemminger wrote:
> >
> > > On Sun, 6 Aug 2023 11:33:43 -0400
> > > Alan Beadle wrote:
> > >
> > > > Hi,
> > > >
> > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > > 5.15 kernel with the same results.
> > > >
> > > > Here is my NIC info from lspci:
> > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > > I219-LM (rev 09)
> > > >
> > > > I built and installed DPDK from source, and applied the following boot
> > > > flags: "intel_iommu=on iommu=pt"
> > > >
> > > > After booting I did the following as root:
> > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > > ifconfig enp0s31f6 down
> > > > dpdk-devbind.py --bind=vfio-pci :00:1f.6
> > > >
> > > > All of this appeared to work.
> > > >
> > > > I tried running the "skeleton" example program and got the following
> > > > output:
> > > > sudo ./build/basicfwd
> > > > EAL: Detected CPU lcores: 16
> > > > EAL: Detected NUMA nodes: 1
> > > > EAL: Detected shared linkage of DPDK
> > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > EAL: Selected IOVA mode 'VA'
> > > > EAL: VFIO support initialized
> > > > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > > group bound to VFIO or unbound
> > > > EAL: Requested device :00:1f.6 cannot be used
> > > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > > EAL: Error - exiting with code: 1
> > > > Cause: Error: number of ports must be even
> > > >
> > > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > > >
> > > > -Alan
> > >
> > > IOMMU groups are when multiple PCI devices share the same channel
> > > in the IOMMU. The group is used to determine what mapping to use when
> > > device does DMA. Since this is a security thing, devices in same IOMMU
> > > group can not be shared between kernel and non-kernel usage.
> > >
> > > The IOMMU group is determined by wiring on the motherboard.
> > > Usually it is things like multiple Ethernet ports sharing the same group.
> > > But can be much more confused.
> > >
> > > The only option is to unbind all devices in the group before using
> > > one with DPDK.
> >
> > More info on IOMMU groups is in kernel documentation:
> > https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> >
> > and in this article
> > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive
Re: Help Running Example
Thanks Stephen. It looks like my memory controller is in the same IOMMU group. I assume this means I won't be able to do this with this NIC? -Alan On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger wrote: > > On Mon, 7 Aug 2023 12:40:21 -0700 > Stephen Hemminger wrote: > > > On Sun, 6 Aug 2023 11:33:43 -0400 > > Alan Beadle wrote: > > > > > Hi, > > > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu > > > 5.15 kernel with the same results. > > > > > > Here is my NIC info from lspci: > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) > > > I219-LM (rev 09) > > > > > > I built and installed DPDK from source, and applied the following boot > > > flags: "intel_iommu=on iommu=pt" > > > > > > After booting I did the following as root: > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > > ifconfig enp0s31f6 down > > > dpdk-devbind.py --bind=vfio-pci :00:1f.6 > > > > > > All of this appeared to work. > > > > > > I tried running the "skeleton" example program and got the following > > > output: > > > sudo ./build/basicfwd > > > EAL: Detected CPU lcores: 16 > > > EAL: Detected NUMA nodes: 1 > > > EAL: Detected shared linkage of DPDK > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > EAL: Selected IOVA mode 'VA' > > > EAL: VFIO support initialized > > > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU > > > group bound to VFIO or unbound > > > EAL: Requested device :00:1f.6 cannot be used > > > TELEMETRY: No legacy callbacks, legacy socket not created > > > EAL: Error - exiting with code: 1 > > > Cause: Error: number of ports must be even > > > > > > I'm not at all familiar with DPDK or VFIO. What might the problem be? > > > > > > -Alan > > > > IOMMU groups are when multiple PCI devices share the same channel > > in the IOMMU. The group is used to determine what mapping to use when > > device does DMA. Since this is a security thing, devices in same IOMMU > > group can not be shared between kernel and non-kernel usage. > > > > The IOMMU group is determined by wiring on the motherboard. > > Usually it is things like multiple Ethernet ports sharing the same group. > > But can be much more confused. > > > > The only option is to unbind all devices in the group before using > > one with DPDK. > > More info on IOMMU groups is in kernel documentation: > https://www.kernel.org/doc/html/latest/driver-api/vfio.html > > and in this article > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive
Re: Help Running Example
In the case of the " number of ports must be even " error, it's a simple one. if you look into the code, you will see that it is checking for the number of ports and it is giving an error if the port number is not even as the skeleton app mainly receives packet on 1 port and forward them to a different port (as far as I remember). So, for the sake of simplicity, if you have odd numbers of ports or just a single port, don't worry, just comment out the port number checking and you can even comment out the tx part ( rte_eth_tx_burst() ) and rather free all received packets. Then run and monitor the packet counting which will give you a kick start and later you can try replying to arp requests to expose IP by which you will be able to establish udp/tcp connection.. I suggest try udp first (I also only worked with UDP in dpdk) and you will get a lot of help about this from "bond" app in the official examples. On Sun, Aug 6, 2023 at 9:34 PM Alan Beadle wrote: > Hi, > > I need some help getting DPDK working. I am running Ubuntu 20.04 with > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu > 5.15 kernel with the same results. > > Here is my NIC info from lspci: > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) > I219-LM (rev 09) > > I built and installed DPDK from source, and applied the following boot > flags: "intel_iommu=on iommu=pt" > > After booting I did the following as root: > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > ifconfig enp0s31f6 down > dpdk-devbind.py --bind=vfio-pci :00:1f.6 > > All of this appeared to work. > > I tried running the "skeleton" example program and got the following > output: > sudo ./build/basicfwd > EAL: Detected CPU lcores: 16 > EAL: Detected NUMA nodes: 1 > EAL: Detected shared linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: VFIO support initialized > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU > group bound to VFIO or unbound > EAL: Requested device :00:1f.6 cannot be used > TELEMETRY: No legacy callbacks, legacy socket not created > EAL: Error - exiting with code: 1 > Cause: Error: number of ports must be even > > I'm not at all familiar with DPDK or VFIO. What might the problem be? > > -Alan >
Re: Help Running Example
On Mon, 7 Aug 2023 12:40:21 -0700 Stephen Hemminger wrote: > On Sun, 6 Aug 2023 11:33:43 -0400 > Alan Beadle wrote: > > > Hi, > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu > > 5.15 kernel with the same results. > > > > Here is my NIC info from lspci: > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) > > I219-LM (rev 09) > > > > I built and installed DPDK from source, and applied the following boot > > flags: "intel_iommu=on iommu=pt" > > > > After booting I did the following as root: > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > ifconfig enp0s31f6 down > > dpdk-devbind.py --bind=vfio-pci :00:1f.6 > > > > All of this appeared to work. > > > > I tried running the "skeleton" example program and got the following output: > > sudo ./build/basicfwd > > EAL: Detected CPU lcores: 16 > > EAL: Detected NUMA nodes: 1 > > EAL: Detected shared linkage of DPDK > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > EAL: Selected IOVA mode 'VA' > > EAL: VFIO support initialized > > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU > > group bound to VFIO or unbound > > EAL: Requested device :00:1f.6 cannot be used > > TELEMETRY: No legacy callbacks, legacy socket not created > > EAL: Error - exiting with code: 1 > > Cause: Error: number of ports must be even > > > > I'm not at all familiar with DPDK or VFIO. What might the problem be? > > > > -Alan > > IOMMU groups are when multiple PCI devices share the same channel > in the IOMMU. The group is used to determine what mapping to use when > device does DMA. Since this is a security thing, devices in same IOMMU > group can not be shared between kernel and non-kernel usage. > > The IOMMU group is determined by wiring on the motherboard. > Usually it is things like multiple Ethernet ports sharing the same group. > But can be much more confused. > > The only option is to unbind all devices in the group before using > one with DPDK. More info on IOMMU groups is in kernel documentation: https://www.kernel.org/doc/html/latest/driver-api/vfio.html and in this article https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive
Re: Help Running Example
On Sun, 6 Aug 2023 11:33:43 -0400 Alan Beadle wrote: > Hi, > > I need some help getting DPDK working. I am running Ubuntu 20.04 with > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu > 5.15 kernel with the same results. > > Here is my NIC info from lspci: > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) > I219-LM (rev 09) > > I built and installed DPDK from source, and applied the following boot > flags: "intel_iommu=on iommu=pt" > > After booting I did the following as root: > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > ifconfig enp0s31f6 down > dpdk-devbind.py --bind=vfio-pci :00:1f.6 > > All of this appeared to work. > > I tried running the "skeleton" example program and got the following output: > sudo ./build/basicfwd > EAL: Detected CPU lcores: 16 > EAL: Detected NUMA nodes: 1 > EAL: Detected shared linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: VFIO support initialized > EAL: :00:1f.6 VFIO group is not viable! Not all devices in IOMMU > group bound to VFIO or unbound > EAL: Requested device :00:1f.6 cannot be used > TELEMETRY: No legacy callbacks, legacy socket not created > EAL: Error - exiting with code: 1 > Cause: Error: number of ports must be even > > I'm not at all familiar with DPDK or VFIO. What might the problem be? > > -Alan IOMMU groups are when multiple PCI devices share the same channel in the IOMMU. The group is used to determine what mapping to use when device does DMA. Since this is a security thing, devices in same IOMMU group can not be shared between kernel and non-kernel usage. The IOMMU group is determined by wiring on the motherboard. Usually it is things like multiple Ethernet ports sharing the same group. But can be much more confused. The only option is to unbind all devices in the group before using one with DPDK.
RE: help
[AMD Official Use Only - General] Like I said earlier, trying with the Intel host I have on VMWARE, specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz With IOMMU enabled, VMXNET3 works with VA as well as PA. [VV] since ` enable_unsafe_iommu: not enabled` on Intel platform, could it be possible the specific version EXSI hypervisor supports the HW IOMMU specific to the platform. My suspicion will be in case on AMD platform changes for required to enable HW iommu might not be available to specific EXSI (hypervior OS) used. I am not an expert on virtio_user PMD, but I can check if it will work with PA with deferred setting for vmx_net3 PMD are ok? From: Igor de Paula Sent: Tuesday, July 25, 2023 8:42 PM To: Varghese, Vipin Cc: Yigit, Ferruh ; Jochen Behrens ; Thomas Monjalon ; users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil ; Ronak Doshi ; Immanni, Venkat ; Chenbo Xia Subject: Re: help Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. Well, Like I said earlier, trying with the Intel host I have on VMWARE, specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz With IOMMU enabled, VMXNET3 works with VA as well as PA. Meaning, PA works regardless if IOMMU is enabled or not. From my experience anyway. That's why I thought that: virtio_user needs VA to work. For some reason VMXNET3 does not work with VA (only on AMD host). On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin mailto:vipin.vargh...@amd.com>> wrote: [AMD Official Use Only - General] Thanks Igor, As suspected the vmx_net3 works with 1. Iommu: disabled 2. enable_unsafe_iommu: enabled 3. dpdk eal iova mode: PA as pointed by you in logs, the virtio_user fails as it expects VA too. Will check and get back. From: Igor de Paula mailto:igord...@gmail.com>> Sent: Tuesday, July 25, 2023 8:16 PM To: Yigit, Ferruh mailto:ferruh.yi...@amd.com>> Cc: Jochen Behrens mailto:jbehr...@vmware.com>>; Thomas Monjalon mailto:tho...@monjalon.net>>; users@dpdk.org<mailto:users@dpdk.org>; Gupta, Nipun mailto:nipun.gu...@amd.com>>; Agarwal, Nikhil mailto:nikhil.agar...@amd.com>>; Ronak Doshi mailto:dos...@vmware.com>>; Immanni, Venkat mailto:venkat.imma...@amd.com>>; Varghese, Vipin mailto:vipin.vargh...@amd.com>>; Chenbo Xia mailto:chenbo@intel.com>> Subject: Re: help Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. Hi, Attaching the logs of EAL when trying to run a configuration with virtio_user port when IOMMU is disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA as PA but the viritui_user needs IOVA as VA. I am also attaching the output of dmesg. I am not sure which kernel logs you wanted... if there is anything else please let me know.. Regarding the ESXI logs, they are HUGE so I will send to you on a separate email. On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit mailto:ferruh.yi...@amd.com>> wrote: On 7/21/2023 12:39 PM, Igor de Paula wrote: > I am trying to use virtio_user for an interface with the > kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html > <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > I think this requires IOVA as va. > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, he may know better. Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=pa'? > It does work with Intel host and IOMMU > enabled. Part of the negotiation when setting it up is getting the IOMMU > number so I thought it has to have IOMMU. > Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below information you provided. > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work. > ESXI version - VMware ESXi, 7.0.0, 16324942 > AMD: AMD EPYC 7452 32-Core Processo > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > Regarding the logs I will try and attach it soon. > Thanks for info, waiting for logs. > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit > mailto:ferruh.yi...@amd.com> > <mailto:ferruh.yi...@amd.com<mailto:ferruh.yi...@amd.com>>> wrote: > > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, >
Re: help
Well, Like I said earlier, trying with the Intel host I have on VMWARE, specifically Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz With IOMMU enabled, VMXNET3 works with VA as well as PA. Meaning, PA works regardless if IOMMU is enabled or not. From my experience anyway. That's why I thought that: virtio_user needs VA to work. For some reason VMXNET3 does not work with VA (only on AMD host). On Tue, Jul 25, 2023 at 4:04 PM Varghese, Vipin wrote: > [AMD Official Use Only - General] > > Thanks Igor, > > > > As suspected the vmx_net3 works with > > > >1. Iommu: disabled >2. enable_unsafe_iommu: enabled >3. dpdk eal iova mode: PA > > > > as pointed by you in logs, the virtio_user fails as it expects VA too. > > > > Will check and get back. > > > > *From:* Igor de Paula > *Sent:* Tuesday, July 25, 2023 8:16 PM > *To:* Yigit, Ferruh > *Cc:* Jochen Behrens ; Thomas Monjalon < > tho...@monjalon.net>; users@dpdk.org; Gupta, Nipun ; > Agarwal, Nikhil ; Ronak Doshi ; > Immanni, Venkat ; Varghese, Vipin < > vipin.vargh...@amd.com>; Chenbo Xia > *Subject:* Re: help > > > > *Caution:* This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > > Hi, > Attaching the logs of EAL when trying to run a configuration with > virtio_user port when IOMMU is > disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA > as PA but the viritui_user needs IOVA as VA. > I am also attaching the output of dmesg. I am not sure which kernel logs > you wanted... if there is anything else please let me know.. > Regarding the ESXI logs, they are HUGE so I will send to you on a separate > email. > > > > On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit wrote: > > On 7/21/2023 12:39 PM, Igor de Paula wrote: > > I am trying to use virtio_user for an interface with the > > kernel: > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > > I think this requires IOVA as va. > > > > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, > he may know better. > > Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and > '--iova-mode=pa'? > > > > It does work with Intel host and IOMMU > > enabled. Part of the negotiation when setting it up is getting the IOMMU > > number so I thought it has to have IOMMU. > > > > Yes, issue looks like related to the IOMMU, and it may be either related > to HW support, or ESXi iommu driver support, we will check using below > information you provided. > > > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > > again, that didn't work. > > ESXI version - VMware ESXi, 7.0.0, 16324942 > > AMD: AMD EPYC 7452 32-Core Processo > > > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > > Regarding the logs I will try and attach it soon. > > > > Thanks for info, waiting for logs. > > > > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit > <mailto:ferruh.yi...@amd.com>> wrote: > > > > Hi Igor, > > > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > > please try with: > > - enable 'enable_unsafe_noiommu_mode' flag > > - Force PA mode via '--iova-mode=pa' eal parameter > > > > > > Also to be able to figure out AMD IOMMU support level, can you please > > provide: > > - AMD part number > > - ESXi/hypervisor version > > - ESXi & VM kernel logs > > > > > > Thanks, > > Ferruh > > > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > > +Ronak from the ESX team > > > > > > > > > > > > > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > > > grub. (Or intel_iommu=off for Intel processors.) > > > > > > > > > > > > Jochen > > > > > > > > > > > > *From: *Thomas Monjalon > <mailto:tho...@monjalon.net>> > > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > > *To: *Igor de Paula mailto:igord...@gmail.com > >> > > > *Cc: *users@dpdk.org <mailto:users@dpdk.org> > <mailto:users@dpdk.org>>, Jochen Behrens > > > mailto:jbehr...@vmware.com>>, Nipun Gupta > > mailto:nipun.gu...@a
RE: help
[AMD Official Use Only - General] Thanks Igor, As suspected the vmx_net3 works with 1. Iommu: disabled 2. enable_unsafe_iommu: enabled 3. dpdk eal iova mode: PA as pointed by you in logs, the virtio_user fails as it expects VA too. Will check and get back. From: Igor de Paula Sent: Tuesday, July 25, 2023 8:16 PM To: Yigit, Ferruh Cc: Jochen Behrens ; Thomas Monjalon ; users@dpdk.org; Gupta, Nipun ; Agarwal, Nikhil ; Ronak Doshi ; Immanni, Venkat ; Varghese, Vipin ; Chenbo Xia Subject: Re: help Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. Hi, Attaching the logs of EAL when trying to run a configuration with virtio_user port when IOMMU is disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA as PA but the viritui_user needs IOVA as VA. I am also attaching the output of dmesg. I am not sure which kernel logs you wanted... if there is anything else please let me know.. Regarding the ESXI logs, they are HUGE so I will send to you on a separate email. On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit mailto:ferruh.yi...@amd.com>> wrote: On 7/21/2023 12:39 PM, Igor de Paula wrote: > I am trying to use virtio_user for an interface with the > kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html > <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > I think this requires IOVA as va. > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, he may know better. Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=pa'? > It does work with Intel host and IOMMU > enabled. Part of the negotiation when setting it up is getting the IOMMU > number so I thought it has to have IOMMU. > Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below information you provided. > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work. > ESXI version - VMware ESXi, 7.0.0, 16324942 > AMD: AMD EPYC 7452 32-Core Processo > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > Regarding the logs I will try and attach it soon. > Thanks for info, waiting for logs. > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit > mailto:ferruh.yi...@amd.com> > <mailto:ferruh.yi...@amd.com<mailto:ferruh.yi...@amd.com>>> wrote: > > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > > > > > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > > grub. (Or intel_iommu=off for Intel processors.) > > > > > > > > Jochen > > > > > > > > *From: *Thomas Monjalon mailto:tho...@monjalon.net> > <mailto:tho...@monjalon.net<mailto:tho...@monjalon.net>>> > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula mailto:igord...@gmail.com> > <mailto:igord...@gmail.com<mailto:igord...@gmail.com>>> > > *Cc: *users@dpdk.org<mailto:users@dpdk.org> > <mailto:users@dpdk.org<mailto:users@dpdk.org>> > mailto:users@dpdk.org> > <mailto:users@dpdk.org<mailto:users@dpdk.org>>>, Jochen Behrens > > mailto:jbehr...@vmware.com> > <mailto:jbehr...@vmware.com<mailto:jbehr...@vmware.com>>>, Nipun Gupta > mailto:nipun.gu...@amd.com> > <mailto:nipun.gu...@amd.com<mailto:nipun.gu...@amd.com>>>, Nikhil Agarwal > > mailto:nikhil.agar...@amd.com> > <mailto:nikhil.agar...@amd.com<mailto:nikhil.agar...@amd.com>>>, Ferruh > Yigit mailto:ferruh.yi...@amd.com> > <mailto:ferruh.yi...@amd.com<mailto:ferruh.yi...@amd.com>>> > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... &g
Re: help
On 7/21/2023 12:39 PM, Igor de Paula wrote: > I am trying to use virtio_user for an interface with the > kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html > <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html> > I think this requires IOVA as va. > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, he may know better. Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=pa'? > It does work with Intel host and IOMMU > enabled. Part of the negotiation when setting it up is getting the IOMMU > number so I thought it has to have IOMMU. > Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below information you provided. > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work. > ESXI version - VMware ESXi, 7.0.0, 16324942 > AMD: AMD EPYC 7452 32-Core Processo > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > Regarding the logs I will try and attach it soon. > Thanks for info, waiting for logs. > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <mailto:ferruh.yi...@amd.com>> wrote: > > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > > > > > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > > grub. (Or intel_iommu=off for Intel processors.) > > > > > > > > Jochen > > > > > > > > *From: *Thomas Monjalon <mailto:tho...@monjalon.net>> > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula mailto:igord...@gmail.com>> > > *Cc: *users@dpdk.org <mailto:users@dpdk.org> <mailto:users@dpdk.org>>, Jochen Behrens > > mailto:jbehr...@vmware.com>>, Nipun Gupta > mailto:nipun.gu...@amd.com>>, Nikhil Agarwal > > mailto:nikhil.agar...@amd.com>>, Ferruh > Yigit mailto:ferruh.yi...@amd.com>> > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... > >> In the Bios I changed to amd_iommu=on and in the host it's the > same for > >> either. > >> > >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon > mailto:tho...@monjalon.net>> wrote: > >> > >> > 20/07/2023 11:35, Igor de Paula: > >> > > The weird thing is that it only happens when I am using a > host with an > >> > AMD > >> > > processor. It doesn't happen when I use a host with an Intel > processor. > >> > > >> > So it's probably a matter of BIOS settings for the IOMMU? > >> > > >> > > >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > mailto:tho...@monjalon.net>> > >> > > wrote: > >> > > > >> > > > +Cc the vmxnet3 maintainer. > >> > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > >> > > > > >> > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > >> > > > > This is because it can't negotiate the IOMMU type with > any port. > >> > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > mailto:tho...@monjalon.net> > >> > > > >> > > > wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must > be even" > >>
Re: help
I am trying to use virtio_user for an interface with the kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html I think this requires IOVA as va. It does work with Intel host and IOMMU enabled. Part of the negotiation when setting it up is getting the IOMMU number so I thought it has to have IOMMU. I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but again, that didn't work. ESXI version - VMware ESXi, 7.0.0, 16324942 AMD: AMD EPYC 7452 32-Core Processo On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz Regarding the logs I will try and attach it soon. Thanks On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit wrote: > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > > > > > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > > grub. (Or intel_iommu=off for Intel processors.) > > > > > > > > Jochen > > > > > > > > *From: *Thomas Monjalon > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula > > *Cc: *users@dpdk.org , Jochen Behrens > > , Nipun Gupta , Nikhil Agarwal > > , Ferruh Yigit > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... > >> In the Bios I changed to amd_iommu=on and in the host it's the same for > >> either. > >> > >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon > wrote: > >> > >> > 20/07/2023 11:35, Igor de Paula: > >> > > The weird thing is that it only happens when I am using a host with > an > >> > AMD > >> > > processor. It doesn't happen when I use a host with an Intel > processor. > >> > > >> > So it's probably a matter of BIOS settings for the IOMMU? > >> > > >> > > >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon < > tho...@monjalon.net> > >> > > wrote: > >> > > > >> > > > +Cc the vmxnet3 maintainer. > >> > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > >> > > > > >> > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > >> > > > > This is because it can't negotiate the IOMMU type with any port. > >> > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon < > tho...@monjalon.net > >> > > > >> > > > wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must be > even" > >> > > > > > > >> > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > >> > > > > > > I am running a VM inside a VMWARE server (vSphere). > >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a > >> > > > virtio_user to > >> > > > > > > interact with the kernel stack. > >> > > > > > > In another app I have it working but instead of virtio_user > I am > >> > > > running > >> > > > > > > KNI, it works in IOVA-PA mode. > >> > > > > > > I am looking to replace the KNI. > >> > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: > >> > > > > > > > >> > > > > > > >> > > > > >> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > < > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > >> > > > > > > I get a error it can't run in PA mode. > &g
Re: help
Hi Igor, VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you please try with: - enable 'enable_unsafe_noiommu_mode' flag - Force PA mode via '--iova-mode=pa' eal parameter Also to be able to figure out AMD IOMMU support level, can you please provide: - AMD part number - ESXi/hypervisor version - ESXi & VM kernel logs Thanks, Ferruh On 7/20/2023 5:21 PM, Jochen Behrens wrote: > +Ronak from the ESX team > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > grub. (Or intel_iommu=off for Intel processors.) > > > > Jochen > > > > *From: *Thomas Monjalon > *Date: *Thursday, July 20, 2023 at 6:00 AM > *To: *Igor de Paula > *Cc: *users@dpdk.org , Jochen Behrens > , Nipun Gupta , Nikhil Agarwal > , Ferruh Yigit > *Subject: *Re: help > > !! External Email > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > 20/07/2023 14:44, Igor de Paula: >> I have enabled it in the host and in the BIOS for AMD... >> In the Bios I changed to amd_iommu=on and in the host it's the same for >> either. >> >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon wrote: >> >> > 20/07/2023 11:35, Igor de Paula: >> > > The weird thing is that it only happens when I am using a host with an >> > AMD >> > > processor. It doesn't happen when I use a host with an Intel processor. >> > >> > So it's probably a matter of BIOS settings for the IOMMU? >> > >> > >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon >> > > wrote: >> > > >> > > > +Cc the vmxnet3 maintainer. >> > > > >> > > > Please Jochen, do you have an idea what's wrong below? >> > > > >> > > > >> > > > 20/07/2023 11:25, Igor de Paula: >> > > > > This is because it can't negotiate the IOMMU type with any port. >> > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > > >> > > > wrote: >> > > > > >> > > > > > Hello, >> > > > > > >> > > > > > The first error is "Cause: Error: number of ports must be even" >> > > > > > >> > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: >> > > > > > > I am running a VM inside a VMWARE server (vSphere). >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a >> > > > virtio_user to >> > > > > > > interact with the kernel stack. >> > > > > > > In another app I have it working but instead of virtio_user I am >> > > > running >> > > > > > > KNI, it works in IOVA-PA mode. >> > > > > > > I am looking to replace the KNI. >> > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: >> > > > > > > >> > > > > > >> > > > >> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path >> > >> > <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path> >> > > > > > > I get a error it can't run in PA mode. >> > > > > > > >> > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I get the >> > following >> > > > > > errors: >> > > > > > > EAL: lib.eal log level changed from info to debug >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 >> > > > > > > EAL: Support maximum 128 logical core(s) by configuration. >> > > > > > > EAL: Detected 2 lcore(s) >> > > > > > > EAL: Detected 1 NUMA nodes >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' >> > > > > > > EAL: Detected static linkage of DPDK >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes >> > > > > > > EAL: Virtual area found at 0x1 (size = 0x7000) >&
Re: help
+Ronak from the ESX team In our usage, we do set amd_iommu=off in the boot command line from grub. (Or intel_iommu=off for Intel processors.) Jochen From: Thomas Monjalon Date: Thursday, July 20, 2023 at 6:00 AM To: Igor de Paula Cc: users@dpdk.org , Jochen Behrens , Nipun Gupta , Nikhil Agarwal , Ferruh Yigit Subject: Re: help !! External Email +Cc some AMD maintainers, they can have an idea about IOMMU settings. 20/07/2023 14:44, Igor de Paula: > I have enabled it in the host and in the BIOS for AMD... > In the Bios I changed to amd_iommu=on and in the host it's the same for > either. > > On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon wrote: > > > 20/07/2023 11:35, Igor de Paula: > > > The weird thing is that it only happens when I am using a host with an > > AMD > > > processor. It doesn't happen when I use a host with an Intel processor. > > > > So it's probably a matter of BIOS settings for the IOMMU? > > > > > > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > > > wrote: > > > > > > > +Cc the vmxnet3 maintainer. > > > > > > > > Please Jochen, do you have an idea what's wrong below? > > > > > > > > > > > > 20/07/2023 11:25, Igor de Paula: > > > > > This is because it can't negotiate the IOMMU type with any port. > > > > > > > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > > > > I am running a VM inside a VMWARE server (vSphere). > > > > > > > My goal it to set up DPDK with two HW ports, and set up a > > > > virtio_user to > > > > > > > interact with the kernel stack. > > > > > > > In another app I have it working but instead of virtio_user I am > > > > running > > > > > > > KNI, it works in IOVA-PA mode. > > > > > > > I am looking to replace the KNI. > > > > > > > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fhowto%2Fvirtio_user_as_exception_path.html%23virtio-user-as-exception-path=05%7C01%7Cjbehrens%40vmware.com%7Cfbb1f6df362e4537d62608db8921432f%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638254548229277606%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=3NuWkbtQx2q8NJ2CzArNfuDjP%2FYIVv1kMc%2Bo4IiHlmM%3D=0<https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path> > > > > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the > > following > > > > > > errors: > > > > > > > EAL: lib.eal log level changed from info to debug > > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > > > > EAL: Detected 2 lcore(s) > > > > > > > EAL: Detected 1 NUMA nodes > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > > > > EAL: Detected static linkage of DPDK > > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > > > EAL: DPAA Bus not present. Skipping. > > > > > > > EAL: VFIO PCI modules not loaded > > > > > > > EAL: Selected IOVA mode 'VA' > > > > > > > EAL: Probing VFIO support... > > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported >
Re: help
+Cc some AMD maintainers, they can have an idea about IOMMU settings. 20/07/2023 14:44, Igor de Paula: > I have enabled it in the host and in the BIOS for AMD... > In the Bios I changed to amd_iommu=on and in the host it's the same for > either. > > On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon wrote: > > > 20/07/2023 11:35, Igor de Paula: > > > The weird thing is that it only happens when I am using a host with an > > AMD > > > processor. It doesn't happen when I use a host with an Intel processor. > > > > So it's probably a matter of BIOS settings for the IOMMU? > > > > > > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > > > wrote: > > > > > > > +Cc the vmxnet3 maintainer. > > > > > > > > Please Jochen, do you have an idea what's wrong below? > > > > > > > > > > > > 20/07/2023 11:25, Igor de Paula: > > > > > This is because it can't negotiate the IOMMU type with any port. > > > > > > > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > > > > I am running a VM inside a VMWARE server (vSphere). > > > > > > > My goal it to set up DPDK with two HW ports, and set up a > > > > virtio_user to > > > > > > > interact with the kernel stack. > > > > > > > In another app I have it working but instead of virtio_user I am > > > > running > > > > > > > KNI, it works in IOVA-PA mode. > > > > > > > I am looking to replace the KNI. > > > > > > > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > > > > > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the > > following > > > > > > errors: > > > > > > > EAL: lib.eal log level changed from info to debug > > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > > > > EAL: Detected 2 lcore(s) > > > > > > > EAL: Detected 1 NUMA nodes > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > > > > EAL: Detected static linkage of DPDK > > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > > > EAL: DPAA Bus not present. Skipping. > > > > > > > EAL: VFIO PCI modules not loaded > > > > > > > EAL: Selected IOVA mode 'VA' > > > > > > > EAL: Probing VFIO support... > > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > > > > EAL: VFIO support initialized > > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > > > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > > > > > > EAL: Setting up physically contiguous memory... > > > > > > > EAL: Setting maximum number of open files to 1048576 > > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > > > > hugepage_sz:1073741824 > > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > > > EAL: Virtual area found at 0x14000 (size = 0x20) > > > > > > > EAL: VA reserved for memseg list at 0x14000, size 20 > > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > > > EAL: Virtual area found at 0x218000 (size = 0x20) > > > > > > > EAL: VA reserved for memseg list at 0x218000, size 20 > > > > > > > EAL: TSC frequency is ~235 KHz > > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > > > > > EAL: Trying to obtain current memory policy. > > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > > > > EAL: Restoring previous memory policy: 0 > > > > > > > EAL: request: mp_malloc_sync > > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > > > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > > EAL:
Re: help
I have enabled it in the host and in the BIOS for AMD... In the Bios I changed to amd_iommu=on and in the host it's the same for either. On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon wrote: > 20/07/2023 11:35, Igor de Paula: > > The weird thing is that it only happens when I am using a host with an > AMD > > processor. It doesn't happen when I use a host with an Intel processor. > > So it's probably a matter of BIOS settings for the IOMMU? > > > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > > wrote: > > > > > +Cc the vmxnet3 maintainer. > > > > > > Please Jochen, do you have an idea what's wrong below? > > > > > > > > > 20/07/2023 11:25, Igor de Paula: > > > > This is because it can't negotiate the IOMMU type with any port. > > > > > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > > > I am running a VM inside a VMWARE server (vSphere). > > > > > > My goal it to set up DPDK with two HW ports, and set up a > > > virtio_user to > > > > > > interact with the kernel stack. > > > > > > In another app I have it working but instead of virtio_user I am > > > running > > > > > > KNI, it works in IOVA-PA mode. > > > > > > I am looking to replace the KNI. > > > > > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the > following > > > > > errors: > > > > > > EAL: lib.eal log level changed from info to debug > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > > > EAL: Detected 2 lcore(s) > > > > > > EAL: Detected 1 NUMA nodes > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > > > EAL: Detected static linkage of DPDK > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > > EAL: DPAA Bus not present. Skipping. > > > > > > EAL: VFIO PCI modules not loaded > > > > > > EAL: Selected IOVA mode 'VA' > > > > > > EAL: Probing VFIO support... > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > > > EAL: VFIO support initialized > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > > > > > EAL: Setting up physically contiguous memory... > > > > > > EAL: Setting maximum number of open files to 1048576 > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > > > hugepage_sz:1073741824 > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > > EAL: Virtual area found at 0x14000 (size = 0x20) > > > > > > EAL: VA reserved for memseg list at 0x14000, size 20 > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > > EAL: Virtual area found at 0x218000 (size = 0x20) > > > > > > EAL: VA reserved for memseg list at 0x218000, size 20 > > > > > > EAL: TSC frequency is ~235 KHz > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > > > > EAL: Trying to obtain current memory policy. > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > > > EAL: Restoring previous memory policy: 0 > > > > > > EAL: request: mp_malloc_sync > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > > > initializing > > > > > > EAL: Requested device :0b:00.0 cannot be used > > > > > > EAL: PCI device :13:00.0 on NUMA socket -1 > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA',
Re: help
20/07/2023 11:35, Igor de Paula: > The weird thing is that it only happens when I am using a host with an AMD > processor. It doesn't happen when I use a host with an Intel processor. So it's probably a matter of BIOS settings for the IOMMU? > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > wrote: > > > +Cc the vmxnet3 maintainer. > > > > Please Jochen, do you have an idea what's wrong below? > > > > > > 20/07/2023 11:25, Igor de Paula: > > > This is because it can't negotiate the IOMMU type with any port. > > > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > wrote: > > > > > > > Hello, > > > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > > I am running a VM inside a VMWARE server (vSphere). > > > > > My goal it to set up DPDK with two HW ports, and set up a > > virtio_user to > > > > > interact with the kernel stack. > > > > > In another app I have it working but instead of virtio_user I am > > running > > > > > KNI, it works in IOVA-PA mode. > > > > > I am looking to replace the KNI. > > > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the following > > > > errors: > > > > > EAL: lib.eal log level changed from info to debug > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > > EAL: Detected 2 lcore(s) > > > > > EAL: Detected 1 NUMA nodes > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > > EAL: Detected static linkage of DPDK > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > EAL: DPAA Bus not present. Skipping. > > > > > EAL: VFIO PCI modules not loaded > > > > > EAL: Selected IOVA mode 'VA' > > > > > EAL: Probing VFIO support... > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > > EAL: VFIO support initialized > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > > > > EAL: Setting up physically contiguous memory... > > > > > EAL: Setting maximum number of open files to 1048576 > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > > hugepage_sz:1073741824 > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > EAL: Virtual area found at 0x14000 (size = 0x20) > > > > > EAL: VA reserved for memseg list at 0x14000, size 20 > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > > EAL: Ask a virtual area of 0x20 bytes > > > > > EAL: Virtual area found at 0x218000 (size = 0x20) > > > > > EAL: VA reserved for memseg list at 0x218000, size 20 > > > > > EAL: TSC frequency is ~235 KHz > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > > > EAL: Trying to obtain current memory policy. > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > > EAL: Restoring previous memory policy: 0 > > > > > EAL: request: mp_malloc_sync > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > > initializing > > > > > EAL: Requested device :0b:00.0 cannot be used > > > > > EAL: PCI device :13:00.0 on NUMA socket -1 > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > > initializing > > > > > EAL: Requested device :13:00.0 cannot be used > > > > > EAL: Bus (pci) probe failed. > > > > > EAL: lib.telemetry log level changed from disabled to warning > > > > > EAL: Error - exiting with code: 1 > > > > > Cause: Error: number of ports must be even > > > > > EAL: request: mp_malloc_sync > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > >
Re: help
The weird thing is that it only happens when I am using a host with an AMD processor. It doesn't happen when I use a host with an Intel processor. On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon wrote: > +Cc the vmxnet3 maintainer. > > Please Jochen, do you have an idea what's wrong below? > > > 20/07/2023 11:25, Igor de Paula: > > This is because it can't negotiate the IOMMU type with any port. > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > wrote: > > > > > Hello, > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > I am running a VM inside a VMWARE server (vSphere). > > > > My goal it to set up DPDK with two HW ports, and set up a > virtio_user to > > > > interact with the kernel stack. > > > > In another app I have it working but instead of virtio_user I am > running > > > > KNI, it works in IOVA-PA mode. > > > > I am looking to replace the KNI. > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the following > > > errors: > > > > EAL: lib.eal log level changed from info to debug > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > EAL: Detected 2 lcore(s) > > > > EAL: Detected 1 NUMA nodes > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > EAL: Detected static linkage of DPDK > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > EAL: DPAA Bus not present. Skipping. > > > > EAL: VFIO PCI modules not loaded > > > > EAL: Selected IOVA mode 'VA' > > > > EAL: Probing VFIO support... > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > EAL: VFIO support initialized > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > > > EAL: Setting up physically contiguous memory... > > > > EAL: Setting maximum number of open files to 1048576 > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > hugepage_sz:1073741824 > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > EAL: Ask a virtual area of 0x20 bytes > > > > EAL: Virtual area found at 0x14000 (size = 0x20) > > > > EAL: VA reserved for memseg list at 0x14000, size 20 > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > > EAL: Ask a virtual area of 0x20 bytes > > > > EAL: Virtual area found at 0x218000 (size = 0x20) > > > > EAL: VA reserved for memseg list at 0x218000, size 20 > > > > EAL: TSC frequency is ~235 KHz > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > > EAL: Trying to obtain current memory policy. > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > EAL: Restoring previous memory policy: 0 > > > > EAL: request: mp_malloc_sync > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > initializing > > > > EAL: Requested device :0b:00.0 cannot be used > > > > EAL: PCI device :13:00.0 on NUMA socket -1 > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > initializing > > > > EAL: Requested device :13:00.0 cannot be used > > > > EAL: Bus (pci) probe failed. > > > > EAL: lib.telemetry log level changed from disabled to warning > > > > EAL: Error - exiting with code: 1 > > > > Cause: Error: number of ports must be even > > > > EAL: request: mp_malloc_sync > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > > > > > > > > > For some reason the HW ports won't setup. From what I understand > > > > net_vmxnet3 should work with VA mode. > > > > I enabled I/OMUU for the VM. > > > > The weird thing even when enabled, I still have the > > > > enable_unsafe_noiommu_mode flag on. > > > > And because it's on the
Re: help
+Cc the vmxnet3 maintainer. Please Jochen, do you have an idea what's wrong below? 20/07/2023 11:25, Igor de Paula: > This is because it can't negotiate the IOMMU type with any port. > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon wrote: > > > Hello, > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > 03/05/2023 18:13, Igor de Paula: > > > I am running a VM inside a VMWARE server (vSphere). > > > My goal it to set up DPDK with two HW ports, and set up a virtio_user to > > > interact with the kernel stack. > > > In another app I have it working but instead of virtio_user I am running > > > KNI, it works in IOVA-PA mode. > > > I am looking to replace the KNI. > > > > > > When I try to set up virtio_user port as in the doc: > > > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > > I get a error it can't run in PA mode. > > > > > > > > > When I try to run as VA mode from a parameter, I get the following > > errors: > > > EAL: lib.eal log level changed from info to debug > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > EAL: Support maximum 128 logical core(s) by configuration. > > > EAL: Detected 2 lcore(s) > > > EAL: Detected 1 NUMA nodes > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > EAL: Checking presence of .so 'librte_eal.so' > > > EAL: Detected static linkage of DPDK > > > EAL: Ask a virtual area of 0x7000 bytes > > > EAL: Virtual area found at 0x1 (size = 0x7000) > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > EAL: DPAA Bus not present. Skipping. > > > EAL: VFIO PCI modules not loaded > > > EAL: Selected IOVA mode 'VA' > > > EAL: Probing VFIO support... > > > EAL: IOMMU type 1 (Type 1) is supported > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > EAL: VFIO support initialized > > > EAL: Ask a virtual area of 0x5b000 bytes > > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > > EAL: Setting up physically contiguous memory... > > > EAL: Setting maximum number of open files to 1048576 > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > hugepage_sz:1073741824 > > > EAL: Ask a virtual area of 0x2000 bytes > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > EAL: Ask a virtual area of 0x20 bytes > > > EAL: Virtual area found at 0x14000 (size = 0x20) > > > EAL: VA reserved for memseg list at 0x14000, size 20 > > > EAL: Ask a virtual area of 0x2000 bytes > > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > > EAL: Ask a virtual area of 0x20 bytes > > > EAL: Virtual area found at 0x218000 (size = 0x20) > > > EAL: VA reserved for memseg list at 0x218000, size 20 > > > EAL: TSC frequency is ~235 KHz > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > EAL: Trying to obtain current memory policy. > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > EAL: Restoring previous memory policy: 0 > > > EAL: request: mp_malloc_sync > > > EAL: Heap on socket 0 was expanded by 1024MB > > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > initializing > > > EAL: Requested device :0b:00.0 cannot be used > > > EAL: PCI device :13:00.0 on NUMA socket -1 > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > initializing > > > EAL: Requested device :13:00.0 cannot be used > > > EAL: Bus (pci) probe failed. > > > EAL: lib.telemetry log level changed from disabled to warning > > > EAL: Error - exiting with code: 1 > > > Cause: Error: number of ports must be even > > > EAL: request: mp_malloc_sync > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > > > > > For some reason the HW ports won't setup. From what I understand > > > net_vmxnet3 should work with VA mode. > > > I enabled I/OMUU for the VM. > > > The weird thing even when enabled, I still have the > > > enable_unsafe_noiommu_mode flag on. > > > And because it's on the this: > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev); > > > > > > return PA mode, and it fails. > > > > > > When I disable it by modifying > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another > > error. > > > The error is that it doesn't find a suitable IOMMU type: > > > Just putting the relevant message: > > > > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > EAL:
Re: help
This is because it can't negotiate the IOMMU type with any port. On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon wrote: > Hello, > > The first error is "Cause: Error: number of ports must be even" > > > 03/05/2023 18:13, Igor de Paula: > > I am running a VM inside a VMWARE server (vSphere). > > My goal it to set up DPDK with two HW ports, and set up a virtio_user to > > interact with the kernel stack. > > In another app I have it working but instead of virtio_user I am running > > KNI, it works in IOVA-PA mode. > > I am looking to replace the KNI. > > > > When I try to set up virtio_user port as in the doc: > > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > I get a error it can't run in PA mode. > > > > > > When I try to run as VA mode from a parameter, I get the following > errors: > > EAL: lib.eal log level changed from info to debug > > EAL: Detected lcore 0 as core 0 on socket 0 > > EAL: Detected lcore 1 as core 0 on socket 0 > > EAL: Support maximum 128 logical core(s) by configuration. > > EAL: Detected 2 lcore(s) > > EAL: Detected 1 NUMA nodes > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > EAL: Checking presence of .so 'librte_eal.so.21' > > EAL: Checking presence of .so 'librte_eal.so' > > EAL: Detected static linkage of DPDK > > EAL: Ask a virtual area of 0x7000 bytes > > EAL: Virtual area found at 0x1 (size = 0x7000) > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > EAL: DPAA Bus not present. Skipping. > > EAL: VFIO PCI modules not loaded > > EAL: Selected IOVA mode 'VA' > > EAL: Probing VFIO support... > > EAL: IOMMU type 1 (Type 1) is supported > > EAL: IOMMU type 7 (sPAPR) is not supported > > EAL: IOMMU type 8 (No-IOMMU) is supported > > EAL: VFIO support initialized > > EAL: Ask a virtual area of 0x5b000 bytes > > EAL: Virtual area found at 0x17000 (size = 0x5b000) > > EAL: Setting up physically contiguous memory... > > EAL: Setting maximum number of open files to 1048576 > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > hugepage_sz:1073741824 > > EAL: Ask a virtual area of 0x2000 bytes > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > EAL: Ask a virtual area of 0x20 bytes > > EAL: Virtual area found at 0x14000 (size = 0x20) > > EAL: VA reserved for memseg list at 0x14000, size 20 > > EAL: Ask a virtual area of 0x2000 bytes > > EAL: Virtual area found at 0x214000 (size = 0x2000) > > EAL: Memseg list allocated at socket 0, page size 0x10kB > > EAL: Ask a virtual area of 0x20 bytes > > EAL: Virtual area found at 0x218000 (size = 0x20) > > EAL: VA reserved for memseg list at 0x218000, size 20 > > EAL: TSC frequency is ~235 KHz > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > EAL: Trying to obtain current memory policy. > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > EAL: Restoring previous memory policy: 0 > > EAL: request: mp_malloc_sync > > EAL: Heap on socket 0 was expanded by 1024MB > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > initializing > > EAL: Requested device :0b:00.0 cannot be used > > EAL: PCI device :13:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > initializing > > EAL: Requested device :13:00.0 cannot be used > > EAL: Bus (pci) probe failed. > > EAL: lib.telemetry log level changed from disabled to warning > > EAL: Error - exiting with code: 1 > > Cause: Error: number of ports must be even > > EAL: request: mp_malloc_sync > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > For some reason the HW ports won't setup. From what I understand > > net_vmxnet3 should work with VA mode. > > I enabled I/OMUU for the VM. > > The weird thing even when enabled, I still have the > > enable_unsafe_noiommu_mode flag on. > > And because it's on the this: > > > > dev_iova_mode = pci_device_iova_mode(dr, dev); > > > > return PA mode, and it fails. > > > > When I disable it by modifying > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another > error. > > The error is that it doesn't find a suitable IOMMU type: > > Just putting the relevant message: > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > EAL: PCI device :0b:00.0 on NUMA socket -1 > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > > EAL: :0b:00.0 failed to select IOMMU
Re: help
Hello, The first error is "Cause: Error: number of ports must be even" 03/05/2023 18:13, Igor de Paula: > I am running a VM inside a VMWARE server (vSphere). > My goal it to set up DPDK with two HW ports, and set up a virtio_user to > interact with the kernel stack. > In another app I have it working but instead of virtio_user I am running > KNI, it works in IOVA-PA mode. > I am looking to replace the KNI. > > When I try to set up virtio_user port as in the doc: > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > I get a error it can't run in PA mode. > > > When I try to run as VA mode from a parameter, I get the following errors: > EAL: lib.eal log level changed from info to debug > EAL: Detected lcore 0 as core 0 on socket 0 > EAL: Detected lcore 1 as core 0 on socket 0 > EAL: Support maximum 128 logical core(s) by configuration. > EAL: Detected 2 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Checking presence of .so 'librte_eal.so.21.3' > EAL: Checking presence of .so 'librte_eal.so.21' > EAL: Checking presence of .so 'librte_eal.so' > EAL: Detected static linkage of DPDK > EAL: Ask a virtual area of 0x7000 bytes > EAL: Virtual area found at 0x1 (size = 0x7000) > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: DPAA Bus not present. Skipping. > EAL: VFIO PCI modules not loaded > EAL: Selected IOVA mode 'VA' > EAL: Probing VFIO support... > EAL: IOMMU type 1 (Type 1) is supported > EAL: IOMMU type 7 (sPAPR) is not supported > EAL: IOMMU type 8 (No-IOMMU) is supported > EAL: VFIO support initialized > EAL: Ask a virtual area of 0x5b000 bytes > EAL: Virtual area found at 0x17000 (size = 0x5b000) > EAL: Setting up physically contiguous memory... > EAL: Setting maximum number of open files to 1048576 > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824 > EAL: Ask a virtual area of 0x2000 bytes > EAL: Virtual area found at 0x100062000 (size = 0x2000) > EAL: Memseg list allocated at socket 0, page size 0x10kB > EAL: Ask a virtual area of 0x20 bytes > EAL: Virtual area found at 0x14000 (size = 0x20) > EAL: VA reserved for memseg list at 0x14000, size 20 > EAL: Ask a virtual area of 0x2000 bytes > EAL: Virtual area found at 0x214000 (size = 0x2000) > EAL: Memseg list allocated at socket 0, page size 0x10kB > EAL: Ask a virtual area of 0x20 bytes > EAL: Virtual area found at 0x218000 (size = 0x20) > EAL: VA reserved for memseg list at 0x218000, size 20 > EAL: TSC frequency is ~235 KHz > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > EAL: Trying to obtain current memory policy. > EAL: Setting policy MPOL_PREFERRED for socket 0 > EAL: Restoring previous memory policy: 0 > EAL: request: mp_malloc_sync > EAL: Heap on socket 0 was expanded by 1024MB > EAL: PCI device :0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 net_vmxnet3 > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing > EAL: Requested device :0b:00.0 cannot be used > EAL: PCI device :13:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 net_vmxnet3 > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing > EAL: Requested device :13:00.0 cannot be used > EAL: Bus (pci) probe failed. > EAL: lib.telemetry log level changed from disabled to warning > EAL: Error - exiting with code: 1 > Cause: Error: number of ports must be even > EAL: request: mp_malloc_sync > EAL: Heap on socket 0 was shrunk by 1024MB > > > > For some reason the HW ports won't setup. From what I understand > net_vmxnet3 should work with VA mode. > I enabled I/OMUU for the VM. > The weird thing even when enabled, I still have the > enable_unsafe_noiommu_mode flag on. > And because it's on the this: > > dev_iova_mode = pci_device_iova_mode(dr, dev); > > return PA mode, and it fails. > > When I disable it by modifying > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get another error. > The error is that it doesn't find a suitable IOMMU type: > Just putting the relevant message: > > > EAL: Heap on socket 0 was expanded by 1024MB > EAL: PCI device :0b:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 net_vmxnet3 > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device) > EAL: :0b:00.0 failed to select IOMMU type > EAL: Requested device :0b:00.0 cannot be used > EAL: PCI device :13:00.0 on NUMA socket -1 > EAL: probe driver: 15ad:7b0 net_vmxnet3 > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No
Re: help with virtio_port
Thanks! It's weird because on the KNI version, I am listening on the port and waiting for a link message. If I run tcpdump there is no packet coming on that port but the rte_handle_kni_request doe's deal with it. When I try to listen on the port, without KNI, when I am using virtio_user I do get a packet when I use tcpdump. I guess when KNI initialized it takes the traffic associated with the functions it supports. Can someone confirm that this is actually the case? Many thanks for the help so far On Tue, May 23, 2023 at 5:23 PM Stephen Hemminger wrote: > > On Tue, 23 May 2023 16:46:24 +0100 > Igor de Paula wrote: > > > Hi, > > I am running the DPDK version: 21.08.0 and Ubuntu 20.04.3 LTS. > > I have an application that uses KNI to interface with the kernel. > > I want to replace it with virtio_user ports as KNI will be deprecated in > > the future. > > Most of the functionality I am able to replace but there is one thing I am > > struggling with. > > In KNI we can add functions that will be called in case the network stack > > makes a request. The following code shows this: > > struct rte_kni *kni; > > struct rte_kni_conf conf; > > struct rte_kni_ops ops; > > struct rte_eth_dev_info dev_info; > > int ret; > > /* Clear conf at first */ > > memset(, 0, sizeof(conf)); > > conf.core_id = 0; > > memset(, 0, sizeof(ops)); > > ops.port_id = ppo->id; > > ops.config_promiscusity = ippe_ppo_set_kni_promiscuous_mode; > > ops.change_mtu = ippe_ppo_set_kni_mtu; > > ops.config_network_if = ippe_ppo_set_kni_interface; > > ops.config_mac_address = ippe_ppo_set_kni_mac_address; > > kni = rte_kni_alloc(pktmbuf_pool[0], , ); > > > > > > And there is a handle_request function supplied by KNI that calls these > > functions when need be, > > I haven't found any documentation on how to replace this functionality. I > > am no expert in how to set up and interact with the kernel stack, Some help > > on how to achieve this would be appreciated. > > If you want to handle changes to kernel network device, then you > will have to build a netlink listener that monitors these changes.
Re: help with virtio_port
Sorry I replied privately. I am replying here again.
WIth the KNI, I have a listener listener as follows:
static void *kni_monitor_system_networking(void *data)
{
struct mnl_socket *nl;
char buf[MNL_SOCKET_BUFFER_SIZE];
int ret;
nl = mnl_socket_open(NETLINK_ROUTE);
if (nl == NULL)
{
perror("mnl_socket_open");
exit(EXIT_FAILURE);
}
if (mnl_socket_bind(nl, RTMGRP_LINK | RTMGRP_IPV4_IFADDR,
MNL_SOCKET_AUTOPID) < 0)
{
perror("mnl_socket_bind");
exit(EXIT_FAILURE);
}
ret = mnl_socket_recvfrom(nl, buf, sizeof(buf));
while (ret > 0)
{
ret = mnl_cb_run(buf, ret, 0, 0, data_cb, NULL);
if (ret <= 0)
break;
ret = mnl_socket_recvfrom(nl, buf, sizeof(buf));
}
if (ret == -1)
{
perror("error");
exit(EXIT_FAILURE);
}
mnl_socket_close(nl);
return 0;
}
But also I have a thread looping over rte_kni_handle_request:
while (1)
{
for (t = kni_tracker; t; t = t->next)
{
struct rte_kni *kni = ( struct rte_kni * )t->kni;
if (ppo->kni)
{
rte_kni_handle_request(kni);
}
usleep(1);
}
The rte_kni_handle_request calls the kni_config_promiscusity.
So from what you are saying, I can replace this call with another monitor
on the netlink ports (the work will be on finding the correct flags and to
implement the callback I guess)?
I from what I see, mnl_socket_bind(nl, RTMGRP_LINK | RTMGRP_IPV4_IFADDR,
MNL_SOCKET_AUTOPID) < 0) this linke should get also requests that are link
related, but it doesn't seem to get any packets when I use tcpdump on the
interface.
On Tue, May 23, 2023 at 5:23 PM Stephen Hemminger <
step...@networkplumber.org> wrote:
> On Tue, 23 May 2023 16:46:24 +0100
> Igor de Paula wrote:
>
> > Hi,
> > I am running the DPDK version: 21.08.0 and Ubuntu 20.04.3 LTS.
> > I have an application that uses KNI to interface with the kernel.
> > I want to replace it with virtio_user ports as KNI will be deprecated in
> > the future.
> > Most of the functionality I am able to replace but there is one thing I
> am
> > struggling with.
> > In KNI we can add functions that will be called in case the network stack
> > makes a request. The following code shows this:
> > struct rte_kni *kni;
> > struct rte_kni_conf conf;
> > struct rte_kni_ops ops;
> > struct rte_eth_dev_info dev_info;
> > int ret;
> > /* Clear conf at first */
> > memset(, 0, sizeof(conf));
> > conf.core_id = 0;
> > memset(, 0, sizeof(ops));
> > ops.port_id = ppo->id;
> > ops.config_promiscusity = ippe_ppo_set_kni_promiscuous_mode;
> > ops.change_mtu = ippe_ppo_set_kni_mtu;
> > ops.config_network_if = ippe_ppo_set_kni_interface;
> > ops.config_mac_address = ippe_ppo_set_kni_mac_address;
> > kni = rte_kni_alloc(pktmbuf_pool[0], , );
> >
> >
> > And there is a handle_request function supplied by KNI that calls these
> > functions when need be,
> > I haven't found any documentation on how to replace this functionality. I
> > am no expert in how to set up and interact with the kernel stack, Some
> help
> > on how to achieve this would be appreciated.
>
> If you want to handle changes to kernel network device, then you
> will have to build a netlink listener that monitors these changes.
>
Re: help with virtio_port
On Tue, 23 May 2023 16:46:24 +0100 Igor de Paula wrote: > Hi, > I am running the DPDK version: 21.08.0 and Ubuntu 20.04.3 LTS. > I have an application that uses KNI to interface with the kernel. > I want to replace it with virtio_user ports as KNI will be deprecated in > the future. > Most of the functionality I am able to replace but there is one thing I am > struggling with. > In KNI we can add functions that will be called in case the network stack > makes a request. The following code shows this: > struct rte_kni *kni; > struct rte_kni_conf conf; > struct rte_kni_ops ops; > struct rte_eth_dev_info dev_info; > int ret; > /* Clear conf at first */ > memset(, 0, sizeof(conf)); > conf.core_id = 0; > memset(, 0, sizeof(ops)); > ops.port_id = ppo->id; > ops.config_promiscusity = ippe_ppo_set_kni_promiscuous_mode; > ops.change_mtu = ippe_ppo_set_kni_mtu; > ops.config_network_if = ippe_ppo_set_kni_interface; > ops.config_mac_address = ippe_ppo_set_kni_mac_address; > kni = rte_kni_alloc(pktmbuf_pool[0], , ); > > > And there is a handle_request function supplied by KNI that calls these > functions when need be, > I haven't found any documentation on how to replace this functionality. I > am no expert in how to set up and interact with the kernel stack, Some help > on how to achieve this would be appreciated. If you want to handle changes to kernel network device, then you will have to build a netlink listener that monitors these changes.
