[dpdk-dev] How to debug packet sends to virtual functions
>>> It's the latter, i.e. two primary processes running DPDK on the same VM but >>> different VF devices. The VF devices belongs to the same physical devices >>> though. >> >> I'll try that as well. > > If you need more help with how we set things up, I'll be happy to help you. I found the bug, it was an illegal arp entry that I had created causing the problem. So this case has been solved. I re-checked this ifconfig issue, and now I can get it to work without ifconfig in the guest. Not sure why it didn't work without it previously, but apparently, there is something else I did that made things work. Sorry about the fuzz, but thanks for your effort! Regards Mats Liljegren
[dpdk-dev] How to debug packet sends to virtual functions
> I found the bug, it was an illegal arp entry that I had created causing the > problem. So this case has been solved. > > I re-checked this ifconfig issue, and now I can get it to work without > ifconfig > in the guest. Not sure why it didn't work without it previously, but > apparently, there is something else I did that made things work. > > Sorry about the fuzz, but thanks for your effort! No problem, glad to hear it's not a DPDK bug :-) Best regards, Anatoly Burakov DPDK SW Engineer -- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
[dpdk-dev] How to debug packet sends to virtual functions
Hi Anatoly, >> The guest starts with loading the igbvf kernel driver, uses ifconfig >> up, >> then loads and binds to igb_uio. After that, DPDK works. If I skip any step >> here it doesn't work. Skipping "ifconfig" step resulted in packets being >> received but I couldn't send them, they just got queued up but was never >> sent. >> > > Hm, never seen this before. Apologies if you have already mentioned this, but > what NIC are you using? I'll see if I can replicate your issues. Not sure if I've mentioned it, but it is a quad I350. Port 0 and 3 on host only, and 1 and 2 are being used by DPDK in a guest. The challenge was to make a ping go from port 0 to 3 and back, via a somewhat modified DPDK l2fwd example. Port 1 and 2 had two virtual functions each, and each such pair was given to an instance of this l2fwd example. >> It's the latter, i.e. two primary processes running DPDK on the same VM but >> different VF devices. The VF devices belongs to the same physical devices >> though. > > I'll try that as well. If you need more help with how we set things up, I'll be happy to help you. Regards Mats Liljegren
[dpdk-dev] How to debug packet sends to virtual functions
Hi Mats > The guest starts with loading the igbvf kernel driver, uses ifconfig up, > then loads and binds to igb_uio. After that, DPDK works. If I skip any step > here it doesn't work. Skipping "ifconfig" step resulted in packets being > received but I couldn't send them, they just got queued up but was never > sent. > Hm, never seen this before. Apologies if you have already mentioned this, but what NIC are you using? I'll see if I can replicate your issues. > It's the latter, i.e. two primary processes running DPDK on the same VM but > different VF devices. The VF devices belongs to the same physical devices > though. I'll try that as well. Best regards, Anatoly Burakov DPDK SW Engineer -- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
[dpdk-dev] How to debug packet sends to virtual functions
>> I finally got things working. I apparently missed an "ifconfig up" in >> the >> guest, before starting dpdk. I'm still confused why this would be needed. Is >> dpdk unable to do a full initialization of the virtual function from the >> guest? >> > You wouldn't be able to call ifconfig on your VF device if you have bound > your guest VF device to igb_uio driver. Have you bound your VF device to > igb_uio? Or you have enabled automatic port unbinding (which is disabled by > default in recent releases)? The guest starts with loading the igbvf kernel driver, uses ifconfig up, then loads and binds to igb_uio. After that, DPDK works. If I skip any step here it doesn't work. Skipping "ifconfig" step resulted in packets being received but I couldn't send them, they just got queued up but was never sent. >> While I got one instance of DPDK running, I got a problem when starting two >> instances of DPDK running against different virtual functions. These virtual >> functions stems from the same physical interfaces. >> >> Starting them one or the other works fine. I have to adapt my static arp >> entries since they have different MAC addresses, but this is only expected. >> When starting them both however, I receive no packets. If one is running >> and currently processing packets, it will stop doing so the instance the >> second >> instance starts. >> > Are you referring to two different VM's each having a separate VF device, or > are you trying to run two primary processes on one VM with different VF > devices? It's the latter, i.e. two primary processes running DPDK on the same VM but different VF devices. The VF devices belongs to the same physical devices though. Regards Mats Liljegren
[dpdk-dev] How to debug packet sends to virtual functions
On Tue, Feb 4, 2014 at 2:40 PM, Burakov, Anatoly wrote: > Hi Mats > > Technically, you can use igb_uio on the host as well (DPDK PMD supports > creating virtual devices since at least release 1.5.0), it's just that you'll > have to set everything up yourself inside your host DPDK application (you > can't use "ip net" to set up VF devices if you're using DPDK drivers). > Unfortunately, I'm not familiar enough with that part of the code to comment > on what exactly you should do to make it work with igb_uio, but I can > certainly ask around if you want. > > I also noticed that you are running KVM with virsh. We always run our VM's by > passing QEMU command line directly, without using virsh, so unfortunately I > cannot be of much help here as I'm not familiar with virsh. Your best bet > would be to get whatever parameters virsh passes to QEMU and modify them to > suit your needs and according to DPDK documentation. Hi Anatoly, I finally got things working. I apparently missed an "ifconfig up" in the guest, before starting dpdk. I'm still confused why this would be needed. Is dpdk unable to do a full initialization of the virtual function from the guest? I'm also curious whether this is how it is intended to work, or am I experiencing a strange type of bug? If not, is it documented that ifconfig is needed in the guest? I thought I've read all documentation thoroughly, but sometimes it is easy to miss the obvious anyway... While I got one instance of DPDK running, I got a problem when starting two instances of DPDK running against different virtual functions. These virtual functions stems from the same physical interfaces. Starting them one or the other works fine. I have to adapt my static arp entries since they have different MAC addresses, but this is only expected. When starting them both however, I receive no packets. If one is running and currently processing packets, it will stop doing so the instance the second instance starts. Anyone knowing what could cause such behavior? Regards Mats Liljegren
[dpdk-dev] How to debug packet sends to virtual functions
Hi Mats > Hi Anatoly, > > I finally got things working. I apparently missed an "ifconfig up" in > the > guest, before starting dpdk. I'm still confused why this would be needed. Is > dpdk unable to do a full initialization of the virtual function from the > guest? > You wouldn't be able to call ifconfig on your VF device if you have bound your guest VF device to igb_uio driver. Have you bound your VF device to igb_uio? Or you have enabled automatic port unbinding (which is disabled by default in recent releases)? > While I got one instance of DPDK running, I got a problem when starting two > instances of DPDK running against different virtual functions. These virtual > functions stems from the same physical interfaces. > > Starting them one or the other works fine. I have to adapt my static arp > entries since they have different MAC addresses, but this is only expected. > When starting them both however, I receive no packets. If one is running > and currently processing packets, it will stop doing so the instance the > second > instance starts. > Are you referring to two different VM's each having a separate VF device, or are you trying to run two primary processes on one VM with different VF devices? Best regards, Anatoly Burakov DPDK SW Engineer -- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
[dpdk-dev] How to debug packet sends to virtual functions
Hi Mats, Yes the virsh xml is created in essentially the same way. And the setup in host is also the same as yours. But no PCI passthru. As I understand, in a PF-VF deployment it makes no sense for PCI passthru, coz the PCI device must be visible and under control of host. (Pls correct me if Im wrong). -Qinglai On Tue, Feb 4, 2014 at 1:31 PM, Mats Liljegren wrote: > Hi Qinglai, > > How did you configure the PCI passthrough in KVM? Did you use command > line parameter as described in the instructions, or do you use virsh > XML files, or maybe using virt-manager? > > The steps that works best (i.e. can receive but not transmit) are: > > 1) sudo modprobe ixgbe max_vfs=2 > 2) sudo ip link set em1 vf 0 mac > 3) sudo ifconfig em1 up > 4) sudo modprobe pci-stub > 5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id > 6) echo ":02:10.0" | sudo tee > /sys/bus/pci/devices/\:02\:10.0/driver/unbind > 7) echo ":02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind > 8) virsh create vm vm.xml > > The guest then does: > > 5) sudo modprobe uio > 6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio :02:10.0 > 7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio > 8) sudo > > The virsh xml file has the following entry for the virtual function: > > > /> > >function='0x0'/> > > > I actually have two virtual functions and two physical functions, but > I just repeat above for each. > > Regards > Mats > > On Tue, Feb 4, 2014 at 12:14 PM, jigsaw wrote: >> Hi Mats, >> >> I didn't have any deviation. What I did is just loading ixgbe (with >> extra params for vf, as you mentioned in your first email), >> and DPDK is up and running in guest. Of course I also followed section >> 8.10 of the DPDK release notes. >> >> I can switch between DPDK and ixgebvf in guest at runtime and >> everything works fine. >> >> Sorry I can't help to debug coz I have only 82599EB at hand. >> >> -Qinglai >> >> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren >> wrote: >>> Hi Qinglai, >>> >>> Thanks for the response! >>> >>> My previous attempt was with ixgbe loaded in the host. I also needed >>> to load ixgbevf, but this seems to be because of a short-coming in >>> libvirt. Maybe loading ixgbevf and then unbind it when running the >>> guest is what causes my problems. I could receive packets with this >>> setup, but not transmit. I used extra debug to syslog, and it showed >>> that packets was placed in the transmit queue, but these packets was >>> never sent. >>> >>> I'll see if I can get this working without loading ixgbevf in the host. >>> >>> What instructions did you follow to get this working? Did you do any >>> deviation from the instructions? >>> >>> Regards >>> Mats >>> >>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: Hi Mats, I've tried vf with 82599EB and it works fine. As long as the VF is visible in guest, DPDK's VF driver should work just as ixgbevf, which shares more or less the same code. I don't understand why you would expect DPDK at guest to work as VF, while the host has no ixgbe loaded. To make further debug I'd suggest compile ixgbe driver with your own syslogs. At least you will be able to see the signalling between vf and ixgbe drivers. -Qinglai On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren wrote: > This is my fourth mail in my desperate attempt to get DPDK running in > KVM and no comments so far, not even any "it works for me". Am I the > only one crazy enough to believe that this can be done? > > Anyway, out of desperation I tried to get it running without having > ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed > the instructions in the Programmer's Guide, chapter "Setting Up a KVM > Virtual Machine Monitor", using the PMD version of the instructions. > > I get as far as being able to see my four virtual functions in the > guest using "lspci". But starting the DPDK application gives me the > following error: > > PMD:The MAC address is not valid. > The most likely cause of this error is that the VM host > has not assigned a valid MAC address to this VF device. > Please consult the DPDK Release Notes (FAQ section) for > a possible solution to this problem. > > This may be true, but without any kernel modules loaded, how am I > supposed to change any MAC addresses? Can this be done from within > DPDK? > > As a side-note, I did try to load ixgbevf in the guest, but it > produced no interfaces. There was no error messages in the syslog > though. > > Is it possible to get X540 working in a guest or should I switch hardware? > > Since the instructions assumes I know the command line to KVM to start > my guest (which I do not), I cannot followed them precisely. I use > virsh and XML file, and maybe
[dpdk-dev] How to debug packet sends to virtual functions
Hi Anatoly, Just realized that the instructions gave two alternatives for the host: DPDK igb_uio or ixgbe. It was the former, DPDK igb_uio that I was referring to as being the non-working alternative. Loading ixgbe enables me to set the mac addresses. This is also what I tried previously, when I both got ixgbevf working in the guest as well as receive to work in DPDK in the guest. The problem is making transmit work in DPDK in the guest. Regards Mats On Tue, Feb 4, 2014 at 12:52 PM, Burakov, Anatoly wrote: >> -Original Message- >> From: Mats Liljegren [mailto:liljegren.mats2 at gmail.com] >> Sent: Tuesday, February 04, 2014 11:48 AM >> To: Burakov, Anatoly >> Cc: dev at dpdk.org >> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions >> >> Hi Anatoly, >> >> Thanks for the clarification. Then I can't use those instructions, which is >> what I >> was starting to suspect. I have to find another way of getting DPDK running >> in >> the guest then. Using ixgbevf in the guest works fine, though. >> >> Regards >> Mats >> > > Hi Mats > > Let me clarify - do you run DPDK on both host and guest? E.g. you load > igb_uio on the host as well? If not, I don't see why you can't use those > instructions - you're creating VF devices on the host anyway, and that's > where you should set their MAC addresses. > > Best regards, > Anatoly Burakov > DPDK SW Engineer > > -- > Intel Shannon Limited > Registered in Ireland > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare > Registered Number: 308263 Business address: Dromore House, East Park, > Shannon, Co. Clare > >
[dpdk-dev] How to debug packet sends to virtual functions
> -Original Message- > From: Mats Liljegren [mailto:liljegren.mats2 at gmail.com] > Sent: Tuesday, February 04, 2014 1:04 PM > To: Burakov, Anatoly > Cc: dev at dpdk.org > Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions > > Hi Anatoly, > > Just realized that the instructions gave two alternatives for the > host: DPDK igb_uio or ixgbe. It was the former, DPDK igb_uio that I was > referring to as being the non-working alternative. > > Loading ixgbe enables me to set the mac addresses. This is also what I tried > previously, when I both got ixgbevf working in the guest as well as receive to > work in DPDK in the guest. The problem is making transmit work in DPDK in > the guest. > > Regards > Mats > > Hi Mats Technically, you can use igb_uio on the host as well (DPDK PMD supports creating virtual devices since at least release 1.5.0), it's just that you'll have to set everything up yourself inside your host DPDK application (you can't use "ip net" to set up VF devices if you're using DPDK drivers). Unfortunately, I'm not familiar enough with that part of the code to comment on what exactly you should do to make it work with igb_uio, but I can certainly ask around if you want. I also noticed that you are running KVM with virsh. We always run our VM's by passing QEMU command line directly, without using virsh, so unfortunately I cannot be of much help here as I'm not familiar with virsh. Your best bet would be to get whatever parameters virsh passes to QEMU and modify them to suit your needs and according to DPDK documentation. Best regards, Anatoly Burakov DPDK SW Engineer -- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
[dpdk-dev] How to debug packet sends to virtual functions
Hi Mats, I didn't have any deviation. What I did is just loading ixgbe (with extra params for vf, as you mentioned in your first email), and DPDK is up and running in guest. Of course I also followed section 8.10 of the DPDK release notes. I can switch between DPDK and ixgebvf in guest at runtime and everything works fine. Sorry I can't help to debug coz I have only 82599EB at hand. -Qinglai On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren wrote: > Hi Qinglai, > > Thanks for the response! > > My previous attempt was with ixgbe loaded in the host. I also needed > to load ixgbevf, but this seems to be because of a short-coming in > libvirt. Maybe loading ixgbevf and then unbind it when running the > guest is what causes my problems. I could receive packets with this > setup, but not transmit. I used extra debug to syslog, and it showed > that packets was placed in the transmit queue, but these packets was > never sent. > > I'll see if I can get this working without loading ixgbevf in the host. > > What instructions did you follow to get this working? Did you do any > deviation from the instructions? > > Regards > Mats > > On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: >> Hi Mats, >> >> I've tried vf with 82599EB and it works fine. As long as the VF is >> visible in guest, DPDK's VF driver should work just as ixgbevf, which >> shares more or less the same code. >> >> I don't understand why you would expect DPDK at guest to work as VF, >> while the host has no ixgbe loaded. >> To make further debug I'd suggest compile ixgbe driver with your own >> syslogs. At least you will be able to see the signalling between vf >> and ixgbe drivers. >> >> -Qinglai >> >> >> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren >> wrote: >>> This is my fourth mail in my desperate attempt to get DPDK running in >>> KVM and no comments so far, not even any "it works for me". Am I the >>> only one crazy enough to believe that this can be done? >>> >>> Anyway, out of desperation I tried to get it running without having >>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed >>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM >>> Virtual Machine Monitor", using the PMD version of the instructions. >>> >>> I get as far as being able to see my four virtual functions in the >>> guest using "lspci". But starting the DPDK application gives me the >>> following error: >>> >>> PMD:The MAC address is not valid. >>> The most likely cause of this error is that the VM host >>> has not assigned a valid MAC address to this VF device. >>> Please consult the DPDK Release Notes (FAQ section) for >>> a possible solution to this problem. >>> >>> This may be true, but without any kernel modules loaded, how am I >>> supposed to change any MAC addresses? Can this be done from within >>> DPDK? >>> >>> As a side-note, I did try to load ixgbevf in the guest, but it >>> produced no interfaces. There was no error messages in the syslog >>> though. >>> >>> Is it possible to get X540 working in a guest or should I switch hardware? >>> >>> Since the instructions assumes I know the command line to KVM to start >>> my guest (which I do not), I cannot followed them precisely. I use >>> virsh and XML file, and maybe I've misunderstood how to translate the >>> pci-assign parameter to XML code. I currently use a entry, >>> but I've also tried . Neither has been >>> working for me so far, though the version got me as far as >>> being able to receive packets at least, but not transmitting. >>> >>> Regards >>> Mats >>> >>> >>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren >>> wrote: Never mind, I was hit by the infamous MAC spoofing... I got it working on both the host and the guest using ixgbevf driver, so apparently the cables are correctly attached. Using DPDK is still no-go. It can receive packets, but when sending the packets the function returns success, but the driver reports nothing (i.e. no errors, no sent packets, no nothing, except for received packets of course). What could cause this behavior? Regards Mats On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren wrote: > I have a follow-up on this: > > ixgbe version 3.13.10-k > ixgbevf version 2.7.12-k > > (These are what was provided by Ubuntu 13.10) > > I tried the following sequence on the host, before starting the guest: > 1) sudo rmmod ixgbe > 2) sudo modprobe ixgbe max_vfs=2 > 3) sudo ifconfig em1 up # This is the physical function > 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function > 5) ping 192.168.2.1 > > I can see that the ping request reaches its target, and a reply is > sent back. But this reply is not received by the ping shell command. > > Why? > > Regards, > Mats > > On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren > wrote:
[dpdk-dev] How to debug packet sends to virtual functions
Hi Anatoly, Thanks for the clarification. Then I can't use those instructions, which is what I was starting to suspect. I have to find another way of getting DPDK running in the guest then. Using ixgbevf in the guest works fine, though. Regards Mats On Tue, Feb 4, 2014 at 12:21 PM, Burakov, Anatoly wrote: >> -Original Message- >> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Mats Liljegren >> Sent: Tuesday, February 04, 2014 10:45 AM >> To: jigsaw >> Cc: dev at dpdk.org >> Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions >> >> Hi Qinglai, >> >> Thanks for the response! >> >> My previous attempt was with ixgbe loaded in the host. I also needed to load >> ixgbevf, but this seems to be because of a short-coming in libvirt. Maybe >> loading ixgbevf and then unbind it when running the guest is what causes my >> problems. I could receive packets with this setup, but not transmit. I used >> extra debug to syslog, and it showed that packets was placed in the transmit >> queue, but these packets was never sent. >> >> I'll see if I can get this working without loading ixgbevf in the host. >> >> What instructions did you follow to get this working? Did you do any >> deviation from the instructions? >> >> Regards >> Mats >> >> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: >> > Hi Mats, >> > >> > I've tried vf with 82599EB and it works fine. As long as the VF is >> > visible in guest, DPDK's VF driver should work just as ixgbevf, which >> > shares more or less the same code. >> > >> > I don't understand why you would expect DPDK at guest to work as VF, >> > while the host has no ixgbe loaded. >> > To make further debug I'd suggest compile ixgbe driver with your own >> > syslogs. At least you will be able to see the signalling between vf >> > and ixgbe drivers. >> > >> > -Qinglai >> > >> > >> > On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren >> > wrote: >> >> This is my fourth mail in my desperate attempt to get DPDK running in >> >> KVM and no comments so far, not even any "it works for me". Am I the >> >> only one crazy enough to believe that this can be done? >> >> >> >> Anyway, out of desperation I tried to get it running without having >> >> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed >> >> the instructions in the Programmer's Guide, chapter "Setting Up a KVM >> >> Virtual Machine Monitor", using the PMD version of the instructions. >> >> >> >> I get as far as being able to see my four virtual functions in the >> >> guest using "lspci". But starting the DPDK application gives me the >> >> following error: >> >> >> >> PMD:The MAC address is not valid. >> >> The most likely cause of this error is that the VM host >> >> has not assigned a valid MAC address to this VF device. >> >> Please consult the DPDK Release Notes (FAQ section) for >> >> a possible solution to this problem. >> >> >> >> This may be true, but without any kernel modules loaded, how am I >> >> supposed to change any MAC addresses? Can this be done from within >> >> DPDK? >> >> >> >> As a side-note, I did try to load ixgbevf in the guest, but it >> >> produced no interfaces. There was no error messages in the syslog >> >> though. >> >> >> >> Is it possible to get X540 working in a guest or should I switch hardware? >> >> >> >> Since the instructions assumes I know the command line to KVM to >> >> start my guest (which I do not), I cannot followed them precisely. I >> >> use virsh and XML file, and maybe I've misunderstood how to translate >> >> the pci-assign parameter to XML code. I currently use a >> >> entry, but I've also tried . Neither has >> >> been working for me so far, though the version got me as >> >> far as being able to receive packets at least, but not transmitting. >> >> >> >> Regards >> >> Mats >> >> >> >> >> >> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren >> >> wrote: >> >>> Never mind, I was hit by the infamous MAC spoofing... I got it >> >>> working on both the host and the guest using ixgbevf driver, so >> >>> apparently the c
[dpdk-dev] How to debug packet sends to virtual functions
Hi Qinglai, How did you configure the PCI passthrough in KVM? Did you use command line parameter as described in the instructions, or do you use virsh XML files, or maybe using virt-manager? The steps that works best (i.e. can receive but not transmit) are: 1) sudo modprobe ixgbe max_vfs=2 2) sudo ip link set em1 vf 0 mac 3) sudo ifconfig em1 up 4) sudo modprobe pci-stub 5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id 6) echo ":02:10.0" | sudo tee /sys/bus/pci/devices/\:02\:10.0/driver/unbind 7) echo ":02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind 8) virsh create vm vm.xml The guest then does: 5) sudo modprobe uio 6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio :02:10.0 7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio 8) sudo The virsh xml file has the following entry for the virtual function: I actually have two virtual functions and two physical functions, but I just repeat above for each. Regards Mats On Tue, Feb 4, 2014 at 12:14 PM, jigsaw wrote: > Hi Mats, > > I didn't have any deviation. What I did is just loading ixgbe (with > extra params for vf, as you mentioned in your first email), > and DPDK is up and running in guest. Of course I also followed section > 8.10 of the DPDK release notes. > > I can switch between DPDK and ixgebvf in guest at runtime and > everything works fine. > > Sorry I can't help to debug coz I have only 82599EB at hand. > > -Qinglai > > On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren > wrote: >> Hi Qinglai, >> >> Thanks for the response! >> >> My previous attempt was with ixgbe loaded in the host. I also needed >> to load ixgbevf, but this seems to be because of a short-coming in >> libvirt. Maybe loading ixgbevf and then unbind it when running the >> guest is what causes my problems. I could receive packets with this >> setup, but not transmit. I used extra debug to syslog, and it showed >> that packets was placed in the transmit queue, but these packets was >> never sent. >> >> I'll see if I can get this working without loading ixgbevf in the host. >> >> What instructions did you follow to get this working? Did you do any >> deviation from the instructions? >> >> Regards >> Mats >> >> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: >>> Hi Mats, >>> >>> I've tried vf with 82599EB and it works fine. As long as the VF is >>> visible in guest, DPDK's VF driver should work just as ixgbevf, which >>> shares more or less the same code. >>> >>> I don't understand why you would expect DPDK at guest to work as VF, >>> while the host has no ixgbe loaded. >>> To make further debug I'd suggest compile ixgbe driver with your own >>> syslogs. At least you will be able to see the signalling between vf >>> and ixgbe drivers. >>> >>> -Qinglai >>> >>> >>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren >>> wrote: This is my fourth mail in my desperate attempt to get DPDK running in KVM and no comments so far, not even any "it works for me". Am I the only one crazy enough to believe that this can be done? Anyway, out of desperation I tried to get it running without having ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed the instructions in the Programmer's Guide, chapter "Setting Up a KVM Virtual Machine Monitor", using the PMD version of the instructions. I get as far as being able to see my four virtual functions in the guest using "lspci". But starting the DPDK application gives me the following error: PMD:The MAC address is not valid. The most likely cause of this error is that the VM host has not assigned a valid MAC address to this VF device. Please consult the DPDK Release Notes (FAQ section) for a possible solution to this problem. This may be true, but without any kernel modules loaded, how am I supposed to change any MAC addresses? Can this be done from within DPDK? As a side-note, I did try to load ixgbevf in the guest, but it produced no interfaces. There was no error messages in the syslog though. Is it possible to get X540 working in a guest or should I switch hardware? Since the instructions assumes I know the command line to KVM to start my guest (which I do not), I cannot followed them precisely. I use virsh and XML file, and maybe I've misunderstood how to translate the pci-assign parameter to XML code. I currently use a entry, but I've also tried . Neither has been working for me so far, though the version got me as far as being able to receive packets at least, but not transmitting. Regards Mats On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren wrote: > Never mind, I was hit by the infamous MAC spoofing... I got it working > on both the host and the guest using ixgbevf driver, so apparently the >
[dpdk-dev] How to debug packet sends to virtual functions
Hi Mats, I've tried vf with 82599EB and it works fine. As long as the VF is visible in guest, DPDK's VF driver should work just as ixgbevf, which shares more or less the same code. I don't understand why you would expect DPDK at guest to work as VF, while the host has no ixgbe loaded. To make further debug I'd suggest compile ixgbe driver with your own syslogs. At least you will be able to see the signalling between vf and ixgbe drivers. -Qinglai On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren wrote: > This is my fourth mail in my desperate attempt to get DPDK running in > KVM and no comments so far, not even any "it works for me". Am I the > only one crazy enough to believe that this can be done? > > Anyway, out of desperation I tried to get it running without having > ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed > the instructions in the Programmer's Guide, chapter "Setting Up a KVM > Virtual Machine Monitor", using the PMD version of the instructions. > > I get as far as being able to see my four virtual functions in the > guest using "lspci". But starting the DPDK application gives me the > following error: > > PMD:The MAC address is not valid. > The most likely cause of this error is that the VM host > has not assigned a valid MAC address to this VF device. > Please consult the DPDK Release Notes (FAQ section) for > a possible solution to this problem. > > This may be true, but without any kernel modules loaded, how am I > supposed to change any MAC addresses? Can this be done from within > DPDK? > > As a side-note, I did try to load ixgbevf in the guest, but it > produced no interfaces. There was no error messages in the syslog > though. > > Is it possible to get X540 working in a guest or should I switch hardware? > > Since the instructions assumes I know the command line to KVM to start > my guest (which I do not), I cannot followed them precisely. I use > virsh and XML file, and maybe I've misunderstood how to translate the > pci-assign parameter to XML code. I currently use a entry, > but I've also tried . Neither has been > working for me so far, though the version got me as far as > being able to receive packets at least, but not transmitting. > > Regards > Mats > > > On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren > wrote: >> Never mind, I was hit by the infamous MAC spoofing... I got it working >> on both the host and the guest using ixgbevf driver, so apparently the >> cables are correctly attached. >> >> Using DPDK is still no-go. It can receive packets, but when sending >> the packets the function returns success, but the driver reports >> nothing (i.e. no errors, no sent packets, no nothing, except for >> received packets of course). >> >> What could cause this behavior? >> >> Regards >> Mats >> >> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren >> wrote: >>> I have a follow-up on this: >>> >>> ixgbe version 3.13.10-k >>> ixgbevf version 2.7.12-k >>> >>> (These are what was provided by Ubuntu 13.10) >>> >>> I tried the following sequence on the host, before starting the guest: >>> 1) sudo rmmod ixgbe >>> 2) sudo modprobe ixgbe max_vfs=2 >>> 3) sudo ifconfig em1 up # This is the physical function >>> 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function >>> 5) ping 192.168.2.1 >>> >>> I can see that the ping request reaches its target, and a reply is >>> sent back. But this reply is not received by the ping shell command. >>> >>> Why? >>> >>> Regards, >>> Mats >>> >>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren >>> wrote: I'm trying to get a modified version of the l2fwd example running, and have problems with packets being silently thrown away. I can receive packets, and my printf's indicates that the packets are being sent to the correct port, using correct MAC address as source address. And still, the packets are lost. Since the port is a virtual function, it seems like I cannot use tcpdump on it to see the network traffic. There is nothing coming out of the cable (activity light not flashing, the receiving end running tcpdump does not see any traffic). I'm using two X540 with two virtual functions each. The application runs in a KVM/Qemu environmen. Any suggestions how to debug this? Regards, Mats
[dpdk-dev] How to debug packet sends to virtual functions
> -Original Message- > From: Mats Liljegren [mailto:liljegren.mats2 at gmail.com] > Sent: Tuesday, February 04, 2014 11:48 AM > To: Burakov, Anatoly > Cc: dev at dpdk.org > Subject: Re: [dpdk-dev] How to debug packet sends to virtual functions > > Hi Anatoly, > > Thanks for the clarification. Then I can't use those instructions, which is > what I > was starting to suspect. I have to find another way of getting DPDK running in > the guest then. Using ixgbevf in the guest works fine, though. > > Regards > Mats > Hi Mats Let me clarify - do you run DPDK on both host and guest? E.g. you load igb_uio on the host as well? If not, I don't see why you can't use those instructions - you're creating VF devices on the host anyway, and that's where you should set their MAC addresses. Best regards, Anatoly Burakov DPDK SW Engineer -- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare
[dpdk-dev] How to debug packet sends to virtual functions
Hi Qinglai, Thanks for the response! My previous attempt was with ixgbe loaded in the host. I also needed to load ixgbevf, but this seems to be because of a short-coming in libvirt. Maybe loading ixgbevf and then unbind it when running the guest is what causes my problems. I could receive packets with this setup, but not transmit. I used extra debug to syslog, and it showed that packets was placed in the transmit queue, but these packets was never sent. I'll see if I can get this working without loading ixgbevf in the host. What instructions did you follow to get this working? Did you do any deviation from the instructions? Regards Mats On Tue, Feb 4, 2014 at 11:26 AM, jigsaw wrote: > Hi Mats, > > I've tried vf with 82599EB and it works fine. As long as the VF is > visible in guest, DPDK's VF driver should work just as ixgbevf, which > shares more or less the same code. > > I don't understand why you would expect DPDK at guest to work as VF, > while the host has no ixgbe loaded. > To make further debug I'd suggest compile ixgbe driver with your own > syslogs. At least you will be able to see the signalling between vf > and ixgbe drivers. > > -Qinglai > > > On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren > wrote: >> This is my fourth mail in my desperate attempt to get DPDK running in >> KVM and no comments so far, not even any "it works for me". Am I the >> only one crazy enough to believe that this can be done? >> >> Anyway, out of desperation I tried to get it running without having >> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed >> the instructions in the Programmer's Guide, chapter "Setting Up a KVM >> Virtual Machine Monitor", using the PMD version of the instructions. >> >> I get as far as being able to see my four virtual functions in the >> guest using "lspci". But starting the DPDK application gives me the >> following error: >> >> PMD:The MAC address is not valid. >> The most likely cause of this error is that the VM host >> has not assigned a valid MAC address to this VF device. >> Please consult the DPDK Release Notes (FAQ section) for >> a possible solution to this problem. >> >> This may be true, but without any kernel modules loaded, how am I >> supposed to change any MAC addresses? Can this be done from within >> DPDK? >> >> As a side-note, I did try to load ixgbevf in the guest, but it >> produced no interfaces. There was no error messages in the syslog >> though. >> >> Is it possible to get X540 working in a guest or should I switch hardware? >> >> Since the instructions assumes I know the command line to KVM to start >> my guest (which I do not), I cannot followed them precisely. I use >> virsh and XML file, and maybe I've misunderstood how to translate the >> pci-assign parameter to XML code. I currently use a entry, >> but I've also tried . Neither has been >> working for me so far, though the version got me as far as >> being able to receive packets at least, but not transmitting. >> >> Regards >> Mats >> >> >> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren >> wrote: >>> Never mind, I was hit by the infamous MAC spoofing... I got it working >>> on both the host and the guest using ixgbevf driver, so apparently the >>> cables are correctly attached. >>> >>> Using DPDK is still no-go. It can receive packets, but when sending >>> the packets the function returns success, but the driver reports >>> nothing (i.e. no errors, no sent packets, no nothing, except for >>> received packets of course). >>> >>> What could cause this behavior? >>> >>> Regards >>> Mats >>> >>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren >>> wrote: I have a follow-up on this: ixgbe version 3.13.10-k ixgbevf version 2.7.12-k (These are what was provided by Ubuntu 13.10) I tried the following sequence on the host, before starting the guest: 1) sudo rmmod ixgbe 2) sudo modprobe ixgbe max_vfs=2 3) sudo ifconfig em1 up # This is the physical function 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function 5) ping 192.168.2.1 I can see that the ping request reaches its target, and a reply is sent back. But this reply is not received by the ping shell command. Why? Regards, Mats On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren wrote: > I'm trying to get a modified version of the l2fwd example running, and > have problems with packets being silently thrown away. I can receive > packets, and my printf's indicates that the packets are being sent to > the correct port, using correct MAC address as source address. And > still, the packets are lost. > > Since the port is a virtual function, it seems like I cannot use > tcpdump on it to see the network traffic. There is nothing coming out > of the cable (activity light not flashing, the receiving end running > tcpdump does not see any
[dpdk-dev] How to debug packet sends to virtual functions
This is my fourth mail in my desperate attempt to get DPDK running in KVM and no comments so far, not even any "it works for me". Am I the only one crazy enough to believe that this can be done? Anyway, out of desperation I tried to get it running without having ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed the instructions in the Programmer's Guide, chapter "Setting Up a KVM Virtual Machine Monitor", using the PMD version of the instructions. I get as far as being able to see my four virtual functions in the guest using "lspci". But starting the DPDK application gives me the following error: PMD:The MAC address is not valid. The most likely cause of this error is that the VM host has not assigned a valid MAC address to this VF device. Please consult the DPDK Release Notes (FAQ section) for a possible solution to this problem. This may be true, but without any kernel modules loaded, how am I supposed to change any MAC addresses? Can this be done from within DPDK? As a side-note, I did try to load ixgbevf in the guest, but it produced no interfaces. There was no error messages in the syslog though. Is it possible to get X540 working in a guest or should I switch hardware? Since the instructions assumes I know the command line to KVM to start my guest (which I do not), I cannot followed them precisely. I use virsh and XML file, and maybe I've misunderstood how to translate the pci-assign parameter to XML code. I currently use a entry, but I've also tried . Neither has been working for me so far, though the version got me as far as being able to receive packets at least, but not transmitting. Regards Mats On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren wrote: > Never mind, I was hit by the infamous MAC spoofing... I got it working > on both the host and the guest using ixgbevf driver, so apparently the > cables are correctly attached. > > Using DPDK is still no-go. It can receive packets, but when sending > the packets the function returns success, but the driver reports > nothing (i.e. no errors, no sent packets, no nothing, except for > received packets of course). > > What could cause this behavior? > > Regards > Mats > > On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren > wrote: >> I have a follow-up on this: >> >> ixgbe version 3.13.10-k >> ixgbevf version 2.7.12-k >> >> (These are what was provided by Ubuntu 13.10) >> >> I tried the following sequence on the host, before starting the guest: >> 1) sudo rmmod ixgbe >> 2) sudo modprobe ixgbe max_vfs=2 >> 3) sudo ifconfig em1 up # This is the physical function >> 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function >> 5) ping 192.168.2.1 >> >> I can see that the ping request reaches its target, and a reply is >> sent back. But this reply is not received by the ping shell command. >> >> Why? >> >> Regards, >> Mats >> >> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren >> wrote: >>> I'm trying to get a modified version of the l2fwd example running, and >>> have problems with packets being silently thrown away. I can receive >>> packets, and my printf's indicates that the packets are being sent to >>> the correct port, using correct MAC address as source address. And >>> still, the packets are lost. >>> >>> Since the port is a virtual function, it seems like I cannot use >>> tcpdump on it to see the network traffic. There is nothing coming out >>> of the cable (activity light not flashing, the receiving end running >>> tcpdump does not see any traffic). >>> >>> I'm using two X540 with two virtual functions each. The application >>> runs in a KVM/Qemu environmen. >>> >>> Any suggestions how to debug this? >>> >>> Regards, >>> Mats
[dpdk-dev] How to debug packet sends to virtual functions
Never mind, I was hit by the infamous MAC spoofing... I got it working on both the host and the guest using ixgbevf driver, so apparently the cables are correctly attached. Using DPDK is still no-go. It can receive packets, but when sending the packets the function returns success, but the driver reports nothing (i.e. no errors, no sent packets, no nothing, except for received packets of course). What could cause this behavior? Regards Mats On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren wrote: > I have a follow-up on this: > > ixgbe version 3.13.10-k > ixgbevf version 2.7.12-k > > (These are what was provided by Ubuntu 13.10) > > I tried the following sequence on the host, before starting the guest: > 1) sudo rmmod ixgbe > 2) sudo modprobe ixgbe max_vfs=2 > 3) sudo ifconfig em1 up # This is the physical function > 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function > 5) ping 192.168.2.1 > > I can see that the ping request reaches its target, and a reply is > sent back. But this reply is not received by the ping shell command. > > Why? > > Regards, > Mats > > On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren > wrote: >> I'm trying to get a modified version of the l2fwd example running, and >> have problems with packets being silently thrown away. I can receive >> packets, and my printf's indicates that the packets are being sent to >> the correct port, using correct MAC address as source address. And >> still, the packets are lost. >> >> Since the port is a virtual function, it seems like I cannot use >> tcpdump on it to see the network traffic. There is nothing coming out >> of the cable (activity light not flashing, the receiving end running >> tcpdump does not see any traffic). >> >> I'm using two X540 with two virtual functions each. The application >> runs in a KVM/Qemu environmen. >> >> Any suggestions how to debug this? >> >> Regards, >> Mats
[dpdk-dev] How to debug packet sends to virtual functions
I have a follow-up on this: ixgbe version 3.13.10-k ixgbevf version 2.7.12-k (These are what was provided by Ubuntu 13.10) I tried the following sequence on the host, before starting the guest: 1) sudo rmmod ixgbe 2) sudo modprobe ixgbe max_vfs=2 3) sudo ifconfig em1 up # This is the physical function 4) sudo ifconfig em1_0 192.168.2.2 # This is the virtual function 5) ping 192.168.2.1 I can see that the ping request reaches its target, and a reply is sent back. But this reply is not received by the ping shell command. Why? Regards, Mats On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren wrote: > I'm trying to get a modified version of the l2fwd example running, and > have problems with packets being silently thrown away. I can receive > packets, and my printf's indicates that the packets are being sent to > the correct port, using correct MAC address as source address. And > still, the packets are lost. > > Since the port is a virtual function, it seems like I cannot use > tcpdump on it to see the network traffic. There is nothing coming out > of the cable (activity light not flashing, the receiving end running > tcpdump does not see any traffic). > > I'm using two X540 with two virtual functions each. The application > runs in a KVM/Qemu environmen. > > Any suggestions how to debug this? > > Regards, > Mats
[dpdk-dev] How to debug packet sends to virtual functions
I'm trying to get a modified version of the l2fwd example running, and have problems with packets being silently thrown away. I can receive packets, and my printf's indicates that the packets are being sent to the correct port, using correct MAC address as source address. And still, the packets are lost. Since the port is a virtual function, it seems like I cannot use tcpdump on it to see the network traffic. There is nothing coming out of the cable (activity light not flashing, the receiving end running tcpdump does not see any traffic). I'm using two X540 with two virtual functions each. The application runs in a KVM/Qemu environmen. Any suggestions how to debug this? Regards, Mats
