Re: ixv device_attach: ixv0 attach returned 5 (was sr-iov issues, reset_hw() failed with error -100)
Jeb, Okay, just wanted to make sure it was known. =] Ultima On Thu, Apr 28, 2016 at 4:36 PM, cramerj wrote: > Yeah, we're running into some issues with our implementation. I'm working > on the VF driver now and hope to get some of these bugs fixed within the > next couple weeks. > > Thanks, > -Jeb > > -Original Message- > From: owner-freebsd-curr...@freebsd.org [mailto: > owner-freebsd-curr...@freebsd.org] On Behalf Of Ultima > Sent: Sunday, April 24, 2016 8:32 PM > Cc: freebsd-curr...@freebsd.org; freebsd-virtualization@freebsd.org; > freebsd-hardw...@freebsd.org > Subject: ixv device_attach: ixv0 attach returned 5 (was sr-iov issues, > reset_hw() failed with error -100) > > The sr-iov vf driver is failing to attach. > > > # pciconf -lv: (filtered to only relevant output) > ix0@pci0:129:0:0: class=0x02 card=0x1458 chip=0x15288086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Ethernet Controller 10-Gigabit X540-AT2' > class = network > subclass = ethernet > ix1@pci0:129:0:1: class=0x02 card=0x1458 chip=0x15288086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Ethernet Controller 10-Gigabit X540-AT2' > class = network > subclass = ethernet > none155@pci0:129:0:129: class=0x02 card=0x1458 chip=0x15158086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = 'X540 Ethernet Controller Virtual Function' > class = network > subclass = ethernet > none156@pci0:129:0:131: class=0x02 card=0x1458 chip=0x15158086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = 'X540 Ethernet Controller Virtual Function' > class = network > subclass = ethernet > > # devctl attach pci0:129:0:129 > devctl: Failed to attach pci0:129:0:129: Input/output error > > # dmesg > ixv0: 1.4.6-k> at device 0.129 numa-domain 1 on pci12 > ixv0: Using MSIX interrupts with 2 vectors > ixv0: ixgbe_reset_hw() failed with error -100 > device_attach: ixv0 attach returned 5 > > # cat /etc/iovctl.conf > PF { > device : ix1; > num_vfs : 31; > } > > DEFAULT { > passthrough : true; > } > VF-0 { > passthrough : false; > } > VF-1 { > passthrough : false; > } > > > Any ideas? > > Ultima > ___ > freebsd-curr...@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
ixv device_attach: ixv0 attach returned 5 (was sr-iov issues, reset_hw() failed with error -100)
The sr-iov vf driver is failing to attach. # pciconf -lv: (filtered to only relevant output) ix0@pci0:129:0:0: class=0x02 card=0x1458 chip=0x15288086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller 10-Gigabit X540-AT2' class = network subclass = ethernet ix1@pci0:129:0:1: class=0x02 card=0x1458 chip=0x15288086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller 10-Gigabit X540-AT2' class = network subclass = ethernet none155@pci0:129:0:129: class=0x02 card=0x1458 chip=0x15158086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'X540 Ethernet Controller Virtual Function' class = network subclass = ethernet none156@pci0:129:0:131: class=0x02 card=0x1458 chip=0x15158086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'X540 Ethernet Controller Virtual Function' class = network subclass = ethernet # devctl attach pci0:129:0:129 devctl: Failed to attach pci0:129:0:129: Input/output error # dmesg ixv0: at device 0.129 numa-domain 1 on pci12 ixv0: Using MSIX interrupts with 2 vectors ixv0: ixgbe_reset_hw() failed with error -100 device_attach: ixv0 attach returned 5 # cat /etc/iovctl.conf PF { device : ix1; num_vfs : 31; } DEFAULT { passthrough : true; } VF-0 { passthrough : false; } VF-1 { passthrough : false; } Any ideas? Ultima ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: sr-iov issues, reset_hw() failed with error -100
Recently upgraded to r297351, once running iovctl -C -f /etc/iovctl.conf everything appears to work as it should. The main network interface ramains connected and pinging works fine. iovctl.conf PF { device : ix1; num_vfs : 31; } DEFAULT { passthrough : true; } VF-0 { passthrough : false; } VF-1 { passthrough : false; } Once vf's have been initialized, I have found none-passthrough vf's are unusable. # devctl attach pci0:129:0:129 devctl: Failed to attach pci0:129:0:129: Input/output error The dmesg spits out the same error as before, so it appears that the errors I was mentioning before is actually the none-passthrough vf's attempting to attach, but fails. /var/log/messages: ixv0: at device 0.131 on pci12 ixv0: Using MSIX interrupts with 2 vectors ixv0: ixgbe_reset_hw() failed with error -100 device_attach: ixv0 attach returned 5 ixv0: at device 0.129 on pci12 ixv0: Using MSIX interrupts with 2 vectors ixv0: ixgbe_reset_hw() failed with error -100 device_attach: ixv0 attach returned 5 On Wed, Mar 9, 2016 at 5:36 PM, Ultima wrote: > I have been interested in this when I first read about it in 2012. =] > > Can this be done in loader.conf? I have vmm_load="YES" > > I'm not sure if the vf's are usable, I have not actually tested the vf's. > The parent ix1 still shows no response. > > kldunload vmm > > kenv hw.vmm.force_iommu=1 > > kldload vmm > > iovctl -Cf /etc/iovctl.conf > > The same error messages appear, I currently on hmm i'm not sure, I > upgraded with git and it doesn't show rev(last time I use git for > source?) 8b372d1(master) > > On Wed, Mar 9, 2016 at 5:00 PM, Eric Joyner wrote: > >> I don't know if you're still interested in this, but did you do "kenv >> hw.vmm.force_iommu=1" before loading the vmm module? I think that might be >> necessary. >> >> On Wed, Feb 24, 2016 at 5:12 PM Ultima wrote: >> >>> Yeah, still getting the -100 error, I do have sendmail disabled. I just >>> tested with sendmail up and running then add the VF's and it still shows >>> the error message. >>> >>> On Wed, Feb 24, 2016 at 8:04 PM, Eric Joyner wrote: >>> >>>> Are you still getting the -100 errors when trying to load the VF driver? >>>> >>>> I've tried SR-IOV on a system here, and I can confirm that traffic >>>> stops passing on the PF interface when you create a VF interface. That >>>> didn't used to happen, so I'm investigating why that is right now. >>>> >>>> On Wed, Feb 24, 2016 at 8:09 AM Ultima wrote: >>>> >>>>> Decided to do some more tests, I actually have a second board with >>>>> sr-iov >>>>> capabilities that I used for awhile with vmware esxi. I decided to test >>>>> this out and unfortunately it won't activate, it is giving the no space >>>>> left on device error message. I double checked bios and all VT-d >>>>> related >>>>> options are enabled and have hw.ix.num_queues="4" in >>>>> /boot/loader.conf. Is >>>>> there anything else that may need to be set? .(It did work on vmware) >>>>> >>>>> For my second test, I moved the X540-AT1 to the board with the >>>>> X540-AT2. >>>>> It functioned with the same issues as the AT2 tho. >>>>> >>>>> >>>>> I don't think I listed the motherboards in question yet so ill list >>>>> them >>>>> now. >>>>> >>>>> S1200BTLRM - >>>>> >>>>> http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM >>>>> MD80-TM0 >>>>> <http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRMMD80-TM0> >>>>> - http://b2b.gigabyte.com/products/product-page.aspx?pid=5146#ov >>>>> >>>>> I'm not sure if it will be of any help tho. >>>>> >>>>> Ultima >>>>> ___ >>>>> freebsd-curr...@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>>>> To unsubscribe, send any mail to " >>>>> freebsd-current-unsubscr...@freebsd.org" >>>>> >>>> >>> > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: sr-iov issues, reset_hw() failed with error -100
I have been interested in this when I first read about it in 2012. =] Can this be done in loader.conf? I have vmm_load="YES" I'm not sure if the vf's are usable, I have not actually tested the vf's. The parent ix1 still shows no response. kldunload vmm kenv hw.vmm.force_iommu=1 kldload vmm iovctl -Cf /etc/iovctl.conf The same error messages appear, I currently on hmm i'm not sure, I upgraded with git and it doesn't show rev(last time I use git for source?) 8b372d1(master) On Wed, Mar 9, 2016 at 5:00 PM, Eric Joyner wrote: > I don't know if you're still interested in this, but did you do "kenv > hw.vmm.force_iommu=1" before loading the vmm module? I think that might be > necessary. > > On Wed, Feb 24, 2016 at 5:12 PM Ultima wrote: > >> Yeah, still getting the -100 error, I do have sendmail disabled. I just >> tested with sendmail up and running then add the VF's and it still shows >> the error message. >> >> On Wed, Feb 24, 2016 at 8:04 PM, Eric Joyner wrote: >> >>> Are you still getting the -100 errors when trying to load the VF driver? >>> >>> I've tried SR-IOV on a system here, and I can confirm that traffic stops >>> passing on the PF interface when you create a VF interface. That didn't >>> used to happen, so I'm investigating why that is right now. >>> >>> On Wed, Feb 24, 2016 at 8:09 AM Ultima wrote: >>> >>>> Decided to do some more tests, I actually have a second board with >>>> sr-iov >>>> capabilities that I used for awhile with vmware esxi. I decided to test >>>> this out and unfortunately it won't activate, it is giving the no space >>>> left on device error message. I double checked bios and all VT-d related >>>> options are enabled and have hw.ix.num_queues="4" in /boot/loader.conf. >>>> Is >>>> there anything else that may need to be set? .(It did work on vmware) >>>> >>>> For my second test, I moved the X540-AT1 to the board with the >>>> X540-AT2. >>>> It functioned with the same issues as the AT2 tho. >>>> >>>> >>>> I don't think I listed the motherboards in question yet so ill list them >>>> now. >>>> >>>> S1200BTLRM - >>>> >>>> http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM >>>> MD80-TM0 >>>> <http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRMMD80-TM0> >>>> - http://b2b.gigabyte.com/products/product-page.aspx?pid=5146#ov >>>> >>>> I'm not sure if it will be of any help tho. >>>> >>>> Ultima >>>> ___ >>>> freebsd-curr...@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to " >>>> freebsd-current-unsubscr...@freebsd.org" >>>> >>> >> ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: sr-iov issues, reset_hw() failed with error -100
Yeah, still getting the -100 error, I do have sendmail disabled. I just tested with sendmail up and running then add the VF's and it still shows the error message. On Wed, Feb 24, 2016 at 8:04 PM, Eric Joyner wrote: > Are you still getting the -100 errors when trying to load the VF driver? > > I've tried SR-IOV on a system here, and I can confirm that traffic stops > passing on the PF interface when you create a VF interface. That didn't > used to happen, so I'm investigating why that is right now. > > On Wed, Feb 24, 2016 at 8:09 AM Ultima wrote: > >> Decided to do some more tests, I actually have a second board with sr-iov >> capabilities that I used for awhile with vmware esxi. I decided to test >> this out and unfortunately it won't activate, it is giving the no space >> left on device error message. I double checked bios and all VT-d related >> options are enabled and have hw.ix.num_queues="4" in /boot/loader.conf. Is >> there anything else that may need to be set? .(It did work on vmware) >> >> For my second test, I moved the X540-AT1 to the board with the X540-AT2. >> It functioned with the same issues as the AT2 tho. >> >> >> I don't think I listed the motherboards in question yet so ill list them >> now. >> >> S1200BTLRM - >> >> http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM >> MD80-TM0 >> <http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRMMD80-TM0> >> - http://b2b.gigabyte.com/products/product-page.aspx?pid=5146#ov >> >> I'm not sure if it will be of any help tho. >> >> Ultima >> ___ >> freebsd-curr...@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org >> " >> > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: sr-iov issues, reset_hw() failed with error -100
Decided to do some more tests, I actually have a second board with sr-iov capabilities that I used for awhile with vmware esxi. I decided to test this out and unfortunately it won't activate, it is giving the no space left on device error message. I double checked bios and all VT-d related options are enabled and have hw.ix.num_queues="4" in /boot/loader.conf. Is there anything else that may need to be set? .(It did work on vmware) For my second test, I moved the X540-AT1 to the board with the X540-AT2. It functioned with the same issues as the AT2 tho. I don't think I listed the motherboards in question yet so ill list them now. S1200BTLRM - http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM MD80-TM0 - http://b2b.gigabyte.com/products/product-page.aspx?pid=5146#ov I'm not sure if it will be of any help tho. Ultima ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: sr-iov issues, reset_hw() failed with error -100
Upgraded to r295920 and used generic kernel. Then rebooted and checked bios thoroughly. I found 3, yes 3 different areas in the bios for enabling sr-iov (some screen shots below). 2 are for vt-d(Forgot to take a screenshot of the 2nt, all were enabled) and one for sr-iov(was disabled). Unfortunately with generic kernel, and all these options enabled, adding only 2vf's resulted in the same behavior as above. Little bit of good news, when removing the vf's and ifconfig ix1 down/up the interface's functionality is restored. I also tested this on MYKERNEL r295920, sr-iov option in bios likely played a role in this. If you have any ideas, I'm willing to test them. Thanks! =] VT-d screen: https://puu.sh/niyiq/4fee92e4a3.jpgn PCI advanced options: https://puu.sh/niyjg/88e71e48d9.jpgn Ultima On Mon, Feb 22, 2016 at 9:46 PM, Ultima wrote: > I forgot to mention my kernel conf I'm not sure if it would cause this > issue, but I'll test again with GENERIC. > > --- /usr/src/sys/amd64/conf/GENERIC 2016-02-22 21:05:37.152953000 -0500 > +++ /root/MYKERNEL-11-CURRENT-AMD64 2015-12-28 19:18:22.893391452 -0500 > @@ -91,6 +91,12 @@ > options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed > options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones > > +### VIMAGE ### > +options VIMAGE > + > +### ROUTE TABLES ### > +options ROUTETABLES=2 > + > # Make an SMP-capable kernel by default > options SMP # Symmetric MultiProcessor Kernel > > The interface is just dead after creating the vf's earlier. Recreating > with only 2, its still dead. > > Running out of ideas, decided to try a tcpdump... > > # dhclient ix1 > DHCPREQUEST on ix1 to 255.255.255.255 port 67 > DHCPREQUEST on ix1 to 255.255.255.255 port 67 > DHCPREQUEST on ix1 to 255.255.255.255 port 67 > DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 7 > DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 13 > DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 14 > > # tcpdump -i ix1 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on ix1, link-type EN10MB (Ethernet), capture size 262144 bytes > 21:41:13.234688 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: > BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 > 21:41:16.236671 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: > BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 > 21:41:23.243242 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: > BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 > 21:41:38.261015 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from -Hidden- (oui Unknown), length 300 > 21:41:45.284752 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from -Hidden- (oui Unknown), length 300 > 21:41:58.292223 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from -Hidden- (oui Unknown), length 300 > > I don't think this is that helpful tho. =/ > tcpdump on the other end, the packets are never received. > > I'v already double checked VT-d, but I'll make it tripple! =] I can't > reset the system right now, but ill send an update when possible with only > 2 vf's and unmodified generic. > > Ultima > > On Mon, Feb 22, 2016 at 8:52 PM, Eric Joyner wrote: > >> I don't really have any ideas on the error -100. Error -100 means there >> was a mailbox error, so something failed in the initial communications >> setup between the PF and VF, but I don't know what exactly went wrong. >> >> I'm grasping at straws, but try using a smaller number of VFs initially, >> like 2? And check to see if VT-d is enabled in your BIOS? (Though I >> would've expected iovctl to fail). >> >> - Eric >> >> >> On Mon, Feb 22, 2016 at 12:01 PM Ultima wrote: >> >>> After reboot... >>> >>> ifconfig ix1 up >>> >>> dhclient ix1 >>> DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 >>> DHCPOFFER from 192.168.1.1 >>> DHCPREQUEST on ix1 to 255.255.255.255 port 67 >>> DHCPACK from 192.168.1.1 >>> bound to 192.168.1.145 -- renewal in 21600 seconds. >>> >>> ix0 down >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms >>> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms >>> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms >>> >>> iovctl -Cf /etc/iovctl.conf >>> >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> ^
Re: sr-iov issues, reset_hw() failed with error -100
I forgot to mention my kernel conf I'm not sure if it would cause this issue, but I'll test again with GENERIC. --- /usr/src/sys/amd64/conf/GENERIC 2016-02-22 21:05:37.152953000 -0500 +++ /root/MYKERNEL-11-CURRENT-AMD64 2015-12-28 19:18:22.893391452 -0500 @@ -91,6 +91,12 @@ options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones +### VIMAGE ### +options VIMAGE + +### ROUTE TABLES ### +options ROUTETABLES=2 + # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel The interface is just dead after creating the vf's earlier. Recreating with only 2, its still dead. Running out of ideas, decided to try a tcpdump... # dhclient ix1 DHCPREQUEST on ix1 to 255.255.255.255 port 67 DHCPREQUEST on ix1 to 255.255.255.255 port 67 DHCPREQUEST on ix1 to 255.255.255.255 port 67 DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 7 DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 13 DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 14 # tcpdump -i ix1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix1, link-type EN10MB (Ethernet), capture size 262144 bytes 21:41:13.234688 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 21:41:16.236671 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 21:41:23.243242 IP 192.168.1.145.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 21:41:38.261015 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 21:41:45.284752 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 21:41:58.292223 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300 I don't think this is that helpful tho. =/ tcpdump on the other end, the packets are never received. I'v already double checked VT-d, but I'll make it tripple! =] I can't reset the system right now, but ill send an update when possible with only 2 vf's and unmodified generic. Ultima On Mon, Feb 22, 2016 at 8:52 PM, Eric Joyner wrote: > I don't really have any ideas on the error -100. Error -100 means there > was a mailbox error, so something failed in the initial communications > setup between the PF and VF, but I don't know what exactly went wrong. > > I'm grasping at straws, but try using a smaller number of VFs initially, > like 2? And check to see if VT-d is enabled in your BIOS? (Though I > would've expected iovctl to fail). > > - Eric > > > On Mon, Feb 22, 2016 at 12:01 PM Ultima wrote: > >> After reboot... >> >> ifconfig ix1 up >> >> dhclient ix1 >> DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 >> DHCPOFFER from 192.168.1.1 >> DHCPREQUEST on ix1 to 255.255.255.255 port 67 >> DHCPACK from 192.168.1.1 >> bound to 192.168.1.145 -- renewal in 21600 seconds. >> >> ix0 down >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms >> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms >> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms >> >> iovctl -Cf /etc/iovctl.conf >> >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 29 packets transmitted, 0 packets received, 100.0% packet loss >> ifconfig ix1 up >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 12 packets transmitted, 0 packets received, 100.0% packet loss >> >> ix1 is no longer usable until a restart... >> >> iovctl -Dd ix1 >> ifconfig ix1 up >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 9 packets transmitted, 0 packets received, 100.0% packet loss >> >> >> >> Is there anything else that maybe useful? >> >> here is my ifconfig at the end (after ifconfig ix0 up) >> >> >> ix0: flags=8943 metric 0 >> mtu 1500 >> >> options=e400b9 >> ether -Hidden- >> inet 192.168.1.8 netmask 0xff00 broadcast 192.168.1.255 >> inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 >> nd6 options=29 >> media: Ethernet autoselect (10Gbase-T ) >> status: active >> ix1: flags=8843 metric 0 mtu 1500 >> >> options=e407bb >> ether -Hidden- &
Re: sr-iov issues, reset_hw() failed with error -100
Yeah, dmesg does show 48. 12 Cores and 24 threads each. 48 total On Mon, Feb 22, 2016 at 4:35 PM, Steven Hartland wrote: > isn't that 48 cores (12 real 12 virtual) per CPU? > > > On 22/02/2016 21:26, Ultima wrote: > >> This system has 24 cores (e5-2670v3)x2 >> >> Ultima >> >> On Mon, Feb 22, 2016 at 3:53 PM, Pieper, Jeffrey E < >> jeffrey.e.pie...@intel.com> wrote: >> >> Just out of curiosity, how many cores does your system have? >>> >>> Jeff >>> >>> -Original Message- >>> From: owner-freebsd-curr...@freebsd.org [mailto: >>> owner-freebsd-curr...@freebsd.org] On Behalf Of Ultima >>> Sent: Monday, February 22, 2016 12:02 PM >>> To: Eric Joyner >>> Cc: freebsd-curr...@freebsd.org; freebsd-virtualization@freebsd.org >>> Subject: Re: sr-iov issues, reset_hw() failed with error -100 >>> >>> After reboot... >>> >>> ifconfig ix1 up >>> >>> dhclient ix1 >>> DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 >>> DHCPOFFER from 192.168.1.1 >>> DHCPREQUEST on ix1 to 255.255.255.255 port 67 >>> DHCPACK from 192.168.1.1 >>> bound to 192.168.1.145 -- renewal in 21600 seconds. >>> >>> ix0 down >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms >>> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms >>> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms >>> >>> iovctl -Cf /etc/iovctl.conf >>> >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> ^C >>> --- 192.168.1.1 ping statistics --- >>> 29 packets transmitted, 0 packets received, 100.0% packet loss >>> ifconfig ix1 up >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> ^C >>> --- 192.168.1.1 ping statistics --- >>> 12 packets transmitted, 0 packets received, 100.0% packet loss >>> >>> ix1 is no longer usable until a restart... >>> >>> iovctl -Dd ix1 >>> ifconfig ix1 up >>> ping 192.168.1.1 >>> PING 192.168.1.1 (192.168.1.1): 56 data bytes >>> ^C >>> --- 192.168.1.1 ping statistics --- >>> 9 packets transmitted, 0 packets received, 100.0% packet loss >>> >>> >>> >>> Is there anything else that maybe useful? >>> >>> here is my ifconfig at the end (after ifconfig ix0 up) >>> >>> >>> ix0: flags=8943 metric 0 >>> mtu 1500 >>> >>> >>> options=e400b9 >>> ether -Hidden- >>> inet 192.168.1.8 netmask 0xff00 broadcast 192.168.1.255 >>> inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 >>> nd6 options=29 >>> media: Ethernet autoselect (10Gbase-T ) >>> status: active >>> ix1: flags=8843 metric 0 mtu 1500 >>> >>> >>> options=e407bb >>> ether -Hidden- >>> inet 192.168.1.145 netmask 0xff00 broadcast 192.168.1.255 >>> nd6 options=29 >>> media: Ethernet autoselect (10Gbase-T ) >>> status: active >>> lo0: flags=8049 metric 0 mtu 16384 >>> options=63 >>> inet6 ::1 prefixlen 128 >>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >>> inet 127.0.0.1 netmask 0xff00 >>> nd6 options=21 >>> groups: lo >>> bridge0: flags=8843 metric 0 mtu >>> 1500 >>> ether -Hidden- >>> nd6 options=9 >>> groups: bridge >>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>> member: ix0 flags=143 >>> ifmaxaddr 0 port 1 priority 128 path cost 2000 >>> member: epair0a flags=143 >>> ifmaxaddr 0 port 5 priority 128 path cost 2000 >>> epair0a: flags=8943 >>> metric >>> 0 mtu 1500 >>> options=8 >>> ether -Hidden- >>> inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5 >>> nd6 options=21 >>> media: Ethernet 10Gbase-T (10Gbase-T ) >>> status: active >>> groups: epair >>> >>> On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner wrote: >>> >>> Did you do an ifconfig up on ix1 before loading the VF driver? >>>> >>>> On Sat, Feb 20, 2016 at 11:57 AM Ultima wrote: >>>> >>&g
Re: sr-iov issues, reset_hw() failed with error -100
This system has 24 cores (e5-2670v3)x2 Ultima On Mon, Feb 22, 2016 at 3:53 PM, Pieper, Jeffrey E < jeffrey.e.pie...@intel.com> wrote: > Just out of curiosity, how many cores does your system have? > > Jeff > > -Original Message- > From: owner-freebsd-curr...@freebsd.org [mailto: > owner-freebsd-curr...@freebsd.org] On Behalf Of Ultima > Sent: Monday, February 22, 2016 12:02 PM > To: Eric Joyner > Cc: freebsd-curr...@freebsd.org; freebsd-virtualization@freebsd.org > Subject: Re: sr-iov issues, reset_hw() failed with error -100 > > After reboot... > > ifconfig ix1 up > > dhclient ix1 > DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 > DHCPOFFER from 192.168.1.1 > DHCPREQUEST on ix1 to 255.255.255.255 port 67 > DHCPACK from 192.168.1.1 > bound to 192.168.1.145 -- renewal in 21600 seconds. > > ix0 down > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms > 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms > 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms > > iovctl -Cf /etc/iovctl.conf > > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 29 packets transmitted, 0 packets received, 100.0% packet loss > ifconfig ix1 up > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 12 packets transmitted, 0 packets received, 100.0% packet loss > > ix1 is no longer usable until a restart... > > iovctl -Dd ix1 > ifconfig ix1 up > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 9 packets transmitted, 0 packets received, 100.0% packet loss > > > > Is there anything else that maybe useful? > > here is my ifconfig at the end (after ifconfig ix0 up) > > > ix0: flags=8943 metric 0 > mtu 1500 > > options=e400b9 > ether -Hidden- > inet 192.168.1.8 netmask 0xff00 broadcast 192.168.1.255 > inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 > nd6 options=29 > media: Ethernet autoselect (10Gbase-T ) > status: active > ix1: flags=8843 metric 0 mtu 1500 > > options=e407bb > ether -Hidden- > inet 192.168.1.145 netmask 0xff00 broadcast 192.168.1.255 > nd6 options=29 > media: Ethernet autoselect (10Gbase-T ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=63 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 > inet 127.0.0.1 netmask 0xff00 > nd6 options=21 > groups: lo > bridge0: flags=8843 metric 0 mtu > 1500 > ether -Hidden- > nd6 options=9 > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: ix0 flags=143 >ifmaxaddr 0 port 1 priority 128 path cost 2000 > member: epair0a flags=143 >ifmaxaddr 0 port 5 priority 128 path cost 2000 > epair0a: flags=8943 metric > 0 mtu 1500 > options=8 > ether -Hidden- > inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5 > nd6 options=21 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > > On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner wrote: > > > Did you do an ifconfig up on ix1 before loading the VF driver? > > > > On Sat, Feb 20, 2016 at 11:57 AM Ultima wrote: > > > >> Decided to do some testing with iovctl to see how sr-iov is coming > along. > >> Currently when adding the vf's there are a couple errors, and the > network > >> no longer function after iovctl is started. My guess is the reset_hw() > >> call > >> that is failing. Any ideas why this call would fail? I tested this on > both > >> ports, ix1 is detached and unused for this test, however inserting a > cable > >> results in an unusable port. iovctl -Dd ix1 removes the vf's, however > >> functionality is still not restored without a system restart. > >> > >> FreeBSD S1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r295736: Wed Feb 17 > >> 21:17:28 EST 2016 root@S1:/usr/obj/usr/src/sys/MYKERNEL amd64 > >> > >> /boot/loader.conf > >> hw.ix.num_queues="4" > >> > >> /etc/iovctl.conf > >> PF { > >> device : ix1; > >> num_vfs : 31; > >> } > >> > >> DEFAULT { > >> passthrough : true; > >> } > >> VF-0 { > >> passthrough : false; > >> } > &
Re: sr-iov issues, reset_hw() failed with error -100
After reboot... ifconfig ix1 up dhclient ix1 DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 DHCPOFFER from 192.168.1.1 DHCPREQUEST on ix1 to 255.255.255.255 port 67 DHCPACK from 192.168.1.1 bound to 192.168.1.145 -- renewal in 21600 seconds. ix0 down ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms iovctl -Cf /etc/iovctl.conf ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes ^C --- 192.168.1.1 ping statistics --- 29 packets transmitted, 0 packets received, 100.0% packet loss ifconfig ix1 up ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes ^C --- 192.168.1.1 ping statistics --- 12 packets transmitted, 0 packets received, 100.0% packet loss ix1 is no longer usable until a restart... iovctl -Dd ix1 ifconfig ix1 up ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes ^C --- 192.168.1.1 ping statistics --- 9 packets transmitted, 0 packets received, 100.0% packet loss Is there anything else that maybe useful? here is my ifconfig at the end (after ifconfig ix0 up) ix0: flags=8943 metric 0 mtu 1500 options=e400b9 ether -Hidden- inet 192.168.1.8 netmask 0xff00 broadcast 192.168.1.255 inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (10Gbase-T ) status: active ix1: flags=8843 metric 0 mtu 1500 options=e407bb ether -Hidden- inet 192.168.1.145 netmask 0xff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (10Gbase-T ) status: active lo0: flags=8049 metric 0 mtu 16384 options=63 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff00 nd6 options=21 groups: lo bridge0: flags=8843 metric 0 mtu 1500 ether -Hidden- nd6 options=9 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: ix0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 2000 member: epair0a flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 epair0a: flags=8943 metric 0 mtu 1500 options=8 ether -Hidden- inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5 nd6 options=21 media: Ethernet 10Gbase-T (10Gbase-T ) status: active groups: epair On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner wrote: > Did you do an ifconfig up on ix1 before loading the VF driver? > > On Sat, Feb 20, 2016 at 11:57 AM Ultima wrote: > >> Decided to do some testing with iovctl to see how sr-iov is coming along. >> Currently when adding the vf's there are a couple errors, and the network >> no longer function after iovctl is started. My guess is the reset_hw() >> call >> that is failing. Any ideas why this call would fail? I tested this on both >> ports, ix1 is detached and unused for this test, however inserting a cable >> results in an unusable port. iovctl -Dd ix1 removes the vf's, however >> functionality is still not restored without a system restart. >> >> FreeBSD S1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r295736: Wed Feb 17 >> 21:17:28 EST 2016 root@S1:/usr/obj/usr/src/sys/MYKERNEL amd64 >> >> /boot/loader.conf >> hw.ix.num_queues="4" >> >> /etc/iovctl.conf >> PF { >> device : ix1; >> num_vfs : 31; >> } >> >> DEFAULT { >> passthrough : true; >> } >> VF-0 { >> passthrough : false; >> } >> VF-1 { >> passthrough : false; >> } >> >> # iovctl -C -f /etc/iovctl.conf >> >> dmesg >> ixv0: > 1.4.6-k> at device 0.129 on pci12 >> ixv0: Using MSIX interrupts with 2 vectors >> ixv0: ixgbe_reset_hw() failed with error -100 >> device_attach: ixv0 attach returned 5 >> ixv0: > 1.4.6-k> at device 0.131 on pci12 >> ixv0: Using MSIX interrupts with 2 vectors >> ixv0: ixgbe_reset_hw() failed with error -100 >> device_attach: ixv0 attach returned 5 >> pci12: at device 0.133 (no driver attached) >> pci12: at device 0.135 (no driver attached) >> pci12: at device 0.137 (no driver attached) >> pci12: at device 0.139 (no driver attached) >> pci12: at device 0.141 (no driver attached) >> pci12: at device 0.143 (no driver attached) >> pci12: at device 0.145 (no driver attached) >> pci12: at device 0.147 (no driver attached) >> pci12: at device 0.149 (no driver attached) >> pci12: at device 0.151 (no driver attached) >> pci12: at device 0.153 (no driver attached) >> pci12: at device 0.155 (no driver attached) >> pci12: at device 0.157 (no drive
sr-iov issues, reset_hw() failed with error -100
class = network subclass = ethernet ppt26@pci0:129:0:185: class=0x02 card=0x1458 chip=0x15158086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'X540 Ethernet Controller Virtual Function' class = network subclass = ethernet ppt27@pci0:129:0:187: class=0x02 card=0x1458 chip=0x15158086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'X540 Ethernet Controller Virtual Function' class = network subclass = ethernet ppt28@pci0:129:0:189: class=0x02 card=0x1458 chip=0x15158086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'X540 Ethernet Controller Virtual Function' class = network subclass = ethernet Ultima ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"