Hey Pavel, In order to support the bypass drive you need a special driver. I'm surprised you weren't supplied with one when you bought the NIC. Without it you can't communicate with the uC that handles all the bypass functionality. Likewise the SDP aren't connected to what the upstream ixgbe driver thinks they should be so "undefined" things could happen.
Thanks, -Don Skidmore <donald.c.skidm...@intel.com> > -----Original Message----- > From: Pavel Odintsov [mailto:pavel.odint...@gmail.com] > Sent: Friday, June 19, 2015 3:43 AM > To: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] VF inside VM with bypassed PCI-E ixgbe > > I have finished my "home work". > > issue reproduced with 4.0.0 kernel and latest 4.1.1 ixgbe driver. > > Same error in dmesg: > [ 221.676388] ixgbe 0000:00:05.0 0000:00:05.0 (uninitialized): Failed to > enable > PCI sriov: -38 > > On Fri, Jun 19, 2015 at 12:58 PM, Pavel Odintsov > <pavel.odint...@gmail.com> wrote: > > Hello! > > > > After some investigation of ixgbe and Linux kernel code. > > > > I found function which return this "error 38", it's sriov_enable from > > Linux Kernel (http://lxr.free- > electrons.com/source/drivers/pci/iov.c?v=3.16#L173). > > > > And error 38 means return of ENOSYS > > (http://lxr.free-electrons.com/source/include/uapi/asm-generic/errno.h > > ?v=3.16#L9) from it. Which means "Function not implemented". > > > > Finally, Linux Kernel really sure about virtual nature of this bypassed NIC. > > > > if (!dev->is_physfn) > > return -ENOSYS; > > > > So, we need dig into kernel code. > > > > On Fri, Jun 19, 2015 at 12:05 PM, Pavel Odintsov > > <pavel.odint...@gmail.com> wrote: > >> Sorry, for the mess. It was very hard day and I post fe silly posts > >> due to my inattention (and incorrectly selected ssh window). > >> > >> Finally, I have got only single issue with ixgbe driver there. > >> > >> I can't create VF inside KVM VM. Not with max_vfs param, not with > >> echo > >> 2 > /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_numvfs. > >> > >> Could you help me? > >> > >> On Fri, Jun 19, 2015 at 11:58 AM, Pavel Odintsov > >> <pavel.odint...@gmail.com> wrote: > >>> Hello! > >>> > >>> Please ignre this: > >>> echo 2 > > >>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/sriov_numvfs > >>> -bash: echo: write error: Function not implemented > >>> > >>> echo 2 > > >>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_numvfs > >>> -bash: echo: write error: Function not implemented > >>> > >>> > >>> Now it works fine. But broken name which refer to /proc/* instead of > >>> /sys/* still an issue. > >>> > >>> > >>> On Fri, Jun 19, 2015 at 11:55 AM, Pavel Odintsov > >>> <pavel.odint...@gmail.com> wrote: > >>>> Hello! > >>>> > >>>> Some details. > >>>> > >>>> Error: > >>>> echo '2' > /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs > >>>> -bash: /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs: Permission > >>>> denied > >>>> > >>>> Definitely related with non existent proc fs path: > >>>> find /proc/|grep sriov_numvfs|wc -l > >>>> 0 > >>>> > >>>> It's definitely a bug. > >>>> > >>>> Because I could find this pseudo files in /sys: > >>>> root@filter ~ # find /sys/|grep vfs > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/sriov_numvfs > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/sriov_totalvfs > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_numvfs > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_totalvfs > >>>> root@filter ~ # find /proc/|grep vfs > >>>> /proc/sys/vm/vfs_cache_pressure > >>>> > >>>> But I still can't use sriov_numvfs from there: > >>>> > >>>> cat > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/sriov_totalvfs > >>>> 63 > >>>> > >>>> cat > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_totalvfs > >>>> 63 > >>>> > >>>> echo 2 > > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/sriov_numvfs > >>>> -bash: echo: write error: Function not implemented > >>>> > >>>> echo 2 > > >>>> /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.1/sriov_numvfs > >>>> -bash: echo: write error: Function not implemented > >>>> > >>>> dmesg|tail -n 100 > >>>> [3004270.392052] ixgbe: Intel(R) 10 Gigabit PCI Express Network > >>>> Driver > >>>> - version 3.19.1-k > >>>> [3004270.392316] ixgbe: Copyright (c) 1999-2014 Intel Corporation. > >>>> [3004295.119478] vfio-pci 0000:03:00.0: Driver doesn't support > >>>> SRIOV configuration via sysfs [3004299.761902] vfio-pci > >>>> 0000:03:00.1: Driver doesn't support SRIOV configuration via sysfs > >>>> > >>>> :( > >>>> > >>>> On Fri, Jun 19, 2015 at 11:37 AM, Pavel Odintsov > >>>> <pavel.odint...@gmail.com> wrote: > >>>>> Hello, folks! > >>>>> > >>>>> I have KVM VM with bypassed PCI-E 82599 two port NIC. > >>>>> > >>>>> Driver version: 4.0.3 at Debian Jessie 3.16 kernel. > >>>>> > >>>>> lspci -v|grep Eth > >>>>> 00:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > >>>>> RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20) > >>>>> 00:05.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>>> SFI/SFP+ Network Connection (rev 01) > >>>>> Subsystem: Intel Corporation Ethernet Server Adapter X520-2 > >>>>> 00:06.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >>>>> SFI/SFP+ Network Connection (rev 01) > >>>>> Subsystem: Intel Corporation Ethernet Server Adapter X520-2 > >>>>> > >>>>> And I want to use VF inside this KVM VM. > >>>>> > >>>>> And I doing following: > >>>>> rmmod ixgbe ixgbevf > >>>>> modprobe ixgbe max_vfs=2,2 > >>>>> modprobe ixgbevf > >>>>> > >>>>> But after this operation I haven't saw virtual functions in lspci > >>>>> output: > >>>>> lspci -tv > >>>>> -[0000:00]-+-00.0 Intel Corporation 440FX - 82441FX PMC [Natoma] > >>>>> +-01.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton > >>>>> II] > >>>>> +-01.1 Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton > >>>>> II] > >>>>> +-01.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI > >>>>> +-02.0 Realtek Semiconductor Co., Ltd. > >>>>> RTL-8100/8101L/8139 PCI Fast Ethernet Adapter > >>>>> +-03.0 Intel Corporation 82801I (ICH9 Family) USB UHCI > Controller #1 > >>>>> +-03.1 Intel Corporation 82801I (ICH9 Family) USB UHCI > Controller #2 > >>>>> +-03.2 Intel Corporation 82801I (ICH9 Family) USB UHCI > Controller #3 > >>>>> +-03.7 Intel Corporation 82801I (ICH9 Family) USB2 > >>>>> EHCI Controller #1 > >>>>> +-04.0 Red Hat, Inc Virtio memory balloon > >>>>> +-05.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > >>>>> Network Connection > >>>>> +-06.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > >>>>> Network Connection > >>>>> \-07.0 Cirrus Logic GD 5446 > >>>>> > >>>>> And haven't saw any additional interfaces: > >>>>> > >>>>> ip link show > >>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > >>>>> UNKNOWN mode DEFAULT group default > >>>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > >>>>> pfifo_fast state UP mode DEFAULT group default qlen 1000 > >>>>> link/ether 52:54:00:ec:be:23 brd ff:ff:ff:ff:ff:ff > >>>>> 19: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state > DOWN > >>>>> mode DEFAULT group default qlen 1000 > >>>>> link/ether 90:e2:ba:83:3f:24 brd ff:ff:ff:ff:ff:ff > >>>>> 20: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > mq > >>>>> state UP mode DEFAULT group default qlen 1000 > >>>>> link/ether 90:e2:ba:83:3f:25 brd ff:ff:ff:ff:ff:ff > >>>>> > >>>>> > >>>>> I saw this error in dmesg: > >>>>> > >>>>> [848550.356391] ixgbe 0000:00:06.0: Enabling SR-IOV VFs using the > >>>>> max_vfs module parameter is deprecated. > >>>>> [848550.356398] ixgbe 0000:00:06.0: Please use the pci sysfs > >>>>> interface instead. Ex: > >>>>> [848550.356403] ixgbe 0000:00:06.0: echo '2' > > >>>>> /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs > >>>>> [848705.672383] ixgbe 0000:00:06.0 (unregistered net_device): > >>>>> Failed to enable PCI sriov: -38 <------- COULD BE IMPORTANT > >>>>> > >>>>> And try another approach: > >>>>> rmmod ixgbe ixgbevf > >>>>> > >>>>> modprobe ixgbe > >>>>> echo '2' > /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs > >>>>> echo '2' > /sys/bus/pci/devices/0000:00:05.0/sriov_numvfs > >>>>> > >>>>> And it's not working at all: > >>>>> echo '2' > /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs > >>>>> -bash: /sys/bus/pci/devices/0000:00:06.0/sriov_numvfs: Permission > >>>>> denied > >>>>> > >>>>> Is it possible to use VF inside KVM VM? > >>>>> > >>>>> -- > >>>>> Sincerely yours, Pavel Odintsov > >>>> > >>>> > >>>> > >>>> -- > >>>> Sincerely yours, Pavel Odintsov > >>> > >>> > >>> > >>> -- > >>> Sincerely yours, Pavel Odintsov > >> > >> > >> > >> -- > >> Sincerely yours, Pavel Odintsov > > > > > > > > -- > > Sincerely yours, Pavel Odintsov > > > > -- > Sincerely yours, Pavel Odintsov > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
