[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On 08/25/2016 02:01 PM, Ferruh Yigit wrote: > On 8/25/2016 5:57 AM, Keren Hochman wrote: >> Hi, >> I there a way to permanently bind a nic port when using DPDK drier with >> kernel < 3.6 ? (In this kernel VFIO driver is not supported)? >> Thanks, Keren >> > > There was a tool from Panu for this purpose: > http://dpdk.org/ml/archives/dev/2015-December/029500.html Yup, but driverctl uses the newer driver_override binding feature which is only in kernel >= 3.16 so it's of no use with kernel < 3.6. - Panu -
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On 8/25/2016 5:57 AM, Keren Hochman wrote: > Hi, > I there a way to permanently bind a nic port when using DPDK drier with > kernel < 3.6 ? (In this kernel VFIO driver is not supported)? > Thanks, Keren > There was a tool from Panu for this purpose: http://dpdk.org/ml/archives/dev/2015-December/029500.html Regards, ferruh
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Hi, I there a way to permanently bind a nic port when using DPDK drier with kernel < 3.6 ? (In this kernel VFIO driver is not supported)? Thanks, Keren
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On 11/13/2015 01:48 PM, Montorsi, Francesco wrote: > Hi John, > >> -Original Message- >> From: Mcnamara, John [mailto:john.mcnamara at intel.com] >> >> The Ubuntu dpdk package for 15.10 contains system scripts with functions for >> reserving hugepages and binding interfaces on bootup: >> >> >> /etc/dpdk/dpdk.conf >> /etc/dpdk/interfaces >> /etc/init.d/dpdk >> /lib/dpdk/dpdk-init >> /lib/systemd/system/dpdk.service >> /sbin/dpdk_nic_bind >> /usr/bin/testpmd >> /usr/share/doc/dpdk/README.Debian >> /usr/share/doc/dpdk/changelog.Debian.gz >> /usr/share/doc/dpdk/copyright >> /usr/share/dpdk/tools/cpu_layout.py >> /usr/share/dpdk/tools/dpdk_nic_bind.py >> /usr/share/dpdk/tools/setup.sh >> /usr/share/python/runtime.d/dpdk.rtupdate >> >> http://packages.ubuntu.com/wily/amd64/dpdk/filelist >> >> If you have the latest version of Ubuntu you can check that out or else >> download and extract the files from the .deb to see how they do it. >> > > This certainly looks very useful. I inspected the package and the > files you mentioned and indeed it looks like a good way to go, specially > if Ubuntu distribution is moving in that direction (hopefully other > distros will follow too). > > Thanks a lot! The downside of the Ubuntu-approach is that the setup is fairly static and does not support automatically binding hotplugged devices, AFAICS. Whether that matters to your use-case is of course en entirely different question. - Panu - > Francesco > >
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On 11/13/2015 01:44 PM, Montorsi, Francesco wrote: > Hi Panu, > >> -Original Message- >> From: Panu Matilainen [mailto:pmatilai at redhat.com] >> I've been looking into this recently, here's what I have so far: >> http://laiskiainen.org/git/?p=driverctl.git >> > Thanks I tried the script (I just had to change /bin/sh into > /bin/bash on first line) and it works fine on my Ubuntu 14.04. Oops, fixed now. Thanks for pointing that out, the line between sh and bash is (too) blurry on Fedora & friends. > Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses > kernel 3.0.76 which I don't know if is compatible with VFIO) and there > the output of "list-devices" is plain empty. I will try to look at > what's happening... list-devices currently only lists devices which support driver overriding[*], which is a relatively new kernel feature and would be missing from 3.0.76: https://www.redhat.com/archives/libvir-list/2014-April/msg00222.html It might well be possible to fall back to the "old style" bind system if the kernel doesn't support driver_override. I've no immediate plans to look into that but lets just say patches would at least be considered :) - Panu -
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Hi John, > -Original Message- > From: Mcnamara, John [mailto:john.mcnamara at intel.com] > > The Ubuntu dpdk package for 15.10 contains system scripts with functions for > reserving hugepages and binding interfaces on bootup: > > > /etc/dpdk/dpdk.conf > /etc/dpdk/interfaces > /etc/init.d/dpdk > /lib/dpdk/dpdk-init > /lib/systemd/system/dpdk.service > /sbin/dpdk_nic_bind > /usr/bin/testpmd > /usr/share/doc/dpdk/README.Debian > /usr/share/doc/dpdk/changelog.Debian.gz > /usr/share/doc/dpdk/copyright > /usr/share/dpdk/tools/cpu_layout.py > /usr/share/dpdk/tools/dpdk_nic_bind.py > /usr/share/dpdk/tools/setup.sh > /usr/share/python/runtime.d/dpdk.rtupdate > > http://packages.ubuntu.com/wily/amd64/dpdk/filelist > > If you have the latest version of Ubuntu you can check that out or else > download and extract the files from the .deb to see how they do it. > This certainly looks very useful. I inspected the package and the files you mentioned and indeed it looks like a good way to go, specially if Ubuntu distribution is moving in that direction (hopefully other distros will follow too). Thanks a lot! Francesco
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Hi Panu, > -Original Message- > From: Panu Matilainen [mailto:pmatilai at redhat.com] > I've been looking into this recently, here's what I have so far: > http://laiskiainen.org/git/?p=driverctl.git > Thanks I tried the script (I just had to change /bin/sh into /bin/bash on first line) and it works fine on my Ubuntu 14.04. Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses kernel 3.0.76 which I don't know if is compatible with VFIO) and there the output of "list-devices" is plain empty. I will try to look at what's happening... Thanks, Francesco
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Panu Matilainen writes:
> On 11/11/2015 06:28 PM, Bruce Richardson wrote:
>> On Wed, Nov 11, 2015 at 04:13:01PM +, Montorsi, Francesco wrote:
>>> Hi,
>>> Is there a way to permanently (i.e., have the configuration automatically
>>> applied after reboot) bind a NIC port to DPDK?
>>>
>>> In case there's none, I'm thinking to save in my software a list of the NIC
>>> ports chosen by the user for use with DPDK and then, upon software startup
>>> to just do
>>> for (int i=0; i < ...; i++)
>>> system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>>> Do you see any problem with that?
>>>
>>> Thanks!
>>> Francesco Montorsi
>>>
>>
>> Hi Francesco,
>>
>> I'm not aware of any way to make the bindings permanent across
>> reboots. What you
>> have suggested will work, but there are probably better ways to do
>> the same thing.
>> For example, a couple of lines in an rc.local script can reapply the
>> bindings at
>> boot for you. I'm sure others can suggest other ways of having the
>> same effect,
>> for example, there may be a way to automatically do this using udev or
>> systemd
>> or some such package.
>
> I've been looking into this recently, here's what I have so far:
> http://laiskiainen.org/git/?p=driverctl.git
>
I've been using this recently, on my systems.
+1 for driverctl :-)
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On 11/11/2015 06:28 PM, Bruce Richardson wrote:
> On Wed, Nov 11, 2015 at 04:13:01PM +, Montorsi, Francesco wrote:
>> Hi,
>> Is there a way to permanently (i.e., have the configuration automatically
>> applied after reboot) bind a NIC port to DPDK?
>>
>> In case there's none, I'm thinking to save in my software a list of the NIC
>> ports chosen by the user for use with DPDK and then, upon software startup
>> to just do
>> for (int i=0; i < ...; i++)
>> system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>> Do you see any problem with that?
>>
>> Thanks!
>> Francesco Montorsi
>>
>
> Hi Francesco,
>
> I'm not aware of any way to make the bindings permanent across reboots. What
> you
> have suggested will work, but there are probably better ways to do the same
> thing.
> For example, a couple of lines in an rc.local script can reapply the bindings
> at
> boot for you. I'm sure others can suggest other ways of having the same
> effect,
> for example, there may be a way to automatically do this using udev or systemd
> or some such package.
I've been looking into this recently, here's what I have so far:
http://laiskiainen.org/git/?p=driverctl.git
For the impatient, "make rpm" should produce something usable for recent
Fedora/RHEL systems, usage looks somewhat like this:
Find devices currently driven by ixgbe driver:
# driverctl -v list-devices | grep ixgbe
:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
Change them to use the vfio-pci driver permanently:
# driverctl set-override :01:00.0 vfio-pci
# driverctl set-override :01:00.1 vfio-pci
Find devices with driver overrides:
[root at wsfd-netdev32 ~]# driverctl -v list-devices|grep \*
:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
Remove the permanent driver override for device :01:00.1:
# driverctl unset-override :01:00.1
In addition it has udev rules to export vfio and uio devices on systemd
level, eg the above looks like this with normal drivers:
# systemctl |grep :01:00
sys-devices-pci:00-:00:03.0-:01:00.0-net-em1.device
loaded active plugged Ethernet 10G 4P X520/I350 rNDC
sys-devices-pci:00-:00:03.0-:01:00.1-net-em2.device
loaded active plugged Ethernet 10G 4P X520/I350 rNDC
When changed to vfio, with upstream systemd/udev rules they would just
disappear entirely, but with the driverctl rules they become:
# systemctl |grep :01:00
sys-devices-pci:00-:00:03.0-:01:00.0-vfio.device
loaded active plugged
/sys/devices/pci:00/:00:03.0/:01:00.0/vfio
sys-devices-pci:00-:00:03.0-:01:00.1-vfio.device
loaded active plugged
/sys/devices/pci:00/:00:03.0/:01:00.1/vfio
- Panu -
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
> -Original Message- > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Montorsi, Francesco > Sent: Wednesday, November 11, 2015 4:13 PM > To: dev at dpdk.org > Subject: [dpdk-dev] Permanently binding NIC ports with DPDK drivers > > Hi, > Is there a way to permanently (i.e., have the configuration automatically > applied after reboot) bind a NIC port to DPDK? Hi, The Ubuntu dpdk package for 15.10 contains system scripts with functions for reserving hugepages and binding interfaces on bootup: /etc/dpdk/dpdk.conf /etc/dpdk/interfaces /etc/init.d/dpdk /lib/dpdk/dpdk-init /lib/systemd/system/dpdk.service /sbin/dpdk_nic_bind /usr/bin/testpmd /usr/share/doc/dpdk/README.Debian /usr/share/doc/dpdk/changelog.Debian.gz /usr/share/doc/dpdk/copyright /usr/share/dpdk/tools/cpu_layout.py /usr/share/dpdk/tools/dpdk_nic_bind.py /usr/share/dpdk/tools/setup.sh /usr/share/python/runtime.d/dpdk.rtupdate http://packages.ubuntu.com/wily/amd64/dpdk/filelist If you have the latest version of Ubuntu you can check that out or else download and extract the files from the .deb to see how they do it. John. --
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Hi Bruce, > -Original Message- > From: Bruce Richardson [mailto:bruce.richardson at intel.com] > I'm not aware of any way to make the bindings permanent across reboots. > What you have suggested will work, but there are probably better ways to > do the same thing. I agree... let's see if somebody else has suggestions :) In any case my idea is to make my software as much independent as possible from troubles with future HW and future DPDK versions. A way to do that would be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py script and just use that (since it will be probably always up to date and correct). My only concern is that (reading the python code) dpdk_nic_bind.py script does not return with an error code != 0 if something bad happens during binding... maybe it may be worth doing such a small change... Just my 2 cents, Francesco
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On Wed, Nov 11, 2015 at 04:13:01PM +, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically
> applied after reboot) bind a NIC port to DPDK?
>
> In case there's none, I'm thinking to save in my software a list of the NIC
> ports chosen by the user for use with DPDK and then, upon software startup to
> just do
> for (int i=0; i < ...; i++)
> system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that?
>
> Thanks!
> Francesco Montorsi
>
Hi Francesco,
I'm not aware of any way to make the bindings permanent across reboots. What you
have suggested will work, but there are probably better ways to do the same
thing.
For example, a couple of lines in an rc.local script can reapply the bindings at
boot for you. I'm sure others can suggest other ways of having the same effect,
for example, there may be a way to automatically do this using udev or systemd
or some such package.
/Bruce
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
Hi,
Is there a way to permanently (i.e., have the configuration automatically
applied after reboot) bind a NIC port to DPDK?
In case there's none, I'm thinking to save in my software a list of the NIC
ports chosen by the user for use with DPDK and then, upon software startup to
just do
for (int i=0; i < ...; i++)
system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
Do you see any problem with that?
Thanks!
Francesco Montorsi
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
In my development environment I set up an at-boot provisioning script that
does it. I recommend using scripts and not shelling out from C code. ;)
On Wed, Nov 11, 2015 at 04:13:01PM +, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically
> applied after reboot) bind a NIC port to DPDK?
>
> In case there's none, I'm thinking to save in my software a list of the NIC
> ports chosen by the user for use with DPDK and then, upon software startup to
> just do
> for (int i=0; i < ...; i++)
> system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that?
>
> Thanks!
> Francesco Montorsi
>
[dpdk-dev] Permanently binding NIC ports with DPDK drivers
On Wed, 11 Nov 2015 16:59:14 + "Montorsi, Francesco" wrote: > Hi Bruce, > > > -Original Message- > > From: Bruce Richardson [mailto:bruce.richardson at intel.com] > > I'm not aware of any way to make the bindings permanent across reboots. > > What you have suggested will work, but there are probably better ways to > > do the same thing. > > I agree... let's see if somebody else has suggestions :) > > In any case my idea is to make my software as much independent as possible > from troubles with future HW and future DPDK versions. A way to do that would > be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py > script and just use that (since it will be probably always up to date and > correct). My only concern is that (reading the python code) dpdk_nic_bind.py > script does not return with an error code != 0 if something bad happens > during binding... maybe it may be worth doing such a small change... > > Just my 2 cents, > Francesco > > > > I would recommend using PCI id's and not depending in anyway on port. If you want, I can submit a patch to add derive the systemd/udev compatiable name from existing DPDK port.
