Re: [systemd-devel] how to bind to other drivers using systemd

[Harald Hoyer]

On 13.10.2015 14:08, Panu Matilainen wrote:
> On 09/30/2015 08:43 PM, Flavio Leitner wrote:
>> On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:
>>> I wonder if this could be handled with a generic Type=oneshot,
>>> ExecStart=driverctl bind foo...
>>
>> Could you elaborate more on that?
>>
>> If driverctl is an utility, we could use scripts, yeah.  The issue
>> is how to identify the device. We probably would need some sort
>> matching mechanism like udev does.
> 
> Since the silence is deafening, and since its usually easier to poke holes in
> something concrete no matter how incomplete and/or broken, here's a beginning
> of an approach to alternative driver assignment in the udev realm:
> http://laiskiainen.org/udev/driverctl/
> 
> # 
> # echo vfio-pci > /etc/driverctl.d/:01:00.0
> # echo 1 > /sys/bus/pci/devices/:01:00.0/remove
> # echo 1 > /sys/bus/pci/rescan
> 
> ...and there you go, udev vfio-pci is forced for that device on the rescan, 
> and
> a real hotplug is handled similarly.
> 
> Since normally the device probing occurs early in the boot, these things also
> need to go to initramfs so a dracut module/patch is needed, and dracut needs 
> to
> be run after adding devices here, ie
> 
> # echo vfio-pci > /etc/driverctl.d/:01:00.0
> # dracut -f
> # reboot
> 
> Having to mess with initramfs is a bit of a PITA but at least some of that
> could be hidden in the driverctl utility, and then overriding the default
> driver early in the boot means we get none of the side-effects with interfaces
> coming and going, possibly wrestling with NetworkManager over it etc.
> 
> For a real-world implementation there are tons of TODOs like
> - support non-pci buses
> - support direct bind/unbind in driverctl
> - million errors situations to check
> - vfio & uio -bound devices disappear from systemd, we'd probably want them
> there in some form to be able to depend on them
> - etc...
> 
> - Panu -
> 

I don't know nothing about driver binding/unbinding, but what happens if the
initial driver binding (with the wrong driver) happens in-between these lines?

if [ -L /sys/$DEVPATH/driver ]; then
echo "$DEV" > /sys/$DEVPATH/driver/unbind
fi
[…]

--->> HERE the other driver binds to the device <<<

[…]
   echo "$DRV" > /sys/$DEVPATH/driver_override
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Harald Hoyer]

On 13.10.2015 14:08, Panu Matilainen wrote:
> On 09/30/2015 08:43 PM, Flavio Leitner wrote:
>> On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:
>>> I wonder if this could be handled with a generic Type=oneshot,
>>> ExecStart=driverctl bind foo...
>>
>> Could you elaborate more on that?
>>
>> If driverctl is an utility, we could use scripts, yeah.  The issue
>> is how to identify the device. We probably would need some sort
>> matching mechanism like udev does.
> 
> Since the silence is deafening, and since its usually easier to poke holes in
> something concrete no matter how incomplete and/or broken, here's a beginning
> of an approach to alternative driver assignment in the udev realm:
> http://laiskiainen.org/udev/driverctl/
> 
> # 
> # echo vfio-pci > /etc/driverctl.d/:01:00.0
> # echo 1 > /sys/bus/pci/devices/:01:00.0/remove
> # echo 1 > /sys/bus/pci/rescan
> 
> ...and there you go, udev vfio-pci is forced for that device on the rescan, 
> and
> a real hotplug is handled similarly.
> 
> Since normally the device probing occurs early in the boot, these things also
> need to go to initramfs so a dracut module/patch is needed, and dracut needs 
> to
> be run after adding devices here, ie
> 
> # echo vfio-pci > /etc/driverctl.d/:01:00.0
> # dracut -f
> # reboot

If both kernel modules are already in the initramfs, you don't have to do this.

> 
> Having to mess with initramfs is a bit of a PITA but at least some of that
> could be hidden in the driverctl utility, and then overriding the default
> driver early in the boot means we get none of the side-effects with interfaces
> coming and going, possibly wrestling with NetworkManager over it etc.
> 
> For a real-world implementation there are tons of TODOs like
> - support non-pci buses
> - support direct bind/unbind in driverctl
> - million errors situations to check
> - vfio & uio -bound devices disappear from systemd, we'd probably want them
> there in some form to be able to depend on them
> - etc...
> 
> - Panu -
> 
> 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Panu Matilainen]

On 10/13/2015 03:24 PM, Harald Hoyer wrote:

On 13.10.2015 14:08, Panu Matilainen wrote:

On 09/30/2015 08:43 PM, Flavio Leitner wrote:

On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:

I wonder if this could be handled with a generic Type=oneshot,
ExecStart=driverctl bind foo...


Could you elaborate more on that?

If driverctl is an utility, we could use scripts, yeah.  The issue
is how to identify the device. We probably would need some sort
matching mechanism like udev does.


Since the silence is deafening, and since its usually easier to poke holes in
something concrete no matter how incomplete and/or broken, here's a beginning
of an approach to alternative driver assignment in the udev realm:
http://laiskiainen.org/udev/driverctl/

# 
# echo vfio-pci > /etc/driverctl.d/:01:00.0
# echo 1 > /sys/bus/pci/devices/:01:00.0/remove
# echo 1 > /sys/bus/pci/rescan

...and there you go, udev vfio-pci is forced for that device on the rescan, and
a real hotplug is handled similarly.

Since normally the device probing occurs early in the boot, these things also
need to go to initramfs so a dracut module/patch is needed, and dracut needs to
be run after adding devices here, ie

# echo vfio-pci > /etc/driverctl.d/:01:00.0
# dracut -f
# reboot

Having to mess with initramfs is a bit of a PITA but at least some of that
could be hidden in the driverctl utility, and then overriding the default
driver early in the boot means we get none of the side-effects with interfaces
coming and going, possibly wrestling with NetworkManager over it etc.

For a real-world implementation there are tons of TODOs like
- support non-pci buses
- support direct bind/unbind in driverctl
- million errors situations to check
- vfio & uio -bound devices disappear from systemd, we'd probably want them
there in some form to be able to depend on them
- etc...

 - Panu -



I don't know nothing about driver binding/unbinding,


Neither do I, that's why I'm asking here ;)


but what happens if the
initial driver binding (with the wrong driver) happens in-between these lines?

 if [ -L /sys/$DEVPATH/driver ]; then
echo "$DEV" > /sys/$DEVPATH/driver/unbind
 fi
 […]

--->> HERE the other driver binds to the device <<<

 […]
echo "$DRV" > /sys/$DEVPATH/driver_override


The wrong thing probably... The unbind probably doesn't belong there at 
all, I added it there for the wrong reason and forgot to remove.


Thanks for pointing that out though.

- Panu -



___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel



___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Panu Matilainen]

On 09/30/2015 08:43 PM, Flavio Leitner wrote:

On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:

I wonder if this could be handled with a generic Type=oneshot,
ExecStart=driverctl bind foo...


Could you elaborate more on that?

If driverctl is an utility, we could use scripts, yeah.  The issue
is how to identify the device. We probably would need some sort
matching mechanism like udev does.


Since the silence is deafening, and since its usually easier to poke 
holes in something concrete no matter how incomplete and/or broken, 
here's a beginning of an approach to alternative driver assignment in 
the udev realm: http://laiskiainen.org/udev/driverctl/


# 
# echo vfio-pci > /etc/driverctl.d/:01:00.0
# echo 1 > /sys/bus/pci/devices/:01:00.0/remove
# echo 1 > /sys/bus/pci/rescan

...and there you go, udev vfio-pci is forced for that device on the 
rescan, and a real hotplug is handled similarly.


Since normally the device probing occurs early in the boot, these things 
also need to go to initramfs so a dracut module/patch is needed, and 
dracut needs to be run after adding devices here, ie


# echo vfio-pci > /etc/driverctl.d/:01:00.0
# dracut -f
# reboot

Having to mess with initramfs is a bit of a PITA but at least some of 
that could be hidden in the driverctl utility, and then overriding the 
default driver early in the boot means we get none of the side-effects 
with interfaces coming and going, possibly wrestling with NetworkManager 
over it etc.


For a real-world implementation there are tons of TODOs like
- support non-pci buses
- support direct bind/unbind in driverctl
- million errors situations to check
- vfio & uio -bound devices disappear from systemd, we'd probably want 
them there in some form to be able to depend on them

- etc...

- Panu -


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Panu Matilainen]

On 10/13/2015 03:28 PM, Harald Hoyer wrote:

On 13.10.2015 14:08, Panu Matilainen wrote:

On 09/30/2015 08:43 PM, Flavio Leitner wrote:

On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:

I wonder if this could be handled with a generic Type=oneshot,
ExecStart=driverctl bind foo...


Could you elaborate more on that?

If driverctl is an utility, we could use scripts, yeah.  The issue
is how to identify the device. We probably would need some sort
matching mechanism like udev does.


Since the silence is deafening, and since its usually easier to poke holes in
something concrete no matter how incomplete and/or broken, here's a beginning
of an approach to alternative driver assignment in the udev realm:
http://laiskiainen.org/udev/driverctl/

# 
# echo vfio-pci > /etc/driverctl.d/:01:00.0
# echo 1 > /sys/bus/pci/devices/:01:00.0/remove
# echo 1 > /sys/bus/pci/rescan

...and there you go, udev vfio-pci is forced for that device on the rescan, and
a real hotplug is handled similarly.

Since normally the device probing occurs early in the boot, these things also
need to go to initramfs so a dracut module/patch is needed, and dracut needs to
be run after adding devices here, ie

# echo vfio-pci > /etc/driverctl.d/:01:00.0
# dracut -f
# reboot


If both kernel modules are already in the initramfs, you don't have to do this.


'dracut -f' is necessary to get any new/changed/removed 
/etc/driverctl.d/ bits into the initramfs though.


I had not considered the modules question at all, which is of course 
another valid concern, these non-default drivers might well not be in 
the initramfs by default.


Thanks,

- Panu -

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Flavio Leitner]

On Tue, Sep 29, 2015 at 12:30:54AM +0300, Mantas Mikulėnas wrote:
> I wonder if this could be handled with a generic Type=oneshot,
> ExecStart=driverctl bind foo...

Could you elaborate more on that?

If driverctl is an utility, we could use scripts, yeah.  The issue
is how to identify the device. We probably would need some sort
matching mechanism like udev does.

Another issue is identifying the target driver.  Ideally you would
have a template that says the device and the driver somehow.
For example something like ``bind-@.service´´, so when you start
bind-eth0@vfio-pci.service it would unbind eth0 from the previous
driver (if needed) and bind the device to vfio-pci driver.

fbl

 
> -- 
> Mantas Mikulėnas
> On Sep 28, 2015 21:48, "Flavio Leitner"  wrote:
> 
> > On Mon, Sep 28, 2015 at 08:06:50PM +0200, Kay Sievers wrote:
> > > On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner 
> > wrote:
> > > > I am looking for guidance on how to properly resolve driver binding
> > > > with systemd (which seems to me the best place to do that).
> > >
> > > This seems to be a too exotic and niche use case, nothing
> > > general-purpose enough to implement high-level knobs in systemd.
> >
> > These alternative drivers might be new yet, so there is no much around
> > them, but we need to start right to avoid problems in the future.
> >
> > Accelerated userspace datapath is one use case that changes the default
> > NIC driver to UIO or VFIO driver. Then you use DPDK applications for
> > fast packet processing or use DPDK integrated Open vSwitch for switching.
> >
> > Consider that Open Stack (WIP), Open Shift and maybe containers can use
> > the above setup, so isn't really a niche use case.  It's pretty much a
> > requirement for NFV projects as well.
> >
> > > As you mention, you could use custom udev rules to actually bind a
> > > specific driver. I don't think we want anything more abstract in
> > > systemd.
> >
> > The problem with that is we can't stop the service to roll back the
> > driver.  Or more importantly, create a dependency to, let's say,
> > start Open vSwitch only after the ports are bound to the right driver.
> > Or start an application that requires the NIC to be bound first.
> >
> > Thanks,
> > fbl
> >
> > ___
> > systemd-devel mailing list
> > systemd-devel@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Panu Matilainen]

On 09/28/2015 09:55 PM, Kay Sievers wrote:

On Mon, Sep 28, 2015 at 8:48 PM, Flavio Leitner  wrote:

On Mon, Sep 28, 2015 at 08:06:50PM +0200, Kay Sievers wrote:

On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner  wrote:

I am looking for guidance on how to properly resolve driver binding
with systemd (which seems to me the best place to do that).


This seems to be a too exotic and niche use case, nothing
general-purpose enough to implement high-level knobs in systemd.


These alternative drivers might be new yet, so there is no much around
them, but we need to start right to avoid problems in the future.

Accelerated userspace datapath is one use case that changes the default
NIC driver to UIO or VFIO driver. Then you use DPDK applications for
fast packet processing or use DPDK integrated Open vSwitch for switching.

Consider that Open Stack (WIP), Open Shift and maybe containers can use
the above setup, so isn't really a niche use case.  It's pretty much a
requirement for NFV projects as well.


As you mention, you could use custom udev rules to actually bind a
specific driver. I don't think we want anything more abstract in
systemd.


The problem with that is we can't stop the service to roll back the
driver.  Or more importantly, create a dependency to, let's say,
start Open vSwitch only after the ports are bound to the right driver.
Or start an application that requires the NIC to be bound first.


Sure, I understand all that. I would still call all that a niche and I
still don't think it's systemd's job to solve such a problem.


Well, sure we could easily hack up some DPDK-specific thing for this, 
but the need for alternative driver binding is in no way specific to DPDK:
- as explained in https://lwn.net/Articles/143397/, distros might need 
to support multiple (versions of) traditional drivers for a single device

- VFIO is used for more than just NIC passthrough in virtualization

So if we go and whip up some DPDK specific hack for this, chances are 
it'll evantually clash with something else. That's why we're asking for 
opinions how to best implement this in a generic fashion systemd/udev.


- Panu -
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Kay Sievers]

On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner  wrote:
> I am looking for guidance on how to properly resolve driver binding
> with systemd (which seems to me the best place to do that).

This seems to be a too exotic and niche use case, nothing
general-purpose enough to implement high-level knobs in systemd.

As you mention, you could use custom udev rules to actually bind a
specific driver. I don't think we want anything more abstract in
systemd.

Kay
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Flavio Leitner]

On Mon, Sep 28, 2015 at 08:06:50PM +0200, Kay Sievers wrote:
> On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner  wrote:
> > I am looking for guidance on how to properly resolve driver binding
> > with systemd (which seems to me the best place to do that).
> 
> This seems to be a too exotic and niche use case, nothing
> general-purpose enough to implement high-level knobs in systemd.

These alternative drivers might be new yet, so there is no much around
them, but we need to start right to avoid problems in the future.

Accelerated userspace datapath is one use case that changes the default
NIC driver to UIO or VFIO driver. Then you use DPDK applications for
fast packet processing or use DPDK integrated Open vSwitch for switching.

Consider that Open Stack (WIP), Open Shift and maybe containers can use
the above setup, so isn't really a niche use case.  It's pretty much a
requirement for NFV projects as well.

> As you mention, you could use custom udev rules to actually bind a
> specific driver. I don't think we want anything more abstract in
> systemd.

The problem with that is we can't stop the service to roll back the
driver.  Or more importantly, create a dependency to, let's say,
start Open vSwitch only after the ports are bound to the right driver.
Or start an application that requires the NIC to be bound first.

Thanks,
fbl

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Kay Sievers]

On Mon, Sep 28, 2015 at 8:48 PM, Flavio Leitner  wrote:
> On Mon, Sep 28, 2015 at 08:06:50PM +0200, Kay Sievers wrote:
>> On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner  wrote:
>> > I am looking for guidance on how to properly resolve driver binding
>> > with systemd (which seems to me the best place to do that).
>>
>> This seems to be a too exotic and niche use case, nothing
>> general-purpose enough to implement high-level knobs in systemd.
>
> These alternative drivers might be new yet, so there is no much around
> them, but we need to start right to avoid problems in the future.
>
> Accelerated userspace datapath is one use case that changes the default
> NIC driver to UIO or VFIO driver. Then you use DPDK applications for
> fast packet processing or use DPDK integrated Open vSwitch for switching.
>
> Consider that Open Stack (WIP), Open Shift and maybe containers can use
> the above setup, so isn't really a niche use case.  It's pretty much a
> requirement for NFV projects as well.
>
>> As you mention, you could use custom udev rules to actually bind a
>> specific driver. I don't think we want anything more abstract in
>> systemd.
>
> The problem with that is we can't stop the service to roll back the
> driver.  Or more importantly, create a dependency to, let's say,
> start Open vSwitch only after the ports are bound to the right driver.
> Or start an application that requires the NIC to be bound first.

Sure, I understand all that. I would still call all that a niche and I
still don't think it's systemd's job to solve such a problem.

Kay
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to bind to other drivers using systemd

[Mantas Mikulėnas]

I wonder if this could be handled with a generic Type=oneshot,
ExecStart=driverctl bind foo...

-- 
Mantas Mikulėnas
On Sep 28, 2015 21:48, "Flavio Leitner"  wrote:

> On Mon, Sep 28, 2015 at 08:06:50PM +0200, Kay Sievers wrote:
> > On Sun, Sep 27, 2015 at 11:37 PM, Flavio Leitner 
> wrote:
> > > I am looking for guidance on how to properly resolve driver binding
> > > with systemd (which seems to me the best place to do that).
> >
> > This seems to be a too exotic and niche use case, nothing
> > general-purpose enough to implement high-level knobs in systemd.
>
> These alternative drivers might be new yet, so there is no much around
> them, but we need to start right to avoid problems in the future.
>
> Accelerated userspace datapath is one use case that changes the default
> NIC driver to UIO or VFIO driver. Then you use DPDK applications for
> fast packet processing or use DPDK integrated Open vSwitch for switching.
>
> Consider that Open Stack (WIP), Open Shift and maybe containers can use
> the above setup, so isn't really a niche use case.  It's pretty much a
> requirement for NFV projects as well.
>
> > As you mention, you could use custom udev rules to actually bind a
> > specific driver. I don't think we want anything more abstract in
> > systemd.
>
> The problem with that is we can't stop the service to roll back the
> driver.  Or more importantly, create a dependency to, let's say,
> start Open vSwitch only after the ports are bound to the right driver.
> Or start an application that requires the NIC to be bound first.
>
> Thanks,
> fbl
>
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel