Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Guillermo]

2018-01-15 8:39 GMT-03:00 Laurent Bercot:
>
>  The change is obviously incompatible with mdevd-0.0.1.0. It appears to
> work for me, but please test it - especially Olivier who was missing
> events with the old coldplug. If no problems are found, I'll cut the
> 0.1.0.0 release.

Worked for me too. All kernel modules loaded, same as when using eudev.

G.

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

I actually already have that trick used to log all events, so I can
tell you that since I booted I've had 640 events processed.

Just ran mdevd-coldplug again, and now I have 1172 events logged, so
that's 532 events generated by mdevd-coldplug just now.

In my log I have basically the output of "date && env && echo ---" and
based on said log, those 532 events are around 120 KB. As for the
640 original events, around 140 KB.

It seems it should all be under the 192 KB of the default buffer, but
as observed with that buffer mdevd runs out of space...

1 MB is propably too much, I should try to see how much is actually
needed. Next time I get to do some reboots I'll see what I can find.

Cheers,

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

And to answer your question, about what could cause such a burst of
events, this time I have an easy answer: mdevd-coldplug !


 Yes, I forgot to mention this.

 The kernel-generated events seem to be significantly heavier than the
mdev-style synthetic ones. The DEVPATH is the full path without
symlinks, which is usually pretty long, and there are also a bunch
of misc variables with every event. It's very common for a kernel-
generated event to reach 250 bytes, which means a 64kB buffer can
only hold 256 events, and probably less because some events are
easily 300 bytes long; so I tripled the buffer size.

 If it's not enough for a run-of-the-mill desktop PC, I should
increase it again. 1 MB sounds really excessive though: it means
it would be able to hold 3000 to 4000 events. Can you run
mdevd-coldplug with a dry-run mdevd to count the number of events,
you're getting and their approximate size? A nice trick to trace
events is to add the following line at the top of your mdev.conf file:

-.* 0:0 0600 ! @env

which will send a bunch of lines to your catch-all logger. :)

 Thanks for testing!

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

On Mon, 15 Jan 2018 11:39:19 +
"Laurent Bercot"  wrote:

>   The change is obviously incompatible with mdevd-0.0.1.0. It appears
> to work for me, but please test it - especially Olivier who was
> missing events with the old coldplug. If no problems are found, I'll
> cut the 0.1.0.0 release.

The good news is, it appears to work fine here as well, even with my
fancy hardware :p

As a side note however, note that when I updated I also reverted to the
default buffer size (which I noticed you upped), but then not only did
my audio card not show up (amongst other things, this one is noticable
because of a service waiting for it...) but some hard disks seemed not
to have been processed either.
A quick look in the log showed my old friend was back:

mdevd: fatal: unable to receive netlink message: No buffer space
available


And to answer your question, about what could cause such a burst of
events, this time I have an easy answer: mdevd-coldplug !

My first try when I noticed the error was simply to re-run
mdevd-coldplug, and I got the buffer error again (and still no
audio/etc)
Easy fix, I upped the buffer back to 1MB and everything went fine this
time. Rebooted since, with the buffer set to 1MB, and everything works
fine right away. So yay (but my fancy hardware requires a
"larger-than-normal" buffer for mdevd it seems).


Anyhow, it all works fine now, with mdevd & mdevd-coldplug.

Thanks!

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

OK, I did this, I compared what I got by reading every
/sys/class/*/*/uevent and /sys/bus/*/devices/*/uevent file (like
mdevd-coldplug does for /sys/dev/*/*/uevent) with what I got by
writing 'add' to them (like udevadm trigger does) and then using
'udevadm monitor --kernel --property' to watch the netlink events, and
there's no difference.


 Ah, but you are too late, Mr. Bond! The change was already in motion,
and all your competence cannot stop me from unleashing it upon the
world. And you will be in a prime spot to witness and experience it
firsthand, ha ha ha!

 Even though it's still possible to synthesize events, I find it
cleaner, as in more architecturally correct, to let the kernel create
them. The kernel is inherently more trustworthy than any userland
process, so it loosens up the requirements on validation: there will
be no fake events, and events will all be well-formed.
 The only reason why I did not do this before is that I was closely
following the "mdev -s" model, for compatibility; but if the model is
wrong, I'll switch to the "udevadm trigger" one in a heartbeat.

 This also solves Olivier's secondary problem, since "start the uevent
manager first, then run the coldplug and have the existing uevent
manager pick up events" becomes the correct - and only - way to make
it work properly.

 The latest mdevd git head implements those changes: mdevd-netlink and
mdevd have been merged, the mdevd-netlink binary has disappeared, and
mdevd-coldplug doesn't print anything on stdout but triggers kernel
events instead, scanning /sys/subsystem/*/devices/* when /sys/subsystem
exists and /sys/class/*/* and /sys/bus/*/devices/* otherwise.
mdevd uses the "-D fd" option to trigger readiness notification, since
the -d switch was already used.

 The change is obviously incompatible with mdevd-0.0.1.0. It appears to
work for me, but please test it - especially Olivier who was missing
events with the old coldplug. If no problems are found, I'll cut the
0.1.0.0 release.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Guillermo]

2018-01-13 12:40 GMT-03:00 Guillermo:
>
> 2018-01-12 23:10 GMT-03:00 Laurent Bercot:
>>
>>  Anyway, I will change to what udev does; but this is annoying because
>> it requires a rearchitecture, so it will take some time.
>>
>>  With the mdev -s model, it was possible to just read from sysfs,
>> synthesize events, and send the synthetic events to a dedicated mdevd
>> instance. With the udevadm trigger model, this is not what happens:
>> instead of reading from sysfs and synthesizing events, the coldplugger
>> actually pokes the kernel, which creates real events; and the netlink
>> listener must be up in order to receive and process them.
>
> I'm not sure you'd need to modify this part. The uevent files can
> still be read, systemd does try to read IFINDEX, MAJOR and MINOR from
> them during enumerator_scan_dir_and_add_devices(), and from what I've
> seen, the information you can read from them is pretty much the same
> as what you get from events triggered by writing to them. I'll tell
> you what, I'm going to try doing both things (using eudev's 'udevadm
> monitor' to check the netlink part) and see if I get the same results.

OK, I did this, I compared what I got by reading every
/sys/class/*/*/uevent and /sys/bus/*/devices/*/uevent file (like
mdevd-coldplug does for /sys/dev/*/*/uevent) with what I got by
writing 'add' to them (like udevadm trigger does) and then using
'udevadm monitor --kernel --property' to watch the netlink events, and
there's no difference.

udevadm monitor showed:

* ACTION=add, as expected. mdevd-coldplug adds it for the synthetic event.
* A DEVPATH starting with /device that corresponds to the /sys/devices
subdirectory. The DEVPATH added by mdevd-coldplug for the synthetic
event starts with /dev and corresponds to the /sys/dev symlink.
* A SEQNUM line that I suppose can be ignored.
* A SUBSYSTEM line. mdevd-coldplug adds it for the synthetic event
using readlinkat() with the 'subsystem' symlink found in the same
directory as the uevent file, and retrieving the last pathname
component. This works with /sys/{class,bus} too.

And everything else was the same as the contents of the uevent file.
So reading /sys/class/*/*/uevent and /sys/bus/*/devices/*/uevent (or
/sys/subsystem/*/devices/*/uevent instead, if it exists), synthesizing
events and sending them to mdevd's standard input would work, it
seems.

G.

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Guillermo]

2018-01-12 23:10 GMT-03:00 Laurent Bercot:
>
>  There are modules for which a /sys/class entry, and an appropriate
> major/minor pair for mdevd to create the device node, only appear when
> the module is loaded, and there's no prior uevent file to trigger
> loading the module: the one I've tested it on is ppp_generic.
> /sys/class/ppp/ppp/uevent only appears once you modprobe ppp_generic, at
> the same time as /sys/dev/char/108:0. So even scanning /sys/class
> does not guarantee you'll get events for all the modules you need.
> It probably has to do with the fact that ppp_generic isn't tied to
> any hardware in particular; whereas in Olivier's instance, I suspect
> the hardware was already visible somewhere under /sys/bus.

I'm leaning towards the same explanation. I don't think the device
manager was ever expected to load *all* needed modules, only the
hardware related ones. After all, systemd also has modules-load.d to
explicitly request loading them:

* https://www.freedesktop.org/software/systemd/man/modules-load.d.html

>  The only explanation I see is that the canonical way of triggering
> uevents changed after landley stopped being involved in sysfs and mdev.
> I think this is likely, since his doc mentions the obsolete (but still
> present) /sys/block hierarchy.

As a side note, /sys/block is duplicated in /sys/class/block, and
caught by the /sys/class/*/* scan. On my computer, disks and disk
partitions, for example, are only found there. I also wondered why
'udevadm trigger' didn't access /sys/block.

>  Anyway, I will change to what udev does; but this is annoying because
> it requires a rearchitecture, so it will take some time.
>
>  With the mdev -s model, it was possible to just read from sysfs,
> synthesize events, and send the synthetic events to a dedicated mdevd
> instance. With the udevadm trigger model, this is not what happens:
> instead of reading from sysfs and synthesizing events, the coldplugger
> actually pokes the kernel, which creates real events; and the netlink
> listener must be up in order to receive and process them.

I'm not sure you'd need to modify this part. The uevent files can
still be read, systemd does try to read IFINDEX, MAJOR and MINOR from
them during enumerator_scan_dir_and_add_devices(), and from what I've
seen, the information you can read from them is pretty much the same
as what you get from events triggered by writing to them. I'll tell
you what, I'm going to try doing both things (using eudev's 'udevadm
monitor' to check the netlink part) and see if I get the same results.

G.

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

It might be worth to look at what systemd [1] and eudev [2] do here.
When 'udevadm trigger --type=devices --action=add' is performed after
the udev daemon starts executing, it looks like what happens is
basically this:


 Thanks a lot for studying this. That's what I was going to do, but
I wouldn't have gotten to it before next week and you saved me a
lot of time.



On the other hand it looks like mdevd-coldplug and BusyBox' 'mdev -s'
want to access the /sys/dev/block/${major}:${minor} and
/sys/dev/char/${major}:${minor} symlinks. If it is the case that some
of these won't exist until the relevant kernel modules are loaded, and
if the device manager is expected to load them, it would mean it can't
reliably use /sys/dev.


 I've done some testing and the results are definitely weird.
 There are modules for which a /sys/class entry, and an appropriate
major/minor pair for mdevd to create the device node, only appear when
the module is loaded, and there's no prior uevent file to trigger
loading the module: the one I've tested it on is ppp_generic.
/sys/class/ppp/ppp/uevent only appears once you modprobe ppp_generic, at
the same time as /sys/dev/char/108:0. So even scanning /sys/class
does not guarantee you'll get events for all the modules you need.
It probably has to do with the fact that ppp_generic isn't tied to
any hardware in particular; whereas in Olivier's instance, I suspect
the hardware was already visible somewhere under /sys/bus.

 The other weird thing is that landley, who actually knew all this
because he wrote this documentation:
 https://landley.net/kdocs/pending/hotplug.txt
where he mentions /sys/bus/*/devices/*/dev and /sys/class/*/*/dev ,
chose to use /sys/dev/block and /sys/dev/char instead in mdev -s.
I assumed he knew what he was doing and simply duplicated functionality
from mdev -s, which was apparently the wrong thing to do.

 The only explanation I see is that the canonical way of triggering
uevents changed after landley stopped being involved in sysfs and mdev.
I think this is likely, since his doc mentions the obsolete (but still
present) /sys/block hierarchy.

 Anyway, I will change to what udev does; but this is annoying because
it requires a rearchitecture, so it will take some time.

 With the mdev -s model, it was possible to just read from sysfs,
synthesize events, and send the synthetic events to a dedicated mdevd
instance. With the udevadm trigger model, this is not what happens:
instead of reading from sysfs and synthesizing events, the coldplugger
actually pokes the kernel, which creates real events; and the netlink
listener must be up in order to receive and process them. This means
that it does not make sense to keep the "mdevd reads from stdin"
interface; mdevd and mdevd-netlink should be merged and be a single,
netlink-listening and uevent-processing daemon, and mdevd-coldplug
should be turned into a copy of "udevadm trigger".

 Sigh. These interfaces are almost as hidden, hermetic, and difficult
to work with as proprietary software.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

On Thu, 11 Jan 2018 23:29:45 -0300
Guillermo  wrote:

> #/bin/execlineb -P
> elglob sysclass /sys/class/*/*/uevent
> elglob sysbus /sys/bus/*/devices/*/uevent
> forx f { $sysclass $sysbus }
> importas -u ueventfile f
> redirfd -w 1 $ueventfile
> echo add

Just tried this, and it worked fine; All my devices were properly
processed. So this might be the way to go indeed, instead of a full
scan of /sys

Cheers,

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Guillermo]

2018-01-09 21:35 GMT-03:00 Laurent Bercot:
>
>  What it looks like is the kernel not assigning a major and a minor
> to the device before you manually trigger the "add". I suspect it's
> just that the relevant module is not loaded, but why would that be
> different from any other hardware managing to make mdevd autoload the
> appropriate module?
> [...]
>  Well I certainly don't want people to need to write their own
> trigger-uevents service, so I'll make sure your hardware is properly
> found by mdevd-coldplug. I'd just like to understand what is happening
> and add the proper fix rather than take a big hammer and scan all of
> /sys/devices just to be sure.

It might be worth to look at what systemd [1] and eudev [2] do here.
When 'udevadm trigger --type=devices --action=add' is performed after
the udev daemon starts executing, it looks like what happens is
basically this:

* If /sys/subsystem exists, an 'add' string is written to every
/sys/subsystem/*/devices/*/uevent file.
* If /sys/subsystem does not exist, an 'add' string is written to
every /sys/class/*/*/uevent and /sys/bus/*/devices/*/uevent file.

I have a 4.9-series kernel and no /sys/subsystem, but I do have
/sys/class/*/* and /sys/bus/*/devices/* files indeed, and they are in
all cases symbolic links to subdirectories of /sys/devices with
'uevent' files. So it kind of looks like a big hammer is used, but
with an easy to generate set of pathnames instead of actually
traversing
the /sys/devices hierarchy. Maybe this can be taken to be the
currently agreed sysfs interface for device managers (or else systemd
breaks).

On the other hand it looks like mdevd-coldplug and BusyBox' 'mdev -s'
want to access the /sys/dev/block/${major}:${minor} and
/sys/dev/char/${major}:${minor} symlinks. If it is the case that some
of these won't exist until the relevant kernel modules are loaded, and
if the device manager is expected to load them, it would mean it can't
reliably use /sys/dev.

For testing purposes, with mdevd-netlink running, either this:

#/bin/execlineb -P
elglob syssubsystem /sys/subsystem/*/devices/*/uevent
forx f { $syssubsystem }
importas -u ueventfile f
redirfd -w 1 $ueventfile
echo add

or this:

#/bin/execlineb -P
elglob sysclass /sys/class/*/*/uevent
elglob sysbus /sys/bus/*/devices/*/uevent
forx f { $sysclass $sysbus }
importas -u ueventfile f
redirfd -w 1 $ueventfile
echo add

could be used to see what happens, depending on the case. I don't have
mdevd installed with a suitable /etc/mdev.conf file at the moment, but
I tried executing the script while eudev's udevd was running (as a
replacement for udevadm trigger), and I did get all kernel modules to
load, and did see new /sys/dev/*/* symlinks after that.

G.

[1]
* https://github.com/systemd/systemd/blob/master/src/udev/udevadm-trigger.c
adm_trigger(), exec_list()
* https://github.com/systemd/systemd/blob/master/src/libudev/libudev-enumerate.c
udev_enumerate_scan_devices()
* 
https://github.com/systemd/systemd/blob/master/src/libsystemd/sd-device/device-enumerator.c
device_enumerator_scan_devices(), enumerator_scan_devices_all(),
enumerator_scan_dir(), enumerator_scan_dir_and_add_devices()
* 
https://github.com/systemd/systemd/blob/master/src/libsystemd/sd-device/sd-device.c
sd_device_new_from_syspath()

[2]
* https://github.com/gentoo/eudev/blob/master/src/udev/udevadm-trigger.c
adm_trigger(), exec_list()
* https://github.com/gentoo/eudev/blob/master/src/libudev/libudev-enumerate.c
udev_enumerate_scan_devices(), scan_devices_all(), scan_dir(),
scan_dir_and_add_devices(), syspath_add()

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

On Wed, 10 Jan 2018 00:35:33 +
"Laurent Bercot"  wrote:

>   Question that may help:
>   - does your /sys/devices/pci:00/:00:01.0/:01:00.1
> directory contain a "dev" file: before you manually trigger the add?
> afterwards?

Nope, afraid not; neither before nor after. What happens is that, in
each of the case I tried (usb mouse, audio card, pc speaker), after I
write "add" to the uevent file, a new directory will have appeared in
the folder, e.g. mouse0 or input/input5/event5, which will be the one
containing a dev file. (Of course by then there's also a corresponding
syslink in /sys/dev/char).

Sorry, not sure how to avoid using a big hammer...


Cheers,

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

Thing is, before writing to that uevent, there was nothing of the sort
in /sys/dev.


 What it looks like is the kernel not assigning a major and a minor
to the device before you manually trigger the "add". I suspect it's
just that the relevant module is not loaded, but why would that be
different from any other hardware managing to make mdevd autoload the
appropriate module? It's weird, and my kernel knowledge is insufficient
to understand this. If someone's more familiar with the internals of
the Linux kernel, especially module loading and hardware detection,
please stand up.

 Question that may help:
 - does your /sys/devices/pci:00/:00:01.0/:01:00.1 directory
contain a "dev" file: before you manually trigger the add? afterwards?



Note that I don't mind just using my trigger-uevents service instead,
it's what I used to do anyway; I just figured, as I switched to mdevd,
since there's a coldplug thing might as well give it a try...


 Well I certainly don't want people to need to write their own
trigger-uevents service, so I'll make sure your hardware is properly
found by mdevd-coldplug. I'd just like to understand what is happening
and add the proper fix rather than take a big hammer and scan all of
/sys/devices just to be sure.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

First off, about the buffer issue, I'm not sure why but I couldn't
reproduce it. :/ Rebooted a few times, with the default buffer, and
mdevd-netlink never complained.
I still have the old messages in my logs, it happened every single
reboot, and not once this time don't know. Maybe I'll try again
later, maybe it's the weather... :D
(I even tried rebooting w/out strace/tee in case, same thing. Damn, I
don't get it, why do things work? uh?... :p)


On Tue, 09 Jan 2018 13:36:48 +
"Laurent Bercot"  wrote:

> >By that I mean, currently if I don't do anything else, some things
> >aren't processed. For instance, my audio card isn't there, neither
> >are a few input devices, such as my (usb) mouse.  
> 
>   I can't reproduce the problem on my system: mdevd-coldplug detects
> all the USB devices. But it's a small system with no fancy hardware.

Not sure I have fancy hardware, but who knows :p

% uname -a
Linux arch.local 4.14.11-1-ARCH #1 SMP PREEMPT Wed Jan 3 07:02:42 UTC
2018 x86_64 GNU/Linux

Not an old kernel, and no firmware involved. What I found out is that,
there are no symlinks in /sys/dev at first no. I'll use my audio card as
example.

So upon boot, nothing in /sys/dev points to it. But if I write "add"
to /sys/devices/pci:00/:00:01.0/:01:00.1/uevent then magic
happens, uevents are triggered, modules loaded, and now my audio card is
there. Also symlinks of related devices do appear in /sys/dev/char:

ls -lR /sys/dev/char|grep 01:00.1
lrwxrwxrwx 1 root root 0 Jan  9 19:58 116:2
-> ../../devices/pci:00/:00:01.0/:01:00.1/sound/card1/controlC1
lrwxrwxrwx 1 root root 0 Jan  9 19:58 116:3
-> ../../devices/pci:00/:00:01.0/:01:00.1/sound/card1/pcmC1D3p
lrwxrwxrwx 1 root root 0 Jan  9 19:58 116:4
-> ../../devices/pci:00/:00:01.0/:01:00.1/sound/card1/hwC1D0
lrwxrwxrwx 1 root root 0 Jan  9 19:58 13:67
-> ../../devices/pci:00/:00:01.0/:01:00.1/sound/card1/input3/event3

Thing is, before writing to that uevent, there was nothing of the sort
in /sys/dev.


% pwd
/sys/devices/pci:00/:00:01.0/:01:00.1

% cat uevent
DRIVER=snd_hda_intel
PCI_CLASS=40300
PCI_ID=1002:AA28
PCI_SUBSYS_ID=1462:AA28
PCI_SLOT_NAME=:01:00.1
MODALIAS=pci:v1002dAA28sv1462sdAA28bc04sc03i00

% ls -l subsystem
lrwxrwxrwx 1 root root 0 Jan  9 20:07 subsystem -> ../../../../bus/pci



I see similar results for e.g. my mouse, not there (or in /dev/input)
until I write "add"
to 
/sys/devices/pci:00/:00:1d.2/usb8/8-1/8-1:1.0/0003:045E:0040.0001/input/input1/uevent;
or the pc speaker, not there until I write "add"
to /sys/devices/platform/pcspkr/uevent, only then does /sys/dev/char/13:66 
(symlink
  to ../../devices/platform/pcspkr/input/input2/event2) appear.


If that's any consolation, `mdev -s` doesn't do any better than
mdevd-coldplug, so maybe it works as expected, and I do have fancy
hardware after all.
(I can't remember if I had tried it before or not, but maybe that was
why I didn't use it, and used my own trigger-uevents service instead.)

Note that I don't mind just using my trigger-uevents service instead,
it's what I used to do anyway; I just figured, as I switched to mdevd,
since there's a coldplug thing might as well give it a try...


Cheers,

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

By that I mean, currently if I don't do anything else, some things
aren't processed. For instance, my audio card isn't there, neither are
a few input devices, such as my (usb) mouse.


 I can't reproduce the problem on my system: mdevd-coldplug detects
all the USB devices. But it's a small system with no fancy hardware.

 What would really be helpful in diagnosing this would be:
 - your kernel version: if you're using an old kernel, maybe /sys
has a subtly different format
 - if you can find them, the contents of the directories in /sys
describing the hardware that is missed by mdevd-coldplug; in particular
   * whether they are accessible via a symlink in /sys/dev/char
(or /sys/dev/block, if mdevd-coldplug is also missing block devices)
   * the contents of their "dev" and "uevent" files
   * whether they have a "subsystem" symlink and what it points to
   * whether that hardware requires loading firmware

 Thanks!

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

On Mon, 08 Jan 2018 19:31:17 +
"Laurent Bercot"  wrote:

> >But, it seems to me that when you're talking about this you expect
> >one to have two different instances of mdevd, one for netlinkd &
> >another one
> >for coldplug. That's not how I made things however: mdevd is a
> >longrun and I simply have both coldplug & netlinkd piped into it.  
> 
>   That can work, but I think it's overly complex.

Well, I don't find it overly complex. It's just two services piped into
the same third one, nothing I find too complex. It's certainly not the
first time I would get different services piped into another common
one. (Even if I don't have helpers as in s6-rc, it's easy enough to do
as well.)


>   One is long-lived, reading from mdevd-netlink, and can be logged ;
> if you want to both supervise it separately from mdevd-netlink and
> log its output, you need a supervision architecture that can pipeline
> more than two longruns. (I wonder where that can be found. ;))
>   The other one is short-lived, reading from mdevd-coldplug which is a
> short-lived program. So you would just have a "mdevd-coldplug | mdevd"
> oneshot running after the mdevd-netlink pipeline has been launched.
> The second mdevd will simply die when the coldplug is over.

Right. I wasn't too worried about running two instances of mdevd at
once, it's just that I like it better when I have the one mdevd
service, and anything it has to say gets logged in the same place,
always, no matter the "source" of the event.

Otherwise it's either the log from mdevd, or coldplug, depending on the
source. But it's not a big deal. I'll see whether I keep things as I
have them, or make mdevd-coldplug piped into its own mdevd.


> >On another topic of sorts: so I start mdevd, netlinkd, and run 
> >coldplug.
> >Cool. But what else would I need to do (upon boot) to ensure
> >everything has been processed correctly?
> >
> >By that I mean, currently if I don't do anything else, some things
> >aren't processed. For instance, my audio card isn't there, neither
> >are a few input devices, such as my (usb) mouse.  
> 
>   Oh? In that case, I would say it's a bug in mdevd-coldplug, which is
> supposed to process everything. Do you have a pattern of stuff that
> isn't processed and should be?

Well, as I said I would always have my audio card & a few input devices
missing. It might be some modules that aren't loaded, and until
then the devices don't show up, hence mdevd-coldplug not finding
them under /sys/dev, but I don't know much else I'm afraid...


>   Wow. That sounds very weird. What in your boot sequence could create
> such a huge burst of events? Can you strace mdevd-netlink to see
> what it's reading from the kernel, and tee its output to check whether
> those are legit events to be sent to mdevd or line noise that is
> improperly filtered at netlink registration time?

The only thing I can think of that would generate a burst of events is
my trigger-uevents script mentionned earlier, but I think I got that
error when it wasn't started on boot...

I'll see if I can strace mdevd-netlink when I get a chance and report
back.


Cheers,

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Colin Booth]

On Mon, Jan 08, 2018 at 07:31:17PM +, Laurent Bercot wrote:
> > #!/usr/bin/execlineb -P
> > pipeline { find /sys -type f -name uevent -print0 }
> > forstdin -0 -p f importas -u f f redirfd -w 1 $f echo add
>  The problem with your script is that it's getting a lot of
> duplicates, since multiple symlinks in /sys point to the same
> directory describing your device. I'm still hoping to avoid scanning
> the entirety of /sys, so I'd like to find a correct pattern, but if
> there's no other way, I'll just scan everything discarding symlinks.
> 
'-type f' should only find regular files, not symlinks. I don't believe
it will even recurse into symlinks by default so deduplication shouldn't
be necessary.

-- 
Colin Booth

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

However, somewhat recently I believe I heard (read) you say on IRC that
the recommended way was actually the other way around, so I switched to
start netlinkd first, and then run coldplug.

 Well, it's really up to you.
 The thing is, if you coldplug first then listen to the netlink, you
have a window where events can appear and won't be picked up: the
netlink listener isn't up yet so the kernel is talking to the hand,
but the coldplugr has already scanned /sys and won't do it again,
so nobody will notice a device needs to be added.

 Listening to the netlink first and then running the coldplug makes
sure there's no race condition where events will be ignored.



But, it seems to me that when you're talking about this you expect one
to have two different instances of mdevd, one for netlinkd & another 
one

for coldplug. That's not how I made things however: mdevd is a longrun
and I simply have both coldplug & netlinkd piped into it.


 That can work, but I think it's overly complex.



If not, does that mean the recommended way is really to have two
instances of mdevd? Or is there another/better way I'm not seeing? I
like to have only a single mdevd, one longrun (logged) service, as (I
feel) it should be.


 mdevd is not a long-lived process per se: it simply reads from its
stdin, and it dies when it reads EOF. Think "cat". The real long-lived
process is mdevd-netlink, which normally never dies, and never closes
its stdout, so an attached mdevd process would never die either.

 So yes, it's okay to have - temporarily - two mdevd instances, that's
the design I had in mind.

 One is long-lived, reading from mdevd-netlink, and can be logged ; if
you want to both supervise it separately from mdevd-netlink and log its
output, you need a supervision architecture that can pipeline more than
two longruns. (I wonder where that can be found. ;))
 The other one is short-lived, reading from mdevd-coldplug which is a
short-lived program. So you would just have a "mdevd-coldplug | mdevd"
oneshot running after the mdevd-netlink pipeline has been launched.
The second mdevd will simply die when the coldplug is over.

 It's not a problem to have two mdevd instances running concurrently.
The worst that can happen is that an event arrives after mdevd-netlink
is running but before mdevd-coldplug has finished scanning /sys; in
that case, the event will be picked up twice, and depending on how
complex and duplicate-resistant the event handling is in mdev.conf,
shenanigans may happen. But:
 - for the majority of devices, nothing special will happen; the
later instance will just fail to perform some action because the earlier
instance has already done it, it will log a warning and ignore the 
event.

 - this problem would also occur with a single mdevd instance; you
would simply send it the same event twice.

 I don't think that corner case is worth worrying about. It is
unavoidable no matter the device manager you're using.


On another topic of sorts: so I start mdevd, netlinkd, and run 
coldplug.

Cool. But what else would I need to do (upon boot) to ensure everything
has been processed correctly?

By that I mean, currently if I don't do anything else, some things
aren't processed. For instance, my audio card isn't there, neither are
a few input devices, such as my (usb) mouse.


 Oh? In that case, I would say it's a bug in mdevd-coldplug, which is
supposed to process everything. Do you have a pattern of stuff that
isn't processed and should be?



#!/usr/bin/execlineb -P
pipeline { find /sys -type f -name uevent -print0 }
forstdin -0 -p f importas -u f f redirfd -w 1 $f echo add


 This is the job of mdevd-coldplug you're duplicating, so yeah, if
you have to do that, then it's better to fix mdevd-coldplug. :)

 The problem with your script is that it's getting a lot of
duplicates, since multiple symlinks in /sys point to the same
directory describing your device. I'm still hoping to avoid scanning
the entirety of /sys, so I'd like to find a correct pattern, but if
there's no other way, I'll just scan everything discarding symlinks.


Now I may be doing something wrong somewhere, but I've always had to 
use

a larger buffer to make sure things worked correctly myself (That is,
even before w/ s6-uevent-listener).
I'm currently using 1 MB (1048576) to be sure, though it should work
with a smaller one, but I know that even with 131072 (so twice your
"large side" default) I'd always get the following error during boot:

mdevd-netlink: fatal: unable to receive netlink message: No buffer
space available


 Wow. That sounds very weird. What in your boot sequence could create
such a huge burst of events? Can you strace mdevd-netlink to see
what it's reading from the kernel, and tee its output to check whether
those are legit events to be sent to mdevd or line noise that is
improperly filtered at netlink registration time?

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Olivier Brunel]

Hey there :)

So I've been using this (instead of your old uevent handler & busybox's
mdev) for a little while now, works great; Thanks!

I do have a few questions though. Originally what I had done was to
start mdevd, run mdevd-coldplug (referred to as coldplug thereafter),
and then start mdevd-netlink (referred to as netlinkd thereafter).

However, somewhat recently I believe I heard (read) you say on IRC that
the recommended way was actually the other way around, so I switched to
start netlinkd first, and then run coldplug.

But, it seems to me that when you're talking about this you expect one
to have two different instances of mdevd, one for netlinkd & another one
for coldplug. That's not how I made things however: mdevd is a longrun
and I simply have both coldplug & netlinkd piped into it.

So for that to work with this new setup/order, I needed to have both do
atomic writes, to ensure things don't get mangled/mixed up. (Wasn't
needed before as I could simply make sure netlinkd only starts after
coldplug, the later being a oneshot.)

Things were good for netlinkd, but not coldplug. So I simply patched it
to flush its output after each uevent. (This being linux and pipe,
I believe atomic writes are ensured so long as we don't go over 4KB,
which I assume won't happen for a single uevent.)

But since that means coldplug now performs a few more write calls, I'm
guessing you might not be interrested in doing such a change upstream,
or am I mistaking?
If not, does that mean the recommended way is really to have two
instances of mdevd? Or is there another/better way I'm not seeing? I
like to have only a single mdevd, one longrun (logged) service, as (I
feel) it should be.


On another topic of sorts: so I start mdevd, netlinkd, and run coldplug.
Cool. But what else would I need to do (upon boot) to ensure everything
has been processed correctly?

By that I mean, currently if I don't do anything else, some things
aren't processed. For instance, my audio card isn't there, neither are
a few input devices, such as my (usb) mouse.

So what I'm doing currently, is have a small trigger-uevents oneshot,
that does this:


#!/usr/bin/execlineb -P
pipeline { find /sys -type f -name uevent -print0 }
forstdin -0 -p f importas -u f f redirfd -w 1 $f echo add


It works, but I'm wondering whether this is the right thing to do, or if
there's a better/more correct way to get there?


Lastly, as a side note, on the doc for netlinkd, regarding the buffer
size, you say "The default is 65536 (which is on the large side)." (Same
as you did before for s6-uevent-listener I believe.)

Now I may be doing something wrong somewhere, but I've always had to use
a larger buffer to make sure things worked correctly myself (That is,
even before w/ s6-uevent-listener).
I'm currently using 1 MB (1048576) to be sure, though it should work
with a smaller one, but I know that even with 131072 (so twice your
"large side" default) I'd always get the following error during boot:

mdevd-netlink: fatal: unable to receive netlink message: No buffer
space available



Thanks!

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

   Excuse me for insisting so long and off-topic. For my excuse let me 
say my only will was to use mdevd *and* Xorg. I'm not fond of politics 
more than you. Let's put an end to this thread.


 It's ok. It's not off-topic: I wrote mdevd, so I should expect those
discussions.

 Casper brought a good point: libudev is also about allowing 
applications

to access the uevent stream, in a dynamic way, i.e. more flexible than
just the mdev.conf configuration file. I will think about a simple way
to do the same.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 16/11/2017 à 12:31, Laurent Bercot a écrit :

   I fully understand that this list is focused on good programming
practice and libudev is considered something disgusting in that
respect. My question was triggered by the frustration, probably
shared by many, of not being able to replace Udev by Mdev or Mdevd,
only because Xorg autoconfig has become a must.


 And do you think it's fair to direct that frustration at people who have
nothing to do with its causes and cannot do anything about it, instead of
the people in charge?

 The issue is not a technical one: it's a political one. I can provide
technical solutions to technical problems, but if you want to tackle
political problems, you will need someone with more political power
and more willingness to do politics.



Using the same API as libudev isn't necessary, because Xorg might be
patched to use a different one, but what is needed is to hide the
unstability of sysfs by some run-time abstraction layer.

   Replacing systemd-udev by something simpler and well done, like
mdev or mdevd is not only a technical, but also a crucial political
issue for software freedom. So hopping somebody with the skills will
come out with a good idea


 Apparently I was not clear enough, so let me repeat: replacing udev with
something simpler and well done is, AFAICT, impossible as long as you
stick to the notion of "hide the unstability of sysfs by some run-time
abstraction layer". The run-time abstraction layer is exactly what
libudev is; if you start with the same concepts as udev and implement
from
there, you *will* end up with udev, there's no way around it.

 Maybe an alternative implementation of libudev would be better; it
probably would, because given freedesktop.org's (and especially Kay
Sievers') history, chances are the current implementation of libudev is
terrible. But a new implementation *would not solve the underlying
issue*. Any gains would only be marginal; a new API could be somewhat
different,
but not different enough from libudev's API to be worth it. At the end
of the day, it would just split up the software ecosystem and incur
maintenance costs, for what? the satisfaction of not having GKH and KS
control uevent management in Xorg, knowing full well that the alternative
does the exact same thing as libudev with the exact same issues,
minus some minor implementation ones?

 When I look at libudev, I see a bunch of bloaty wrappers and generic
functionality put in a nice package and enforced on people. I don't
like it any more than you do, but the real fix is to do away with the
need for that bloat in the first place; it definitely is not to go NIH
and say "ok, we'll do our own libudev, which will be just as useless, but
at least it won't be yours, nyah nyah nyah".

 I'm interested in writing code that does real things, not in writing
bloaty wrappers, even if I know that MY bloaty wrappers would be the
best bloaty wrappers ever.

 Tell you what, I do have an idea for a libudev alternative: in your
application, you write your code *exactly* as you would if you were
accessing the files in sysfs directly, except you don't access files
under /sys, you access files under /definitelynotsys instead. And
you defer the responsibility of the /sys to /definitelynotsys mapping
to a third-party layer; I can even be that third party, if you want.
Nobody can blame you: you're not accessing sysfs, you're just using
my API. And it's nobody's business if a common implementation of my API
is making /definitelynotsys a symlink to /sys.

--
 Laurent


Dear Laurent.

Excuse me for insisting so long and off-topic. For my excuse let me 
say my only will was to use mdevd *and* Xorg. I'm not fond of politics 
more than you. Let's put an end to this thread.


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

   I fully understand that this list is focused on good programming 
practice and libudev is considered something disgusting in that 
respect. My question was triggered by the frustration, probably shared 
by many, of not being able to replace Udev by Mdev or Mdevd, only 
because Xorg autoconfig has become a must.


 And do you think it's fair to direct that frustration at people who 
have
nothing to do with its causes and cannot do anything about it, instead 
of

the people in charge?

 The issue is not a technical one: it's a political one. I can provide
technical solutions to technical problems, but if you want to tackle
political problems, you will need someone with more political power
and more willingness to do politics.


Using the same API as libudev isn't necessary, because Xorg might be 
patched to use a different one, but what is needed is to hide the 
unstability of sysfs by some run-time abstraction layer.


   Replacing systemd-udev by something simpler and well done, like mdev 
or mdevd is not only a technical, but also a crucial political issue 
for software freedom. So hopping somebody with the skills will come out 
with a good idea


 Apparently I was not clear enough, so let me repeat: replacing udev 
with

something simpler and well done is, AFAICT, impossible as long as you
stick to the notion of "hide the unstability of sysfs by some run-time
abstraction layer". The run-time abstraction layer is exactly what
libudev is; if you start with the same concepts as udev and implement 
from

there, you *will* end up with udev, there's no way around it.

 Maybe an alternative implementation of libudev would be better; it
probably would, because given freedesktop.org's (and especially Kay
Sievers') history, chances are the current implementation of libudev is
terrible. But a new implementation *would not solve the underlying 
issue*. Any gains would only be marginal; a new API could be somewhat 
different,

but not different enough from libudev's API to be worth it. At the end
of the day, it would just split up the software ecosystem and incur
maintenance costs, for what? the satisfaction of not having GKH and KS
control uevent management in Xorg, knowing full well that the 
alternative

does the exact same thing as libudev with the exact same issues,
minus some minor implementation ones?

 When I look at libudev, I see a bunch of bloaty wrappers and generic
functionality put in a nice package and enforced on people. I don't
like it any more than you do, but the real fix is to do away with the
need for that bloat in the first place; it definitely is not to go NIH
and say "ok, we'll do our own libudev, which will be just as useless, 
but

at least it won't be yours, nyah nyah nyah".

 I'm interested in writing code that does real things, not in writing
bloaty wrappers, even if I know that MY bloaty wrappers would be the
best bloaty wrappers ever.

 Tell you what, I do have an idea for a libudev alternative: in your
application, you write your code *exactly* as you would if you were
accessing the files in sysfs directly, except you don't access files
under /sys, you access files under /definitelynotsys instead. And
you defer the responsibility of the /sys to /definitelynotsys mapping
to a third-party layer; I can even be that third party, if you want.
Nobody can blame you: you're not accessing sysfs, you're just using
my API. And it's nobody's business if a common implementation of my API
is making /definitelynotsys a symlink to /sys.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 16/11/2017 à 10:25, Casper Ti. Vector a écrit :

On Wed, Nov 15, 2017 at 02:53:21PM +, Laurent Bercot wrote:

  Is it really the only place in Xorg that depends on libudev?
I'd think it would be much, much more entangled with libudev than
this.

Unfortunately, just as noted by Guillermo, libudev is also, in more
complicated ways, needed by quite a few x11 drivers and (perhaps most
importantly) xserver itself.  I apologise for (again) not thoroughly
investigating the issues before replying :(


  So, yeah. If you want a patch for xf86-input-evdev that will
make it stop depend on libudev, I can write one in an hour. But
that patch will likely never go upstream, because it goes against
policy; and complying with the policy would basically amount to
rewriting libudev and making a new udevd. Which I believe I'm quite
able to do, and better than the existing ones, but it would still be
bad software design, and I have no interest in contributing to the
problem.

(Again, I am not someone fluent in low level programming, so please feel
free to correct me if I make stupid mistakes in the following.)

 From my cursory glance at the x11 code that use libudev, and the libudev
documentation, the main functionalities of libudev seem to fall into two
almost orthogonal categories: those that abstract the sysfs interface,
and those that handle the udev event queue.  I have not come up with a
good idea on how to provide the latter in a vender-neutral way, but a
neutral design of the former seems obvious: if sysfs was really never
intended to be a stable interface at all, at least make a minimalist
library outside udev that provide the necessary functionalities.

However, as you noted, "sysfs has not changed interfaces to the point of
breaking stuff in 12 years and counting", so the intended instability is
not really an excuse; and even if we pretend that sysfs is so unstable,
stating that sysfs is "a private export only to be consumed by udev" [1]
(ie. not mdev, nldev, vdev, etc.) has no technical basis.  But noticing
that the actors in the drama include GKH and KS, this is unsurprising;
this is also the reason I refered to the conspiracy theory again.

[1] 

I agree that there is a conspiracy. The mails exchanged by Rob 
Landley are very explicit.


I don't think Linus Torvalds would tolerate on the long term that a 
group of people develop a private kernel interface for the purpose of 
forcing systemd to all Linux users. In fact I guess he is aware of this, 
and even sensitive to the bad quality of their code, but he will not 
move because only one person complains.


Rob Landley would have a stronger argument if a large number of 
users had their desktops running mdev and undocumented changes in sysfs 
breaks it.


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Casper Ti. Vector]

On Wed, Nov 15, 2017 at 02:53:21PM +, Laurent Bercot wrote:
>  Is it really the only place in Xorg that depends on libudev?
> I'd think it would be much, much more entangled with libudev than
> this.

Unfortunately, just as noted by Guillermo, libudev is also, in more
complicated ways, needed by quite a few x11 drivers and (perhaps most
importantly) xserver itself.  I apologise for (again) not thoroughly
investigating the issues before replying :(

>  So, yeah. If you want a patch for xf86-input-evdev that will
> make it stop depend on libudev, I can write one in an hour. But
> that patch will likely never go upstream, because it goes against
> policy; and complying with the policy would basically amount to
> rewriting libudev and making a new udevd. Which I believe I'm quite
> able to do, and better than the existing ones, but it would still be
> bad software design, and I have no interest in contributing to the
> problem.

(Again, I am not someone fluent in low level programming, so please feel
free to correct me if I make stupid mistakes in the following.)

>From my cursory glance at the x11 code that use libudev, and the libudev
documentation, the main functionalities of libudev seem to fall into two
almost orthogonal categories: those that abstract the sysfs interface,
and those that handle the udev event queue.  I have not come up with a
good idea on how to provide the latter in a vender-neutral way, but a
neutral design of the former seems obvious: if sysfs was really never
intended to be a stable interface at all, at least make a minimalist
library outside udev that provide the necessary functionalities.

However, as you noted, "sysfs has not changed interfaces to the point of
breaking stuff in 12 years and counting", so the intended instability is
not really an excuse; and even if we pretend that sysfs is so unstable,
stating that sysfs is "a private export only to be consumed by udev" [1]
(ie. not mdev, nldev, vdev, etc.) has no technical basis.  But noticing
that the actors in the drama include GKH and KS, this is unsurprising;
this is also the reason I refered to the conspiracy theory again.

[1] 

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 16/11/2017 à 03:01, Guillermo a écrit :

2017-11-15 14:11 GMT-03:00 Didier Kryn:

 I don't know what evdev is (I guess some virtual device) and what's its
place in the big Xorg picture, but I think some of you guys know it. Any
usefull link?

X11 comes in pieces; there is the server, normally installed as
/usr/bin/X and / or /usr/bin/Xorg, and modules, that e.g. Devuan
installs in /usr/lib/xorg/modules, I believe, and that include the
video drivers (/usr/lib/xorg/modules/drivers) and input drivers
(/usr/lib/xorg/modules/input). Evdev an input driver (i.e. it handles
the keyboard, mouse, etc.) that IIUC uses Linux' event interface
(CONFIG_INPUT_EVDEV, "Say Y here if you want your input device events
be accessible under char device 13:64+ - /dev/input/eventX in a
generic way").

You can see which modules are loaded by X at startup in its log file
(normally /var/log/Xorg*.log), and if your computer has the package
that provides the evdev driver installed, see 'man evdev'.

2017-11-15 11:53 GMT-03:00 Laurent Bercot:

  Is it really the only place in Xorg that depends on libudev?
I'd think it would be much, much more entangled with libudev than
this.

Indeed it is. The X server itself:

* https://cgit.freedesktop.org/xorg/xserver/tree/config/udev.c

The modesetting video driver:

* 
https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/drivers/modesetting/drmmode_display.c
(drmmode_*uevent*())

The Intel video driver:

* 
https://cgit.freedesktop.org/xorg/driver/xf86-video-intel/tree/src/uxa/intel_driver.c
(I830*UEvent*())

The amdgpu video driver:

* 
https://cgit.freedesktop.org/xorg/driver/xf86-video-amdgpu/tree/src/drmmode_display.c
(drmmode_*uevent*())

And probably more. Although it seems that in most cases the dependency
on the libudev API is selectable by an option at compile time.

G.


IIUC evdev merges inputs from various devices, keyboard(s), 
mouse(s) and touchpad(s). But autoconfig has mostly to do with display 
size and shape, and this is rather an output device. Sorry, this is 
becoming off-topic.


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 15/11/2017 à 19:27, Laurent Bercot a écrit :

   "Applications should not access sysfs" doesn't mean there should
be only one version of the intermediate library. There's already
Systemd-Udev and Eudev versions, at least. For the case of Xorg, it
just means sysfs access should not be hard-coded in the source of
Xorg, but there should be some library in between, so that it is not
necessary to recompile Xorg for every version of the kernel. So any
replacement for the needed subset of libudev would do it.


 Yes, but that's the whole point: the needed information is available
as is in sysfs, and putting a library in-between is 100% bloat. It's
just about testing something in the device path, and the device path
is just that - a path in sysfs. The way libudev proceeds to get
that information is entirely more complex than necessary - it even adds
operations that can fail, such as memory allocation, when it is
entirely useless here; but with the way the libudev architecture is
designed, it is the only way, and there's no escaping the bloat,
added SPOF, added failure points.

 There is _no good way_ to implement the interesting parts of
EvdevDeviceIsVirtual() without directly accessing sysfs. It's either
sysfs, or libudev, or a libudev replacement that would, by design,
necessarily be just as bad as libudev.

--
 Laurent

I fully understand that this list is focused on good programming 
practice and libudev is considered something disgusting in that respect. 
My question was triggered by the frustration, probably shared by many, 
of not being able to replace Udev by Mdev or Mdevd, only because Xorg 
autoconfig has become a must. Using the same API as libudev isn't 
necessary, because Xorg might be patched to use a different one, but 
what is needed is to hide the unstability of sysfs by some run-time 
abstraction layer.


Replacing systemd-udev by something simpler and well done, like 
mdev or mdevd is not only a technical, but also a crucial political 
issue for software freedom. So hopping somebody with the skills will 
come out with a good idea :-)


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Guillermo]

2017-11-15 14:11 GMT-03:00 Didier Kryn:
>
> I don't know what evdev is (I guess some virtual device) and what's its
> place in the big Xorg picture, but I think some of you guys know it. Any
> usefull link?

X11 comes in pieces; there is the server, normally installed as
/usr/bin/X and / or /usr/bin/Xorg, and modules, that e.g. Devuan
installs in /usr/lib/xorg/modules, I believe, and that include the
video drivers (/usr/lib/xorg/modules/drivers) and input drivers
(/usr/lib/xorg/modules/input). Evdev an input driver (i.e. it handles
the keyboard, mouse, etc.) that IIUC uses Linux' event interface
(CONFIG_INPUT_EVDEV, "Say Y here if you want your input device events
be accessible under char device 13:64+ - /dev/input/eventX in a
generic way").

You can see which modules are loaded by X at startup in its log file
(normally /var/log/Xorg*.log), and if your computer has the package
that provides the evdev driver installed, see 'man evdev'.

2017-11-15 11:53 GMT-03:00 Laurent Bercot:
>
>  Is it really the only place in Xorg that depends on libudev?
> I'd think it would be much, much more entangled with libudev than
> this.

Indeed it is. The X server itself:

* https://cgit.freedesktop.org/xorg/xserver/tree/config/udev.c

The modesetting video driver:

* 
https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/drivers/modesetting/drmmode_display.c
(drmmode_*uevent*())

The Intel video driver:

* 
https://cgit.freedesktop.org/xorg/driver/xf86-video-intel/tree/src/uxa/intel_driver.c
(I830*UEvent*())

The amdgpu video driver:

* 
https://cgit.freedesktop.org/xorg/driver/xf86-video-amdgpu/tree/src/drmmode_display.c
(drmmode_*uevent*())

And probably more. Although it seems that in most cases the dependency
on the libudev API is selectable by an option at compile time.

G.

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

   "Applications should not access sysfs" doesn't mean there should be 
only one version of the intermediate library. There's already 
Systemd-Udev and Eudev versions, at least. For the case of Xorg, it 
just means sysfs access should not be hard-coded in the source of Xorg, 
but there should be some library in between, so that it is not 
necessary to recompile Xorg for every version of the kernel. So any 
replacement for the needed subset of libudev would do it.


 Yes, but that's the whole point: the needed information is available
as is in sysfs, and putting a library in-between is 100% bloat. It's
just about testing something in the device path, and the device path
is just that - a path in sysfs. The way libudev proceeds to get
that information is entirely more complex than necessary - it even adds
operations that can fail, such as memory allocation, when it is
entirely useless here; but with the way the libudev architecture is
designed, it is the only way, and there's no escaping the bloat,
added SPOF, added failure points.

 There is _no good way_ to implement the interesting parts of
EvdevDeviceIsVirtual() without directly accessing sysfs. It's either
sysfs, or libudev, or a libudev replacement that would, by design,
necessarily be just as bad as libudev.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 15/11/2017 à 15:53, Laurent Bercot a écrit :

Linus has said, multiple times, that sysfs wasn't supposed to be a
stable interface, so applications should just not access sysfs.


"Applications should not access sysfs" doesn't mean there should be 
only one version of the intermediate library. There's already 
Systemd-Udev and Eudev versions, at least. For the case of Xorg, it just 
means sysfs access should not be hard-coded in the source of Xorg, but 
there should be some library in between, so that it is not necessary to 
recompile Xorg for every version of the kernel. So any replacement for 
the needed subset of libudev would do it.


I don't know what evdev is (I guess some virtual device) and what's 
its place in the big Xorg picture, but I think some of you guys know it. 
Any usefull link?


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

The only fragment in xf86-input-evdev that requires libudev is
(from src/evdev.c, with some changes in whitespaces):


 Is it really the only place in Xorg that depends on libudev?
I'd think it would be much, much more entangled with libudev than
this.

 Because if it's that single piece of code, it's indeed very easy
to replace... at least from a technical standpoint.
 From a political standpoint, it's a wholly different matter. The
problem is that "people" - I don't know exactly who, but kernel
folks seem to be in on this - have decided that applications should
never access sysfs directly, and instead they should contact an
"intermediary layer" that is the only one allowed to interact with
sysfs.
 In theory that makes sense, and the reason why kernel folks are
behind it is it allows them to more or less break sysfs at will.
Linus has said, multiple times, that sysfs wasn't supposed to be a
stable interface, so applications should just not access sysfs.

 The problem is that it basically requires the intermediary layer
to fully duplicate the sysfs data in userspace, and provide users
with a way to access the cached data. That is exactly what udevd and
libudev do, and from a software design point, it's a terrible idea:
 - it duplicates data, and duplication leads to inconsistencies
 - it introduces a SPOF in the form of udevd
 - it politically enforces libudev as the de facto standard that
everyone needs to use, whether you like it or not (this is the
vendor lock-in I'm talking about)
 - the presence of a wrapper layer generally introduces more
bloat than it solves problems; sysfs has not changed interfaces to the
point of breaking stuff in 12 years and counting
 - ...

 So, yeah. If you want a patch for xf86-input-evdev that will
make it stop depend on libudev, I can write one in an hour. But
that patch will likely never go upstream, because it goes against
policy; and complying with the policy would basically amount to
rewriting libudev and making a new udevd. Which I believe I'm quite
able to do, and better than the existing ones, but it would still be
bad software design, and I have no interest in contributing to the
problem.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Casper Ti. Vector]

After googling `LNXSYSTEM', I found this to be seemingly doable using
regular Unix filesystem mechanisms on sysfs.  But sysfs seems to be
controlled by the systemd people [1], so this is more of a political
issue than of a technical issue.  (Or perhaps this is just a part of the
grand conspiracy, if you prefer ;)

[1] 

On Wed, Nov 15, 2017 at 09:38:44AM +0800, Casper Ti. Vector wrote:
> which seems not quite difficult to replace...

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Casper Ti. Vector]

The only fragment in xf86-input-evdev that requires libudev is
(from src/evdev.c, with some changes in whitespaces):
> ...
> #include 
> ...
> 
> static BOOL
> EvdevDeviceIsVirtual(const char* devicenode)
> {
> struct udev *udev = NULL;
> struct udev_device *device = NULL;
> struct stat st;
> int rc = FALSE;
> const char *devpath;
> 
> udev = udev_new();
> if (!udev) goto out;
> if (stat(devicenode, &st) == -1) goto out;
> 
> device = udev_device_new_from_devnum(udev, 'c', st.st_rdev);
> if (!device) goto out;
> 
> devpath = udev_device_get_devpath(device);
> if (!devpath) goto out;
> if (strstr(devpath, "LNXSYSTM")) rc = TRUE;
> 
> out:
> udev_device_unref(device);
> udev_unref(udev);
> return rc;
> }
which seems not quite difficult to replace...

On Tue, Nov 14, 2017 at 11:00:06PM +0100, Didier Kryn wrote:
> A most desired addition to mdev would be to provide a means for Xorg to
> autoconfigure, which it does now, IIUC, with the help of libudev. This
> doesn't means using the same API as libudev because I bet Xorg people would
> be open to an alternative.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Didier Kryn]

Le 12/11/2017 à 20:24, Laurent Bercot a écrit :

Will there be a plan to add something roughly isomorphic to libudev?


 Not in the short or medium term. The problem of libudev, and this
is shared by everything from freedesktop.org as well as systemd, is
that its design and structure cannot be achieved without a monolithic
implementation, and are a stealthy way to enforce vendor lock-in.

 In other words: if you try to implement something like libudev, you
necessarily end up writing your own udevd, which is a useless pursuit -
and there is no way to make it more modular.



A most desired addition to mdev would be to provide a means for 
Xorg to autoconfigure, which it does now, IIUC, with the help of 
libudev. This doesn't means using the same API as libudev because I bet 
Xorg people would be open to an alternative.


Not saying this is easy, but it's not the same thing as pursuing 
the whole udev...


Didier

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

Will there be a plan to add something roughly isomorphic to libudev?


 Not in the short or medium term. The problem of libudev, and this
is shared by everything from freedesktop.org as well as systemd, is
that its design and structure cannot be achieved without a monolithic
implementation, and are a stealthy way to enforce vendor lock-in.

 In other words: if you try to implement something like libudev, you
necessarily end up writing your own udevd, which is a useless pursuit -
and there is no way to make it more modular.

 I'm still trying (using systemd as a study case, since I have a lot
more material to work with for the init system than for hotplug
management) to think of a way to provide similar functionality in a
more Unixish fashion, but this is extremely difficult, because the
political agenda is deeply interwoven with the technical aspects, and
disentangling them is nigh impossible.

--
 Laurent

Re: [announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Casper Ti. Vector]

Will there be a plan to add something roughly isomorphic to libudev?

On Mon, Oct 23, 2017 at 08:02:33AM +, Laurent Bercot wrote:
> mdevd is an uevent manager reading a sequence of uevents and handling
> them without forking, that understands the full /etc/mdev.conf format.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

[announce] mdevd-0.0.1.0 - a mdev-compatible uevent manager

[Laurent Bercot]

 Hello,

 About two years ago, there was some talk on the Busybox mailing-list
about the need for a version of "mdev" that would not use a separate
process for every uevent (as a hotplug manager does) but that would act
as a daemon, able to handle a series of uevents - typically read from
the netlink.
 One of the goals of such a program was to reduce boot time on slow /
resource-constrained devices that don't like creating hundreds of
processes at the same time - especially when they contend for a
sequence number.

 I took a quick look at the time, but came to the conclusion that the
way mdev was coded made it very difficult. Basically, mdev gets its
uevent variables from the environment, then reads and processes its
config file, performing actions as it goes. A quick hack to add
"daemon mode" support to mdev would still make it process the config
file for every event, similarly to what "mdev -s" does; this would
remove the forks but still be pretty inefficient, not to mention
particularly ugly. To implement "daemon mode mdev" in a clean way, a
full rewrite was needed. So I shelved the idea at the time.

 Until now.
 mdevd is an uevent manager reading a sequence of uevents and handling
them without forking, that understands the full /etc/mdev.conf format.

 "mdevd-coldplug | mdevd" is equivalent to "mdev -s".
 "mdevd-netlink | mdevd" is a daemon that listens to the netlink and
processes uevents sequentially, without the need for mdev.seq hacks
coming from the kernel spawning hotplug managers in parallel.

 You can find it here:

 https://skarnet.org/software/mdevd/
 git://git.skarnet.org/mdevd
 https://github.com/skarnet/mdevd

 Since it's a full rewrite with a very different architecture from mdev
and little code reusability with the rest of Busybox, it did not make
sense to include it in Busybox, which is why it's provided as a
separate package.

 Bug-reports welcome.
 mdevd is still considered beta for some functionality I could not
extensively test, such as firmware loading. If your setup uses firmware
loading or otherwise obscure mdev features, I'm especially interested
in your reports and/or comments. (mdevd comes with a dry run mode, so
you don't have to be a reckless cowboy to test it.)

 Enjoy,

--
 Laurent