Interrupts ... no trouble, everybody agree you need only one unblocked
interrupt source, but never ask for the detail which one ... :)
Hi Laurent !
I'm sorry if I came across as dismissive or strongly opposed to the
idea.
It was not my intent. My intent, as always, is to try and make sure that
potential new code 1. is worth writing at all, and 2. is designed the
right way. I don't object to discourage you, I object to generate
discussion.
Which has been working so far. ;)
ACK
I understand your point. If I don't modify my mdev.conf, everything
will still work the same and I can script the functionality if I prefer;
but if you prefer centralizing the mounts and other stuff, it will also
be possible to do it in mdev.conf.
This is my primary intention.
It is very reasonable. The only questions are:
- what are the costs of adding your functionality to mdev.conf ?
- are the benefits worth the costs ?
I don't know the exact cost ahead, but they should be not massive. look,
we need an mkdir and symlink plus setting owner/permissions, and we need
to setup an arg vector to call mount. The rest will be some rework of
the parser, but I don't consider this into cost. So this shouldn't be
much of increasing for the extended syntax.
Ok may be an include option in mdev.conf has some extra cost ... but
benefit would be for all who like to split mdev.conf into separate
files. Could be a BB config option "allow include in mdev.conf".
Some more cost will result from possibility to use netlink, but the
benefit will be not parsing the table for every event. So some cost is
acceptable for me. Either part hotplug handler oder netlink reader may
be excluded on BB config. I see no trouble grouping the code and
excluding it when the option is deselected. Minor changes may persist,
but shall not blow out the principles Busybox is based on, else I do
have a big trash can ...
As a preamble, let me just say that if you manage to make your syntax
extensions a compile-time option and I can deactivate them to get the
same mdev binary as I have today, then I have no objection, apart from
the fact that I don't think it's good design - see below.
Deactivating unwanted stuff in BB config is my intention. Deactivation
will not result in same binary, due to some intended parser changes, but
cost for this shouldn't be much notable ...
... and yes, I know to be picky on size restricted development. I
started my programming practice on an 8008 with 512 Byte ROM and 128
Byte RAM ... in addition to a Zuse Z31 computer (one of the first
computer models using transistors) with 2000 words magnetic core memory
of each 11 decimal digits ... :)
My point, which I didn't make clear in my previous message because I was
too busy poking fun at you (my apologies, banter should never override
clarity), is that I find it bad design to bloat the parser for a hotplug
helper configuration file in order to add functionality for a /dev
initialization program, that has *nothing to do* with hotplug helper
configuration.
Ok, here we agree. I would highly deprecate blowing up the hotplug
scanning, but here we come to the reason I stumbled about and asked if
we can avoid this extra parsing all together, speeding up all. So I was
back at netlink.
The confusion between the two comes from two places:
- the applet name, of course; I find it unfortunate that "mdev" and
"mdev -s" share the same name, and I'd rather have them called "mdev"
and "mdev-init", for instance.
ACK, this could be done, for all the functionality ... but this will be
a philosophical discussion. As mdev the hotplug helper does not use the
code of mdev -s, there is no time cost to have that in one binary.
- the fact that "mdev"'s activity as a hotplug helper is about creating
nodes, fixing permissions, and generally doing stuff around /dev, which
is, as you say, logically close to what you need to do at initialization
time, so at first sight the configuration file looks like a tempting
place to put that initialization time work.
But I maintain that mdev.conf should be for mdev, not for mdev -s.
mdev -s is just doing the equivalent of calling mdev for every device
that has been created since boot. If you make a special section in
mdev.conf for mdev -s, this 1. blurs the lines even more, which is not
desirable, and 2. bloats the parser for mdev with things it does not
need after the first mdev -s invocation.
To 1. -> ACK, see below
To 2. -> as I intent to avoid parsing on every event, that is I like to
parse all rules into memory table and then scan only memory table for
each arriving event, The extra code in the parser does not matter to
your concern. Beside this I tried to chose syntax carefully, to produce
not much overhead. We have two checks, one to the last char of the regex
(slash means directory, atsign means symlink -> ignored on hotplug) and
the second check is to the percent sign of the mount file system type.
As I want to rework the parser into a straight forward type of parser,
working with chars and not moving strings around, I expect an overall
speedup, even with my included checks :)
I splitted of this work into mdev -i and mdev -s, may be we call them
mdev-init and mdev-scan. So the init part would use a different file
than /etc/mdev.conf in your description, but what of those who like to
have all this in one file? Test in mdev-init for /etc/mdev-init.conf and
fallback to /etc/mdev.conf? Same to mdev-scan, try /etc/mdev-scan.conf
and fallback to /etc/mdev.conf (would allow to split coldplug and
hotplug rules)? Could be benefit without much cost.
Sure, all that can be done with some shell scripts, scattering around
all that information at different places, or the need to setup and
read/process different configuration files. My intention is to get
this information which depend on system setup in a single, relatively
simple to use or modifiable location.
I agree with that goal. I just don't think mdev.conf is the place to
do it.
See my suggestion above. Could be a good decision.
And what else are a few lines, in mdev.conf, describing those mounts,
other then placing them in a separate "early mount points file"? ...
ok, lets go one step further on this. Let us add an "include" option
to mdev.conf, which allow to split the mdev configuration in
different files, and/or place mdev.conf lines in separate files in a
/etc/mdev.d/ directory. You like to have an "early mount points
file"? Put those mount informations in a file and add an include for
that file in mdev.conf ... ?
Well, my suggestion would be to keep mdev.conf as it is, and to
have an additional mdev-init.conf (or however you want to name it) file
with the syntax of your choice, specifically for the things you want
mdev -s to do that are not purely mdev invocations. One config file
for the hotplug helper, and a separate one for the one-time
initialization.
Ok, I agree a separate file for both operation may fit the needs of some
users better, we could have both with minmal overhead when using same
syntax and trying different names on open, than fallback to /etc/mdev.conf.
... beside this you insist on the overhead for the hotplug handler.
Which I like to eliminate at the same step.
... and what is wrong about calling mount from mdev? Setting up the
device files system includes mount operations, so mdev should be able
to call mount, when and how it is required.
You're not going to call mount from mdev as a hotplug helper.
You're only going to call mount from mdev -s. Nothing wrong with that
if it's what you want, but I still believe you should keep the mount
stuff outside of mdev.conf.
Yes. Please, go back and change "mdev" in my text in "mdev -s", that was
my intention, I was a bit sloppy.
My intention is to fork only one process, parse the conf
table once and then read sanitized events from stdin, scanning the in
memory conf table and invoking the right operations.
I went ahead and implemented the "netlink listener forking a
single process" thing, with the protocol I suggested in my previous
message:
http://skarnet.org/software/s6-linux-utils/s6-uevent-listener.html
(To get the s6-devd behaviour, I also implemented a s6-uevent-spawner
program, which reads from s6-uevent-listener and spawns a program for
every event. But that is the part you don't want. You want to write
a long-lived program that would be spawned by s6-uevent-listener.)
Feel free to try it, torture it, copy it and create your own stuff
based on it - this goes for Nathanaël too. I honestly don't think you
can get a much better netlink uevent listener than this,
resource-wise (it even uses posix_spawn() instead of fork(), for
noMMU systems) and simplicity-wise.
Thanks for this work. I did not look to this yet, but I expect to help
doing the job on mdev. As I like to use as much existing and pretested
code as possible, I really appreciate this.
No, it is not unnecessary. The long lived instance tries to stay at
low memory usage, then forks and second instance read mdev.conf into
the memory table. When the event system goes idle, as it is most of
the time, the second instance dies and memory is freed to the
system.
I'm still not sold on that.
Even if the event system is idle most of the time, when an event
arrives, you *need* to handle it, and to handle it *now*. If you
don't have the resources when you need to spawn the second instance,
what do you do ?
This question always exist. You fixate on the hotplug helper usage, IMO
spawning a process for each event is brain damaged. Would be wise if
kernel spawns a single hotplug helper process if none is running and
then send messages serialized via pipe to that helper until the helper
dies or kernel decide it is no longer required (close pipe). I can't
change the kernel stuff, so next to this get's the netlink usage.
... but I don't want to block hotplug helper usage. The kernel does
spawn a process per event, so we need to live with this. So how to
optimize? We can speed things when the hotplug helper does it's per
event job faster.
So do it faster: Just grab your event information, serialize the message
calls, and then write the sanitized event request to a fifo. Done & Exit.
May be the startup of a parser process takes minimal longer than current
hotplug helper / parser combination, but as soon as the process is
running, it processes events faster than spawning on every event.
Therefor the expected overall performance is better than now.
And a word to resource usage. When hotplug helper and parser is coupled,
each instance (on each event call) needs to read in the table and this
requires memory. On the other side, if you have one instance with the
requirement to read the table and may be several small hotplug helper
with low memory usage, you need less memory. So I expect finally less
resource constrains (time & memory), not more, especially on event
bursts (and events tend to happen in bursts).
If you're going to have a long-lived program at all, then I think
it's better to fully commit the resources you need to handle events
properly.
This is a philosophical question. Is it better to reserve all required
memory for event handling, blocking the usage of the memory in all other
cases, or drop as much memory as possible and start up the event handler
only when required. Both has pros & cons.
IMO it's better to free as much resources as possible, to give the
system the ability for a better overall performance, accepting the risk
for problems on system overload.
We are on a Unix based platform which is not a real time system. The
only proven method of avoiding trouble due to resource constrains is
adding enough amount of extra resources for reserve ... beside
minimizing resource usage.
... and you just reinvented udevd. Congratulations ! ;)
Laurent this is not very friendly.
Sorry, you're right, it wasn't. Hey, with s6-uevent-listener,
I kinda reinvented udevd too! The easy part, at least.
... and that kind of work may now make it into the mdev system.
As I told, I'm in the phase of brain storming and planing the project
details, not hacking any code before I got an idea which functionality
is required.
Right, adding more to the conf language adds complexity there, but
removes complexity of the surrounding scripts and collect system
specific information at a central place (or places), usually hidden
deep in the startup scripts or some config files they use.
Again: I agree with you here, I just don't think mdev.conf is the
place to do initialization-but-not-hotplug operations.
Admin decision, but otherwise ACK, it highly depends on what you want to
achieve. So we should have the possibility of separate conf files, and
may be separate commands?
OT an idea from above: Could we have hidden commands in Busybox? e.g.
applet names starting with a dot, like .mdev-fifo, .mdev-parser, etc.
Those commands shall neither be listed on the BB --help function
listing, nor shall automatic symlinks be created. Otherwise you can
still call them using full syntax "busybox .mdev/parser" or via the
preferred applet feature from inside BB applets/shell scripts. With no
difference then hiding.
This way we can create separate applets for different purposes and do
not need to do things like merging "mdev" and "mdev -s". Hidden commands
may help avoiding unnecessary cluttering, while splitting into
functional modules (which may easily be called from scripts).
I can live with both, but I appreciate clarity.
IMO you are going off topic. Now you are talking about udev, and
modifying / using udev. We are most likely looking at systems using
mdev and are based on BB, not the upstream tools.
Well yes, it's off-topic for your practical initialization need here.
I can easily go on the udevd tangent, however, because I believe
that's the whole crux of the problem: no matter how well-designed
the hotplug helpers are, and no matter the mechanisms (hotplug or
netlink, short-lived or long-lived...), the difficult part of device
management is the actual userspace library, the list of actions
to take when such and such device gets plugged/removed/transmogrified.
This is absolute grunt work, which so far has been done for udevd,
and nothing else - not even mdev; and it's a shame, because it
perpetuates the reliance on the horrible mess that is udevd, and
marginalizes users of mdev or other solutions. If we wanted to
advertise a workable, credible alternative to udevd in the mainstream
world, we would need to come up with that huge device management
database. And this makes me recoil.
I'm willing to extend and rewrite the mdev parser, so we may be able to
add extra features to simplify things, but my problem is, I'm coming
from the world of statically defined setups and use only minimalistic
hot plugging features. So what are the details of "user actions" you are
grumbling about. I did not hop on them until now, but let us discuss and
collect the wishes, then see what may be of benefit for Busybox users
and wish ones are accepted for integration. I don't expect BB mdev to
ever extend to a full fledged desktop device manager, but let us look
what may be of value to implement ... and may be someone can later on
grab and extend a separate version to a better fitting full "udev" rework :)
FYI: Writing specialized language parser/compiler and pseudo code
interpreter is one of my favors, with success on several professionally
used systems. Even if I dislike to do such things, my knowledge would
allow me to rewrite a usable mdev from scratch in acceptable time. As I
retired, I'm willing to spend the time to give this device management
stuff a push. I did not expect to go so far, as I started the project to
get in some extensions, but lets have a look, where we could go. Then
decide which route our journey take. Ideas welcome.
--
Harald
_______________________________________________
busybox mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/busybox
