Re: RFD: Rework/extending functionality of mdev
Hi Laurent ! 1) Starting up a system with mdev usually involves the same steps to mount proc, sys and a tmpfs on dev, then adding some subdirectories to /dev, setting owner, group and permissions, symlinking some entries and possibly mounting more virtual file systems. This work need to be done by the controlling startup script, so we reinvent the wheel in every system setup. The thing is, this work doesn't always need to be done, and when it does, it does whether you use udevd, mdev, or anything else. /dev preparation is one of the possible building blocks of an init routine, and it's quite separate from mounting /proc and /sys, and it's also separate from populating /dev with mdev -s. You missed the fact, I told everything stays under control of the admin or system setup. Nobody shall be forced to do things in a specific way. I want just give some extended functionality and flexibility to do this setup in an very easy and IMO straight looking way. If you like/need to do those mounts in a specific way, just put them in your script and leave out the mount lines from mdev.conf, otherwise if you need to setup those virtual file systems with specific options you can specify them in the same line of configuration. ... I dislike the idea of integrating early init functions into mdev, because those functions are geographically (as in the place where they're performed) close to mdev, but they have nothing to do with mdev logically. Sorry, I don't agree. mdev's purpose is to setup the device file system infrastructure, or at least to help with setting up this. Ok, at first leave out proc and sysfs, there you may be right, but what about devpts on /dev/pts, or /dev/mqueue, etc., and finally the tmpfs for /dev itself. Do you really say they do not belong to the device file system infrastructure? Or all those symlinks like /dev/fd, /dev/stdin, /dev/stdout, etc., do you consider them not to belong to mdev? Not talking about setting owner informations and permissions for those entries in the device file system. Sure, all that can be done with some shell scripts, scattering around all that information at different places, or the need to setup and read/process different configuration files. My intention is to get this information which depend on system setup in a single, relatively simple to use or modifiable location. The startup script itself just invokes mdev, which gets the information from /etc/mdev.conf, which calls the necessary commands to do the operation. That is, it frees the distro manager to write script code to parse the system specific information into the startup script. I consider Busybox to be a set of tools, anybody may use to setup a system to there own wishes, not to force anybody to do things in the way any person may feel being it's the best way to do things, but I want to enable functionality for those who like to collect this information and put them in a central place. Information usually scattered around and hidden deep in the scripts controlling the startup. MOUNTPOINT UID:GID PERMISSIONS %FSTYPE [=DEVICE] [OPTIONS] I can't help but think this somehow duplicates the work of mount with /etc/fstab. If you really want to integrate the functionality in some binary instead of scripting it, I think the right way would be to patch the mount applet so it accepts any file (instead of hardcoding /etc/fstab), so you could have a early mount points file, distinct from your /etc/fstab, that you would call mount on before running mdev -s. Laurent you didn't look at the examples. I do not want to hard code any *functionality* in mdev. I want add extra functionality to the current mdev.conf syntax to allow to do some more stuff, which is usually done geographically close to calling mdev, and done in so many systems in similar matter. Look at the usual usage of /etc/fstab, on how many systems do you find there information about your virtual file systems? Usually fstab is used for the disk devices. In addition what about creating the mount points, setting owner, group and permissions? This is not done by mount and not specified in fstab. So changing anything there would mean to modify fstab syntax, possibility breaking other programs and scripts, reading and modifying fstab. Neither do I want to code any special functionality in a binary, nor do I try to duplicate the operation of mount. I just want to extend the mdev.conf syntax to add simple configuration information for those close to mdev operations at a central place, parse this information and call the usual commands, e.g. mount with the right options, as shell scripts do. And what else are a few lines, in mdev.conf, describing those mounts, other then placing them in a separate early mount points file? ... ok, lets go one step further on this. Let us add an include option to mdev.conf, which allow to split the mdev configuration in different files, and/or place
Re: RFD: Rework/extending functionality of mdev
Hi Harald, I'm sorry if I came across as dismissive or strongly opposed to the idea. It was not my intent. My intent, as always, is to try and make sure that potential new code 1. is worth writing at all, and 2. is designed the right way. I don't object to discourage you, I object to generate discussion. Which has been working so far. ;) You missed the fact, I told everything stays under control of the admin or system setup. Nobody shall be forced to do things in a specific way. I want just give some extended functionality and flexibility to do this setup in an very easy and IMO straight looking way. I understand your point. If I don't modify my mdev.conf, everything will still work the same and I can script the functionality if I prefer; but if you prefer centralizing the mounts and other stuff, it will also be possible to do it in mdev.conf. It is very reasonable. The only questions are: - what are the costs of adding your functionality to mdev.conf ? - are the benefits worth the costs ? You may have noticed that I'm very conservative as far as code is concerned ;) You're saying I like centralized information, the benefits are obvious to me and I'm answering I prefer scripting, so I don't see the benefits to your change. It's a debate of taste, and we'll never get anywhere like this. We need to dig a little more. As a preamble, let me just say that if you manage to make your syntax extensions a compile-time option and I can deactivate them to get the same mdev binary as I have today, then I have no objection, apart from the fact that I don't think it's good design - see below. Sorry, I don't agree. mdev's purpose is to setup the device file system infrastructure, or at least to help with setting up this. Ok, at first leave out proc and sysfs, there you may be right, but what about devpts on /dev/pts, or /dev/mqueue, etc., and finally the tmpfs for /dev itself. Do you really say they do not belong to the device file system infrastructure? Or all those symlinks like /dev/fd, /dev/stdin, /dev/stdout, etc., do you consider them not to belong to mdev? Not talking about setting owner informations and permissions for those entries in the device file system. OK, now this is interesting. I firmly believe that most of our disagreement comes from the fact that mdev and mdev -s are the same binary, but *not* the same functionality at all. To me, mdev.conf is a configuration file for mdev functionality, i.e. configuration for a hotplug helper. Since mdev can be invoked on every hotplug event, it's very important that mdev.conf parsing remain fast. You are saying that you would benefit from user interface improvements to the /dev *initialization* functionality, i.e. mdev -s. I understand, and I agree that a one-stop-shop for /dev initialization would be nice. My point, which I didn't make clear in my previous message because I was too busy poking fun at you (my apologies, banter should never override clarity), is that I find it bad design to bloat the parser for a hotplug helper configuration file in order to add functionality for a /dev initialization program, that has *nothing to do* with hotplug helper configuration. The confusion between the two comes from two places: - the applet name, of course; I find it unfortunate that mdev and mdev -s share the same name, and I'd rather have them called mdev and mdev-init, for instance. - the fact that mdev's activity as a hotplug helper is about creating nodes, fixing permissions, and generally doing stuff around /dev, which is, as you say, logically close to what you need to do at initialization time, so at first sight the configuration file looks like a tempting place to put that initialization time work. But I maintain that mdev.conf should be for mdev, not for mdev -s. mdev -s is just doing the equivalent of calling mdev for every device that has been created since boot. If you make a special section in mdev.conf for mdev -s, this 1. blurs the lines even more, which is not desirable, and 2. bloats the parser for mdev with things it does not need after the first mdev -s invocation. Sure, all that can be done with some shell scripts, scattering around all that information at different places, or the need to setup and read/process different configuration files. My intention is to get this information which depend on system setup in a single, relatively simple to use or modifiable location. I agree with that goal. I just don't think mdev.conf is the place to do it. The startup script itself just invokes mdev, which gets the information from /etc/mdev.conf No: the startup script invokes mdev -s, which invokes (or does the equivalent of invoking) mdev, which gets the information from mdev.conf. But the functionality you're planning to add is specific to mdev -s: the actions would be taken by mdev -s but outside of its series of mdev invocations. And what else are a few lines, in mdev.conf, describing those mounts, other
Re: RFD: Rework/extending functionality of mdev
On Tue, Mar 10, 2015 at 01:37:41PM +0100, Harald Becker wrote: Hi Laurent ! ... I dislike the idea of integrating early init functions into mdev, because those functions are geographically (as in the place where they're performed) close to mdev, but they have nothing to do with mdev logically. Sorry, I don't agree. mdev's purpose is to setup the device file system infrastructure, or at least to help with setting up this. Ok, at first leave out proc and sysfs, there you may be right, but what about devpts on /dev/pts, or /dev/mqueue, etc., and finally the tmpfs for /dev itself. Do you really say they do not belong to the device file system infrastructure? Or all those symlinks like /dev/fd, /dev/stdin, /dev/stdout, etc., do you consider them not to belong to mdev? Not talking about setting owner informations and permissions for those entries in the device file system. In my humble opinion, these do not belong to mdev the busybox applet. Do one thing means *method*, not just problem area - just like cut, head, and tail are separate programs, despite all selecting portions of a text input based on mathematical criteria. And mdev takes care of setting up devices by looking at information from the kernel (via uevent or environment; the two are very similar). If you don't make this distinction, how do you argue that lspci should not query the PCI ID database via DNS (-Q), or that it's out of scope for an init system to fold in the device manager? Now, if you added support for using busybox as pre-compiled init scripts, it would be reasonable for an applet to do this. (Yes, I've heard of such things being done with busybox.) ... but here it's different. s6-devd spawns a helper program for every event, like /proc/sys/kernel/hotplug would do, but it reads the events from the netlink so they're serialized. Spawning a process for every event produce massive slowdown on system startup. My intention is to fork only one process, parse the conf table once and then read sanitized events from stdin, scanning the in memory conf table and invoking the right operations. If you use a similar format to uevent (eg, VAR1=VAL1\0VAR2=VAL2\0\0 or something similar) this sounds desireable to me. I suppose this could be mdev -i. However, I'd suggest figuring out how to make environment variable hooks work the same in mdev -s/-i and just plain mdev first. Otherwise, mdev -i would be at a disadvantage. Note: while it's trivial to swap out the environment with each event, this is probably a Bad Idea for a longer-lived process, due to the potential for leaks and other chaos. Factors that make me want this: * mdev can noticeably slow down boot (I've set up mdev as hotplug helper with Debian, and notice that the resulting initrd takes a second or two more than udev to load.) I would assume that loading all the modules in Debian results in a fork bomb. * hotplug is a rare event, but with modern hardware it frequently causes a sudden burst of activity. A flash drive frequently triggers 5+ events, and plugging in a phone/ turning on mass storage may cause a dozen or more. Even plugging in an SD card may cause 3 or more events. In your design, the long-lived process forks a unique helper that reads mdev.conf... so it's not meant to be used with any program compatible with /proc/sys/kernel/hotplug - it's only meant to be used with mdev. So, why fork at all ? Your temporary instance is unnecessary - just have a daemon that parses mdev.conf, then listens to the netlink, and handles the events itself. No, it is not unnecessary. The long lived instance tries to stay at low memory usage, then forks and second instance read mdev.conf into the memory table. When the event system goes idle, as it is most of the time, the second instance dies and memory is freed to the system. When I'm doing something particularly memory intensive (in my case, *usually* linking a large program), I sometimes stop every service and program I can. This has allowed me to build at least one program I could not have built otherwise. There may be times when there's memory pressure that the user is unaware of, but frequently the user controls both memory pressure and hotplug. The long lived instance probably needs to stay out of busybox to make this really minimize memory use, though. Certainly it shouldn't be in the same binary as mdev. (Some people use multiple busybox binaries, which is probably more optimal.) Seriously, udev is a hard problem. udevd is bloated and its configuration is too complex; mdev is close to the point where parsing a config file for every invocation will become too expensive - I believe that it has not reached that point yet, but if you add more stuff to the conf language, it will. Right, adding more to the conf language adds complexity there, but removes complexity of the surrounding scripts and collect system specific information at a central place (or places), usually hidden
Re: RFD: Rework/extending functionality of mdev
Hi, getting hints and ideas from Laurent and Natanael, I found we can get most flexibility, when when try to do some modularization of the steps done by mdev. At fist there are two different kinds of work to deal with: 1) The overall operation and usage of netlink 2) Extending the mdev.conf syntax Both are independent so first look at the overall operation ... and we are currently looking at operation/functionality. This neither means they are all separate programs/applets. We may put several functionalities in one applet and distinguish by some options. First look at, what to do, then decide how to implement: mdev needs to do the following steps: - on startup the sys file system is scanned for device entries - as a hotplug handler for each event a process is forked passing information in environment variables - when using netlink a long lived daemon read event messages from a network socket to assemble same information as the hotplug handler - when all information for an event has been gathered, mdev needs to search it's configuration table for the required entry, and then ... - ... do the required operations for the device entry That is scanning the sys file system, hotplug event handler and netlink event daemon trigger operation of the mdev parser. Forking a conf file parser for each event is much overhead on system startup, when many event arrive in short amount of time. There we would benefit from a single process reading from a pipe, dieing when there are no more events and reestablish when new events arrive. Both the sys file system scanner and a netlink daemon could easily establish a pipe and then send device commands to the parser. The parser reads mdev.conf once and creates a in memory table, then reads commands from the pipe and scans the memory table for the right entry. When EOF on pipe read, the parser can exit successfully. The sys file system scanner (startup = mdev -s) can create the pipe, then scan the sysfs and send the commands to the parser. When done, the pipe can be closed and after waiting for the parser process just exit. The netlink daemon, can establish the netlink socket, then read events an sanitize the messages. When there is any message for the parser, a pipe is created and messages can be passed to the parser. When netlink is idle for some amount of time, it can close the pipe and check the child status. Confusion arise only on the hotplug handler part, as here a new process is started for every event by the kernel. Forking a pipe to send this to the parser would double the overhead. But leaving the parser running for some amount of time, would only work with a named fifo, startup of the parser when required and adds timeout management to the parser ... ... but ok, do we look at an alternative: Consider a small long lived daemon, which create a named fifo and then poll this fifo until data get available. On hotplug event a small helper is started, which read it's information, serializes then write the command to the fifo and exit. The long lived daemon sees the data (but does not read), then forks a parser and gives the read end of the pipe to the the parser. The parser reads mdev.conv once, then processes commands from the fifo. Now we are at the situation where the timeout needs to be checked in the parser. When there are no more events on the fifo the parser just dies successfully (freeing used memory). This will be detected by the small long lived daemon, which check the exit status and can act on failures (e.g. run a failure script). On successful exit of the parser the daemon starts again waiting for data on the fifo (which he still hold open for reading and writing). This way the hotplug helper will benefit from a single run parser on startup, but memory used by the conf parser is freed during normal system operation. The doubling of the timeout management in netlink daemon and parser can be intentional when different timeouts are used. Where a small duration for the idle timeout of netlink can be chosen, the parser itself does use a higher timeout, which only triggers when the hotplug helper method is used. yes, there are some rough corners, but we are at the phase of brain storming. Beside those corners do we get a modular system, which avoid respawning/rereading the conf table for every event, but frees memory when there are no more events. Even the hotplug helper method will benefit, as the helper process can exit, as soon as the command has been written to the fifo. The parser reads serialized commands from the pipe and process required actions. May be we should consider using that small parser helper daemon and the named fifo in all cases. sys file system scanner, hotplug helper and netlink daemon will then just use the fifo. This would even allow to use the same fifo to activate the mdev parser from a user space program (including single parser start for multiple events).
mdev and usb device node creation
Greetings: I am using an older version of busybox (1.7.0) with a newer linux kernel (3.10) and trying to get it to discover and dynamically create usb device nodes underneath /dev/bus/usb. This doesn't seem to work when I execute mdev with the -s option. Can someone please tell me if this capability is supported in newer versions of busybox? Thanks, Dallas ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: mdev and usb device node creation
On Tue, Mar 10, 2015 at 4:48 PM, Harald Becker ra...@gmx.de wrote: Hi Dallas ! I am using an older version of busybox (1.7.0) with a newer linux kernel (3.10) and trying to get it to discover and dynamically create usb device nodes underneath /dev/bus/usb. This doesn't seem to work when I execute mdev with the -s option. And were is the rest of your setup? What matching entries in /etc/mdev.conf belong to your concern? Can someone please tell me if this capability is supported in newer versions of busybox? mdev -s scans the sys file system for devices and setup the /dev entries. If you want dynamic setup you should use the hotplug feature, but the device node handling of both depend on the entries in /etc/mdev.conf. So what happens and when, what is wrong in your eyes? May be you just need to enter your correct device setup. -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox Hi Harald, I was sort of expecting a mdev scan to discover and create device nodes for usb. Sounds like I need some rules defined in /etc/mdev.conf. I just have a default one. In the meantime, I have been dynamically adding / removing the usb device nodes triggered off of hotplug events. Seems to be working fine so far. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: busybox bugzilla down?
On Tue, 2015-03-10 at 22:49 +0100, Bernhard Reutner-Fischer wrote: On March 10, 2015 10:13:42 PM GMT+01:00, Bernhard Reutner-Fischer rep.dot@gmail.com wrote: On March 10, 2015 9:55:07 PM GMT+01:00, Jim Keniston jkeni...@linux.vnet.ibm.com wrote: When I try to visit https://bugs.busybox.net , I see: - Software error: Can't connect to the database. Error: Can't connect to MySQL server on 'db1.osuosl.org' (110) Is your database installed and up and running? Do you have the correct username and password selected in localconfig? For help, please send mail to the webmaster (webmas...@codepoet.org), giving this error message and the time and date of the error. - and mail to webmas...@codepoet.org bounces. Apparently the db-server had HW issues, our bugzilla should be in the backups and restored ASAP. I'll take care of updating the wrong webmaster email addr. Thanks, I see that it's up now. Thanks! Jim ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: mdev and usb device node creation
Hi Dallas ! Hi Harald, I was sort of expecting a mdev scan to discover and create device nodes for usb. Sounds like I need some rules defined in /etc/mdev.conf. I just have a default one. What consider you a default one? There are so many distros and systems out in the wild, even the Busybox snapshot archives contain two or three different mdev configurations (as examples). SO I can't tell you anything specific, without seeing what your configuration describes. mdev either scans /sys/class/... for devices entries or picks up hotplug event messages, then search the /etc/mdev.conf for a matching device entry. Then it does create/remove device nodes according to the given information. This includes moving location (e.g. into a subdirectory), creating symlinks, setting owner, group and permissions. Everything mdev does is controlled by the rules in mdev.conf, but it may contain catch all rules to do some default action. A short search on the net gave me the following line: $DEVNAME=bus/usb/([0-9]+)/([0-9]+) root:plugdev 0660 =bus/usb/%1/%2 Put this in your mdev.conf, may be it catches the names of your kernel. .. Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
