Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
On Thu, Mar 17, 2022 at 10:17:50AM -0400, Mouse wrote: > > ZFS is already architecturally unclean, in that it means that > /etc/fstab no longer describes the set of standardly mounted > filesystems. (I was not aware of this property of ZFS; I find it > astonishing - and depressing - that nobody built a shim to fix it. > That would be one of the first things I would do if I were setting up > such a thing.) > There is no need - you can set a property on the file system not to automatically mount and perform a "legacy" mount in /etc/fstab if you so choose. -- Brett Lymn -- Sent from my NetBSD device. "We are were wolves", "You mean werewolves?", "No we were wolves, now we are something else entirely", "Oh"
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
>> ZFS is already architecturally unclean, in that it means that >> /etc/fstab no longer describes the set of standardly mounted >> filesystems. (I was not aware of this property of ZFS; [...].) > [Y]ou can set a property on the file system not to automatically > mount and perform a "legacy" mount in /etc/fstab if you so choose. Then...I guess I don't understand what the kerfuffle is all about. Those who are bothered as I am by zfs can do it the way they like and not have to deal with it; those who aren't can do it the way _they_ like. Sounds to me good all around. Unless I've Missed Something (which seems reasonably likely) Though it seems like a good discussion in that some real progress was made, in that the lack of a "noerror" or "allowfail" or whatever flag in fstab was highlighted and may someday get added. I'm tempted to try doing it in my own code trees /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
On 3/17/2022 5:52 AM, Taylor R Campbell wrote:
> Does anyone actually do this -- have local mounts on top of remote
mounts?
> I keep hearing about the theoretical possibility of /usr on nfs and
/usr/src or /usr/local on local ffs.
Not directly relevant to NetBSD, but the standard (IIRC) setup on the
NeXT had a shared NFS-mounted / with a UFS /private mounted on top of
it. /etc, /var and /tmp were symlinks to /private/{etc,var,tmp} , so an
individual NeXT could be in a separate administrative domain from the
file server. MacOS retains /private, and though I have never set one up
that way, I can easily imagine it being the preferred setup for a
compute farm.
Take care,
-Konead
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
> Does anyone actually do this -- have local mounts on top of remote > mounts? At the moment I don't, but I have in the past and have no real doubt I will in the future. > I keep hearing about the theoretical possibility of /usr on nfs and > /usr/src or /usr/local on local ffs. Back when I worked with a smallish (10-20) farm of mostly-diskless machines, it was not uncommon to have most machines set up with an fstab that looked something like (to use NetBSD syntax) server:/nfs/root/myname / nfs rw server:/nfs/usr/myarch /usr nfs rw server:/nfs/home /home nfs rw kernfs /kern kernfs rw procfs /proc procfs rw and then one or two of them with real disk adding a line or two like /dev/sd0a /project ffs rw or /dev/wd0a /home/someone/something ffs rw Such machines could, of course, be set up as fully diskful. But in an environment like that the benefits of keeping the machines' setups _almost_ identical can easily outweigh the benefits of going diskful. > But you'd really have to go out of your way to set that up -- > certainly sysinst won't do that for you. Sysinst won't do lots of things many people want. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
On 3/17/22 15:17, Mouse wrote: 3. Move all ZFS mounts to /etc/rc.d/mountcritlocal . 3 is the only thing here I object to because it is architecturally unclean, giving special semantics to zfs. ZFS is already architecturally unclean, in that it means that /etc/fstab no longer describes the set of standardly mounted filesystems. (I was not aware of this property of ZFS; I find it astonishing - and depressing - that nobody built a shim to fix it. That would be one of the first things I would do if I were setting up such a thing.) I feel the same...
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
>> 3. Move all ZFS mounts to /etc/rc.d/mountcritlocal . > 3 is the only thing here I object to because it is architecturally > unclean, giving special semantics to zfs. ZFS is already architecturally unclean, in that it means that /etc/fstab no longer describes the set of standardly mounted filesystems. (I was not aware of this property of ZFS; I find it astonishing - and depressing - that nobody built a shim to fix it. That would be one of the first things I would do if I were setting up such a thing.) >> And somewhat related that came out of this discussion - add a >> "noerror" or "notfatal" or some similar keyword [...] >> Although Ignatios had an interesting solution to this with "noauto" >> and using @reboot crontab entries. > Sure, you work around the lack of the feature, but that doesn't mean > a simpler way is bad. Well put. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: inetd tests failing
>> This is possible for only a restricted set of services (those that >> are at least conceptually datagram services, more or less). > I don't think it precludes tcp as long as they are stateless - http > is an obvious example Ish. HTTP is stateless in the sense that consecutive requests/responses do not depend on one another. But at least some HTTP servers change their behaviour based on on getpeername(), and at least some HTTP requests/responses depend on reaching EOF on the socket (which as far as I know is not undoable and thus requires a new socket for the next request/response). And, of course, there's state in the sense of "I've read a partial request [or sent a partial response], now I have to read [send] the rest", but that is a rather different sense. But there are probably a few for which the request and response are self-delimiting. I couldn't think of any offhand; on looking at /usr/src/etc/inetd.conf, I see identd and not much else. There are others whose requests are a single line, such as finger, but, in all the cases I see, the response is not self-delimiting and thus requires either at least a tiny bit of additional protocol (and a daemon designed for it) or a new socket for each request/response (and thus either additional protocol or a new process for each request). On the question of "is it good" instead of "is it possible" Plenty of such servers may exist in the wild. But are any of them used heavily enough that the code complexity of preforking is worth bothering with? I would guess (and a guess is all it is) that they are rare enough to handle them the way HTTP is often handled, with a long-running daemon (which, since it often isn't execing, can easily use a preforked pool of processes). I'm also not fond of inetd getting as complicated as these imply. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
On Thu, Mar 17, 2022 at 12:52:03PM +, Taylor R Campbell wrote: > Does anyone actually do this -- have local mounts on top of remote > mounts? I do, but all machines affected have / on NFS which makes the setup trivial. Martin
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
Taylor R Campbell writes: >> Date: Thu, 17 Mar 2022 08:32:40 -0400 >> From: Greg Troxel >> >> Simon Burge writes: >> >> > 5. Move all local mounts to /etc/rc.d/mountcritlocal (ala >> > FreeBSD) and possibly rename this to /etc/rc.d/mountlocal . >> >> I think the only thing we lose with this is the ability to mount local >> filesystems on top of mount points that are in remote filesystems, >> without playing the noauto/rc.local game. I am ok with this personally, >> but it feels hard to be sure it won't cause trouble, and I do expect >> some trouble. > > Does anyone actually do this -- have local mounts on top of remote > mounts? > > I keep hearing about the theoretical possibility of /usr on nfs and > /usr/src or /usr/local on local ffs. But you'd really have to go out > of your way to set that up -- certainly sysinst won't do that for you. > Not sure it's too much to ask that if you do set something up like > that you use noauto/rc.local or a local rc.d script to manage it. > > Is this obscure use case actually worth worrying about enough to add > new knobs, invent new NetBSD-specific zfs properties, and/or keep a > confusing series of mount stages in rc.d? > > I think it would be better to just nix the `critical' distinction: > mount root, then mount all local, then start networking and mount all > remote. Keep it simple unless there's a strong reason not to. It is really hard to assess the costs of these two paths over all users and uses. I don't object to mounting all local in mountcritlocal (and I don't care if it is renamed to mounlocal). I don't want to commit that personally, since I don't want to bite off responsibility for fixing problems I didn't anticipate. Right now, mountcritremote comes before servers/etc. and the rest can happen later, in mountall. Right now for almost everyone mountcritremote does nothing. That does't have a compelling reason for change, so there's only risk of breaking things we don't understand, and I'm inclined to leave it. signature.asc Description: PGP signature
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
> Date: Thu, 17 Mar 2022 08:32:40 -0400 > From: Greg Troxel > > Simon Burge writes: > > > 5. Move all local mounts to /etc/rc.d/mountcritlocal (ala > > FreeBSD) and possibly rename this to /etc/rc.d/mountlocal . > > I think the only thing we lose with this is the ability to mount local > filesystems on top of mount points that are in remote filesystems, > without playing the noauto/rc.local game. I am ok with this personally, > but it feels hard to be sure it won't cause trouble, and I do expect > some trouble. Does anyone actually do this -- have local mounts on top of remote mounts? I keep hearing about the theoretical possibility of /usr on nfs and /usr/src or /usr/local on local ffs. But you'd really have to go out of your way to set that up -- certainly sysinst won't do that for you. Not sure it's too much to ask that if you do set something up like that you use noauto/rc.local or a local rc.d script to manage it. Is this obscure use case actually worth worrying about enough to add new knobs, invent new NetBSD-specific zfs properties, and/or keep a confusing series of mount stages in rc.d? I think it would be better to just nix the `critical' distinction: mount root, then mount all local, then start networking and mount all remote. Keep it simple unless there's a strong reason not to.
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
Simon Burge writes: > Lots of interesting discussion! Thanks all. As a loud ranter I'll comment briefly but thanks for the summary and I think we're heading for a good place. > Broadly I think I can summarise to the following options: > > 1. The existing critical_filesystems_zfs rc.conf variable, which > mixes ZFS configuration in both rc.conf and with ZFS itself. If this works, which it by all accounts seems to, the only real problem is that some people (and I can see why) would like to manage the critical property with a zfs property to keep all their zfs config in the same place. Am I misperceiving? > 2. Add ZFS "critical" properties for filesystems and mount those > in /etc/rc.d/mountcritlocal . > 3. Move all ZFS mounts to /etc/rc.d/mountcritlocal . 3 is the only thing here I object to because it is architecturally unclean, giving special semantics to zfs. > 4. Optionally move all ZFS mounts to /etc/rc.d/mountcritlocal > controlled by an rc.conf variable (eg zfs_critical). I view this as syntactic sugar for putting all zfs mounts in the critical_filesystems_zfs variable. It's a little to close to option 3 for my taste, but I won't say I object. > 5. Move all local mounts to /etc/rc.d/mountcritlocal (ala > FreeBSD) and possibly rename this to /etc/rc.d/mountlocal . I think the only thing we lose with this is the ability to mount local filesystems on top of mount points that are in remote filesystems, without playing the noauto/rc.local game. I am ok with this personally, but it feels hard to be sure it won't cause trouble, and I do expect some trouble. > 6. "Intelligent" ordering based on prefixes, tsort, etc. > In terms of what to do for the up-and-coming netbsd-10 (although the > actual release will still be a little whiles away), I think there > appeared to be broad concensus on that option 2 was reasonable. I will > try to get an implementation of that ready. If that's not ready in time > for netbsd-10 I think option 4 is probably least intrusive fallback > especially because it's optional. It may be that after that is done, there is no real reason based on the zfs concerns to change anything else. > critical_filesystems_zfs is still available now, and will be available > going forward. > > Option 3 doesn't seem to buy anything if option 4 is available and > option 5 seems like it could be more impactful to existing mixed local > and NFS setups. Agreed. > Option 6 I think is out of scope of what I want to get root-on-ZFS > working. That doesn't stop others from fleshing this out of course! > > Thoughts? Anything I've missed? I think you got it exactly right. > And somewhat related that came out of this discussion - add a "noerror" > or "notfatal" or some similar keyword that says a failed fsck or mount > doesn't automatically fail the boot and fall back to single user mode. That would be good. > Although Ignatios had an interesting solution to this with "noauto" and > using @reboot crontab entries. Sure, you work around the lack of the feature, but that doesn't mean a simpler way is bad. signature.asc Description: PGP signature
Re: ZFS - mounting filesystems (Was CVS commit: src/etc)
Lots of interesting discussion! Thanks all. Broadly I think I can summarise to the following options: 1. The existing critical_filesystems_zfs rc.conf variable, which mixes ZFS configuration in both rc.conf and with ZFS itself. 2. Add ZFS "critical" properties for filesystems and mount those in /etc/rc.d/mountcritlocal . 3. Move all ZFS mounts to /etc/rc.d/mountcritlocal . 4. Optionally move all ZFS mounts to /etc/rc.d/mountcritlocal controlled by an rc.conf variable (eg zfs_critical). 5. Move all local mounts to /etc/rc.d/mountcritlocal (ala FreeBSD) and possibly rename this to /etc/rc.d/mountlocal . 6. "Intelligent" ordering based on prefixes, tsort, etc. In terms of what to do for the up-and-coming netbsd-10 (although the actual release will still be a little whiles away), I think there appeared to be broad concensus on that option 2 was reasonable. I will try to get an implementation of that ready. If that's not ready in time for netbsd-10 I think option 4 is probably least intrusive fallback especially because it's optional. critical_filesystems_zfs is still available now, and will be available going forward. Option 3 doesn't seem to buy anything if option 4 is available and option 5 seems like it could be more impactful to existing mixed local and NFS setups. Option 6 I think is out of scope of what I want to get root-on-ZFS working. That doesn't stop others from fleshing this out of course! Thoughts? Anything I've missed? And somewhat related that came out of this discussion - add a "noerror" or "notfatal" or some similar keyword that says a failed fsck or mount doesn't automatically fail the boot and fall back to single user mode. Although Ignatios had an interesting solution to this with "noauto" and using @reboot crontab entries. This seems like a useful idea. Anyone up for implementing it? :) Cheers, Simon.
Re: inetd tests failing
On Thu, Mar 17, 2022 at 04:58:56PM +1030, Brett Lymn wrote: > Perhaps a shim program that manages a socket between it and the > underlying daemon - the shim can talk to inetd to coordinate the handoff > of an incoming connection and also being put back onto the idle pool > when the connection closes. My point was: it is pretty simple and efficient to do in setups like httpd where the child and parent are the same and also can cooperate in special ways (like the preford children having a socketpair open to the parent and get their real stdin passed later on wakeup). But in general cases like inetd and child programs not specially written for this setup, it is impossible (or needs special tricks that will kill all the benefits like a shim program). The prefork idea is to have children ready for service as quickly as possible after a new connection has been established - avoiding delay by the OS for creating a new process and initialization time for execing the new servicing binary (and its internal initial initialization time). I don't see how that could be done in inetd context without adding special new kernel support. E.g. we could create a posix_spawn() flag that makes the child process stop immediately after loading and create a new syscall that (if called by the parent) replaces a set of filedescriptors for procs in this state and unblocks them. But maybe I am missing something or misunderstanding the optimization. Martin
Re: inetd tests failing
On Wed, Mar 16, 2022 at 07:42:26AM +0100, Martin Husemann wrote: > > How does inetd replace their stdin once the connection is established? Perhaps a shim program that manages a socket between it and the underlying daemon - the shim can talk to inetd to coordinate the handoff of an incoming connection and also being put back onto the idle pool when the connection closes. > Many deamons will do socket operations (e.g. querying the peer) early > on startup - that would fail (or deliver bogus data) in the pre-fork > case. > I am going to start handwaving here. Perhaps extend kevent to snoop on the setting of socket options on the daemon socket and replicate those settings on an incoming connection. Waving hands even faster, this may not work for all daemons, make it optional so if it blows up then just go back to the exec per connection method. -- Brett Lymn -- Sent from my NetBSD device. "We are were wolves", "You mean werewolves?", "No we were wolves, now we are something else entirely", "Oh"
Re: inetd tests failing
On Tue, Mar 15, 2022 at 05:45:48PM -0400, Mouse wrote: > > This is possible for only a restricted set of services (those that are > at least conceptually datagram services, more or less). For example, I > have a machine with an inetd entry which uses a script running nc to > forward connections to a VNC server on another machine. Given the VNC > protocol, this cannot possibly work with a scheme such as you outline. > I don't think it precludes tcp as long as they are stateless - http is an obvious example > Was the discussion restricted to only such a subset of services? If > so, my apologies; that was not clear to me. > I believe so, it would be an option to pre-fork. -- Brett Lymn -- Sent from my NetBSD device. "We are were wolves", "You mean werewolves?", "No we were wolves, now we are something else entirely", "Oh"
