Re: Exposing a hierarchy of ZFS datasets inside multiple jails
On Tue, Jun 21, 2011 at 08:51:22PM +0200, Paul Schenkeveld wrote: > Hi, > > On Fri, Jun 17, 2011 at 02:46:59PM -0400, Lars Kellogg-Stedman wrote: > > Hello all, > > > > Hi there, > > > > I am trying to expose a hierarchy of home directories to a number of > > FreeBSD jails. The home directories are configured such that each is a > > unique ZFS dataset. The jails are used for development work and hence > > are created and destroyed on a regular basis. > > > > My first thought was simply to use nullfs to mount /home inside the > > jail, but nullfs doesn't provide any way to access subordinate > > filesystems. > > > > My second thought was to export the directories via NFS and then run > > the automounter daemon (amd) inside each jail. This would have Just > > Worked...if it were possible to perform NFS mounts inside a jail. But > > it's not. > > > > My third thought was to run amd on the host and provision nullfs > > mounts into the jails...but amd support for nullfs doesn't exist. > > > > My fourth thought was to go back to exporting the directories using > > NFS, because of course amd works with NFS, right? Unfortunately, > > rather than mounting a directory on the target mountpoint, amd likes > > to mount things in a temporary location (/.amd_mnt/...) and then > > create a symlink...which, of course, is useless inside the jail > > environment.t > > > > So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to > > the jail? No! This brings us back to my first attempt, in which we > > find that there is no way to access subordinate filesystems using > > nullfs. > > > > And then my head exploded. > > > > Is there a good solution for what I'm trying to do? A bad solution > > would be to run a script after booting the jail that would create > > multiple nullfs mountpoints for all the home directories, but this is > > pretty clunky -- it would need to be run periodically to take into > > account new directories or removed directories. So basically I would > > have to write a poorly designed automounter. > > > > There must be a better way. How are other folks solving this? > > > > It looks like there are discussions going back several years about > > setting the VFCF_JAIL on NFS filesystems, but it these haven't > > resulted in any changes to the released code. Is this the best way to > > go? In theory, if I build a kernel under which NFS is jail friendly I > > can go ahead and run amd inside the jail > > Probably not a good solution but to stir the pool of thoughts a bit... > > Nullfs mounts and NFS mounts operate on filesystems (or datasets) and > do not include subordinates. Smbfs operates on directory (sub)trees > so have /home and /home/user[123...] datasets outside the jails, run > samba there with a share called [home] (not to be confused with the > [homes] share that comes with smb.conf.sample) and mount this share > using mount_smbfs inside every jail (from fstab.). > mount_smbfs needing rpcbind for the host that jails are being hosted on and also rpcbind for each jail, your going to run into a problem here due to the fact that rpcbind listens on the *:N without going in and making some specicial modifications for each jail and the host itself by editing code and recompiling for each jail. This is a known problem with RPC type services and jails and will only escalate by the number of jails that you would have to implement this on. Instead of making each userdir a dataset I would suggest just creating them as regular directories within a base dataset if this is the approach you would like to take with nullfs. ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
Hi, On Fri, Jun 17, 2011 at 02:46:59PM -0400, Lars Kellogg-Stedman wrote: > Hello all, > > Hi there, > > I am trying to expose a hierarchy of home directories to a number of > FreeBSD jails. The home directories are configured such that each is a > unique ZFS dataset. The jails are used for development work and hence > are created and destroyed on a regular basis. > > My first thought was simply to use nullfs to mount /home inside the > jail, but nullfs doesn't provide any way to access subordinate > filesystems. > > My second thought was to export the directories via NFS and then run > the automounter daemon (amd) inside each jail. This would have Just > Worked...if it were possible to perform NFS mounts inside a jail. But > it's not. > > My third thought was to run amd on the host and provision nullfs > mounts into the jails...but amd support for nullfs doesn't exist. > > My fourth thought was to go back to exporting the directories using > NFS, because of course amd works with NFS, right? Unfortunately, > rather than mounting a directory on the target mountpoint, amd likes > to mount things in a temporary location (/.amd_mnt/...) and then > create a symlink...which, of course, is useless inside the jail > environment.t > > So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to > the jail? No! This brings us back to my first attempt, in which we > find that there is no way to access subordinate filesystems using > nullfs. > > And then my head exploded. > > Is there a good solution for what I'm trying to do? A bad solution > would be to run a script after booting the jail that would create > multiple nullfs mountpoints for all the home directories, but this is > pretty clunky -- it would need to be run periodically to take into > account new directories or removed directories. So basically I would > have to write a poorly designed automounter. > > There must be a better way. How are other folks solving this? > > It looks like there are discussions going back several years about > setting the VFCF_JAIL on NFS filesystems, but it these haven't > resulted in any changes to the released code. Is this the best way to > go? In theory, if I build a kernel under which NFS is jail friendly I > can go ahead and run amd inside the jail Probably not a good solution but to stir the pool of thoughts a bit... Nullfs mounts and NFS mounts operate on filesystems (or datasets) and do not include subordinates. Smbfs operates on directory (sub)trees so have /home and /home/user[123...] datasets outside the jails, run samba there with a share called [home] (not to be confused with the [homes] share that comes with smb.conf.sample) and mount this share using mount_smbfs inside every jail (from fstab.). Just my $.02 Regards, Paul Schenkeveld ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
Christian, Thank you for your answer. I've read through your email a few times, and I think that there is a problem with your suggestion. Please let me know if I've misunderstood something. If I create a ZFS dataset: zfs create tank/nullfs And then mount this somewhere else via nullfs: mount_nullfs /tank/nullfs /mnt/nfs_home And then create a new ZFS dataset below tank/nullfs: zfs create tank/nullfs/user1 And then create some files in that dataset: $ touch /tank/nullfs/user1/file1 $ touch /tank/nullfs/user1/file2 $ find /tank/nullfs/user1 /tank/nullfs/user1 /tank/nullfs/user1/file1 /tank/nullfs/user1/file2 The only thing I will find in /mnt/nfs_home is an empty directory named "user1": $ find /mnt/nfs_home /mnt/nfs_home/ /mnt/nfs_home/user1 The nullfs mount of /tank/nullfs to /mnt/nfs_home only exposes files and directories contained in the "nullfs" dataset, but not in any subordinate datasets. This is exactly my original problem (otherwise I would simply have nullfs mounted /home inside my jails). > teufelchen# zfs create tank/nullfs/dataset1 > teufelchen# touch /mnt/tank/nfs_home/dataset1/newfile > teufelchen# jexec 14 ls /mnt/nfs_home/dataset1/ > newfile > > Is this what you are trying todo? I think that there may be a problem with your example here. When you run: teufelchen# touch /mnt/tank/nfs_home/dataset1/newfile ...I don't think you're creating the file where you think you are. Take a look at /mnt/tank/nullfs/dataset1; I suspect you won't find it there. What you've accomplished is to use the "nfs_home" dataset exclusively, ignoring any subordinate datasets. That is, you're treating it like you do this: zfs create tank/nullfs mkdir /mnt/tank/nullfs/dataset1 Rather than: zfs create tank/nullfs zfs create tank/nullfs/dataset1 The difference is crucial to this problem. Cheers, -- Lars ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
Moin, I assume the easiest solution would be to make the many mountpoints (datasets) invisiable for NFS. Here we go. teufelchen# /etc/rc.d/jail stop alcatraz ## This will be the dataset which contains home's for example teufelchen# zfs create tank/nullfs ## Now I create a dir which will be shared to the jails ## My idea is to make the mountpoints invisiable for NFS teufelchen# mkdir /mnt/tank/nfs_home teufelchen# mount_nullfs /mnt/tank/nullfs/ /mnt/tank/nfs_home/ teufelchen# mkdir /mnt/tank/alcatraz/mnt/nfs_home teufelchen# cat /mnt/tank/freebsd/fstab.alcatraz 192.168.1.10:/mnt/tank/nfs_home /mnt/tank/alcatraz/mnt/nfs_home nfs rw 0 0 teufelchen# /etc/rc.d/jail start alcatraz ### IIRC ### You like to create datasets on the host, which apear in the jails just in ### time? teufelchen# zfs create tank/nullfs/dataset1 teufelchen# touch /mnt/tank/nfs_home/dataset1/newfile teufelchen# jexec 14 ls /mnt/nfs_home/dataset1/ newfile Is this what you are trying todo? Christian Degen ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
On Fri, 17 Jun 2011 14:46:59 -0400 Lars Kellogg-Stedman wrote: > Hello all, > > Hi there, > > I am trying to expose a hierarchy of home directories to a number of > FreeBSD jails. The home directories are configured such that each is a > unique ZFS dataset. The jails are used for development work and hence > are created and destroyed on a regular basis. > > My first thought was simply to use nullfs to mount /home inside the > jail, but nullfs doesn't provide any way to access subordinate > filesystems. > > My second thought was to export the directories via NFS and then run > the automounter daemon (amd) inside each jail. This would have Just > Worked...if it were possible to perform NFS mounts inside a jail. But > it's not. See the second part of the patch at http://www.leidinger.net/FreeBSD/current-patches/sys:nfsclient.diff for how to allow NFS mounts in a jail. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
> Why dont you create a NFS-loopback? You can create a fstab for each jail and > use this to mount the nfs-shares. > from rc.conf > jail_alcatraz_fstab="/mnt/tank/freebsd/fstab.alcatraz" > > fstab.alcatraz > 127.0.0.1:/foo/bar /var/jails/alcatraz/mnt/bar nfs rw 0 0 The problem with this solution -- if I understand you correctly -- is that it doesn't deal well with a population of home directories that may change while the jail is running. This is what I described as "writing a poorly designed automounter". I could... - Iterate over /home - Generate an appropriate /etc/fstab. entry for each directory - Start the jail ...but the only way to pick up changes would be to reboot the jail. This is certainly a functional fallback position, but I'd love a more graceful solution. -- Lars Kellogg-Stedman Senior Technologist Harvard University SEAS Academic and Research Computing (ARC) ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Exposing a hierarchy of ZFS datasets inside multiple jails
Moin, On 06/17/2011 18:46, Lars Kellogg-Stedman wrote: My first thought was simply to use nullfs to mount /home inside the jail, but nullfs doesn't provide any way to access subordinate filesystems. Why dont you create a NFS-loopback? You can create a fstab for each jail and use this to mount the nfs-shares. from rc.conf jail_alcatraz_fstab="/mnt/tank/freebsd/fstab.alcatraz" fstab.alcatraz 127.0.0.1:/foo/bar/var/jails/alcatraz/mnt/barnfsrw00 yours, bubulein ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Exposing a hierarchy of ZFS datasets inside multiple jails
Hello all, Hi there, I am trying to expose a hierarchy of home directories to a number of FreeBSD jails. The home directories are configured such that each is a unique ZFS dataset. The jails are used for development work and hence are created and destroyed on a regular basis. My first thought was simply to use nullfs to mount /home inside the jail, but nullfs doesn't provide any way to access subordinate filesystems. My second thought was to export the directories via NFS and then run the automounter daemon (amd) inside each jail. This would have Just Worked...if it were possible to perform NFS mounts inside a jail. But it's not. My third thought was to run amd on the host and provision nullfs mounts into the jails...but amd support for nullfs doesn't exist. My fourth thought was to go back to exporting the directories using NFS, because of course amd works with NFS, right? Unfortunately, rather than mounting a directory on the target mountpoint, amd likes to mount things in a temporary location (/.amd_mnt/...) and then create a symlink...which, of course, is useless inside the jail environment.t So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to the jail? No! This brings us back to my first attempt, in which we find that there is no way to access subordinate filesystems using nullfs. And then my head exploded. Is there a good solution for what I'm trying to do? A bad solution would be to run a script after booting the jail that would create multiple nullfs mountpoints for all the home directories, but this is pretty clunky -- it would need to be run periodically to take into account new directories or removed directories. So basically I would have to write a poorly designed automounter. There must be a better way. How are other folks solving this? It looks like there are discussions going back several years about setting the VFCF_JAIL on NFS filesystems, but it these haven't resulted in any changes to the released code. Is this the best way to go? In theory, if I build a kernel under which NFS is jail friendly I can go ahead and run amd inside the jail -- Lars Kellogg-Stedman ___ freebsd-jail@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
