the pam_namespace sounds really interesting and seems to do the equivalent and much more, I will keep that in mind for next unshare related works and forget about pam_unshare :).
Regards, Matthieu 2012/12/17 Mark A. Grondona <[email protected]> > > > You may also be able to use pam_namespace(8) which should exist > in most distros already. > > I also have a little lua library that binds the unshare(2) > and chroot(2) system calls, which might be useful with the > spank/lua plugin (We use it to create per-job /tmp and /dev/shm > directories, and to create private SysV IPC.) Let me know > if you'd like to see the library. It isn't posted anywhere > public yet. > > mark > > > Matthieu Hautreux <[email protected]> writes: > > > Some times ago, I wrote a pam_unshare to help using the unshare syscall > in > > different tools using PAM. It was a beta dev that is still beta as I > still > > not find time to really use/validate it... > > > > I have just tried to do what Mark proposed using it in Slurm and it seems > > to do the job correctly. You will find it enclosed. > > > > Here is the example I tried : > > > > --- > > [root@leaf ~]# dd if=/dev/zero of=/tmp/toto bs=1M count=100 > > 100+0 records in > > 100+0 records out > > 104857600 bytes (105 MB) copied, 0.474764 s, 221 MB/s > > [root@leaf ~]# mount -o loop /tmp/toto /mnt/test > > mount: you must specify the filesystem type > > [root@leaf ~]# mkfs.ext3 /tmp/toto > > mke2fs 1.41.14 (22-Dec-2010) > > /tmp/toto is not a block special device. > > Proceed anyway? (y,n) y > > Filesystem label= > > OS type: Linux > > Block size=1024 (log=0) > > Fragment size=1024 (log=0) > > Stride=0 blocks, Stripe width=0 blocks > > 25688 inodes, 102400 blocks > > 5120 blocks (5.00%) reserved for the super user > > First data block=1 > > Maximum filesystem blocks=67371008 > > 13 block groups > > 8192 blocks per group, 8192 fragments per group > > 1976 inodes per group > > Superblock backups stored on blocks: > > 8193, 24577, 40961, 57345, 73729 > > > > Writing inode tables: done > > Creating journal (4096 blocks): done > > Writing superblocks and filesystem accounting information: done > > > > This filesystem will be automatically checked every 27 mounts or > > 180 days, whichever comes first. Use tune2fs -c or -i to override. > > [root@leaf ~]# mount -o loop /tmp/toto /mnt/test > > [root@leaf ~]# ls /mnt/test > > lost+found > > [root@leaf ~]# grep -i pam /etc/slurm/slurm.conf > > UsePAM=no > > [root@leaf ~]# srun ls /mnt/test/ > > lost+found > > [root@leaf ~]# emacs -nw /etc/slurm/slurm.conf > > [root@leaf ~]# grep -i pam /etc/slurm/slurm.conf > > UsePAM=yes > > [root@leaf ~]# cat /etc/pam.d/slurm > > auth required pam_localuser.so > > auth required pam_shells.so > > account required pam_unix.so > > account required pam_access.so > > session required pam_unix.so > > session required pam_unshare.so debug flags=CLONE_NEWNS > > prolog=/etc/slurm/unshare.prolog > > [root@leaf ~]# cat /etc/slurm/unshare.prolog > > #!/bin/bash > > > > mount --make-rprivate / > > umount /mnt/test > > > > exit 0 > > [root@leaf ~]# service slurm restart > > Restarting slurm (via systemctl): [ OK ] > > [root@leaf ~]# srun ls /mnt/test/ > > [root@leaf ~]# ls /mnt/test > > lost+found > > [root@leaf ~]# > > > > as a user : > > > > [mat@leaf ~]$ srun ls /mnt/test/ > > [mat@leaf ~]$ ls /mnt/test/ > > lost+found > > [mat@leaf ~]$ > > > > > > --- > > > > It needs more testing and there are probably some bugs in the code, but > if > > you want to try it, I would be pleased by any feedback. I should > certainly > > create a repo on github if someone wants to use it and report bugs... let > > me know if it sounds useful and I will do that. It should be usable > > directly in openssh too, if you want to do the same for remote > connections. > > > > Regards, > > Matthieu > > > > > > 2012/12/17 Mark A. Grondona <[email protected]> > > > >> > >> > >> > >> You shouldn't need to muck with slurm cgroups code nor use a full > >> > >> chroot solution to remove access to certain fileystems for unauthorized > >> > >> jobs. You can write a simple plugin to put the job in a private > >> > >> namespace via the unshare(2) system call, mark the new filesystem > >> > >> tree private, then unmount unauthorized filesystems. > >> > >> > >> > >> Since you've marked the mount tree in the current namespace as > >> > >> private, the unmount shouldn't propagate to other namespaces, > >> > >> so your Lustre filesystems will stay mounted in the default > >> > >> namespace. > >> > >> > >> > >> You would need to do this in a spank plugin so that the unshare(2) > >> > >> and mount(8) commands run in the context of the processes which > >> > >> you are launching. > >> > >> > >> > >> Here's an example of the process using the unshare(1) shell utility: > >> > >> > >> > >> > >> > >> # hype356 /root > df /p/lcrater2 > >> > >> Filesystem 1K-blocks Used Available Use% Mounted on > >> > >> zwicky-mds2-ib0@o2ib9:/lc2 > >> > >> 218665928880 125545872548 93120053348 58% > >> > >> /p/lcrater2 > >> > >> # hype356 /root > unshare -m bash > >> > >> hype356@root:df /p/lcrater2 > >> > >> Filesystem 1K-blocks Used Available Use% Mounted on > >> > >> zwicky-mds2-ib0@o2ib9:/lc2 > >> > >> 218665928880 125545872548 93120053348 58% > >> > >> /p/lcrater2 > >> > >> hype356@root:mount --make-rprivate / > >> > >> hype356@root:umount /p/lcrater2 > >> > >> hype356@root:df /p/lcrater2 > >> > >> Filesystem 1K-blocks Used Available Use% Mounted on > >> > >> /dev/sda3 1919906792 26320708 1796060428 2% / > >> > >> > >> > >> > >> > >> Invoking a new login on this node shows that /p/lcrater2 is actually > >> > >> still mounted in the default namespace: > >> > >> > >> > >> grondo@hype356:~$ df /p/lcrater2 > >> > >> Filesystem 1K-blocks Used Available Use% Mounted on > >> > >> 218665928880 125545872548 93120053348 58% /p/lcrater2 > >> > >> > >> > >> > >> > >> mark > >> > >> > >> > >> > >> > >> Moe Jette <[email protected]> writes: > >> > >> > >> > >> > If you define 20 lustre licenses in Slurm and every job using those > >> > >> > resources requests a license that should work. Linux cgroups could > >> > >> > prevent file system for jobs without a license, but that would require > >> > >> > some non-trivial changes to the Slurm code. > >> > >> > > >> > >> > > >> > >> > Quoting Marcin Stolarek <[email protected]>: > >> > >> > > >> > >> >> 2012/12/16 Aaron Knister <[email protected]> > >> > >> >> > >> > >> >>> Hi Marcin, > >> > >> >>> > >> > >> >>> Hi Aaron, > >> > >> >> > >> > >> >> > >> > >> >>> Could you describe the use case for preventing access when the > lustre > >> > >> >>> license isn't specified? That might help me offer a better solution. > >> > >> >>> > >> > >> >> > >> > >> >> For instance, I know that my lustre installation can support > efficiently > >> > >> >> only 20 running jobs, but I have more CPU resources so I'd like to > >> prevent > >> > >> >> running more than 20 jobs on lustre, but allow other jobs (using only > >> local > >> > >> >> scratch or even do not using scratch at all). > >> > >> >> > >> > >> >>> > >> > >> >>> Off the top of my head I could see using prolog/epilog scripts to > mount > >> > >> >>> and unmount lustre as required however this only works if your nodes > >> are > >> > >> >>> job-exclusive (one job per node). > >> > >> >>> > >> > >> >> > >> > >> >> This is not my case, I have heterogenous installation with 64 cores > in > >> a > >> > >> >> few nodes on which are sometimes running 64 separate jobs :) > >> > >> >> > >> > >> >>> > >> > >> >>> Another option would be to use a chroot environment for each job and > >> only > >> > >> >>> bind-mount required filesystems. This approach has a lot of > overhead, > >> > >> >>> though and I'm not entirely sure of the implementation details. You > >> could > >> > >> >>> also try the cgroup namespace subsystem. I did some reading this > >> morning > >> > >> >>> and can't find documentation for this but I had some vague > recollection > >> > >> >>> that you could control which filesystems are accessible to each > >> namespace. > >> > >> >>> > >> > >> >> chroot proposition sounds interesting, I don't know how to use > cgroups > >> for > >> > >> >> filesystem IO operations limiting. > >> > >> >> > >> > >> >> thank you, > >> > >> >> marcin > >> > >> >> > >> > >> >>> > >> > >> >>> Sent from my iPhone > >> > >> >>> > >> > >> >>> On Dec 16, 2012, at 10:12 AM, Marcin Stolarek < > >> [email protected]> > >> > >> >>> wrote: > >> > >> >>> > >> > >> >>> Hi all, > >> > >> >>> > >> > >> >>> I'm thinking about describing lustre as a licence in our slurm > >> > >> >>> installation. Do you know any possibility of preventing job from > >> writing > >> > >> >>> on/reading from a specified file system if user haven't specified > >> > >> >>> appropriate licence string? > >> > >> >>> > >> > >> >>> thanks, > >> > >> >>> marcin > >> > >> >>> > >> > >> >>> > >> > >> >> > >> >
