You may also be able to use pam_namespace(8) which should exist
in most distros already.
I also have a little lua library that binds the unshare(2)
and chroot(2) system calls, which might be useful with the
spank/lua plugin (We use it to create per-job /tmp and /dev/shm
directories, and to create private SysV IPC.) Let me know
if you'd like to see the library. It isn't posted anywhere
public yet.
mark
Matthieu Hautreux <[email protected]> writes:
> Some times ago, I wrote a pam_unshare to help using the unshare syscall in
> different tools using PAM. It was a beta dev that is still beta as I still
> not find time to really use/validate it...
>
> I have just tried to do what Mark proposed using it in Slurm and it seems
> to do the job correctly. You will find it enclosed.
>
> Here is the example I tried :
>
> ---
> [root@leaf ~]# dd if=/dev/zero of=/tmp/toto bs=1M count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB) copied, 0.474764 s, 221 MB/s
> [root@leaf ~]# mount -o loop /tmp/toto /mnt/test
> mount: you must specify the filesystem type
> [root@leaf ~]# mkfs.ext3 /tmp/toto
> mke2fs 1.41.14 (22-Dec-2010)
> /tmp/toto is not a block special device.
> Proceed anyway? (y,n) y
> Filesystem label=
> OS type: Linux
> Block size=1024 (log=0)
> Fragment size=1024 (log=0)
> Stride=0 blocks, Stripe width=0 blocks
> 25688 inodes, 102400 blocks
> 5120 blocks (5.00%) reserved for the super user
> First data block=1
> Maximum filesystem blocks=67371008
> 13 block groups
> 8192 blocks per group, 8192 fragments per group
> 1976 inodes per group
> Superblock backups stored on blocks:
> 8193, 24577, 40961, 57345, 73729
>
> Writing inode tables: done
> Creating journal (4096 blocks): done
> Writing superblocks and filesystem accounting information: done
>
> This filesystem will be automatically checked every 27 mounts or
> 180 days, whichever comes first. Use tune2fs -c or -i to override.
> [root@leaf ~]# mount -o loop /tmp/toto /mnt/test
> [root@leaf ~]# ls /mnt/test
> lost+found
> [root@leaf ~]# grep -i pam /etc/slurm/slurm.conf
> UsePAM=no
> [root@leaf ~]# srun ls /mnt/test/
> lost+found
> [root@leaf ~]# emacs -nw /etc/slurm/slurm.conf
> [root@leaf ~]# grep -i pam /etc/slurm/slurm.conf
> UsePAM=yes
> [root@leaf ~]# cat /etc/pam.d/slurm
> auth required pam_localuser.so
> auth required pam_shells.so
> account required pam_unix.so
> account required pam_access.so
> session required pam_unix.so
> session required pam_unshare.so debug flags=CLONE_NEWNS
> prolog=/etc/slurm/unshare.prolog
> [root@leaf ~]# cat /etc/slurm/unshare.prolog
> #!/bin/bash
>
> mount --make-rprivate /
> umount /mnt/test
>
> exit 0
> [root@leaf ~]# service slurm restart
> Restarting slurm (via systemctl): [ OK ]
> [root@leaf ~]# srun ls /mnt/test/
> [root@leaf ~]# ls /mnt/test
> lost+found
> [root@leaf ~]#
>
> as a user :
>
> [mat@leaf ~]$ srun ls /mnt/test/
> [mat@leaf ~]$ ls /mnt/test/
> lost+found
> [mat@leaf ~]$
>
>
> ---
>
> It needs more testing and there are probably some bugs in the code, but if
> you want to try it, I would be pleased by any feedback. I should certainly
> create a repo on github if someone wants to use it and report bugs... let
> me know if it sounds useful and I will do that. It should be usable
> directly in openssh too, if you want to do the same for remote connections.
>
> Regards,
> Matthieu
>
>
> 2012/12/17 Mark A. Grondona <[email protected]>
>
>>
>>
>>
>> You shouldn't need to muck with slurm cgroups code nor use a full
>>
>> chroot solution to remove access to certain fileystems for unauthorized
>>
>> jobs. You can write a simple plugin to put the job in a private
>>
>> namespace via the unshare(2) system call, mark the new filesystem
>>
>> tree private, then unmount unauthorized filesystems.
>>
>>
>>
>> Since you've marked the mount tree in the current namespace as
>>
>> private, the unmount shouldn't propagate to other namespaces,
>>
>> so your Lustre filesystems will stay mounted in the default
>>
>> namespace.
>>
>>
>>
>> You would need to do this in a spank plugin so that the unshare(2)
>>
>> and mount(8) commands run in the context of the processes which
>>
>> you are launching.
>>
>>
>>
>> Here's an example of the process using the unshare(1) shell utility:
>>
>>
>>
>>
>>
>> # hype356 /root > df /p/lcrater2
>>
>> Filesystem 1K-blocks Used Available Use% Mounted on
>>
>> zwicky-mds2-ib0@o2ib9:/lc2
>>
>> 218665928880 125545872548 93120053348 58%
>>
>> /p/lcrater2
>>
>> # hype356 /root > unshare -m bash
>>
>> hype356@root:df /p/lcrater2
>>
>> Filesystem 1K-blocks Used Available Use% Mounted on
>>
>> zwicky-mds2-ib0@o2ib9:/lc2
>>
>> 218665928880 125545872548 93120053348 58%
>>
>> /p/lcrater2
>>
>> hype356@root:mount --make-rprivate /
>>
>> hype356@root:umount /p/lcrater2
>>
>> hype356@root:df /p/lcrater2
>>
>> Filesystem 1K-blocks Used Available Use% Mounted on
>>
>> /dev/sda3 1919906792 26320708 1796060428 2% /
>>
>>
>>
>>
>>
>> Invoking a new login on this node shows that /p/lcrater2 is actually
>>
>> still mounted in the default namespace:
>>
>>
>>
>> grondo@hype356:~$ df /p/lcrater2
>>
>> Filesystem 1K-blocks Used Available Use% Mounted on
>>
>> 218665928880 125545872548 93120053348 58% /p/lcrater2
>>
>>
>>
>>
>>
>> mark
>>
>>
>>
>>
>>
>> Moe Jette <[email protected]> writes:
>>
>>
>>
>> > If you define 20 lustre licenses in Slurm and every job using those
>>
>> > resources requests a license that should work. Linux cgroups could
>>
>> > prevent file system for jobs without a license, but that would require
>>
>> > some non-trivial changes to the Slurm code.
>>
>> >
>>
>> >
>>
>> > Quoting Marcin Stolarek <[email protected]>:
>>
>> >
>>
>> >> 2012/12/16 Aaron Knister <[email protected]>
>>
>> >>
>>
>> >>> Hi Marcin,
>>
>> >>>
>>
>> >>> Hi Aaron,
>>
>> >>
>>
>> >>
>>
>> >>> Could you describe the use case for preventing access when the lustre
>>
>> >>> license isn't specified? That might help me offer a better solution.
>>
>> >>>
>>
>> >>
>>
>> >> For instance, I know that my lustre installation can support efficiently
>>
>> >> only 20 running jobs, but I have more CPU resources so I'd like to
>> prevent
>>
>> >> running more than 20 jobs on lustre, but allow other jobs (using only
>> local
>>
>> >> scratch or even do not using scratch at all).
>>
>> >>
>>
>> >>>
>>
>> >>> Off the top of my head I could see using prolog/epilog scripts to mount
>>
>> >>> and unmount lustre as required however this only works if your nodes
>> are
>>
>> >>> job-exclusive (one job per node).
>>
>> >>>
>>
>> >>
>>
>> >> This is not my case, I have heterogenous installation with 64 cores in
>> a
>>
>> >> few nodes on which are sometimes running 64 separate jobs :)
>>
>> >>
>>
>> >>>
>>
>> >>> Another option would be to use a chroot environment for each job and
>> only
>>
>> >>> bind-mount required filesystems. This approach has a lot of overhead,
>>
>> >>> though and I'm not entirely sure of the implementation details. You
>> could
>>
>> >>> also try the cgroup namespace subsystem. I did some reading this
>> morning
>>
>> >>> and can't find documentation for this but I had some vague recollection
>>
>> >>> that you could control which filesystems are accessible to each
>> namespace.
>>
>> >>>
>>
>> >> chroot proposition sounds interesting, I don't know how to use cgroups
>> for
>>
>> >> filesystem IO operations limiting.
>>
>> >>
>>
>> >> thank you,
>>
>> >> marcin
>>
>> >>
>>
>> >>>
>>
>> >>> Sent from my iPhone
>>
>> >>>
>>
>> >>> On Dec 16, 2012, at 10:12 AM, Marcin Stolarek <
>> [email protected]>
>>
>> >>> wrote:
>>
>> >>>
>>
>> >>> Hi all,
>>
>> >>>
>>
>> >>> I'm thinking about describing lustre as a licence in our slurm
>>
>> >>> installation. Do you know any possibility of preventing job from
>> writing
>>
>> >>> on/reading from a specified file system if user haven't specified
>>
>> >>> appropriate licence string?
>>
>> >>>
>>
>> >>> thanks,
>>
>> >>> marcin
>>
>> >>>
>>
>> >>>
>>
>> >>
>>
