This isn't exactly what you're looking for but I'll chime in anyway with
how we do things. We decided to buy a few slightly beefier interactive
nodes and set up cgroups, /tmp and /dev/shm namespaces (/tmp and
/dev/shm are per-user), cputime limits, /tmp quotas, etc to sanely
oversubscribe resources. This ended up being cheaper than other options
and it has worked really well. We currently use LVS to load balance
between interactive nodes but may switch to something else at some point.
We allow users to edit files, compile code, transfer files around, etc.
and also test their code for a little while. Anything beyond that
requires submitting a job. We limit users to 1/4 of the RAM on the node
and only 60 CPU-minutes per process via ulimit. The cpu cgroup (not
cpuset) is used to set a "soft limit" of just a few cores for each user
but allows them to burst to 100% of the cores when there is no
contention on the node.
It would take at least four users all using the maximum amount of RAM
plus some extra use before the node crashes. The memory per node ratio
and other settings could easily be changed if necessary.
In practice, these settings have made it so that no user has crashed an
interactive node since everything was deployed. Obviously I didn't
answer the original question about SLURM but this is an alternative
approach that has worked well for us. If you're interested in the code
we used to set everything up, it is available at:
https://github.com/BYUHPC/uft
Ryan
On 03/24/2014 01:32 PM, Olli-Pekka Lehto wrote:
I can foresee the screen issue as well. One could fairly simply add a check
when the user logs in to see if the the user has a node assigned to them
already and force the session to use that node. It could perhaps even prompt if
they want to do access this session or get a new one.
The immediate issue we encountered when testing with screen, however, is the
fact that when you detach and exit the interactive session SLURM faithfully
cleans all the processes. In most cases this would be preferred but in this
case I want the screen session (and the associated interactive job) to persist.
Any ideas how to do this?
In our case there is no real time limit on the current interactive use nodes so
setting runtime as unlimited is probably the way to go, at least initially. Of
course one needs to have a sufficiently large oversubscription factor of slots
in this case.
Olli-Pekka
On Mar 24, 2014, at 3:48 PM, "Schmidtmann, Carl"
<[email protected]> wrote:
We considered this option as well but the problem we saw with it is what
happens when a user tries to use screen? Many of our users login, start screen,
do some work and then disconnect. Whenever they reconnect they can pick up from
where they left off. If you are allocated to a compute node based on loads, you
likely won't be on the same node where your last session was. This is
inconvenient for the users but then also leaves screen sessions open, at least
until the time limit expires, on compute nodes.
The other issue is the time limit. Do you make it 1 hour, 4 hours, 8 hours? How
long does a user get to be logged in? If the time limit expires, what happens
to the open editor session? Can this be recovered on a different compute node?
We are still looking for a good way to balance users on login nodes. Right now we are
working on a method of redirecting ssh logins based on user IDs which feels extremely
"hacky" as well.
Carl
--
Carl Schmidtmann
Center for Integrated Research Computing
University of Rochester
On Mar 24, 2014, at 5:44 AM, Olli-Pekka Lehto wrote:
Dear devs,
We are testing a concept where we are dynamically allocating a portion of our
compute nodes with oversubscribed interactive nodes for low-intensity use. To
make the use as simple as possible, we are testing redirecting user login
sessions directly to these nodes via SLURM.
Basically the shell initialization on the actual login node contains a SLURM srun command
to spawn an interactive session and the user gets "transparently" dropped into
a shell session on a compute node.
This would offer more flexibility than physically setting up a set of login
nodes. Furthermore, SLURM should be able make better decisions on where to
assign each incoming session based on resource usage than a more naive
round-robin load balancer. This way also all interactive use can be tracked
with SLURM's accounting.
Based on simple initial testing this seems to work but it's still a bit hacky.
My question is has anyone been doing similar things and what are your
experiences? Are there some caveats that we should be aware of?
Best regards,
Olli-Pekka
--
Olli-Pekka Lehto
Development Manager
Computing Platforms
CSC - IT Center for Science Ltd.
E-Mail: [email protected]
Tel: +358 50 381 8604
skype: oplehto // twitter: ople
--
Ryan Cox
Operations Director
Fulton Supercomputing Lab
Brigham Young University
http://tech.ryancox.net
