See srun man page, sections on --exclusive and --relative:
--exclusive
This option has two slightly different meanings for job and job
step allocations....
This option can also be used when initiating more than one job
step within an existing resource allocation, where you want sep‐
arate processors to be dedicated to each job step. If sufficient
processors are not available to initiate the job step, it will
be deferred. This can be thought of as providing resource man‐
agement for the job within it's allocation. Note that all CPUs
allocated to a job are available to each job step unless the
--exclusive option is used plus task affinity is configured.
Since resource management is provided by processor, the --ntasks
option must be specified, but the following options should NOT
be specified --relative, --distribution=arbitrary. See EXAMPLE
below.
Quoting Andy Wettstein <[email protected]>:
>
> Hi,
>
> I was thinking about ways for users to run a high throughput type of job
> without needing to submit thousands of jobs. It occured to me to try GNU
> parallel with srun. So I did something like this:
>
> I ran an interactive slurm session with -n16 on a 16 core node. I
> started a parallel run like this:
>
> seq 128 | ./parallel -j $SLURM_NTASKS "srun -n1 ./runtask arg1:{1}"
>
> -j $SLURM_NTASKS is the number of processes parallel will keep running
> at once.
>
> The runtask script looks like this:
> #!/bin/sh
>
> sleepsecs=$[ ( $RANDOM % 10 ) + 1 ]s
>
> echo task $1 sleep:$sleepsecs host:$(hostname)
> sleep $sleepsecs
>
> Output looks like this:
> task arg1:2 sleep:1s host:midway097
> task arg1:5 sleep:1s host:midway097
> task arg1:7 sleep:2s host:midway097
> ...
> task arg1:126 sleep:10s host:midway097
>
> So this appears to works as I would expect on a single node and parallel
> keeps 16 tasks running at a time.
>
> It does not work as expected if I create an allocation with -N2
> -n32. srun appears to only start these job steps on the node I am
> actually using, so with SLURM_NTASKS=32 srun also start double the steps
> that should be on the node. I can force the host by using -w in the srun
> command, so if there was a way to determine which tasks on which hosts
> were free I think it would be possible to wrap the srun command with a
> -w argument to make this work.
>
> Before I get any further, I guess that brings up a couple of questions.
> First, is this expected behavior from srun when multiple nodes are
> allocated to a job? Second, would this actually be useful to run high
> throughput jobs? IOW, is running thousands of job steps in this manner
> more efficient than thousands of jobs?
>
> Andy
>
> --
> andy wettstein
> hpc system administrator
> research computing center
> university of chicago
> 773.702.1104
