yep, the --exclusive did it. I'll have to test it out some and see how
it works.
On Tue, Sep 25, 2012 at 01:14:04PM -0600, Danny Auble wrote:
>
> Andy, try the --exclusive option with each srun. It should get you what
> you want.
>
> Thousands of steps is more efficient than thousands of jobs. With 2.4+
> the difference might not be that noticeable though to a user.
>
> Danny
>
> On 09/25/12 08:32, Andy Wettstein wrote:
> > Hi,
> >
> > I was thinking about ways for users to run a high throughput type of job
> > without needing to submit thousands of jobs. It occured to me to try GNU
> > parallel with srun. So I did something like this:
> >
> > I ran an interactive slurm session with -n16 on a 16 core node. I
> > started a parallel run like this:
> >
> > seq 128 | ./parallel -j $SLURM_NTASKS "srun -n1 ./runtask arg1:{1}"
> >
> > -j $SLURM_NTASKS is the number of processes parallel will keep running
> > at once.
> >
> > The runtask script looks like this:
> > #!/bin/sh
> >
> > sleepsecs=$[ ( $RANDOM % 10 ) + 1 ]s
> >
> > echo task $1 sleep:$sleepsecs host:$(hostname)
> > sleep $sleepsecs
> >
> > Output looks like this:
> > task arg1:2 sleep:1s host:midway097
> > task arg1:5 sleep:1s host:midway097
> > task arg1:7 sleep:2s host:midway097
> > ...
> > task arg1:126 sleep:10s host:midway097
> >
> > So this appears to works as I would expect on a single node and parallel
> > keeps 16 tasks running at a time.
> >
> > It does not work as expected if I create an allocation with -N2
> > -n32. srun appears to only start these job steps on the node I am
> > actually using, so with SLURM_NTASKS=32 srun also start double the steps
> > that should be on the node. I can force the host by using -w in the srun
> > command, so if there was a way to determine which tasks on which hosts
> > were free I think it would be possible to wrap the srun command with a
> > -w argument to make this work.
> >
> > Before I get any further, I guess that brings up a couple of questions.
> > First, is this expected behavior from srun when multiple nodes are
> > allocated to a job? Second, would this actually be useful to run high
> > throughput jobs? IOW, is running thousands of job steps in this manner
> > more efficient than thousands of jobs?
> >
> > Andy
> >
--
andy wettstein
hpc system administrator
research computing center
university of chicago
773.702.1104
