This is how I recommend to users of our cluster to pass variable parameters to
a slurm array.
In your batch file set a variable that reads a line of a data file for
parameters.
e.g.
PARAMETERS=$(awk -v line=${SLURM_ARRAY_TASK_ID} '{if (NR == line) { print $0;
};}' ./data.dat)
What this does is it uses awk to read the line that matches this array elements
number, if the array element is 1 it will read the first line of data.dat and
set that as the PARAMETERS variable. Then you can use that variable in your
command line for your input data.
executable {common_args} $PARAMETERS
Then you just have to structure the lines of data.dat to resemble what you
would want to pass your executable, then submit your array.
sbatch --array=1-4 batch.sbatch
Will submit a batch file with array elements 1,2,3,and 4 and it will read the
respective lines of data.dat and then use those for the executable, it will
request the same resources for each of these elements of the array and they
will get thrown into the scheduler just like a normal job, it just allows you
to define your job inputs in a more convenient way. Potentially you could
submit thousands of jobs with one submission file in one command by using a job
array. There is a mechanism for sys-admins to set a max array size so there may
be an arbitrary upper limit for how many array elements you can have, but I'm
not certain if this is not set what the default value is.
If you are iterating the same executable over a several input files data.dat
can be easily built by moving all your inputs into one directory then piping
the file listing to the data.dat file.
ls > data.dat
Hope this is helpful, the array switch is pretty well documented in sbatch's
man page also,
Buddy.
-----Original Message-----
From: Trevor Gale [mailto:[email protected]]
Sent: Friday, May 08, 2015 2:34 PM
To: slurm-dev
Subject: [slurm-dev] RE: launching a variable number of jobs in slurm
It would be very nice if i could use a job array but I have to pass in a
different plain text file to the simulator call in each job to specify the
fault. Is it possible to do something like this with job arrays/how would you
do it? or do all the jobs have to be identical?
Thanks,
Trevor
> On May 8, 2015, at 2:44 PM, Scharfenberg, Buddy Lee <[email protected]> wrote:
>
> Trevor,
>
> It does depend a bit on the configuration of your cluster, however it sounds
> like what you need to do is create a job submission file that requests enough
> resources for one of your jobs and then submit them as an array. Read the man
> page on sbatch to determine what switches you need to use in a batch script
> to request what you need. I would also refer to a resource guide for your
> resource if one exists, there should at least be a quick start guide
> somewhere for your specific resources.
>
> Once your script is created you can pass the --array=##-## switch to sbatch
> and it will submit however many jobs are defined in the array and set the
> environment variable SLURM_ARRAY_TASK_ID to whichever element in the array it
> is.
>
> If you need it to run different commands or needs different inputs you can
> use this variable to do things like read a corresponding line in an input
> file, or execute different commands inside the submission script itself with
> if statements that are dependent on the value of the variable.
>
> Good luck,
> Buddy.
>
>
> -----Original Message-----
> From: Trevor Gale [mailto:[email protected]]
> Sent: Friday, May 08, 2015 1:07 PM
> To: slurm-dev
> Subject: [slurm-dev] launching a variable number of jobs in slurm
>
>
> Hello everyone,
>
> I’m developing a piece of software that runs fault injection simulations on a
> cluster running slurm and am trying to figure out the ideal method for
> launching a potentially massive amount of jobs. I’m not very familiar with
> slurm, and had a questions about how slurm allocates resources
> -if I simply call sbatch without specifying and resource requirements,
> does slurm automatically allocate sufficient resources? or does it depend
> upon the configuration of the cluster?
>
> Thanks,
> Trevor=
