To work around some performance issues with our storage server, I need my users to work in a local scratch directory, copying files in at the beginning of a job and out at the end as needed. We're doing something similar to the suggestions on these sites, but would like to write the output and error files to the local scratch directory as well. https://www.princeton.edu/researchcomputing/faq/how-do-i-use-local-scratc/ http://www.ceci-hpc.be/slurm_faq.html#Q11
Something like the following fails. It does not look like SLURM evaluates environment variables in SBATCH directives. #SBATCH --output="$SCRATCH/%A/%a/stdout.txt" #SBATCH --error="$SCRATCH/%A/%a/stderr.txt" Something like the following fails, because the directory in which SLURM tries to create the output and error files does not exist when it tries to create the files. #SBATCH --output="/localscratch/netid/%A/%a/stdout.txt" #SBATCH --error="/localscratch/netid/%A/%a/stderr.txt" export SCRATCH="/localscratch/netid/$SLURM_ARRAY_JOB_ID/$SLURM_ARRAY_TASK_ID" rm -rf $SCRATCH mkdir -p $SCRATCH Creating a local scratch directory in a Prolog script works. However, Prolog scripts that run on compute nodes do not have the SLURM_ARRAY_JOB_ID or SLURM_ARRAY_TASK_ID environment variables. They can only get SLURM_JOB_ID. This makes it difficult to create a unique local scratch directory for each array index in the job arrays that most of my users submit. Does anyone know of a way to get job array or array task ids from a Prolog script or a way to set an environment variable in a Prolog script that could be used in a batch script? Thanks, Neile Havens Scientific Computing System Administrator Wheaton College
