Hi, everyone. I am going to invoke GNU parallel on a multi-machine (nodes)
cluster via "qsub". a toy pbs script looks like:
#!/bin/sh
#PBS -l nodes=2:ppn=2
(...skip..)
cat parameters.txt | parallel bash MYSCRIPT.sh {}
However, the GNU parallel appears to not accurately spawn across nodes (i.e.
all MYSCRIPT.sh are invoked on the same nodes and consume cores/processes more
than I request by PBS). Following the tutorial, I add "-S" option and manually
specify the hosts and cores. Now the command looks like:
(...skip...)
HOSTS=`cat $PBS_NODEFILE | uniq -c | awk 'BEGIN{OFS=""}{print $1,"/",$2}'|tr
'\n' ','|sed 's/,$/ /'`
cat parameters.txt | parallel -S $HOSTS MYSCRIPT.sh {}
Seems this naive modification works for me. But my questions are:
(1) what is a more optimal setting to invoke parallel on cluster via PBS?
Since the job management system (like PBS) have assigned cores for the job, It
should be better to just use these resources.
(2) similar as the question 1. Does the parallel just spawn jobs on the cores
assigned by PBS, or randomly spawn on all available cores? I wonder it would
occupied cores not assigned by PBS so that influence other jobs or result in a
waste of resources (because the cores are have been assigned to this job).
any recommendation and answer are appreciated!!
Best regards,
Hiu
