I have an sshloginfile with 12 hostnames in it. All machines are identical. I
used parallel like this:
$ parallel --controlmaster --sshdelay 0.3 --joblog joblog.txt --sshloginfile
~/etc/parallel/hosts 'echo {} | /shared/nfs/process-file' :::: filelist.txt |
tee -a result.txt
/shared/nfs/process-file is a shell script and is accessible on all hosts
filelist.txt is a file with 6247 paths to .gz files in it. These .gz files are
processed by the process-file script.
While the parallel was running I noticed that all of the jobs were being
allocated to one of the hosts in the hostfile, host05. In fact, the first time
a host other than host05 was allocated a job was on sequence number 622.
The final distribution of how many jobs ran on each of the 12 hosts is:
$ cat joblog.txt | awk '{ print $2 }' | sort | uniq -c 2 host01
538 host02 1 host03 323 host04 4696 host05 1 host07
2 host08 2 host09 493 host10 3 host11 3 host12
1 Host$
As you can see, the distribution is very imbalanced.
Later in the job run, I did start seeing messages like the following:
