I have an sshloginfile with 12 hostnames in it. All machines are identical. I 
used parallel like this:
$ parallel --controlmaster --sshdelay 0.3  --joblog joblog.txt --sshloginfile 
~/etc/parallel/hosts  'echo {} | /shared/nfs/process-file' :::: filelist.txt | 
tee -a result.txt
/shared/nfs/process-file is a shell script and is accessible on all hosts
filelist.txt is a file with 6247 paths to .gz files in it. These .gz files are 
processed by the process-file script.
While the parallel was running I noticed that all of the jobs were being 
allocated to one of the hosts in the hostfile, host05.  In fact, the first time 
a host other than host05 was allocated a job was on sequence number 622. 
The final distribution of how many jobs ran on each of the 12 hosts is:
$ cat joblog.txt  | awk '{ print $2 }' | sort | uniq -c         2 host01     
538 host02         1 host03     323 host04   4696 host05         1 host07       
  2 host08         2 host09     493 host10         3 host11         3 host12    
     1 Host$
As you can see, the distribution is very imbalanced. 
Later in the job run, I did start seeing messages like the following:

Reply via email to