Hi John I associate a PE with each of the two hostgroups, say orte_mx and orte_ib. The user can then submit with -pe orte\* thus leaving the selection of the hostgroup to SGE. The PE startup script would be the same and by passing it SGE variables like $host etc the script can figure out what to set for OMPI_MCA_btl. See the example in Reuti's reply. I don't think the problem is so much selecting the host group as getting information from the PE start script to the mpirun command in the job script. Best wishes Henk
________________________________ From: users-boun...@open-mpi.org on behalf of John Hearns Sent: Sat 6/7/2008 12:07 AM To: Open MPI Users Subject: Re: [OMPI users] using OpenMPI + SGE in a heterogeneous network On Fri, 2008-06-06 at 17:56 +0100, SLIM H.A. wrote: > Hi > > I want to use SGE to run jobs on a cluster with mx and infiniband nodes. > By dividing the nodes into two host groups SGE will submit to either > interconnect. > > The interconnect can be specified in the mpirun command with the --mca > btl parameter. However users would then have to decide at runtime which > of the two interconnects should be used, but if the job has to queue > they just want the first available interconnect. If you are just using hostgroups, how do you stop SGE from assigning jobs across two hostgroups then? IE. if the user does not make an explicit request for @infinihosts or @myrihosts SGE will allocate a set of free hosts - and that could be a mix of the two types. As Reuti says you need those wildcard PEs. Or, as you say, just let the user decide when submitting the job. ps. there is another way of doing the selection, other than hostgroups. Set up two complexes, infiniband and myrinet (say) which are Boolean. Then for each exec host set complex_values infiniband=true complex_values myrinet=true The user submits with -l infiniband I'm not sure if the syntax -l infiniband | myrinet meaning infiniband or myrinet is valid. Reuti? _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
