Hi Orion,
I wonder if you can try this:
use the PE_HOSTFILE that is generated by the qsh -pe my_pe for tight
integration, so the host names may be the ones without the "x". But then
in your mpirun command, specify the interfaces you would like to
exclude, e.g. all except for the gigE interface. See if that works instead.
-mca btl_tcp_if_exclude lo,all_my_non_gigE_if
Orion Poplawski wrote:
Pak Lui wrote:
Hi Orion and Reuti,
Let me see if I can understand the issue by breaking them down first:
(1) First, I am curious to know why you would need to create a
PE_HOSTFILE yourself, because that file is generated by SGE/N1GE when
you specify you are running a parallel job under SGE/N1GE, by doing
something like this with qsub/qsh/qrsh, etc:
In our setup (which I don't believe is very unique) the nodes are
connected by two networks: an "admin" network which allows for
connections from outside the cluster and an "MPI" network that is a
private GigE network connecting the nodes for MPI traffic:
+---------admin net (192.168.0.X)--------+
| | |
+-----------+ +--------+ +--------+
| SGE Master| | coop00 | | coop01 |
| | | coop00x| | coop01x|
+-----------+ +--------+ +--------+
| |
+------------+
MPI net (192.168.1.X)
So the "x" suffix names are the addresses on the MPI network.
Currently (loose integration), we create machines files like:
coop00x.cora.nwra.com cpu=2
coop01x.cora.nwra.com cpu=2
which makes the MPI traffic travel over the MPI network. I'm trying to
duplicate this under "tight" integration.
(2) As for the following error message:
> error: commlib error: access denied (client IP resolved to host name
> "coop01x.cora.nwra.com". This is not identical to clients host name
> "coop01.cora.nwra.com")
As you mentioned in your setup, each node has 2 interfaces. And this
message is an SGE error and it seems to tell you that SGE cannot
resolve the host name.
No, it says that the IP resolved to "coop01x.cora.nwra.com", but the
machines' primary hostname is "coop01.cora.nwra.com".
(4) As for what you have mentioned here:
> Now, looking at the OpenMPI gridengine code, it looks like it gets the
> node name from the first entry in the pe_hostfile, and never really
uses
> the queue name for anything.
>
> ptr = strtok_r(buf, " \n", &tok);
> num = strtok_r(NULL, " \n", &tok);
> queue = strtok_r(NULL, " \n", &tok);
> arch = strtok_r(NULL, " \n", &tok);
> ...
> node->node_name = strdup(ptr);
> node->node_arch = strdup(arch);
>
> Perhaps it can be modified it uses the queue name hostname when doing
> SGE/qrsh calls, but the first hostname when doing MPI communication.
> Not really sure what the intent of the two fields in SGE's pe_hostfile
> is, or if OpenMPI can handle the idea of two hostnames for different
> purposes.
>
Once it is in a parallel environment of SGE (e.g. when you have
started a parallel job with "qsh/qsub/qrsh -pe name_of_pe"), in ORTE
would use the -inherit flag of qrsh to tell qrsh to start a task in a
already scheduled parallel job, therefore we cannot assign another
queue to the job
I don't want to assign another queue to the job. Assuming a pe_hostfile
of:
coop01x.cora.nwra.com 2 coo...@coop01.cora.nwra.com <NULL>
coop00x.cora.nwra.com 2 coo...@coop00.cora.nwra.com <NULL>
I would like ORTE to use "coop01.cora.nwra.com" and
"coop00.cora.nwra.com" (taken from the hostname parts of the queue
names) for use with qrsh, and "coop01x.cora.nwra.com" and
"coop00x.cora.nwra.com" as the hostnames to use for MPI traffic.
Again, this is just a stab at some way to hack what I want to achieve,
which is SGE/admin traffic traveling over one network, and MPI traffic
traveling over another. I don't care how it is done.
Thanks!
- Orion
--
Thanks,
- Pak Lui
pak....@sun.com
