Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Jeff Squyres]

On Jul 8, 2011, at 7:34 PM, Steve Kargl wrote:

>> We unfortunately don't have access to any BSD machines to test this
>> on, ourselves.  It works on other OS's, so I'm curious as to why it
>> doesn't seem to work for you.  :-(
> 
> I can arrange access on the cluster in question. ;-)

Actually, we have a fully-automated test suite that is run in a distributed 
fashion (e.g., I run it via cron on my MPI dev cluster at Cisco; my OMPI peer 
organizations do similar things).  Perhaps we could convince you to run small 
test jobs on a not-infrequent basis...?

Ping me over on the devel list if you're interested (should probably followup 
with the if.c/BTL TCP debugging over there, too).  We've already strayed pretty 
far outside of the "my ssh password setup doesn't work!" typical user questions 
for this list.  :-)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Fri, Jul 08, 2011 at 07:03:13PM -0400, Jeff Squyres wrote:
> Sorry -- I got distracted all afternoon...

No problem.  We all have obligations that we prioritize.

> In addition to what Ralph said (i.e., I'm not sure if the
> CIDR notation stuff made it over to the v1.5 branch or not,
> but it is available from the nightly SVN trunk tarballs:
> http://www.open-mpi.org/nightly/trunk/), here's a few points
> from other mails in this thread...

I try this out sometime next week.

> 1. Gus is correct that OMPI is complaining that bge1 doesn't
> exist on all nodes.  The MCA parameters that you pass on the
> command line get shipped to *all* MPI processes, and therefore
> generally need to work on all of them.  If you have per-host
> MCA parameter values, you can set them a few different ways:
> 
> - have a per-host MCA param file, usually in
> $prefix/etc/openmpi-mca-params.conf
> - have your shell startup files intelligently determine which
> host you're on and set the corresponding MCA environment variable
> as appropriate (e.g., on the head node, set the env variable
> OMPI_MCA_btl_tcp_if_include to bge1, and set it to bge0 on the others)
> 
> Those are a little klunky, but having a heterogeneous setup like this
> is not common, so we haven't really optimized the ability to set
> different MCA params on different servers.

There is no compelling reason for me to keep bge0 on the 10.208.
subnet and bge1 on the 192.168 subnet on node10.  If I switch
the two, so all bge0 nics are on 192.168., then I suppose
that --mca btl_tcp_if_include bge0 should work.  I'll try
this next weekr; if I can kick everyone off the cluster for
a few minutes. 

> 2. I am curious to figure out why the automatic reachability
> computations isn't working for you.  Unfortunately, the code
> to compute the reachability is pretty gnarly.  :-\  The code
> to find the IP interfaces on your machines is in opal/util/if.c.
> That *should* be working -- there's *BSD-specific code in there
> that has been verified by others in the past... but who knows?
> Perhaps it has bit-rotted...?

I'm running a Feb 2011 version of the bleeding edge FreeBSD,
which will become FreeBSD 9.0 is a few months.  Perhaps, 
something has changed in FreeBSD's networking code.  I'll
see if I can understand opal/util/if.c sufficiently to see
what's happening.

> The code to take these IP interfaces
> and figure out if a given peer is reachable is in
> ompi/mca/btl/tcp/btl_tcp_proc.c:mca_btl_tcp_proc_insert().
> This requires a little explanation...

(snip to keep this short)

> This was a long explanation -- I hope it helps...
> Is there any chance you could dig into this to see what's going on?

Thanks, I'll see what I can ferret out of the syste

> We unfortunately don't have access to any BSD machines to test this
> on, ourselves.  It works on other OS's, so I'm curious as to why it
> doesn't seem to work for you.  :-(

I can arrange access on the cluster in question. ;-)

-- 
Steve

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Jeff Squyres]

Sorry -- I got distracted all afternoon...

In addition to what Ralph said (i.e., I'm not sure if the CIDR notation stuff 
made it over to the v1.5 branch or not, but it is available from the nightly 
SVN trunk tarballs: http://www.open-mpi.org/nightly/trunk/), here's a few 
points from other mails in this thread...

1. Gus is correct that OMPI is complaining that bge1 doesn't exist on all 
nodes.  The MCA parameters that you pass on the command line get shipped to 
*all* MPI processes, and therefore generally need to work on all of them.  If 
you have per-host MCA parameter values, you can set them a few different ways:

- have a per-host MCA param file, usually in $prefix/etc/openmpi-mca-params.conf
- have your shell startup files intelligently determine which host you're on 
and set the corresponding MCA environment variable as appropriate (e.g., on the 
head node, set the env variable OMPI_MCA_btl_tcp_if_include to bge1, and set it 
to bge0 on the others)

Those are a little klunky, but having a heterogeneous setup like this is not 
common, so we haven't really optimized the ability to set different MCA params 
on different servers.

2. I am curious to figure out why the automatic reachability computations isn't 
working for you.  Unfortunately, the code to compute the reachability is pretty 
gnarly.  :-\  The code to find the IP interfaces on your machines is in 
opal/util/if.c.  That *should* be working -- there's *BSD-specific code in 
there that has been verified by others in the past... but who knows?  Perhaps 
it has bit-rotted...?  The code to take these IP interfaces and figure out if a 
given peer is reachable is in 
ompi/mca/btl/tcp/btl_tcp_proc.c:mca_btl_tcp_proc_insert().  This requires a 
little explanation...

- There is one TCP BTL "component".  Think of this as the plugin that is 
dlopen'd into the process itself.  It contains some high-level information 
about the plugin itself (e.g., the version number, ...etc.).

- There is one TCP BTL "module" per IP interface that is used for MPI 
communications.  So your head node will have 2 TCP BTL modules and the others 
will only have one TCP BTL module.  A module is a struct with a bunch of 
function pointers and some meta data (e.g., which IP interface it "owns", etc.).

- During the BTL module's initialization, btl_tcp.c:mca_btl_tcp_add_procs() is 
called to notify the module of all of its peers (an ompi_proc_t instance is 
used to describe a peer process -- note: a *process*, not any particular 
communications method or IP address of that process).  btl_tcp_add_procs() 
takes the array of ompi_proc_t instances (that correspond to all the MPI 
processes in MPI_COMM_WORLD) and tries to figure out if this particular TCP BTL 
module can "reach" that peer, per the algorithm described in the FAQ that I 
cited earlier.

- mca_btl_tcp_add_procs() calls mca_btl_tcp_proc_insert() to do the 
reachability computation.  If _insert() succeeds, then _add_procs() assumes 
that this module can reach that process and proceeds accordingly.  If _insert() 
fails, then _add_procs() assumes that this module cannot reach that peer and 
proceeds accordingly.

- mca_btl_tcp_proc_insert() has previously learned about all the IP addresses 
of all the peer MPI processes via a different mechanism called the modex (which 
I won't go into here).  It basically checks the one peer process in question, 
looks up that peer's IP addresses (aka "endpoints", from that peer's TCP BTL 
modules), and tries to find the best quality match that it can.  It basically 
makes a 2D graph of weights of how "good" the connection is to each of the peer 
process' endpoints.  It then finds the best connection and uses that one.

- We unfortunately do not have good debugging output in _proc_insert(), so you 
might need to step through this with a debugger.  :-(  I have a long-languished 
branch that adds lots of debugging output in this reachability computation 
area, but I have never finished it (it has some kind of bug in it that prevents 
it from working, which is why I haven't merged it into the mainline).  

This was a long explanation -- I hope it helps...  Is there any chance you 
could dig into this to see what's going on?  The short version is that all this 
code *should* automatically figure out that the 10.x interface should 
effectively end up getting ignored because it can't be used to commuicate with 
any of its TCP BTL module peers in the other processes on the other servers.

We unfortunately don't have access to any BSD machines to test this on, 
ourselves.  It works on other OS's, so I'm curious as to why it doesn't seem to 
work for you.  :-(


On Jul 8, 2011, at 5:37 PM, Ralph Castain wrote:

> We've been moving to provide support for including values as CIDR notation 
> instead of names - e.g., 192.168.0/16 instead of bge0 or bge1 - but I don't 
> think that has been put into the 1.4 release series. If you need it now, you 
> might try using the developer's trunk - I know 

Re: [OMPI users] Error using hostfile

[Mohan, Ashwin]

Thanks Ralph.

 

I have emailed the network admin on the firewall issue. 

 

About the PATH and LIBRARY PATH issue, is it sufficient evidence that
the path are set alright if I am able to compile and run successfully on
individual nodes mentioned in the machine file. 

 

Thanks,
Ashwin.

 

From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Ralph Castain
Sent: Friday, July 08, 2011 1:58 PM
To: Open MPI Users
Subject: Re: [OMPI users] Error using hostfile

 

Is there a firewall in the way? The error indicates that daemons were
launched on the remote machines, but failed to communicate back.

 

Also, check that your remote PATH and LD_LIBRARY_PATH are being set to
the right place to pickup this version of OMPI. Lots of systems deploy
with default versions that may not be compatible, so if you wind up
running a daemon on the remote node that comes from another
installation, things won't work.

 

 

On Jul 8, 2011, at 10:52 AM, Mohan, Ashwin wrote:





Hi,

I am following up on a previous error posted. Based on the previous
recommendation, I did set up a password less SSH login.

 

I created a hostfile comprising of 4 nodes (w/ each node having 4
slots). I tried to run my job on 4 slots but get no output. Hence, I end
up killing the job. I am trying to run a simple MPI program on 4 nodes
and trying to figure out what could be the issue.  What could I check to
ensure that I can run jobs on 4 nodes (each node has 4 slots)

 

Here is the simple MPI program I am trying to execute on 4 nodes

**

if (my_rank != 0)

{

sprintf(message, "Greetings from the process %d!", my_rank);

dest = 0;

MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag,
MPI_COMM_WORLD);

}

else

{

for (source = 1;source < p; source++)

{

MPI_Recv(message, 100, MPI_CHAR, source, tag, MPI_COMM_WORLD,
);

printf("%s\n", message);

}

 



My hostfile looks like this:

 

[amohan@myocyte48 ~]$ cat hostfile

myocyte46

myocyte47

myocyte48

myocyte49

***

 

I use the following run command: : mpirun --hostfile hostfile -np 4
new46

And receive a blank screen. Hence, I have to kill the job.

 

OUTPUT ON KILLING JOB:

mpirun: killing job...


--

mpirun noticed that the job aborted, but has no info as to the process

that caused that situation.


--


--

mpirun was unable to cleanly terminate the daemons on the nodes shown

below. Additional manual cleanup may be required - please refer to

the "orte-clean" tool for assistance.


--

myocyte46 - daemon did not report back when launched

myocyte47 - daemon did not report back when launched

myocyte49 - daemon did not report back when launched

 

Thanks,

Ashwin.

 

 

From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Ralph Castain
Sent: Wednesday, July 06, 2011 6:46 PM
To: Open MPI Users
Subject: Re: [OMPI users] Error using hostfile

 

Please see http://www.open-mpi.org/faq/?category=rsh#ssh-keys

 

 

On Jul 6, 2011, at 5:09 PM, Mohan, Ashwin wrote:






Hi,

 

I use the following command (mpirun --prefix /usr/local/openmpi1.4.3 -np
4 hello) to successfully execute a simple hello world command on a
single node.  Each node has 4 slots.  Following the successful execution
on one node, I wish to employ 4 nodes and for this purpose wrote a
hostfile. I submitted my job using the following command:

 

mpirun --prefix /usr/local/openmpi1.4.3 -np 4 --hostfile hostfile hello

 

Copied below is the output. How do I go about fixing this issue.

 

**

 

amohan@myocyte48's password: amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte48's password:

Permission denied, please try again.

amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte48's password:

 

Permission denied (publickey,gssapi-with-mic,password).


--

A daemon (pid 22085) died unexpectedly with status 255 while attempting

to launch so we are aborting.

 

There may be more information reported by the environment (see above).

 

This may be because the daemon was unable to find all the needed shared

libraries on the remote node. You may set your LD_LIBRARY_PATH to have
the

location of the shared libraries on the remote nodes and this will

automatically be forwarded to the remote nodes.


--

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Ralph Castain]

We've been moving to provide support for including values as CIDR notation 
instead of names - e.g., 192.168.0/16 instead of bge0 or bge1 - but I don't 
think that has been put into the 1.4 release series. If you need it now, you 
might try using the developer's trunk - I know it works there.


On Jul 8, 2011, at 2:49 PM, Steve Kargl wrote:

> On Fri, Jul 08, 2011 at 04:26:35PM -0400, Gus Correa wrote:
>> Steve Kargl wrote:
>>> On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:
 The easiest way to fix this is likely to use the btl_tcp_if_include
 or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly
 which interfaces to use:
 
   http://www.open-mpi.org/faq/?category=tcp#tcp-selection
 
>>> 
>>> Perhaps, I'm again misreading the output, but it appears that
>>> 1.4.4rc2 does not even see the 2nd nic.
>>> 
>>> hpc:kargl[317] ifconfig bge0 
>>> bge0: flags=8843 metric 0 mtu 1500
>>>   options=8009b
>>>   ether 00:e0:81:40:48:92
>>>   inet 10.208.78.111 netmask 0xff00 broadcast 10.208.78.255
>>> hpc:kargl[318] ifconfig bge1
>>> bge1: flags=8843 metric 0 mtu 1500
>>>   options=8009b
>>>   ether 00:e0:81:40:48:93
>>>   inet 192.168.0.10 netmask 0xff00 broadcast 192.168.0.255
>>> 
>>> kargl[319] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 30 \
>>> --mca btl_tcp_if_include bge1 -machinefile mf1 ./z
>>> 
>>> hpc:kargl[320] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 
>>> 10 --mca btl_tcp_if_include bge1 -machinefile mf1 ./z
>>> [hpc.apl.washington.edu:12295] mca: base: components_open: Looking for btl 
>>> [node11.cimu.org:21878] select: init of component self returned success
>>> [node11.cimu.org:21878] select: initializing btl component sm
>>> [node11.cimu.org:21878] select: init of component sm returned success
>>> [node11.cimu.org:21878] select: initializing btl component tcp
>>> [node11.cimu.org][[13916,1],1][btl_tcp_component.c:468:mca_btl_tcp_component_create_instances]
>>>  invalid interface "bge1"
>>> [node11.cimu.org:21878] select: init of component tcp returned success
>>> --
>>> At least one pair of MPI processes are unable to reach each other for
>>> MPI communications.  This means that no Open MPI device has indicated
>>> that it can be used to communicate between these processes.  This is
>>> an error; Open MPI requires that all MPI processes be able to reach
>>> each other.  This error can sometimes be the result of forgetting to
>>> specify the "self" BTL.
>>> 
>> Hi Steve
>> 
>> It is complaining that bge1 is not valid on node11, not on node10/hpc,
>> where you ran ifconfig.
>> 
>> Would the names of the interfaces and the matching subnet/IP
>> vary from node to node?
>> (E.g. bge0 be associated to 192.168.0.11 on node11, instead of bge1.)
>> 
>> Would it be possible that only on node10 bge1 is on the 192.168.0.0 
>> subnet, but on the other nodes it is bge0 that connects
>> to the 192.168.0.0 subnet perhaps?
> 
> node10 has bge0 = 10.208.x.y and bge1 = 192.168.0.10.
> node11 through node21 use bge0 = 192.168.0.N where N = 11, ..., 21.
> 
>> If you're including only bge1 on your mca btl switch,
>> supposedly all nodes are able to reach
>> each other via an interface called bge1.
>> Is this really the case?
>> You may want to run ifconfig on all nodes to check.
>> 
>> Alternatively, you could exclude node10 from your host file
>> and try to run the job on the remaining nodes
>> (and maybe not restrict the interface names with any btl switch).
> 
> Completely exclude node10 would appear to work.  Of course,
> this then loses the 4 cpus and 16 GB of memory that are
> in node.
> 
> The question to me is why does 1.4.2 work without a
> problem, and 1.4.3 and 1.4.4 have problems with a
> node with 2 NICs.
> 
> I suppose a follow-on question is: Is there some
> way to get 1.4.4 to exclusive use bge1 on node10
> while using bge0 on the other nodes?
> 
> -- 
> Steve
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Fri, Jul 08, 2011 at 04:26:35PM -0400, Gus Correa wrote:
> Steve Kargl wrote:
> >On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:
> >>The easiest way to fix this is likely to use the btl_tcp_if_include
> >>or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly
> >>which interfaces to use:
> >>
> >>http://www.open-mpi.org/faq/?category=tcp#tcp-selection
> >>
> >
> >Perhaps, I'm again misreading the output, but it appears that
> >1.4.4rc2 does not even see the 2nd nic.
> >
> >hpc:kargl[317] ifconfig bge0 
> >bge0: flags=8843 metric 0 mtu 1500
> >
> > options=8009b
> >ether 00:e0:81:40:48:92
> >inet 10.208.78.111 netmask 0xff00 broadcast 10.208.78.255
> >hpc:kargl[318] ifconfig bge1
> >bge1: flags=8843 metric 0 mtu 1500
> >
> > options=8009b
> >ether 00:e0:81:40:48:93
> >inet 192.168.0.10 netmask 0xff00 broadcast 192.168.0.255
> >
> >kargl[319] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 30 \
> >  --mca btl_tcp_if_include bge1 -machinefile mf1 ./z
> >
> >hpc:kargl[320] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 
> >10 --mca btl_tcp_if_include bge1 -machinefile mf1 ./z
> >[hpc.apl.washington.edu:12295] mca: base: components_open: Looking for btl 
> >[node11.cimu.org:21878] select: init of component self returned success
> >[node11.cimu.org:21878] select: initializing btl component sm
> >[node11.cimu.org:21878] select: init of component sm returned success
> >[node11.cimu.org:21878] select: initializing btl component tcp
> >[node11.cimu.org][[13916,1],1][btl_tcp_component.c:468:mca_btl_tcp_component_create_instances]
> > invalid interface "bge1"
> >[node11.cimu.org:21878] select: init of component tcp returned success
> >--
> >At least one pair of MPI processes are unable to reach each other for
> >MPI communications.  This means that no Open MPI device has indicated
> >that it can be used to communicate between these processes.  This is
> >an error; Open MPI requires that all MPI processes be able to reach
> >each other.  This error can sometimes be the result of forgetting to
> >specify the "self" BTL.
> >
> Hi Steve
> 
> It is complaining that bge1 is not valid on node11, not on node10/hpc,
> where you ran ifconfig.
> 
> Would the names of the interfaces and the matching subnet/IP
> vary from node to node?
> (E.g. bge0 be associated to 192.168.0.11 on node11, instead of bge1.)
> 
> Would it be possible that only on node10 bge1 is on the 192.168.0.0 
> subnet, but on the other nodes it is bge0 that connects
> to the 192.168.0.0 subnet perhaps?

node10 has bge0 = 10.208.x.y and bge1 = 192.168.0.10.
node11 through node21 use bge0 = 192.168.0.N where N = 11, ..., 21.

> If you're including only bge1 on your mca btl switch,
> supposedly all nodes are able to reach
> each other via an interface called bge1.
> Is this really the case?
> You may want to run ifconfig on all nodes to check.
> 
> Alternatively, you could exclude node10 from your host file
> and try to run the job on the remaining nodes
> (and maybe not restrict the interface names with any btl switch).

Completely exclude node10 would appear to work.  Of course,
this then loses the 4 cpus and 16 GB of memory that are
in node.

The question to me is why does 1.4.2 work without a
problem, and 1.4.3 and 1.4.4 have problems with a
node with 2 NICs.

I suppose a follow-on question is: Is there some
way to get 1.4.4 to exclusive use bge1 on node10
while using bge0 on the other nodes?

-- 
Steve

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Gus Correa]

Steve Kargl wrote:

On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:

The easiest way to fix this is likely to use the btl_tcp_if_include
or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly
which interfaces to use:

http://www.open-mpi.org/faq/?category=tcp#tcp-selection



Perhaps, I'm again misreading the output, but it appears that
1.4.4rc2 does not even see the 2nd nic.

hpc:kargl[317] ifconfig bge0 
bge0: flags=8843 metric 0 mtu 1500

options=8009b
ether 00:e0:81:40:48:92
inet 10.208.78.111 netmask 0xff00 broadcast 10.208.78.255
inet6 fe80::2e0:81ff:fe40:4892%bge0 prefixlen 64 scopeid 0x3 
nd6 options=29

media: Ethernet autoselect (1000baseT )
status: active
hpc:kargl[318] ifconfig bge1
bge1: flags=8843 metric 0 mtu 1500
options=8009b
ether 00:e0:81:40:48:93
inet 192.168.0.10 netmask 0xff00 broadcast 192.168.0.255
inet6 fe80::2e0:81ff:fe40:4893%bge1 prefixlen 64 scopeid 0x4 
nd6 options=29

media: Ethernet autoselect (1000baseT )
status: active

kargl[319] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 30 \
  --mca btl_tcp_if_include bge1 -machinefile mf1 ./z

hpc:kargl[320] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 10 
--mca btl_tcp_if_include bge1 -machinefile mf1 ./z
[hpc.apl.washington.edu:12295] mca: base: components_open: Looking for btl 
components
[hpc.apl.washington.edu:12295] mca: base: components_open: opening btl 
components
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component self
[hpc.apl.washington.edu:12295] mca: base: components_open: component self has 
no register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component self open 
function successful
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component sm
[hpc.apl.washington.edu:12295] mca: base: components_open: component sm has no 
register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component sm open 
function successful
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component tcp
[hpc.apl.washington.edu:12295] mca: base: components_open: component tcp has no 
register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component tcp open 
function successful
[hpc.apl.washington.edu:12295] select: initializing btl component self
[hpc.apl.washington.edu:12295] select: init of component self returned success
[hpc.apl.washington.edu:12295] select: initializing btl component sm
[hpc.apl.washington.edu:12295] select: init of component sm returned success
[hpc.apl.washington.edu:12295] select: initializing btl component tcp
[hpc.apl.washington.edu:12295] select: init of component tcp returned success
[node11.cimu.org:21878] mca: base: components_open: Looking for btl components
[node11.cimu.org:21878] mca: base: components_open: opening btl components
[node11.cimu.org:21878] mca: base: components_open: found loaded component self
[node11.cimu.org:21878] mca: base: components_open: component self has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component self open 
function successful
[node11.cimu.org:21878] mca: base: components_open: found loaded component sm
[node11.cimu.org:21878] mca: base: components_open: component sm has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component sm open function 
successful
[node11.cimu.org:21878] mca: base: components_open: found loaded component tcp
[node11.cimu.org:21878] mca: base: components_open: component tcp has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component tcp open function 
successful
[node11.cimu.org:21878] select: initializing btl component self
[node11.cimu.org:21878] select: init of component self returned success
[node11.cimu.org:21878] select: initializing btl component sm
[node11.cimu.org:21878] select: init of component sm returned success
[node11.cimu.org:21878] select: initializing btl component tcp
[node11.cimu.org][[13916,1],1][btl_tcp_component.c:468:mca_btl_tcp_component_create_instances]
 invalid interface "bge1"
[node11.cimu.org:21878] select: init of component tcp returned success
--
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.


Hi Steve

It is 

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Fri, Jul 08, 2011 at 12:09:09PM -0700, Steve Kargl wrote:
> On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:
> > 
> > The easiest way to fix this is likely to use the btl_tcp_if_include
> > or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly
> > which interfaces to use:
> > 
> > http://www.open-mpi.org/faq/?category=tcp#tcp-selection
> > 
> 
> Perhaps, I'm again misreading the output, but it appears that
> 1.4.4rc2 does not even see the 2nd nic.
> 

So, now, I'm very confused!  Using '--mca btl_tcp_if_include bge1,bge0'
seems to work even though openmpi says that bge1 is invalid, and if I
reverse the interfaces to '--mca btl_tcp_if_include bge0,bge1' the 
process appears stuck. :(


hpc:kargl[341] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 10 \
  --mca btl_tcp_if_include bge1,bge0 --mca btl tcp,self -machinefile mf1 ./z
...
[node11.cimu.org][[13885,1],1][btl_tcp_component.c:468:\
mca_btl_tcp_component_create_instances] invalid interface "bge1"
[node11.cimu.org:22024] select: init of component tcp returned success
0: hpc.apl.washington.edu
1: node11.cimu.org
Latency: 0.73644
Sync Time: 0.000147468
Now starting main loop
  0: 0 bytes 16384 times -->0.00 Mbps in 0.73622 sec
  1: 1 bytes 16384 times -->0.10 Mbps in 0.73617 sec
  2: 2 bytes 3395  times -->0.21 Mbps in 0.73634 sec
  3: 3 bytes 1697  times -->0.31 Mbps in 0.73611 sec
...
126:  12582914 bytes3  times -->  720.84 Mbps in 0.133178830 sec
[hpc.apl.washington.edu:12390] mca: base: close: component self closed
[hpc.apl.washington.edu:12390] mca: base: close: unloading component self
[hpc.apl.washington.edu:12390] mca: base: close: component tcp closed
[hpc.apl.washington.edu:12390] mca: base: close: unloading component tcp
[node11.cimu.org:22024] mca: base: close: component self closed
[node11.cimu.org:22024] mca: base: close: unloading component self
[node11.cimu.org:22024] mca: base: close: component tcp closed
[node11.cimu.org:22024] mca: base: close: unloading component tcp


hpc:kargl[342] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 10 \
--mca btl_tcp_if_include bge0,bge1 --mca btl tcp,self -machinefile mf1 ./z
...
[node11.cimu.org][[13868,1],1][btl_tcp_component.c:468:\
mca_btl_tcp_component_create_instances] invalid interface "bge1"
[node11.cimu.org:22048] select: init of component tcp returned success
0: hpc.apl.washington.edu
1: node11.cimu.org

and nothing!

-- 
Steve

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:
> 
> The easiest way to fix this is likely to use the btl_tcp_if_include
> or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly
> which interfaces to use:
> 
> http://www.open-mpi.org/faq/?category=tcp#tcp-selection
> 

Perhaps, I'm again misreading the output, but it appears that
1.4.4rc2 does not even see the 2nd nic.

hpc:kargl[317] ifconfig bge0 
bge0: flags=8843 metric 0 mtu 1500
options=8009b
ether 00:e0:81:40:48:92
inet 10.208.78.111 netmask 0xff00 broadcast 10.208.78.255
inet6 fe80::2e0:81ff:fe40:4892%bge0 prefixlen 64 scopeid 0x3 
nd6 options=29
media: Ethernet autoselect (1000baseT )
status: active
hpc:kargl[318] ifconfig bge1
bge1: flags=8843 metric 0 mtu 1500
options=8009b
ether 00:e0:81:40:48:93
inet 192.168.0.10 netmask 0xff00 broadcast 192.168.0.255
inet6 fe80::2e0:81ff:fe40:4893%bge1 prefixlen 64 scopeid 0x4 
nd6 options=29
media: Ethernet autoselect (1000baseT )
status: active

kargl[319] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 30 \
  --mca btl_tcp_if_include bge1 -machinefile mf1 ./z

hpc:kargl[320] /usr/local/openmpi-1.4.4/bin/mpiexec --mca btl_base_verbose 10 
--mca btl_tcp_if_include bge1 -machinefile mf1 ./z
[hpc.apl.washington.edu:12295] mca: base: components_open: Looking for btl 
components
[hpc.apl.washington.edu:12295] mca: base: components_open: opening btl 
components
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component self
[hpc.apl.washington.edu:12295] mca: base: components_open: component self has 
no register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component self open 
function successful
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component sm
[hpc.apl.washington.edu:12295] mca: base: components_open: component sm has no 
register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component sm open 
function successful
[hpc.apl.washington.edu:12295] mca: base: components_open: found loaded 
component tcp
[hpc.apl.washington.edu:12295] mca: base: components_open: component tcp has no 
register function
[hpc.apl.washington.edu:12295] mca: base: components_open: component tcp open 
function successful
[hpc.apl.washington.edu:12295] select: initializing btl component self
[hpc.apl.washington.edu:12295] select: init of component self returned success
[hpc.apl.washington.edu:12295] select: initializing btl component sm
[hpc.apl.washington.edu:12295] select: init of component sm returned success
[hpc.apl.washington.edu:12295] select: initializing btl component tcp
[hpc.apl.washington.edu:12295] select: init of component tcp returned success
[node11.cimu.org:21878] mca: base: components_open: Looking for btl components
[node11.cimu.org:21878] mca: base: components_open: opening btl components
[node11.cimu.org:21878] mca: base: components_open: found loaded component self
[node11.cimu.org:21878] mca: base: components_open: component self has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component self open 
function successful
[node11.cimu.org:21878] mca: base: components_open: found loaded component sm
[node11.cimu.org:21878] mca: base: components_open: component sm has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component sm open function 
successful
[node11.cimu.org:21878] mca: base: components_open: found loaded component tcp
[node11.cimu.org:21878] mca: base: components_open: component tcp has no 
register function
[node11.cimu.org:21878] mca: base: components_open: component tcp open function 
successful
[node11.cimu.org:21878] select: initializing btl component self
[node11.cimu.org:21878] select: init of component self returned success
[node11.cimu.org:21878] select: initializing btl component sm
[node11.cimu.org:21878] select: init of component sm returned success
[node11.cimu.org:21878] select: initializing btl component tcp
[node11.cimu.org][[13916,1],1][btl_tcp_component.c:468:mca_btl_tcp_component_create_instances]
 invalid interface "bge1"
[node11.cimu.org:21878] select: init of component tcp returned success
--
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

-- 
Steve

Re: [OMPI users] Error using hostfile

[Ralph Castain]

Is there a firewall in the way? The error indicates that daemons were launched 
on the remote machines, but failed to communicate back.

Also, check that your remote PATH and LD_LIBRARY_PATH are being set to the 
right place to pickup this version of OMPI. Lots of systems deploy with default 
versions that may not be compatible, so if you wind up running a daemon on the 
remote node that comes from another installation, things won't work.


On Jul 8, 2011, at 10:52 AM, Mohan, Ashwin wrote:

> Hi,
> I am following up on a previous error posted. Based on the previous 
> recommendation, I did set up a password less SSH login.
>  
> I created a hostfile comprising of 4 nodes (w/ each node having 4 slots). I 
> tried to run my job on 4 slots but get no output. Hence, I end up killing the 
> job. I am trying to run a simple MPI program on 4 nodes and trying to figure 
> out what could be the issue.  What could I check to ensure that I can run 
> jobs on 4 nodes (each node has 4 slots)
>  
> Here is the simple MPI program I am trying to execute on 4 nodes
> **
> if (my_rank != 0)
> {
> sprintf(message, "Greetings from the process %d!", my_rank);
> dest = 0;
> MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag, 
> MPI_COMM_WORLD);
> }
> else
> {
> for (source = 1;source < p; source++)
> {
> MPI_Recv(message, 100, MPI_CHAR, source, tag, MPI_COMM_WORLD, 
> );
> printf("%s\n", message);
> }
>  
> 
> My hostfile looks like this:
>  
> [amohan@myocyte48 ~]$ cat hostfile
> myocyte46
> myocyte47
> myocyte48
> myocyte49
> ***
>  
> I use the following run command: : mpirun --hostfile hostfile -np 4 new46
> And receive a blank screen. Hence, I have to kill the job.
>  
> OUTPUT ON KILLING JOB:
> mpirun: killing job...
> --
> mpirun noticed that the job aborted, but has no info as to the process
> that caused that situation.
> --
> --
> mpirun was unable to cleanly terminate the daemons on the nodes shown
> below. Additional manual cleanup may be required - please refer to
> the "orte-clean" tool for assistance.
> --
> myocyte46 - daemon did not report back when launched
> myocyte47 - daemon did not report back when launched
> myocyte49 - daemon did not report back when launched
>  
> Thanks,
> Ashwin.
>  
>  
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On 
> Behalf Of Ralph Castain
> Sent: Wednesday, July 06, 2011 6:46 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Error using hostfile
>  
> Please see http://www.open-mpi.org/faq/?category=rsh#ssh-keys
>  
>  
> On Jul 6, 2011, at 5:09 PM, Mohan, Ashwin wrote:
> 
> 
> Hi,
>  
> I use the following command (mpirun --prefix /usr/local/openmpi1.4.3 -np 4 
> hello) to successfully execute a simple hello world command on a single node. 
>  Each node has 4 slots.  Following the successful execution on one node, I 
> wish to employ 4 nodes and for this purpose wrote a hostfile. I submitted my 
> job using the following command:
>  
> mpirun --prefix /usr/local/openmpi1.4.3 -np 4 --hostfile hostfile hello
>  
> Copied below is the output. How do I go about fixing this issue.
>  
> **
>  
> amohan@myocyte48's password: amohan@myocyte47's password:
> Permission denied, please try again.
> amohan@myocyte48's password:
> Permission denied, please try again.
> amohan@myocyte47's password:
> Permission denied, please try again.
> amohan@myocyte47's password:
> Permission denied, please try again.
> amohan@myocyte48's password:
>  
> Permission denied (publickey,gssapi-with-mic,password).
> --
> A daemon (pid 22085) died unexpectedly with status 255 while attempting
> to launch so we are aborting.
>  
> There may be more information reported by the environment (see above).
>  
> This may be because the daemon was unable to find all the needed shared
> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
> location of the shared libraries on the remote nodes and this will
> automatically be forwarded to the remote nodes.
> --
> --
> mpirun noticed that the job aborted, but has no info as to the process
> that caused that situation.
> --
> --
> mpirun was unable to cleanly terminate the 

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Fri, Jul 08, 2011 at 02:19:27PM -0400, Jeff Squyres wrote:
> On Jul 8, 2011, at 1:31 PM, Steve Kargl wrote:
> 
> > It seems that openmpi-1.4.4 compiled code is trying to use the
> > wrong nic.  My /etc/hosts file has
> > 
> > 10.208.78.111   hpc.apl.washington.edu hpc
> > 192.168.0.10node10.cimu.org node10 n10 master
> > 192.168.0.11node11.cimu.org node11 n11
> > 192.168.0.12node12.cimu.org node12 n12
> > ... down to ...
> > 192.168.0.21node21.cimu.org node21 n21
> > 
> > Note, node10 and hpc are the same system (2 different NICs). 
> 
> Don't confuse the machinefile with the NICs that OMPI will try
> to use.  The machinefile is only hosts on which OMPI will launch.
> Specifically: the machinefile does not influence which NICs OMPI
> will use for MPI communications.

Ah, okay.  I did not realize that a machinefile did not
limit OMPI to a set of IP address.

> > hpc:kargl[268] cat mf_ompi_1
> > node10.cimu.org slots=1
> > node16.cimu.org slots=1
> > hpc:kargl[267] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf_ompi_1 
> > ./z
> > 0: hpc.apl.washington.edu
> > 1: node16.cimu.org
> 
> What function is netmpi.c using to get the hostname that
> is printed?  It might be using MPI_Get_processor_name()
> or gethostname() -- both of which may return whatever hostname(1) returns.  

After reading the code, this appears to have misled me.  The
code uses MPI_Get_processor_name().

> > (gdb) bt
> > #0  0x0003c0bedb9c in kevent () from /lib/libc.so.7
> > #1  0x0052d648 in kq_dispatch ()
> > #2  0x0052c6c3 in opal_event_base_loop ()
> > #3  0x005260cb in opal_progress ()
> > #4  0x00491d1c in mca_pml_ob1_send ()
> > #5  0x0043c753 in PMPI_Send ()
> > #6  0x0041a112 in Sync (p=0x7fffd4d0) at netmpi.c:573
> > #7  0x0041a3cf in DetermineLatencyReps (p=0x3) at netmpi.c:593
> > #8  0x0041a4fe in TestLatency (p=0x3) at netmpi.c:630
> > #9  0x0041a958 in main (argc=1, argv=0x7fffd6a0) at netmpi.c:213
> > (gdb) quit
> 
> The easiest way to fix this is likely to use the btl_tcp_if_include
> or btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly which
> interfaces to use:
> 
> http://www.open-mpi.org/faq/?category=tcp#tcp-selection

Thanks for the pointer.  I'll try this solution later.

> Hypothetically, however, OMPI should be able to determine that
> 192.168.0.x is not reachable from the 10.x network (assuming
> your netmasks are set right), and automatically not use the
> 10.x network to reach any of the non-node10 machines.

The assumption is correct.  192.x is independent of 10.x.

> It's curious that this is not happening; I wonder if this
> is some kind of quirk of OMPI's reachability algorithms
> (http://www.open-mpi.org/faq/?category=tcp#tcp-routability)
>  on FreeBSD...?

I just rebuilt 1.4.4rc2 with '-O -g' to get debugging symbols
into openmpi's libraries and executables.  Is there any
particulare function(s) that I should inspect?

-- 
Steve

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Jeff Squyres]

On Jul 8, 2011, at 1:31 PM, Steve Kargl wrote:

> It seems that openmpi-1.4.4 compiled code is trying to use the
> wrong nic.  My /etc/hosts file has
> 
> 10.208.78.111   hpc.apl.washington.edu hpc
> 192.168.0.10node10.cimu.org node10 n10 master
> 192.168.0.11node11.cimu.org node11 n11
> 192.168.0.12node12.cimu.org node12 n12
> ... down to ...
> 192.168.0.21node21.cimu.org node21 n21
> 
> Note, node10 and hpc are the same system (2 different NICs). 

Don't confuse the machinefile with the NICs that OMPI will try to use.  The 
machinefile is only hosts on which OMPI will launch.  Specifically: the 
machinefile does not influence which NICs OMPI will use for MPI communications.

> hpc:kargl[252] /usr/local/openmpi-1.4.4/bin/mpif90 -o z -g -O ring_f90.f90 
> hpc:kargl[253] cat > mf1
> node10 slots=1
> node11 slots=1
> node12 slots=1
> hpc:kargl[254] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf1 ./z
> Process 0 sending   10  to1  tag  201  (  
>  3  processes in ring)
> 
> in another xterm if I attach to the process on node10, I see
> with gdb.
> 
> (gdb) bt
> #0  0x0003c10f9b9c in kevent () from /lib/libc.so.7
> #1  0x0052ca18 in kq_dispatch ()
> #2  0x0052ba93 in opal_event_base_loop ()
> #3  0x0052549b in opal_progress ()
> #4  0x0048fcfc in mca_pml_ob1_send ()
> #5  0x00428873 in PMPI_Send ()
> #6  0x0041a890 in pmpi_send__ ()
> #7  0x0041a3f0 in ring () at ring_f90.f90:34
> #8  0x0041a640 in main (argc=, 
>argv=) at ring_f90.f90:10
> #9  0x0041a1cc in _start ()
> (gdb) quit
> 
> Now, eliminating node10 from the machine file, I see:
> 
> hpc:kargl[255] cat > mf2
> node11 slots=1
> node12 slots=1
> node13 slots=1
> hpc:kargl[256] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf2 ./z
> Process 0 sending   10  to1  tag  201  (  
>  3  processes in ring)
> Process 0 sent to1
> Process 0 decremented value:   9
> Process 0 decremented value:   8
> Process 0 decremented value:   7
> Process 0 decremented value:   6
> Process 0 decremented value:   5
> Process 0 decremented value:   4
> Process 0 decremented value:   3
> Process 0 decremented value:   2
> Process 0 decremented value:   1
> Process 0 decremented value:   0
> Process0  exiting
> Process1  exiting
> Process2  exiting
> 
> I also have a simple mpi test program netmpi.c from Argonne.
> It shows
> 
> hpc:kargl[263] /usr/local/openmpi-1.4.4/bin/mpicc -o z -g -O GetOpt.c netmpi.c
> hpc:kargl[264] cat mf_ompi_3 
> node11.cimu.org slots=1
> node16.cimu.org slots=1
> hpc:kargl[265] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf_ompi_3 ./z
> 1: node16.cimu.org
> 0: node11.cimu.org
> Latency: 0.73617
> Sync Time: 0.000147234
> Now starting main loop
>  0: 0 bytes 16384 times -->0.00 Mbps in 0.73612 sec
>  1: 1 bytes 16384 times -->0.10 Mbps in 0.73612 sec
>  2: 2 bytes 3396 times -->0.21 Mbps in 0.73611 sec
>  3: 3 bytes 1698 times -->0.31 Mbps in 0.73609 sec
>  4: 5 bytes 2264 times -->0.52 Mbps in 0.73610 sec
>  5: 7 bytes 1358 times -->0.73 Mbps in 0.73608 sec
> 
> 
> hpc:kargl[268] cat mf_ompi_1
> node10.cimu.org slots=1
> node16.cimu.org slots=1
> hpc:kargl[267] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf_ompi_1 ./z
> 0: hpc.apl.washington.edu
> 1: node16.cimu.org

What function is netmpi.c using to get the hostname that is printed?  It might 
be using MPI_Get_processor_name() or gethostname() -- both of which may return 
whatever hostname(1) returns.  

Again -- this is not an indicator of which NIC Open MPI is using.

> (gdb) bt
> #0  0x0003c0bedb9c in kevent () from /lib/libc.so.7
> #1  0x0052d648 in kq_dispatch ()
> #2  0x0052c6c3 in opal_event_base_loop ()
> #3  0x005260cb in opal_progress ()
> #4  0x00491d1c in mca_pml_ob1_send ()
> #5  0x0043c753 in PMPI_Send ()
> #6  0x0041a112 in Sync (p=0x7fffd4d0) at netmpi.c:573
> #7  0x0041a3cf in DetermineLatencyReps (p=0x3) at netmpi.c:593
> #8  0x0041a4fe in TestLatency (p=0x3) at netmpi.c:630
> #9  0x0041a958 in main (argc=1, argv=0x7fffd6a0) at netmpi.c:213
> (gdb) quit

The easiest way to fix this is likely to use the btl_tcp_if_include or 
btl_tcp_if_exclude MCA parameters -- i.e., tell OMPI exactly which interfaces 
to use:

http://www.open-mpi.org/faq/?category=tcp#tcp-selection

Hypothetically, however, OMPI should be able to determine that 192.168.0.x is 
not reachable from the 10.x network (assuming your netmasks are set right), and 
automatically not use the 10.x network to reach any of the non-node10 machines. 
 It's curious that this is not happening; I 

Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD

[Steve Kargl]

On Thu, Jul 07, 2011 at 08:38:56PM -0400, Jeff Squyres wrote:
> On Jul 5, 2011, at 4:24 PM, Steve Kargl wrote:
> > On Tue, Jul 05, 2011 at 01:14:06PM -0700, Steve Kargl wrote:
> >> I have an application that appears to function as I expect
> >> when compiled with openmpi-1.4.2 on FreeBSD 9.0.  But, it
> >> appears to hang during communication between nodes.  What
> >> follows is the long version.
> > 
> > Argh I messed up.  It should read "But, it appears to hang
> > during communication between nodes when using 1.4.3 or 1.4.4."
> > 
> Are you able to run simple MPI applications with 1.4.3 or 1.4.4
> on your OS?  E.g., the "ring_c" program in the example/ directory?
> This might be a good test to see if OMPI's TCP is working at all.
> 
> Assuming that works... Have you tried attaching debuggers to see
> where your process is hanging?  There might be a logic issue in
> your app that isn't-quite-legal-MPI but works with some amount
>  of buffering, but fails if the amount of buffering is reduced.

It seems that openmpi-1.4.4 compiled code is trying to use the
wrong nic.  My /etc/hosts file has

10.208.78.111   hpc.apl.washington.edu hpc
192.168.0.10node10.cimu.org node10 n10 master
192.168.0.11node11.cimu.org node11 n11
192.168.0.12node12.cimu.org node12 n12
... down to ...
192.168.0.21node21.cimu.org node21 n21

Note, node10 and hpc are the same system (2 different NICs). 

hpc:kargl[252] /usr/local/openmpi-1.4.4/bin/mpif90 -o z -g -O ring_f90.f90 
hpc:kargl[253] cat > mf1
node10 slots=1
node11 slots=1
node12 slots=1
hpc:kargl[254] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf1 ./z
 Process 0 sending   10  to1  tag  201  (   
3  processes in ring)

in another xterm if I attach to the process on node10, I see
with gdb.

(gdb) bt
#0  0x0003c10f9b9c in kevent () from /lib/libc.so.7
#1  0x0052ca18 in kq_dispatch ()
#2  0x0052ba93 in opal_event_base_loop ()
#3  0x0052549b in opal_progress ()
#4  0x0048fcfc in mca_pml_ob1_send ()
#5  0x00428873 in PMPI_Send ()
#6  0x0041a890 in pmpi_send__ ()
#7  0x0041a3f0 in ring () at ring_f90.f90:34
#8  0x0041a640 in main (argc=, 
argv=) at ring_f90.f90:10
#9  0x0041a1cc in _start ()
(gdb) quit

Now, eliminating node10 from the machine file, I see:

hpc:kargl[255] cat > mf2
node11 slots=1
node12 slots=1
node13 slots=1
hpc:kargl[256] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf2 ./z
 Process 0 sending   10  to1  tag  201  (   
3  processes in ring)
 Process 0 sent to1
 Process 0 decremented value:   9
 Process 0 decremented value:   8
 Process 0 decremented value:   7
 Process 0 decremented value:   6
 Process 0 decremented value:   5
 Process 0 decremented value:   4
 Process 0 decremented value:   3
 Process 0 decremented value:   2
 Process 0 decremented value:   1
 Process 0 decremented value:   0
 Process0  exiting
 Process1  exiting
 Process2  exiting

I also have a simple mpi test program netmpi.c from Argonne.
It shows

hpc:kargl[263] /usr/local/openmpi-1.4.4/bin/mpicc -o z -g -O GetOpt.c netmpi.c
hpc:kargl[264] cat mf_ompi_3 
node11.cimu.org slots=1
node16.cimu.org slots=1
hpc:kargl[265] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf_ompi_3 ./z
1: node16.cimu.org
0: node11.cimu.org
Latency: 0.73617
Sync Time: 0.000147234
Now starting main loop
  0: 0 bytes 16384 times -->0.00 Mbps in 0.73612 sec
  1: 1 bytes 16384 times -->0.10 Mbps in 0.73612 sec
  2: 2 bytes 3396 times -->0.21 Mbps in 0.73611 sec
  3: 3 bytes 1698 times -->0.31 Mbps in 0.73609 sec
  4: 5 bytes 2264 times -->0.52 Mbps in 0.73610 sec
  5: 7 bytes 1358 times -->0.73 Mbps in 0.73608 sec


hpc:kargl[268] cat mf_ompi_1
node10.cimu.org slots=1
node16.cimu.org slots=1
hpc:kargl[267] /usr/local/openmpi-1.4.4/bin/mpiexec -machinefile mf_ompi_1 ./z
0: hpc.apl.washington.edu
1: node16.cimu.org

(gdb) bt
#0  0x0003c0bedb9c in kevent () from /lib/libc.so.7
#1  0x0052d648 in kq_dispatch ()
#2  0x0052c6c3 in opal_event_base_loop ()
#3  0x005260cb in opal_progress ()
#4  0x00491d1c in mca_pml_ob1_send ()
#5  0x0043c753 in PMPI_Send ()
#6  0x0041a112 in Sync (p=0x7fffd4d0) at netmpi.c:573
#7  0x0041a3cf in DetermineLatencyReps (p=0x3) at netmpi.c:593
#8  0x0041a4fe in TestLatency (p=0x3) at netmpi.c:630
#9  0x0041a958 in main (argc=1, argv=0x7fffd6a0) at netmpi.c:213
(gdb) quit

Why is hpc.apl.washington.edu appearing instead of node10.cimu.org?

-- 
Steve

Re: [OMPI users] Pinning of openmpi to certain defined cores possible

[Ralph Castain]

Look at "mpirun -h"  or "man mpirun" - you'll see options for binding processes 
to cores etc.


On Jul 8, 2011, at 10:13 AM, Vlad Popa wrote:

> Hello!
> 
> We habe a shared memory system based on  4CPUs of 12-core Opteron with a 
> total of 256Gb RAM .
> 
> Are there any switches, which we could set , so that  when running   
> mpirun-jobs  we'd be able to pin them down to a defined certain memory area 
> and defined processor cores so that they stay there?
> 
> best regards
> 
> Vlad Popa
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] InfiniBand, different OpenFabrics transport types

[Bill Johnstone]

Hello, and thanks for the reply.



- Original Message -
> From: Jeff Squyres 
> Sent: Thursday, July 7, 2011 5:14 PM
> Subject: Re: [OMPI users] InfiniBand, different OpenFabrics transport types
> 
> On Jun 28, 2011, at 1:46 PM, Bill Johnstone wrote:
> 
>>  I have a heterogeneous network of InfiniBand-equipped hosts which are all 
> connected to the same backbone switch, an older SDR 10 Gb/s unit.
>> 
>>  One set of nodes uses the Mellanox "ib_mthca" driver, while the 
> other uses the "mlx4" driver.
>> 
>>  This is on Linux 2.6.32, with Open MPI 1.5.3 .
>> 
>>  When I run Open MPI across these node types, I get an error message of the 
> form:
>> 
>>  Open MPI detected two different OpenFabrics transport types in the same 
> Infiniband network. 
>>  Such mixed network trasport configuration is not supported by Open MPI.
>> 
>>  Local host: compute-chassis-1-node-01
>>  Local adapter: mthca0 (vendor 0x5ad, part ID 25208) 
>>  Local transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN 
> 
> Wow, that's cool ("UNKNOWN").  Are you using an old version of 
> OFED or something?

No, clean local build of OFED 1.5.3 packages, but I don't have the full huge 
complement of OFED packages installed, since our setup is not using IPoIB, SDP, 
etc.

ibdiagnet, and all the usual suspects work as expected, and I'm able to do 
large scale Open MPI runs just fine, so long as I don't cross Mellanox HCA 
types.


> Mellanox -- how can this happen?
> 
>>  Remote host: compute-chassis-3-node-01
>>  Remote Adapter: (vendor 0x2c9, part ID 26428) 
>>  Remote transport type: MCA_BTL_OPENIB_TRANSPORT_IB
>> 
>>  Two questions:
>> 
>>  1. Why is this occurring if both adapters have all the OpenIB software set 
> up?  Is it because Open MPI is trying to use functionality such as ConnectX 
> with 
> the newer hardware, which is incompatible with older hardware, or is it 
> something more mundane?
> 
> It's basically a mismatch of IB capabilities -- Open MPI is trying to use 
> more advanced features in some nodes and not in others.

I also tried looking in the adapter-specific settings in the .ini file under 
/etc, but the only difference I found was in MTU, and I think that's configured 
on the switch.
 
>>  2. How can I use IB amongst these heterogeneous nodes?
> 
> Mellanox will need to answer this question...  It might be able to be done, 
> but 
> I don't know how offhand.  The first issue is to figure out why you're 
> getting TRANSPORT_UNKNOWN on the one node.

OK, please let me know what other things to try or what other info I can 
provide.

Re: [OMPI users] Error using hostfile

[Mohan, Ashwin]

Hi,

I am following up on a previous error posted. Based on the previous
recommendation, I did set up a password less SSH login. 

 

I created a hostfile comprising of 4 nodes (w/ each node having 4
slots). I tried to run my job on 4 slots but get no output. Hence, I end
up killing the job. I am trying to run a simple MPI program on 4 nodes
and trying to figure out what could be the issue.  What could I check to
ensure that I can run jobs on 4 nodes (each node has 4 slots)

 

Here is the simple MPI program I am trying to execute on 4 nodes

**

if (my_rank != 0)

{

sprintf(message, "Greetings from the process %d!", my_rank);

dest = 0;

MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag,
MPI_COMM_WORLD);

}

else

{

for (source = 1;source < p; source++)

{

MPI_Recv(message, 100, MPI_CHAR, source, tag, MPI_COMM_WORLD,
);

printf("%s\n", message);

}

 



My hostfile looks like this:

 

[amohan@myocyte48 ~]$ cat hostfile

myocyte46

myocyte47

myocyte48

myocyte49

***

 

I use the following run command: : mpirun --hostfile hostfile -np 4
new46

And receive a blank screen. Hence, I have to kill the job. 

 

OUTPUT ON KILLING JOB:

mpirun: killing job...


--

mpirun noticed that the job aborted, but has no info as to the process

that caused that situation.


--


--

mpirun was unable to cleanly terminate the daemons on the nodes shown

below. Additional manual cleanup may be required - please refer to

the "orte-clean" tool for assistance.


--

myocyte46 - daemon did not report back when launched

myocyte47 - daemon did not report back when launched

myocyte49 - daemon did not report back when launched

 

Thanks,

Ashwin.

 

 

From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Ralph Castain
Sent: Wednesday, July 06, 2011 6:46 PM
To: Open MPI Users
Subject: Re: [OMPI users] Error using hostfile

 

Please see http://www.open-mpi.org/faq/?category=rsh#ssh-keys

 

 

On Jul 6, 2011, at 5:09 PM, Mohan, Ashwin wrote:





Hi,

 

I use the following command (mpirun --prefix /usr/local/openmpi1.4.3 -np
4 hello) to successfully execute a simple hello world command on a
single node.  Each node has 4 slots.  Following the successful execution
on one node, I wish to employ 4 nodes and for this purpose wrote a
hostfile. I submitted my job using the following command:

 

mpirun --prefix /usr/local/openmpi1.4.3 -np 4 --hostfile hostfile hello

 

Copied below is the output. How do I go about fixing this issue.

 

**

 

amohan@myocyte48's password: amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte48's password:

Permission denied, please try again.

amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte47's password:

Permission denied, please try again.

amohan@myocyte48's password:

 

Permission denied (publickey,gssapi-with-mic,password).


--

A daemon (pid 22085) died unexpectedly with status 255 while attempting

to launch so we are aborting.

 

There may be more information reported by the environment (see above).

 

This may be because the daemon was unable to find all the needed shared

libraries on the remote node. You may set your LD_LIBRARY_PATH to have
the

location of the shared libraries on the remote nodes and this will

automatically be forwarded to the remote nodes.


--


--

mpirun noticed that the job aborted, but has no info as to the process

that caused that situation.


--


--

mpirun was unable to cleanly terminate the daemons on the nodes shown

below. Additional manual cleanup may be required - please refer to

the "orte-clean" tool for assistance.


--

myocyte47 - daemon did not report back when launched

myocyte48 - daemon did not report back when launched

 

**

 

Thanks,

Ashwin.

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

 

[OMPI users] pinning processes

[Vlad Popa]

Hi !
I've forgotten to mention that we're using openmpi 1.5.3 on Ubuntu 
Server 11.04 64 bit..


Greetings from Salzburg/Austria

Vlad Popa

[OMPI users] Pinning of openmpi to certain defined cores possible

[Vlad Popa]

Hello!

We habe a shared memory system based on  4CPUs of 12-core Opteron with a 
total of 256Gb RAM .


Are there any switches, which we could set , so that  when running   
mpirun-jobs  we'd be able to pin them down to a defined certain memory 
area and defined processor cores so that they stay there?


best regards

Vlad Popa

[OMPI users] Error-Open MPI over Infiniband: polling LP CQ with status LOCAL LENGTH ERROR

[yanyg]

Hi all,

The message says :

[[17549,1],0][btl_openib_component.c:3224:handle_wc] from 
gulftown to: gulftown error polling LP CQ with status LOCAL 
LENGTH ERROR status number 1 for wr_id 492359816 opcode 
32767  vendor error 105 qp_idx 3

This is very arcane to me, the same test ran when only one MPI 
process on each node, but when we switch to two MPI processes 
on each node, then this error message comes up. Anything I could 
do? Anything related to infiniband configuration, as guessed form 
the string "vendor error 105 qp_idx 3"?

Thanks,
Yiguang

Re: [OMPI users] [WARNING: SPOOFED E-MAIL--Non-Aerospace Sender] Re: Problem with prebuilt ver 1.5.3 for windows

[Shiqing Fan]

Hi Jeff,

Sorry for answering late. These emails were hidden in another thread in 
my email client, so I didn't catch it until now.


The prebuild version of Open MPI was based on Windows 2008, which has 
InterlockedCompareExchange64 natively.


Windows XP doesn't have support of this function, so the prebuild Open 
MPI is not possible to be used under Windows XP. The only solution now 
is to build Open MPI on your XP from source.



Regards,
Shiqing

On 2011-07-08 1:28 PM, Jeffrey A Cummings wrote:
I've been following this list for several months now and have been 
quite impressed by the helpfulness of the list experts in response to 
most questions.  So how come the pregnant silence in response to my 
question?  I could really use some help here.


- Jeff



From: Jeffrey A Cummings 
To: Open MPI Users 
Date: 06/29/2011 04:18 PM
Subject: [WARNING: SPOOFED E-MAIL--Non-Aerospace Sender] Re: [OMPI 
users] Problem with prebuilt ver 1.5.3 for windows

Sent by: users-boun...@open-mpi.org




Anyone (Shiqing perhaps) have any more thoughts on this problem?

- Jeff




From: Damien 
To: Open MPI Users 
Date: 06/24/2011 03:59 PM
Subject: Re: [OMPI users] Problem with prebuilt ver 1.5.3 for windows
Sent by: users-boun...@open-mpi.org




Yeah, and I'm wrong too, InterlockedCompareExchange64 is available on 
32-bit.  I think this is one for Shiqing.


You could build OpenMPI yourself if you have VS2008.  It's pretty easy 
to do.


Damien

On 24/06/2011 1:51 PM, Jeffrey A Cummings wrote:
Damien -

I'm using the 32 bit version of OpenMPI.  I think the 64 refers to the 
size of integer that the function works on, not the operating system 
version.  I didn't have this problem with VS 2008, so I think they've 
changed something in VS 2010.  I just don't know how to fix it.


- Jeff




From: Damien __ 
To: Open MPI Users __ 
Date: 06/24/2011 02:35 PM
Subject: Re: [OMPI users] Problem with prebuilt ver 1.5.3 for windows
Sent by: _users-bounces@open-mpi.org_ 




Jeff,

InterlockedCompareExchange64 is a 64-bit-only instruction.  Are you 
running XP 32-bit (I think you are b/c I don't think there was a XP64 
SP3...).  You need the 32-bit OpenMPI version.  If you are running a 
64-bit OS, but building a 32-bit executable, that instruction isn't 
available in 32-bit and you still need to link with 32-bit OpenMPI.


Damien

On 24/06/2011 12:16 PM, Jeffrey A Cummings wrote:
I'm having a problem using the prebuilt Windows version 1.5.3 with my 
app built with MS VisualStudio 2010.  I get an error message (for each 
node) that says: "The procedure entry point 
InterlockedCompareExchange64 could not be located in the dynamic link 
library KERNEL32.dll".  I'm running Windows XP, sp 3.


- Jeff Cummings


___
users mailing list_
__users@open-mpi.org_ _
__http://www.open-mpi.org/mailman/listinfo.cgi/users
users mailing list_
__users@open-mpi.org_ _
__http://www.open-mpi.org/mailman/listinfo.cgi/users_


___
users mailing list_
__users@open-mpi.org_ _
__http://www.open-mpi.org/mailman/listinfo.cgi/users
users mailing list
users@open-mpi.org_
__http://www.open-mpi.org/mailman/listinfo.cgi/users
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
---
Shiqing Fan
High Performance Computing Center Stuttgart (HLRS)
Tel: ++49(0)711-685-87234  Nobelstrasse 19
Fax: ++49(0)711-685-65832  70569 Stuttgart
http://www.hlrs.de/organization/people/shiqing-fan/
email: f...@hlrs.de

Re: [OMPI users] [WARNING: SPOOFED E-MAIL--Non-Aerospace Sender] Re: Problem with prebuilt ver 1.5.3 for windows

[Jeffrey A Cummings]

I've been following this list for several months now and have been quite 
impressed by the helpfulness of the list experts in response to most 
questions.  So how come the pregnant silence in response to my question? I 
could really use some help here.

- Jeff



From:   Jeffrey A Cummings 
To: Open MPI Users 
List-Post: users@lists.open-mpi.org
Date:   06/29/2011 04:18 PM
Subject:[WARNING: SPOOFED E-MAIL--Non-Aerospace Sender] Re: [OMPI 
users] Problem with prebuilt ver 1.5.3 for windows
Sent by:users-boun...@open-mpi.org



Anyone (Shiqing perhaps) have any more thoughts on this problem? 

- Jeff 




From:Damien  
To:Open MPI Users  
List-Post: users@lists.open-mpi.org
Date:06/24/2011 03:59 PM 
Subject:Re: [OMPI users] Problem with prebuilt ver 1.5.3 for 
windows 
Sent by:users-boun...@open-mpi.org 



Yeah, and I'm wrong too, InterlockedCompareExchange64 is available on 
32-bit.  I think this is one for Shiqing.

You could build OpenMPI yourself if you have VS2008.  It's pretty easy to 
do.

Damien

On 24/06/2011 1:51 PM, Jeffrey A Cummings wrote: 
Damien - 

I'm using the 32 bit version of OpenMPI.  I think the 64 refers to the 
size of integer that the function works on, not the operating system 
version.  I didn't have this problem with VS 2008, so I think they've 
changed something in VS 2010.  I just don't know how to fix it. 

- Jeff 




From:Damien  
To:Open MPI Users  
List-Post: users@lists.open-mpi.org
Date:06/24/2011 02:35 PM 
Subject:Re: [OMPI users] Problem with prebuilt ver 1.5.3 for 
windows 
Sent by:users-boun...@open-mpi.org 



Jeff,

InterlockedCompareExchange64 is a 64-bit-only instruction.  Are you 
running XP 32-bit (I think you are b/c I don't think there was a XP64 
SP3...).  You need the 32-bit OpenMPI version.  If you are running a 
64-bit OS, but building a 32-bit executable, that instruction isn't 
available in 32-bit and you still need to link with 32-bit OpenMPI.

Damien

On 24/06/2011 12:16 PM, Jeffrey A Cummings wrote: 
I'm having a problem using the prebuilt Windows version 1.5.3 with my app 
built with MS VisualStudio 2010.  I get an error message (for each node) 
that says: "The procedure entry point InterlockedCompareExchange64 could 
not be located in the dynamic link library KERNEL32.dll".  I'm running 
Windows XP, sp 3. 

- Jeff Cummings 


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users 


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users 
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] a question about network connection of open-mpi

[zhuangchao]

hello all :


   I   run  the following command :   

/data1/cluster/openmpi/bin/mpirun  -d  -machinefile  /tmp/nodes.10515.txt   -np 
 3  /data1/cluster/mpiblast-pio-1.6/bin/mpiblast   -p blastn -i 
/data1/cluster/sequences/seq_4.txt -d Baculo_Nucleotide -o 
/data1/cluster/blast.out/blast.out.10515  -g T -m  0 -F F 

  Then  I  get  the following  error  from  openmpi:

[node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.0.5 failed: No route to host (113)
[node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 159.226.126.15 failed: No route to host (113)

 The  machinefile  is defined  as  following :

 node1
 node5
 node7

 192.168.0.5  is  not  the  ip  of  hosts in the  machinefile .
159.226.126.15  is  the   another ip of  node1 .  But  hostname node1

corresponds to   11.11.11.1  in  the /etc/hosts .

why   do  I  get  the error ?  Can  you  help me ? 

   Thank you !