Re: [OMPI devel] mtt IBM SPAWN error

[Lenny Verkhovsky]

I saw it. But I think it something else, since it works if I run it with
hostlist

#mpirun -np 3 -H witch2,witch3  dynamic/spawn
#


On Mon, Jun 30, 2008 at 4:03 PM, Ralph H Castain  wrote:

> Well, that error indicates that it was unable to launch the daemon on
> witch3
> for some reason. If you look at the error reported by bash, you will see
> that the ³orted² binary wasn¹t found!
>
> Sounds like a path error ­ you might check to see if witch3 has the
> binaries
> installed, and if they are where you told the system to look...
>
> Ralph
>
>
>
> On 6/30/08 5:21 AM, "Lenny Verkhovsky"  wrote:
>
> > I am not familiar with spawn test of IBM, but maybe this is right
> behavior,
> > if spawn test allocates 3 ranks on the node, and then allocates another 3
> > then this test suppose to fail due to max_slots=4.
> >
> > But it fails with the fallowing hostfile as well BUT WITH A DIFFERENT
> ERROR.
> >
> > #cat hostfile2
> > witch2 slots=4 max_slots=4
> > witch3 slots=4 max_slots=4
> > witch1:/home/BENCHMARKS/IBM #
> /home/USERS/lenny/OMPI_ORTE_18772/bin/mpirun -np
> > 3 -hostfile hostfile2 dynamic/spawn
> > bash: orted: command not found
> > [witch1:22789]
> >
> --
> > A daemon (pid 22791) died unexpectedly with status 127 while attempting
> > to launch so we are aborting.
> > There may be more information reported by the environment (see above).
> > This may be because the daemon was unable to find all the needed shared
> > libraries on the remote node. You may set your LD_LIBRARY_PATH to have
> the
> > location of the shared libraries on the remote nodes and this will
> > automatically be forwarded to the remote nodes.
> >
> --
> > [witch1:22789]
> >
> --
> > mpirun was unable to cleanly terminate the daemons on the nodes shown
> > below. Additional manual cleanup may be required - please refer to
> > the "orte-clean" tool for assistance.
> >
> --
> > witch3 - daemon did not report back when launched
> >
> > On Mon, Jun 30, 2008 at 9:38 AM, Lenny Verkhovsky <
> lenny.verkhov...@gmail.com>
> > wrote:
> >> Hi,
> >> trying to run mtt I failed to run IBM spawn test. It fails only when
> using
> >> hostfile, and not when using host list.
> >> ( OMPI from TRUNK )
> >>
> >> This is working :
> >> #mpirun -np 3 -H witch2 dynamic/spawn
> >>
> >> This Fails:
> >> # cat hostfile
> >> witch2 slots=4 max_slots=4
> >> #mpirun -np 3 -hostfile hostfile dynamic/spawn
> >> [witch1:12392]
> >>
> --
> >> There are not enough slots available in the system to satisfy the 3
> slots
> >> that were requested by the application:
> >>   dynamic/spawn
> >>
> >> Either request fewer slots for your application, or make more slots
> available
> >> for use.
> >>
> --
> >> [witch1:12392]
> >>
> --
> >> A daemon (pid unknown) died unexpectedly on signal 1  while attempting
> to
> >> launch so we are aborting.
> >>
> >> There may be more information reported by the environment (see above).
> >>
> >> This may be because the daemon was unable to find all the needed shared
> >> libraries on the remote node. You may set your LD_LIBRARY_PATH to have
> the
> >> location of the shared libraries on the remote nodes and this will
> >> automatically be forwarded to the remote nodes.
> >>
> --
> >> mpirun: clean termination accomplished
> >>
> >>
> >> Using hostfile1 also works
> >> #cat hostfile1
> >> witch2
> >> witch2
> >> witch2
> >>
> >>
> >> Best Regards
> >> Lenny.
> >>
> >
>
>
>
>

Re: [OMPI devel] mtt IBM SPAWN error

[Ralph H Castain]

Well, that error indicates that it was unable to launch the daemon on witch3
for some reason. If you look at the error reported by bash, you will see
that the ³orted² binary wasn¹t found!

Sounds like a path error ­ you might check to see if witch3 has the binaries
installed, and if they are where you told the system to look...

Ralph



On 6/30/08 5:21 AM, "Lenny Verkhovsky"  wrote:

> I am not familiar with spawn test of IBM, but maybe this is right behavior,
> if spawn test allocates 3 ranks on the node, and then allocates another 3
> then this test suppose to fail due to max_slots=4.
>  
> But it fails with the fallowing hostfile as well BUT WITH A DIFFERENT ERROR.
>  
> #cat hostfile2 
> witch2 slots=4 max_slots=4
> witch3 slots=4 max_slots=4
> witch1:/home/BENCHMARKS/IBM # /home/USERS/lenny/OMPI_ORTE_18772/bin/mpirun -np
> 3 -hostfile hostfile2 dynamic/spawn
> bash: orted: command not found
> [witch1:22789] 
> --
> A daemon (pid 22791) died unexpectedly with status 127 while attempting
> to launch so we are aborting.
> There may be more information reported by the environment (see above).
> This may be because the daemon was unable to find all the needed shared
> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
> location of the shared libraries on the remote nodes and this will
> automatically be forwarded to the remote nodes.
> --
> [witch1:22789] 
> --
> mpirun was unable to cleanly terminate the daemons on the nodes shown
> below. Additional manual cleanup may be required - please refer to
> the "orte-clean" tool for assistance.
> --
> witch3 - daemon did not report back when launched
>  
> On Mon, Jun 30, 2008 at 9:38 AM, Lenny Verkhovsky 
> wrote:
>> Hi, 
>> trying to run mtt I failed to run IBM spawn test. It fails only when using
>> hostfile, and not when using host list.
>> ( OMPI from TRUNK )
>>  
>> This is working :
>> #mpirun -np 3 -H witch2 dynamic/spawn
>>  
>> This Fails:
>> # cat hostfile
>> witch2 slots=4 max_slots=4
>> #mpirun -np 3 -hostfile hostfile dynamic/spawn
>> [witch1:12392] 
>> --
>> There are not enough slots available in the system to satisfy the 3 slots
>> that were requested by the application:
>>   dynamic/spawn
>> 
>> Either request fewer slots for your application, or make more slots available
>> for use.
>> --
>> [witch1:12392] 
>> --
>> A daemon (pid unknown) died unexpectedly on signal 1  while attempting to
>> launch so we are aborting.
>> 
>> There may be more information reported by the environment (see above).
>> 
>> This may be because the daemon was unable to find all the needed shared
>> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
>> location of the shared libraries on the remote nodes and this will
>> automatically be forwarded to the remote nodes.
>> --
>> mpirun: clean termination accomplished
>>  
>>  
>> Using hostfile1 also works
>> #cat hostfile1
>> witch2
>> witch2
>> witch2
>>  
>>  
>> Best Regards
>> Lenny.
>> 
> 

Re: [OMPI devel] mtt IBM SPAWN error

[Ralph H Castain]

That¹s correct ­ and is precisely the behavior it should exhibit. The
reasons:

1. when you specify ­host, we assume max_slots is infinite since you cannot
provide any info to the contrary. We therefore allow you to oversubscribe
the node to your heart¹s desire. However, note one problem: if your original
launch is only one proc, we will set it to be aggressive in terms of
yielding the processor. Your subsequent comm_spawn¹d procs will therefore
suffer degraded performance if they oversubscribe the node.

Can¹t be helped - there is no way to pass enough info with -host for us to
do better.


2. when you run with -hostfile, your hostfile is telling us to allow no more
than 4 procs on the node. You used three in your original launch, leaving
only one slot available. Since each of the procs in the IBM test attempts to
spawn another, your job will fail.

We can always do more to improve the error messaging...
Ralph


On 6/30/08 12:38 AM, "Lenny Verkhovsky"  wrote:

> Hi, 
> trying to run mtt I failed to run IBM spawn test. It fails only when using
> hostfile, and not when using host list.
> ( OMPI from TRUNK )
>  
> This is working :
> #mpirun -np 3 -H witch2 dynamic/spawn
>  
> This Fails:
> # cat hostfile
> witch2 slots=4 max_slots=4
> #mpirun -np 3 -hostfile hostfile dynamic/spawn
> [witch1:12392] 
> --
> There are not enough slots available in the system to satisfy the 3 slots
> that were requested by the application:
>   dynamic/spawn
> 
> Either request fewer slots for your application, or make more slots available
> for use.
> --
> [witch1:12392] 
> --
> A daemon (pid unknown) died unexpectedly on signal 1  while attempting to
> launch so we are aborting.
> 
> There may be more information reported by the environment (see above).
> 
> This may be because the daemon was unable to find all the needed shared
> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
> location of the shared libraries on the remote nodes and this will
> automatically be forwarded to the remote nodes.
> --
> mpirun: clean termination accomplished
>  
>  
> Using hostfile1 also works
> #cat hostfile1
> witch2
> witch2
> witch2
>  
>  
> Best Regards
> Lenny.
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] mtt IBM SPAWN error

[Lenny Verkhovsky]

I am not familiar with spawn test of IBM, but maybe this is right behavior,
if spawn test allocates 3 ranks on the node, and then allocates another 3
then this test suppose to fail due to max_slots=4.

But it fails with the fallowing hostfile as well BUT WITH A DIFFERENT ERROR.

#cat hostfile2
witch2 slots=4 max_slots=4
witch3 slots=4 max_slots=4
witch1:/home/BENCHMARKS/IBM # /home/USERS/lenny/OMPI_ORTE_18772/bin/mpirun
-np 3 -hostfile hostfile2 dynamic/spawn
bash: orted: command not found
[witch1:22789]
--
A daemon (pid 22791) died unexpectedly with status 127 while attempting
to launch so we are aborting.
There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
[witch1:22789]
--
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
--
witch3 - daemon did not report back when launched

On Mon, Jun 30, 2008 at 9:38 AM, Lenny Verkhovsky <
lenny.verkhov...@gmail.com> wrote:

> Hi,
> trying to run mtt I failed to run IBM spawn test. It fails only when using
> hostfile, and not when using host list.
> ( OMPI from TRUNK )
>
> This is working :
> #mpirun -np 3 -H witch2 dynamic/spawn
>
> This Fails:
> # cat hostfile
> witch2 slots=4 max_slots=4
>
> #mpirun -np 3 -hostfile hostfile dynamic/spawn
> [witch1:12392]
> --
> There are not enough slots available in the system to satisfy the 3 slots
> that were requested by the application:
>   dynamic/spawn
>
> Either request fewer slots for your application, or make more slots
> available
> for use.
> --
> [witch1:12392]
> --
> A daemon (pid unknown) died unexpectedly on signal 1  while attempting to
> launch so we are aborting.
>
> There may be more information reported by the environment (see above).
> This may be because the daemon was unable to find all the needed shared
> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
> location of the shared libraries on the remote nodes and this will
> automatically be forwarded to the remote nodes.
> --
> mpirun: clean termination accomplished
>
>
> Using hostfile1 also works
> #cat hostfile1
> witch2
> witch2
> witch2
>
>
> Best Regards
> Lenny.
>

[OMPI devel] mtt IBM SPAWN error

[Lenny Verkhovsky]

Hi,
trying to run mtt I failed to run IBM spawn test. It fails only when using
hostfile, and not when using host list.
( OMPI from TRUNK )

This is working :
#mpirun -np 3 -H witch2 dynamic/spawn

This Fails:
# cat hostfile
witch2 slots=4 max_slots=4

#mpirun -np 3 -hostfile hostfile dynamic/spawn
[witch1:12392]
--
There are not enough slots available in the system to satisfy the 3 slots
that were requested by the application:
  dynamic/spawn

Either request fewer slots for your application, or make more slots
available
for use.
--
[witch1:12392]
--
A daemon (pid unknown) died unexpectedly on signal 1  while attempting to
launch so we are aborting.

There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
mpirun: clean termination accomplished


Using hostfile1 also works
#cat hostfile1
witch2
witch2
witch2


Best Regards
Lenny.

Re: [OMPI devel] multiple GigE interfaces...

[Muhammad Atif]

Thanks for the detailed reply. 

Just one question, I am a bit confused with this comment, or perhaps i phrased 
the question in a bad way and you miss understood me. I will rephrase my 
question and scenario. I hope I can make is terrible scenario easier to 
understand with less confusion.

>> related question is what if I want to run 8 process job (on 2x4
>> cluster) and want to pin a process to an network interface. OpenMPI to
>> my understanding does not give any control of allocating IP to a
>> process (like MPICH)

>You could just say btl_if_include=ethX, thus giving you the right
>network interface. Obviously, this requires separate networks.


I want to pin each process to a distinct ethernet interface. My understanding 
of  btl_if_include = ethX,ethY is that it would let every process to use  ethX 
and ethY in roundrobin fashion. What I want is that if my application is a.out 
(a modified OSU bandwidth benchmark) and can have 4 processes. rank 0 and rank 
2 will run on machineA whereas rank 1 and rank 3 on machineB.  Rank 0 will  
communicate with rank 1, and rank 2 will communicate with rank 3. I want rank 0 
to use eth1, rank 1 to use eth2 (for machine B it will be rank 2 to use eth1 
and rank 3 uses eth2).  In normal case, OMPI send packets in roundrobin through 
both the intefaces. Therefore rank 0 uses eth1 and eth2, and my requirement is 
otherwise. 
To my mind this can be done through routing tables. When I add routes, the OMPI 
just sits there waiting for something to happen, whereas MPICH gives me results 
with same routing table configuration.  Initially I had eth1, and eth2 on same 
subnet, but even with different subnets I am facing the same issue. MPICH works 
with both subnet configurations.

 

Another related issue and different configuration is the case of Xen. Here eth0 
is connected to default shared memory sort of a thing of xen  (in another case 
it will be my very own Xensocket btl for OMPI) and is very fast as compared to 
normal GigE. Eth0 will only be fast, if the machines (and hence the processes)  
are on same "physical hardware". In case the two machines (and therefore the 
processes) are on "different physical hardware", then eth0 will route the 
packets through a physical switch and will be a touch slower than eth1 (i know 
its kinda hard to follow, but I am sorry) .  Eth1 here is normal GigE interface 
and both eth0 and eth1 are on different subnets. While running NAS benchmarks 
etc I want that if the two machines can use eth0, they should only use eth0 and 
even if btl sends a messages through eth1, it should get routed through eth0. 
For this I setup routing tables and see the same issue as above i.e. OMPI sits 
there (especially for LU
 and IS benchmarks), whereas MPICH gives me the results the way I want. 

Is there something related to routing tables that I am overlooking in case of 
OMPI? Why would MPICH give me the results as I am expecting whereas OMPI sits 
there waiting for me to kill the job. 


Best Regards,
Muhammad Atif

PS: I think I did a terrible job in explaining the scenario again :)


- Original Message 
From: Adrian Knoth 
To: Open MPI Developers 
Sent: Monday, June 23, 2008 10:15:53 PM
Subject: Re: [OMPI devel] multiple GigE interfaces...

On Wed, Jun 18, 2008 at 05:13:28PM -0700, Muhammad Atif wrote:

>  Hi again... I was on a break from Xensocket stuff This time some
>  general questions...

Hi.

> question. What if I have multiple Ethernet cards (say 5) on two of my
> quad core machines.  The IP addresses (and the subnets of course) are 
> Machine A   Machine B
> eth0 is y.y.1.a y.y.1.z
> eth1 is y.y.4.by.y.4.y
> eth2 is y.y.4.c   ...
> eth3 is y.y.4.d   ...
> 
>  ...

This sounds pretty weird. And I guess your netmasks don't allow to
separate the NICs, do they?

> from the FAQ's/Some emails in user lists  it is clear that if I want
> to run a job on multiple ethernets, I can use --mca btl_tcp_if_include
> eth0,eth1. This

You can, but you don't have to. If you don't specify something, OMPI
will choose "something right".

> will run the job on two of the subnets utilizing both the Ethernet
> cards. Is it doing some sort of load balancing? or some round robin
> mechanism? What part of code is responsible for this work?

As far as I know, it's handled by OB1 (PML), which does striping across
several BTL instances.

So in other words, as long as both segments are equally fast, the load
balancing should do fine. If they differ in performance, the OB1 doesn't
find an optimal solution. If you're hitting this case, ask htor, he has
an auto-tuning replacement, but that's not going to be part of OMPI.

> eth1,eth2,eth3,eth4. Notice that all of these ethNs are on same subnet.
> Even in the FAQ's (which mostly answers our lame questions)  its not
> entirely