Re: [OMPI users] Open MPI error when using MPI_Comm_spawn

[Ralph H Castain]

Hi Prakash

I can't really test this solution as the Torque dynamic host allocator
appears to be something you are adding to that system (so it isn't part of
the released code). However, the attached code should cleanly add any nodes
to any existing allocation known to OpenRTE.

I hope to resume working on the "next-gen" OpenRTE allocation system in the
fairly near future. This will include support for dynamically obtaining and
updating host allocations via (hopefully!) convenient APIs. Since I expect
that won't be included in the Open MPI releases until at least 1.3 (release
date not known yet, but sometime in the not-near future), I will post a
patch on the OpenRTE web site in the interim - I'll announce that here as
well so people can go pick it up if they need that capability.

Please let me know how this works and/or any changes you have to make to it
to get it to work. Feel free to holler for help.

Ralph



On 4/2/07 11:07 AM, "Prakash Velayutham" 
wrote:

> Thanks Ralph. I will wait for your Torque dynamic host addition solution.
> 
> Prakash
> 
 r...@lanl.gov 04/02/07 1:00 PM >>>
> Hi Prakash
> 
> This is telling you that you have an error in the comm_spawn command itself.
> I am no expert there, so I'll have to let someone else identify it for you.
> 
> There are no limits to launching on nodes in a hostfile - they are all
> automatically considered "allocated" when the file is read. If you had the
> node name in the file, then there is no "dynamic" addition of nodes going
> on.
> 
> Meantime, I am going to send you a different solution to dynamically adding
> nodes under Torque (or any other resource manager).
> 
> Ralph
> 
> 
> On 4/2/07 10:53 AM, "Prakash Velayutham" 
> wrote:
> 
>> Hello,
>> 
>> Thanks for the patch. I still do not know the internals of Open MPI, so can't
>> test this right away. But here is another test I ran and that failed too.
>> 
>> I have now removed Torque from the equation. I am NOT requesting nodes
>> through
>> Torque. I SSH to a compute node and start up the code as below.
>> 
>> prakash@wins04:~/thesis/CS/Samples>mpirun -np 4 --bynode --hostfile
>> machinefile ./parallel.laplace
>> 
>> [wins01:17699] *** An error occurred in MPI_Comm_spawn
>> [wins01:17699] *** on communicator MPI_COMM_WORLD
>> [wins01:17699] *** MPI_ERR_ARG: invalid argument of some other kind
>> [wins01:17699] *** MPI_ERRORS_ARE_FATAL (goodbye)
>> mpirun noticed that job rank 1 with PID 25074 on node wins02 exited on signal
>> 15 (Terminated).
>> 2 additional processes aborted (not shown)
>> 
>> What happened here? Why would orted not let me spawn on new nodes? What kind
>> of restrictions apply in this case? I even have the new node name in the
>> hostfile (machinefile), just in case.
>> 
>> Thanks,
>> Prakash
>> 
>> 
> jbuis...@irisa.fr 04/02/07 12:34 PM >>>
>> Ralph Castain a écrit :
>>> The runtime underneath Open MPI (called OpenRTE) will not allow you to spawn
>>> processes on nodes outside of your allocation. This is for several reasons,
>>> but primarily because (a) we only know about the nodes that were allocated,
>>> so we have no idea how to spawn a process anywhere else, and (b) most
>>> resource managers wouldn't let us do it anyway.
>>> 
>>> I gather you have some node that you know about and have hard-coded into
>>> your application? How do you know the name of the node if it isn't in your
>>> allocation??
>> 
>> Because I can give that names to OpenMPI (or OpenRTE, or whatever). I
>> also would like to do the same, and I don't want OpenMPI to restrict to
>> what it thinks to be the allocation, while I'm sure to know better than
>> it what I am doing.
>> The concept of nodes being in allocations fixed at launch-time is really
>> rigid; and it prevents the application (or whatever else) to modify the
>> allocation at runtime, which may be quite nice.
>> 
>> Here is an ugly patch I've quickly done for my own use, which changes
>> the round-robin rmaps such that is first allocates the hosts to the
>> rmgr, as a copy of some code in the dash_host ras component. It's
>> far from being bugfree, but it can be a startpoint to hack.
>> 
>> Jeremy
>> 
>>> Ralph
>>> 
>>> 
>>> On 4/2/07 10:05 AM, "Prakash Velayutham" 
>>> wrote:
>>> 
 Hello,
 
 I have built Open MPI (1.2) with run-time environment enabled for Torque
 (2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU each)
 from Torque. The from inside of my MPI code I am trying to spawn more
 processes to nodes outside of Torque-assigned nodes using
 MPI_Comm_spawn, but this is failing with an error below:
 
 [wins04:13564] *** An error occurred in MPI_Comm_spawn
 [wins04:13564] *** on communicator MPI_COMM_WORLD
 [wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
 [wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
 mpirun noticed that job rank 1 with PID 15070 

Re: [OMPI users] Open MPI error when using MPI_Comm_spawn

[Ralph Castain]

Hi Prakash

This is telling you that you have an error in the comm_spawn command itself.
I am no expert there, so I'll have to let someone else identify it for you.

There are no limits to launching on nodes in a hostfile - they are all
automatically considered "allocated" when the file is read. If you had the
node name in the file, then there is no "dynamic" addition of nodes going
on.

Meantime, I am going to send you a different solution to dynamically adding
nodes under Torque (or any other resource manager).

Ralph


On 4/2/07 10:53 AM, "Prakash Velayutham" 
wrote:

> Hello,
> 
> Thanks for the patch. I still do not know the internals of Open MPI, so can't
> test this right away. But here is another test I ran and that failed too.
> 
> I have now removed Torque from the equation. I am NOT requesting nodes through
> Torque. I SSH to a compute node and start up the code as below.
> 
> prakash@wins04:~/thesis/CS/Samples>mpirun -np 4 --bynode --hostfile
> machinefile ./parallel.laplace
> 
> [wins01:17699] *** An error occurred in MPI_Comm_spawn
> [wins01:17699] *** on communicator MPI_COMM_WORLD
> [wins01:17699] *** MPI_ERR_ARG: invalid argument of some other kind
> [wins01:17699] *** MPI_ERRORS_ARE_FATAL (goodbye)
> mpirun noticed that job rank 1 with PID 25074 on node wins02 exited on signal
> 15 (Terminated). 
> 2 additional processes aborted (not shown)
> 
> What happened here? Why would orted not let me spawn on new nodes? What kind
> of restrictions apply in this case? I even have the new node name in the
> hostfile (machinefile), just in case.
> 
> Thanks,
> Prakash
> 
> 
 jbuis...@irisa.fr 04/02/07 12:34 PM >>>
> Ralph Castain a écrit :
>> The runtime underneath Open MPI (called OpenRTE) will not allow you to spawn
>> processes on nodes outside of your allocation. This is for several reasons,
>> but primarily because (a) we only know about the nodes that were allocated,
>> so we have no idea how to spawn a process anywhere else, and (b) most
>> resource managers wouldn't let us do it anyway.
>> 
>> I gather you have some node that you know about and have hard-coded into
>> your application? How do you know the name of the node if it isn't in your
>> allocation??
> 
> Because I can give that names to OpenMPI (or OpenRTE, or whatever). I
> also would like to do the same, and I don't want OpenMPI to restrict to
> what it thinks to be the allocation, while I'm sure to know better than
> it what I am doing.
> The concept of nodes being in allocations fixed at launch-time is really
> rigid; and it prevents the application (or whatever else) to modify the
> allocation at runtime, which may be quite nice.
> 
> Here is an ugly patch I've quickly done for my own use, which changes
> the round-robin rmaps such that is first allocates the hosts to the
> rmgr, as a copy of some code in the dash_host ras component. It's
> far from being bugfree, but it can be a startpoint to hack.
> 
> Jeremy
> 
>> Ralph
>> 
>> 
>> On 4/2/07 10:05 AM, "Prakash Velayutham" 
>> wrote:
>> 
>>> Hello,
>>> 
>>> I have built Open MPI (1.2) with run-time environment enabled for Torque
>>> (2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU each)
>>> from Torque. The from inside of my MPI code I am trying to spawn more
>>> processes to nodes outside of Torque-assigned nodes using
>>> MPI_Comm_spawn, but this is failing with an error below:
>>> 
>>> [wins04:13564] *** An error occurred in MPI_Comm_spawn
>>> [wins04:13564] *** on communicator MPI_COMM_WORLD
>>> [wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
>>> [wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
>>> mpirun noticed that job rank 1 with PID 15070 on node wins03 exited on
>>> signal 15 (Terminated).
>>> 2 additional processes aborted (not shown)
>>> 
>>> #
>>> 
>>> MPI_Info info;
>>> MPI_Comm comm, *intercomm;
>>> ...
>>> ...
>>> char *key, *value;
>>> key = "host";
>>> value = "wins08";
>>> rc1 = MPI_Info_create();
>>> rc1 = MPI_Info_set(info, key, value);
>>> rc1 = MPI_Comm_spawn(slave,MPI_ARGV_NULL, 1, info, 0,
>>> MPI_COMM_WORLD, intercomm, arr);
>>> ...
>>> }
>>> 
>>> ###
>>> 
>>> Would this work as it is or is something wrong with my assumption? Is
>>> OpenRTE stopping me from spawning processes outside of the initially
>>> allocated nodes through Torque?
>>> 
>>> Thanks,
>>> Prakash
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 

Re: [OMPI users] Open MPI error when using MPI_Comm_spawn

[Jeremy Buisson]

Ralph Castain a écrit :
> The runtime underneath Open MPI (called OpenRTE) will not allow you to spawn
> processes on nodes outside of your allocation. This is for several reasons,
> but primarily because (a) we only know about the nodes that were allocated,
> so we have no idea how to spawn a process anywhere else, and (b) most
> resource managers wouldn't let us do it anyway.
> 
> I gather you have some node that you know about and have hard-coded into
> your application? How do you know the name of the node if it isn't in your
> allocation??

Because I can give that names to OpenMPI (or OpenRTE, or whatever). I
also would like to do the same, and I don't want OpenMPI to restrict to
what it thinks to be the allocation, while I'm sure to know better than
it what I am doing.
The concept of nodes being in allocations fixed at launch-time is really
rigid; and it prevents the application (or whatever else) to modify the
allocation at runtime, which may be quite nice.

Here is an ugly patch I've quickly done for my own use, which changes
the round-robin rmaps such that is first allocates the hosts to the
rmgr, as a copy of some code in the dash_host ras component. It's
far from being bugfree, but it can be a startpoint to hack.

Jeremy

> Ralph
> 
> 
> On 4/2/07 10:05 AM, "Prakash Velayutham" 
> wrote:
> 
>> Hello,
>>
>> I have built Open MPI (1.2) with run-time environment enabled for Torque
>> (2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU each)
>> from Torque. The from inside of my MPI code I am trying to spawn more
>> processes to nodes outside of Torque-assigned nodes using
>> MPI_Comm_spawn, but this is failing with an error below:
>>
>> [wins04:13564] *** An error occurred in MPI_Comm_spawn
>> [wins04:13564] *** on communicator MPI_COMM_WORLD
>> [wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
>> [wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
>> mpirun noticed that job rank 1 with PID 15070 on node wins03 exited on
>> signal 15 (Terminated).
>> 2 additional processes aborted (not shown)
>>
>> #
>>
>> MPI_Info info;
>> MPI_Comm comm, *intercomm;
>> ...
>> ...
>> char *key, *value;
>> key = "host";
>> value = "wins08";
>> rc1 = MPI_Info_create();
>> rc1 = MPI_Info_set(info, key, value);
>> rc1 = MPI_Comm_spawn(slave,MPI_ARGV_NULL, 1, info, 0,
>> MPI_COMM_WORLD, intercomm, arr);
>> ...
>> }
>>
>> ###
>>
>> Would this work as it is or is something wrong with my assumption? Is
>> OpenRTE stopping me from spawning processes outside of the initially
>> allocated nodes through Torque?
>>
>> Thanks,
>> Prakash
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

diff -ru openmpi-1.2/ompi/mca/btl/tcp/btl_tcp.c 
openmpi-1.2-custom/ompi/mca/btl/tcp/btl_tcp.c
--- openmpi-1.2/ompi/mca/btl/tcp/btl_tcp.c  2006-11-09 19:53:44.0 
+0100
+++ openmpi-1.2-custom/ompi/mca/btl/tcp/btl_tcp.c   2007-03-28 
14:02:10.0 +0200
@@ -117,8 +117,8 @@
 tcp_endpoint->endpoint_btl = tcp_btl;
 rc = mca_btl_tcp_proc_insert(tcp_proc, tcp_endpoint);
 if(rc != OMPI_SUCCESS) {
-OBJ_RELEASE(tcp_endpoint);
 OPAL_THREAD_UNLOCK(_proc->proc_lock);
+OBJ_RELEASE(tcp_endpoint);
 continue;
 }
 
diff -ru openmpi-1.2/opal/threads/mutex.c 
openmpi-1.2-custom/opal/threads/mutex.c
--- openmpi-1.2/opal/threads/mutex.c2006-11-09 19:53:32.0 +0100
+++ openmpi-1.2-custom/opal/threads/mutex.c 2007-03-28 15:59:25.0 
+0200
@@ -54,6 +54,8 @@
 #elif OMPI_ENABLE_DEBUG && OMPI_HAVE_PTHREAD_MUTEX_ERRORCHECK
 /* set type to ERRORCHECK so that we catch recursive locks */
 pthread_mutexattr_settype(, PTHREAD_MUTEX_ERRORCHECK);
+#else
+pthread_mutexattr_settype(, PTHREAD_MUTEX_RECURSIVE);
 #endif
 
 pthread_mutex_init(>m_lock_pthread, );
diff -ru openmpi-1.2/opal/threads/mutex_unix.h 
openmpi-1.2-custom/opal/threads/mutex_unix.h
--- openmpi-1.2/opal/threads/mutex_unix.h   2006-11-09 19:53:32.0 
+0100
+++ openmpi-1.2-custom/opal/threads/mutex_unix.h2007-03-28 
15:36:13.0 +0200
@@ -76,7 +76,7 @@
 
 static inline int opal_mutex_trylock(opal_mutex_t *m)
 {
-#if OMPI_ENABLE_DEBUG
+#if 1 // OMPI_ENABLE_DEBUG
 int ret = pthread_mutex_trylock(>m_lock_pthread);
 if (ret == EDEADLK) {
 errno = ret;
@@ -91,7 +91,7 @@
 
 static inline void opal_mutex_lock(opal_mutex_t *m)
 {
-#if OMPI_ENABLE_DEBUG
+#if 1 // OMPI_ENABLE_DEBUG
 int ret = pthread_mutex_lock(>m_lock_pthread);
 if (ret == EDEADLK) {
 errno = ret;
diff -ru 

Re: [OMPI users] Open MPI error when using MPI_Comm_spawn

[Prakash Velayutham]

Thanks for the info, Ralph. It is as I thought, but was hoping wouldn't
be that way.
I am requesting more nodes from the resource manager from inside of my
application code using the RM's API. when I know they are available
(allocated by the RM), I am trying to split the application data across
the newly allocated nodes from inside of MPI.

Any ideas?

Prakash

>>> r...@lanl.gov 04/02/07 12:11 PM >>>
The runtime underneath Open MPI (called OpenRTE) will not allow you to
spawn
processes on nodes outside of your allocation. This is for several
reasons,
but primarily because (a) we only know about the nodes that were
allocated,
so we have no idea how to spawn a process anywhere else, and (b) most
resource managers wouldn't let us do it anyway.

I gather you have some node that you know about and have hard-coded into
your application? How do you know the name of the node if it isn't in
your
allocation??

Ralph


On 4/2/07 10:05 AM, "Prakash Velayutham" 
wrote:

> Hello,
> 
> I have built Open MPI (1.2) with run-time environment enabled for
Torque
> (2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU
each)
> from Torque. The from inside of my MPI code I am trying to spawn more
> processes to nodes outside of Torque-assigned nodes using
> MPI_Comm_spawn, but this is failing with an error below:
> 
> [wins04:13564] *** An error occurred in MPI_Comm_spawn
> [wins04:13564] *** on communicator MPI_COMM_WORLD
> [wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
> [wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
> mpirun noticed that job rank 1 with PID 15070 on node wins03 exited on
> signal 15 (Terminated).
> 2 additional processes aborted (not shown)
> 
> #
> 
> MPI_Info info;
> MPI_Comm comm, *intercomm;
> ...
> ...
> char *key, *value;
> key = "host";
> value = "wins08";
> rc1 = MPI_Info_create();
> rc1 = MPI_Info_set(info, key, value);
> rc1 = MPI_Comm_spawn(slave,MPI_ARGV_NULL, 1, info, 0,
> MPI_COMM_WORLD, intercomm, arr);
> ...
> }
> 
> ###
> 
> Would this work as it is or is something wrong with my assumption? Is
> OpenRTE stopping me from spawning processes outside of the initially
> allocated nodes through Torque?
> 
> Thanks,
> Prakash
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Open MPI error when using MPI_Comm_spawn

[Ralph Castain]

The runtime underneath Open MPI (called OpenRTE) will not allow you to spawn
processes on nodes outside of your allocation. This is for several reasons,
but primarily because (a) we only know about the nodes that were allocated,
so we have no idea how to spawn a process anywhere else, and (b) most
resource managers wouldn't let us do it anyway.

I gather you have some node that you know about and have hard-coded into
your application? How do you know the name of the node if it isn't in your
allocation??

Ralph


On 4/2/07 10:05 AM, "Prakash Velayutham" 
wrote:

> Hello,
> 
> I have built Open MPI (1.2) with run-time environment enabled for Torque
> (2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU each)
> from Torque. The from inside of my MPI code I am trying to spawn more
> processes to nodes outside of Torque-assigned nodes using
> MPI_Comm_spawn, but this is failing with an error below:
> 
> [wins04:13564] *** An error occurred in MPI_Comm_spawn
> [wins04:13564] *** on communicator MPI_COMM_WORLD
> [wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
> [wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
> mpirun noticed that job rank 1 with PID 15070 on node wins03 exited on
> signal 15 (Terminated).
> 2 additional processes aborted (not shown)
> 
> #
> 
> MPI_Info info;
> MPI_Comm comm, *intercomm;
> ...
> ...
> char *key, *value;
> key = "host";
> value = "wins08";
> rc1 = MPI_Info_create();
> rc1 = MPI_Info_set(info, key, value);
> rc1 = MPI_Comm_spawn(slave,MPI_ARGV_NULL, 1, info, 0,
> MPI_COMM_WORLD, intercomm, arr);
> ...
> }
> 
> ###
> 
> Would this work as it is or is something wrong with my assumption? Is
> OpenRTE stopping me from spawning processes outside of the initially
> allocated nodes through Torque?
> 
> Thanks,
> Prakash
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Open MPI error when using MPI_Comm_spawn

[Prakash Velayutham]

Hello,

I have built Open MPI (1.2) with run-time environment enabled for Torque
(2.1.6) resource manager. Initially I am requesting 4 nodes (1 CPU each)
from Torque. The from inside of my MPI code I am trying to spawn more
processes to nodes outside of Torque-assigned nodes using
MPI_Comm_spawn, but this is failing with an error below:

[wins04:13564] *** An error occurred in MPI_Comm_spawn
[wins04:13564] *** on communicator MPI_COMM_WORLD
[wins04:13564] *** MPI_ERR_ARG: invalid argument of some other kind
[wins04:13564] *** MPI_ERRORS_ARE_FATAL (goodbye)
mpirun noticed that job rank 1 with PID 15070 on node wins03 exited on
signal 15 (Terminated). 
2 additional processes aborted (not shown)

#

MPI_Info info;
MPI_Comm comm, *intercomm;
...
...
char *key, *value;
key = "host";
value = "wins08";
rc1 = MPI_Info_create();
rc1 = MPI_Info_set(info, key, value);
rc1 = MPI_Comm_spawn(slave,MPI_ARGV_NULL, 1, info, 0,
MPI_COMM_WORLD, intercomm, arr);
...
}

###

Would this work as it is or is something wrong with my assumption? Is
OpenRTE stopping me from spawning processes outside of the initially
allocated nodes through Torque?

Thanks,
Prakash