Re: [OMPI users] process mapping

2019-06-21 Thread Noam Bernstein via users
> On Jun 21, 2019, at 5:02 PM, Ralph Castain  wrote:
> 
> 
> 
> Too many emails to track :-(
> 
> Should just be “--map-by core --rank-by core” - nothing fancy required. 
> Sounds like you are getting --map-by node, or at least --rank-by node, which 
> means somebody has set an MCA param either in the system config file or the 
> environment.
> 

Yeay, that worked.  Apparently I didn’t try every combination.  I feel like the 
documentation could be better, since I clearly wasn’t able to figure this out, 
but I’m not sure what particular wording to suggest.  Let me think about it.

thanks,
Noam___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] process mapping

2019-06-21 Thread Ralph Castain via users
On Jun 21, 2019, at 1:52 PM, Noam Bernstein mailto:noam.bernst...@nrl.navy.mil> > wrote:



On Jun 21, 2019, at 4:45 PM, Ralph Castain mailto:r...@open-mpi.org> > wrote:

Hilarious - I wrote that code and I have no idea who added that option or what 
it is supposed to do. I can assure, however, that it isn’t implemented anywhere.

Not really a big deal, since the documentation doesn’t explain them, and I was 
just grasping at straws. Are rankfiles implemented?  Maybe I could use those 
(although binding/mapping command line arguments would definitely be easier).


Perhaps if you tell us what pattern you are trying to get, we can advise you on 
the proper cmd line to get there?

I thought that was in the original email.  Basically, I have 2 18 core CPUs, 
and I want ranks 0-17 on slots 0-17 in cpu 0 and 18-35 on slots 0-17 in cpu 1.  
I’d have thought that would be straightforward, but everything I’ve tried ends 
up with i_task%2 == i_cpu, i.e. 0,2,4,… on cpu 0 and 1,3,5… on cpu 1.

Too many emails to track :-(

Should just be “--map-by core --rank-by core” - nothing fancy required. Sounds 
like you are getting --map-by node, or at least --rank-by node, which means 
somebody has set an MCA param either in the system config file or the 
environment.

Noam


 

 
|
|
 
|
U.S. NAVAL
|
 
|
_RESEARCH_
|
 
LABORATORY

Noam Bernstein, Ph.D.
Center for Materials Physics and Technology
U.S. Naval Research Laboratory
T +1 202 404 8628  F +1 202 404 7546
https://www.nrl.navy.mil


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] process mapping

2019-06-21 Thread Noam Bernstein via users


> On Jun 21, 2019, at 4:45 PM, Ralph Castain  wrote:
> 
> Hilarious - I wrote that code and I have no idea who added that option or 
> what it is supposed to do. I can assure, however, that it isn’t implemented 
> anywhere.

Not really a big deal, since the documentation doesn’t explain them, and I was 
just grasping at straws. Are rankfiles implemented?  Maybe I could use those 
(although binding/mapping command line arguments would definitely be easier).

> 
> Perhaps if you tell us what pattern you are trying to get, we can advise you 
> on the proper cmd line to get there?

I thought that was in the original email.  Basically, I have 2 18 core CPUs, 
and I want ranks 0-17 on slots 0-17 in cpu 0 and 18-35 on slots 0-17 in cpu 1.  
I’d have thought that would be straightforward, but everything I’ve tried ends 
up with i_task%2 == i_cpu, i.e. 0,2,4,… on cpu 0 and 1,3,5… on cpu 1.

Noam


||
|U.S. NAVAL|
|_RESEARCH_|
LABORATORY
Noam Bernstein, Ph.D.
Center for Materials Physics and Technology
U.S. Naval Research Laboratory
T +1 202 404 8628  F +1 202 404 7546
https://www.nrl.navy.mil 
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] process mapping

2019-06-21 Thread Ralph Castain via users
Hilarious - I wrote that code and I have no idea who added that option or what 
it is supposed to do. I can assure, however, that it isn’t implemented anywhere.

Perhaps if you tell us what pattern you are trying to get, we can advise you on 
the proper cmd line to get there?


On Jun 21, 2019, at 1:43 PM, Noam Bernstein mailto:noam.bernst...@nrl.navy.mil> > wrote:

On Jun 21, 2019, at 4:04 PM, Ralph Castain via users mailto:users@lists.open-mpi.org> > wrote:

I’m unaware of any “map-to cartofile” option, nor do I find it in mpirun’s help 
or man page. Are you seeing it somewhere?

>From "mpirun —help”:

tin 1431 : mpirun --help mapping
mpirun (Open MPI) 4.0.1

Usage: mpirun [OPTION]...  [PROGRAM]...
Start the given program using Open RTE

   -cf|--cartofile   
                         Provide a cartography file
followed by all the other mapping-related options.

Maybe what I want is best described as not doing round-robin, but I see no way 
to do that either.

Noam

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] process mapping

2019-06-21 Thread Noam Bernstein via users
> On Jun 21, 2019, at 4:04 PM, Ralph Castain via users 
>  wrote:
> 
> I’m unaware of any “map-to cartofile” option, nor do I find it in mpirun’s 
> help or man page. Are you seeing it somewhere?

From "mpirun —help”:

tin 1431 : mpirun --help mapping
mpirun (Open MPI) 4.0.1

Usage: mpirun [OPTION]...  [PROGRAM]...
Start the given program using Open RTE

   -cf|--cartofile   
 Provide a cartography file
followed by all the other mapping-related options.

Maybe what I want is best described as not doing round-robin, but I see no way 
to do that either.

Noam___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] process mapping

2019-06-21 Thread Ralph Castain via users
I’m unaware of any “map-to cartofile” option, nor do I find it in mpirun’s help 
or man page. Are you seeing it somewhere?


On Jun 21, 2019, at 12:43 PM, Noam Bernstein via users 
mailto:users@lists.open-mpi.org> > wrote:

Hi - are there any examples of the cartofile format?  Or is there some combo of 
—map, —rank, or —bind to achieve this mapping?
[BB/..][../..]
[../BB][../..]
[../..][BB/..]
[../..][../BB]

I tried everything I could think of for —bind-to, —map-by, and —rank-by, and I 
can’t get it to happen.  I can get
[BB/..][../..]
[../../][BB/..]
[../BB][../../]
[../../][../BB]
but that’s not quite what I want.

thanks,
Noam

___
users mailing list
users@lists.open-mpi.org  
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] process mapping

2019-06-21 Thread Noam Bernstein via users
Hi - are there any examples of the cartofile format?  Or is there some combo of 
—map, —rank, or —bind to achieve this mapping?
[BB/..][../..]
[../BB][../..]
[../..][BB/..]
[../..][../BB]

I tried everything I could think of for —bind-to, —map-by, and —rank-by, and I 
can’t get it to happen.  I can get
[BB/..][../..]
[../../][BB/..]
[../BB][../../]
[../../][../BB]
but that’s not quite what I want.

thanks,
Noam

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] Process mapping and affinity on the devel trunk

2011-12-11 Thread Ralph Castain
Hello all

If you are using the developer's trunk or nightly tarball, or are interested in 
new mapping and binding options that will be included in the next feature 
series (1.7), then please read on. If not, then please ignore.

People have raised the question of "the trunk isn't binding processes any more" 
a couple of times recently. OMPI's mapping, ranking, and binding options 
underwent a major change on the developer's trunk a few weeks ago. This was 
done to provide a greater range of options for process placement and binding. 
Although this was mentioned on the devel mailing list awhile ago, I thought a 
general message might be in order, especially for those users out there who are 
working with the trunk.

Most importantly, under the new system, opal_paffinity_alone (and its 
pseudonym, mpi_paffinity_alone) was disabled - it no longer does anything. I 
have added a warning so that any setting of that parameter will warn you of 
this situation. This is more than likely the reason why you are not seeing 
processes bound.

That option has been replaced by the --bind-to  option, where  can be 
none, hardware thread (hwthread), core, L1 cache (l1cache), L2 cache (l2cache), 
L3 cache (l3cache), socket, or numa region. This can also be set as an MCA 
parameter "hwloc_base_binding_policy". There are two allowed qualifiers to the 
binding option:

* if-supported - binding will be done if the system supports it. If the system 
does not support it, the application will execute unbound without issuing a 
warning - otherwise, an error message will be emitted and the execution aborted.

* overload-allowed - if the binding results in more processes than cpus being 
bound to a resource (e.g., if 4 processes are bound to a socket that only has 2 
cpus), then execution will be terminated with an error unless this qualifier is 
provided.


Mapping was also expanded to support mapping by all the same locations via the 
--map-by  option, plus two additional locations: slot (default) and node. 
The option is also available as MCA parameter "rmaps_base_mapping_policy". The 
mapping option has three qualifiers:

* span - treat all allocated nodes as if they were a single node - i.e., map 
across all specified resources before looping around and placing the next layer 
of processes on them. The default is to loop across all resources on each node 
until that node is completely filled before moving to the next node, so the 
"span" qualifier acts to balance the load across the allocation.

* oversubscribe - allow more processes than allocated slots to be mapped onto a 
node. This is the default for user-specified allocations (i.e., by hostfile or 
-host).

* nooversubscribe - error out if more processes than allocated slots are mapped 
onto a node. This is the default for resource managed allocations (e.g., 
specified by SLURM or MOAB).

Another mapper was also added to the system. The "ppr" mapper takes a string 
argument detailing the number of processes to be placed on each resource, with 
the supported resources again including all those specified above. For example, 
a string of "4:node,2:socket,1:core" would tell the mapper to place one process 
on every core in the allocation, with a maximum of 2 on each socket and 4 on 
each node.


Assigning process ranks has a corresponding --rank-by  option, with all 
the same values for  as found for mapping (including the use of "slot" as 
the default). This option is available thru the MCA parameter 
"rmaps_base_ranking_policy". The ranking option has two qualifiers:

* span - similar to the mapping qualifier, this causes the ranks to be assigned 
across all specified resources as if they were a single node

* fill - assign ranks sequentially to all processes on the given resource 
before moving to the next one, filling all such resources on each node before 
moving to the next.


Please note that several convenience options were retained for backward 
compatibility:

*  --pernode, --npernode N, --npersocket N: the npersocket option now binds the 
processes to their mapped socket unless another binding option was specified

*  --bind-to-core,  --bind-to-socket

* --bynode, --byslot


All three options (mapping, ranking, binding) can be used in any combination. 
Thus, you can assign a mapping pattern, pick any option for assigning ranks, 
and pick any option for binding. For example, you could map-by socket, rank-by 
core, and bind-to numa. As a result, there are a very large number of ways to 
arrange your application.

I realize all this flexibility can be confusing and a little overwhelming. I am 
working to provide more documentation on the OMPI wiki site, but it isn't done 
yet. I will let people know when it is completed.

HTH
Ralph