Re: [OMPI users] EXTERNAL: Re: How is hwloc used by OpenMPI

[Rayson Ho]

On Thu, Nov 8, 2012 at 11:07 AM, Jeff Squyres  wrote:
> Correct.  PLPA was a first attempt at a generic processor affinity solution.  
> hwloc is a 2nd generation, much Much MUCH better solution than PLPA (we 
> wholly killed PLPA
> after the INRIA guys designed hwloc).

Edwin,

We ported OGS/Grid Engine to hwloc 1.5 years ago (the original core
binding code in Grid Engine uses PLPA).

http://gridscheduler.sourceforge.net/projects/hwloc/GridEnginehwloc.html


>From an API consumer (both PLPA & hwloc) point of view, some of the
important hwloc advantages are:

1) Grid Engine now can use the same piece of code for different
platforms: Linux, Solaris, AIX, Mac OS X, FreeBSD, Tru64, HP-UX,
Windows. Before with PLPA, we only have support for Linux & Solaris.

2) Support for newer CPU architectures & hardware. As the development
of PLPA stopped a few years ago, many of the newer architectures did
not get recognized properly. We switched over to hwloc when the
original Grid Engine core binding code stopped working on the AMD
Magny-Cours (Opteron 6100 series).

To be fair to PLPA, had the development continued, then it should have
no issues with those new architectures. But then, the data structures
of hwloc seem to be able to handle newer hardware components more
nicely!


We now use information from hwloc to optimize job placement on AMD
Bulldozers (including Piledriver). Currently hwloc just treats each of
the Bulldozer module as 2 cores, so we still have to code a bit of
logic in the Grid Engine code to do what we need.

http://blogs.scalablelogic.com/2012/07/optimizing-grid-engine-for-amd.html

Rayson

==
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/



>
>> Re: layering, I believe you are saying that the relationship to libnuma is 
>> not one where hwloc is adding higher-level functionalities to libnuma, but 
>> rather hwloc is a much improved alternative except for a few system calls it 
>> makes via libnuma out of necessity or convenience.
>
> Correct.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html

Re: [OMPI users] EXTERNAL: Re: How is hwloc used by OpenMPI

[Jeff Squyres]

On Nov 8, 2012, at 10:17 AM, Blosch, Edwin L wrote:

> Thanks, I definitely appreciate the new, hotness of hwloc.  I just couldn't 
> tell from the documentation or the web page how or if it was being used by 
> OpenMPI.
> 
> I still work with OpenMPI 1.4.x and now that I've looked into the builds, I 
> think I understand that PLPA is used in 1.4 and hwloc is brought in as an MCA 
> module in 1.6.x.

Correct.  PLPA was a first attempt at a generic processor affinity solution.  
hwloc is a 2nd generation, much Much MUCH better solution than PLPA (we wholly 
killed PLPA after the INRIA guys designed hwloc).

> Re: layering, I believe you are saying that the relationship to libnuma is 
> not one where hwloc is adding higher-level functionalities to libnuma, but 
> rather hwloc is a much improved alternative except for a few system calls it 
> makes via libnuma out of necessity or convenience.

Correct.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] EXTERNAL: Re: How is hwloc used by OpenMPI

[Blosch, Edwin L]

Thanks, I definitely appreciate the new, hotness of hwloc.  I just couldn't 
tell from the documentation or the web page how or if it was being used by 
OpenMPI.

I still work with OpenMPI 1.4.x and now that I've looked into the builds, I 
think I understand that PLPA is used in 1.4 and hwloc is brought in as an MCA 
module in 1.6.x.

Re: layering, I believe you are saying that the relationship to libnuma is not 
one where hwloc is adding higher-level functionalities to libnuma, but rather 
hwloc is a much improved alternative except for a few system calls it makes via 
libnuma out of necessity or convenience.

Thanks