I apologize for coming to this late - IU's email forwarding jammed up
yesterday, so I'm only now getting around to reading the backlog.
There has been some off-list discussion about advanced paffinity
mappings/bindings. As I noted there, I am in the latter stages of
completing a new mapper that allows users to easily specify #cpus to
"bind" to each processor.
As part of that effort, we have interfaced to the slurm cpus_per_task
and cpuset envars. So we should (once this gets done) pretty much
handle the slurm environment in that regard.
Having worked on the paffinity issue for some time, I am somewhat
strongly opinionated that PLPA is doing exactly what it should do. It
is up to OMPI/ORTE to identify what cpusets were assigned to the job
and figure out the mappings - the PLPA is there to tell us how many
processors are available, how many are in each socket, etc., and to do
the mechanics of binding the specified process to the specified cpus.
I would tend to oppose any change in the relative responsibilities of
OMPI/ORTE and PLPA. It is a good division as it stands, and is working
well. I haven't read anything in this chain that would change my
opinion.
Just my $0.0002
Ralph
On Jul 22, 2009, at 11:22 AM, Jeff Squyres wrote:
On Jul 22, 2009, at 11:17 AM, Sylvain Jeaugey wrote:
I'm interested in joining the effort, since we will likely have the
same
problem with SLURM's cpuset support.
Ok.
> But as to why it's getting EINVAL, that could be wonky. We might
want to
> take this to the PLPA list and have you run some small, non-MPI
examples to
> ensure that PLPA is parsing your /sys tree properly, etc.
I don't see the /sys implication here. Can you be more precise on
which
files are read to determine placement ?
Check in opal/mca/paffinity/linux/plpa/src/libplpa/
plpa_map.c:load_cache().
IIRC, when you are inside a cpuset, you can see all cpus (/sys
should be
unmodified) but calling set_schedaffinity with a mask containing a
cpu
outside the cpuset will return EINVAL.
Ah, that could be the issue.
The only solution I see to solve
this would be to get the "allowed" cpus with sched_getaffinity,
which should be set according to the cpuset mask.
There are two issues here:
- what should OMPI do
- what should PLPA do
PLPA currently does two things:
1. provide a portable set/get affinity API (to isolate you from
whatever version you have in your linux install)
2. provide topology mapping information (sockets, cores)
PLPA does not currently deal with cpusets. If we want to expand
PLPA to somehow interact with cpusets, that should probably be
brought up on the PLPA mailing lists (someone made this suggestion
to me about a month or two ago and I haven't had a chance to follow
up on it :-( ).
OMPI (as a whole -- meaning: including the ORTE layer) does the
following:
1. decide whether to bind MPI processes or not
2. if we do bind, use the paffinity module to bind processes to
specific processors (the linux paffinity module uses PLPA to do the
actual binding -- PLPA is wholly embedded inside OMPI's linux
paffinity module)
And there's two layers involved here:
- the main ORTE logic saying both "yes, bind" and making the
decision as to which processors to bind to
- the linux paffinity component does a thin layer of translation
between ORTE's/OMPI's requests and calling the back-end PLPA library
As Ralph described, OMPI is currently fairly "dumb" about how it
chooses which processors it uses -- 0 to N-1. I think the issue
here is to make OMPI smarter about how it chooses which processors
to use. It could be in ORTE itself, or it could be in the linux
paffinity translation layer (e.g., linux paffinity component could
report only as many processors as are available in the cpuset...?
And binding could be relative to the cpuset...?).
--
Jeff Squyres
jsquy...@cisco.com
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
