Jeff Squyres wrote:
On Aug 29, 2008, at 5:52 PM, Eugene Loh wrote:
I'm looking at the sm BTL.
Excellent! I hope you had a good dash of parmesan with that
spaghetti code in there (the sm btl is among the hairiest sections
in OMPI...). :-)
There's probably some law of software engineering that applies here.
Basically, upon first read, I was filled with bitter resentment against
those who had written the code. :^) Then, as I began to feel mastery
over its, um, intricacies -- to feel that I, too, was becoming a member
of the inner cabal -- I began to feel pride and a desire to protect the
code against intrusive overhaul. :^)
I did peek at some of the Open MPI papers, and they talked about Open
MPI's modular design. The idea is that someone should be able to play
with one component of the architecture without having to become an
expert in the whole thing. The reality I seem to be facing is that to
understand one part (like the sm BTL), I have to understand many parts
(mpool, allocator, common, etc.) and the only way to do so is to read
code, step through with debugger, and ask experts.
I believe the main rationale for doing page-line alignments was for
memory affinity, since (at least on Linux, I don't know about
solaris) you can only affinity-ize pages.
Solaris maps on a per-page basis.
On your big 512 proc machines, I'm assuming that the page memory
affinity will matter...?
You mean for latency? I could imagine so, but don't know for sure. I'm
no expert on this stuff. Theoretically, I could imagine a system where
some of this stuff might fly from cache-to-cache, with the location of
the backing memory not being relevent.
If locality did matter, I could imagine two reasonable choices: FIFOs
being local to the sender or to the receiver -- with the best choice
depending on the system.
That being said, we're certainly open to making things better. E.g.,
if a few procs share a memory locality (can you detect that in
Solaris?), have them share a page or somesuch...?
Yes, I believe you can detect these things in Solaris.
I could imagine splitting the global shared memory segment up per
process. This might have two advantages:
*) If the processes are bound and there is some sort of first-touch
policy, you could manage memory locality just by having the right
process make the allocation. No need for page alignment of tiny
allocations.
*) You wouldn't need to control memory allocations with a lock (except
for multithreaded apps). I haven't looked at this too closely yet, but
the 3*n*n memory allocations in shared memory during MPI_Init are
currently serialized, which sounds disturbing when n is 100 to 500 local
processes.
Graham, Richard L. wrote:
I have not looked at the code in a long time, so not sure how many things have
changed ... In general what you are suggesting is reasonable. However,
especially on large machines you also need to worry about memory locality, so
should allocate from memory pools that are appropriately located. I expect
that memory allocated on a per-socket basis would do.
Is this what "maffinity" and "memory nodes" are about? If so, I would
think memory locality should be handled there rather than in page
alignment of individual 12-byte and 64-byte allocations.
Having said that, I have no clue how easy this is to implement within the
current code base,
Yeah, we start in the sm BTL, but then go into mpool, class, common, and
allocator.
but expect you can rely on first-touch after the procs are locked down to
simplify the implementation.
Thanks for the discussion and insights.
