Jeff Squyres wrote:
Yes, I think the mmap code in the sm btl actually has a sync point inside add_procs that when the
root allocs and sets up the area, it'll locally broadcast a "yes, we're good -- mmap attach
and let's continue" or "bad things happened; sm btl is broke" message.
Yes, that's great. But my point was that (it used to be that) after
that point, processes would start eating chunks out of that shared area
and for large proc counts the last allocations would fail. (The size of
the shared area was poorly chosen and happened to be insufficient.) So,
despite the sync point you describe, some procs would succeed at
mca_btl_sm_add_procs() while others would not. This particular case is
now, I believe, resolved. It just seemed at the time like a case where
the upper layers were making assumptions that were inconsistent with
what the sm BTL was providing.
But I am not confident about the other BTLs.
On Jun 2, 2010, at 12:51 PM, Eugene Loh wrote:
George Bosilca wrote:
We did assume that at least the errors are symmetric, i.e. if A fails
to connect to B then B will fail when trying to connect to A.
I've not been following this thread closely, but thought I'd add a comment.
It used to be that the sm BTL could fail asymmetrically. A shared
memory could be allocated and processes start to allocate resources
within shared memory. At some point, the shared area would be
exhausted. So, some processes were set up to communicate to others, but
the others would not be able to communicate back via the same BTL. I
think this led to much brokenness. (E.g., how would a process return a
sm fragment to a sender?)
At this point, my recollection of those issues is very fuzzy.
In any case, I think those issues went away with the shared-memory work
I did a while back. The size of the area is now computed to be large
enough that each process's initial allocation would succeed.
