On Dec 10, 2008, at 1:11 PM, Eugene Loh wrote:
For shared memory communications, each on-node connection (non-self,
sender-receiver pair) gets a circular buffer during MPI_Init().
Each CB requires the following allocations:
*) ompi_cb_fifo_wrapper_t (roughly 64 bytes)
*) ompi_cb_fifo_ctl_t head (roughly 12 bytes)
*) ompi_cb_fifo_ctl_t tail (roughly 12 bytes)
*) queue (roughly 1024 bytes)
Importantly, the current code lays these four allocations out on
three separate pages. (The tail and queue are aggregated
together.) So, for example, that "head" allocation (12 bytes) ends
up consuming a full page.
As one goes to more and more on-node processes -- say, for a large
SMP or a multicore system -- the number of non-self connections
grows as n*(n-1). So, these circular-buffer allocations end up
consuming a lot of shared memory.
For example, for a 4K pagesize and n=512 on-node processes, the
circular buffers consume 3 Gbyte of memory -- 90% of which is empty
and simply used for page alignment.
I'd like to aggregate more of these allocations so that:
*) shared-memory consumption is reduced
*) the number of allocations (and hence the degree of lock
contention) during MPI_Init is reduced
This certainly seems like a good idea to me.
I'd like to understand the original rationale for these page
alignments. I expect this is related to memory placement of pages.
So, I imagine three scenarios. Which is it?
A) There really is a good reason for each allocation to have its own
page and any attempt to aggregate is doomed.
B) There is actual benefit for placing things carefully in memory,
but substantial aggregation is still possible. That is, for n
processes, we need at most n different allocations -- not 3*n*(n-1).
C) There is no actual justification for having everything on
different pages. That is, allowing different parts of a FIFO CB to
be mapped differently to physical memory sounded to someone like a
good idea at the time, but no one really did any performance
measurements to justify this. Or, if they did, it was only on one
platform and we have no evidence that the same behavior exists on
all platforms. Personally, I've played with some simple experiments
on one (or more?) platforms and found no performance variations due
to placement of shared variables that two processes use for
communication. I guess it's possible that data is moving cache-to-
cache and doesn't care where the backing memory is.
Note that I only want to reduce the number of page-aligned
allocations. I'd preserve cacheline alignment. So, no worry about
false sharing due to a sender thrashing on one end of a FIFO and a
receiver on the other.
I thought that Rich had prior data from other architectures to justify
doing it this way, but I don't know that for a fact...
I *think* the rationale was that if you have a very busy producer/
consumer, the benefit was that they could operate independently and
not affect the other. Since they were both operating on different
pages, writes to one page would not affect reads from the other.
Perhaps this has not worked out in practice...?
--
Jeff Squyres
Cisco Systems
