For shared memory communications, each on-node connection (non-self,
sender-receiver pair) gets a circular buffer during MPI_Init(). Each CB
requires the following allocations:
*) ompi_cb_fifo_wrapper_t (roughly 64 bytes)
*) ompi_cb_fifo_ctl_t head (roughly 12 bytes)
*) ompi_cb_fifo_ctl_t tail (roughly 12 bytes)
*) queue (roughly 1024 bytes)
Importantly, the current code lays these four allocations out on three
separate pages. (The tail and queue are aggregated together.) So, for
example, that "head" allocation (12 bytes) ends up consuming a full page.
As one goes to more and more on-node processes -- say, for a large SMP
or a multicore system -- the number of non-self connections grows as
n*(n-1). So, these circular-buffer allocations end up consuming a lot
of shared memory.
For example, for a 4K pagesize and n=512 on-node processes, the circular
buffers consume 3 Gbyte of memory -- 90% of which is empty and simply
used for page alignment.
I'd like to aggregate more of these allocations so that:
*) shared-memory consumption is reduced
*) the number of allocations (and hence the degree of lock contention)
during MPI_Init is reduced
Any comments?
I'd like to understand the original rationale for these page
alignments. I expect this is related to memory placement of pages. So,
I imagine three scenarios. Which is it?
A) There really is a good reason for each allocation to have its own
page and any attempt to aggregate is doomed.
B) There is actual benefit for placing things carefully in memory, but
substantial aggregation is still possible. That is, for n processes, we
need at most n different allocations -- not 3*n*(n-1).
C) There is no actual justification for having everything on different
pages. That is, allowing different parts of a FIFO CB to be mapped
differently to physical memory sounded to someone like a good idea at
the time, but no one really did any performance measurements to justify
this. Or, if they did, it was only on one platform and we have no
evidence that the same behavior exists on all platforms. Personally,
I've played with some simple experiments on one (or more?) platforms and
found no performance variations due to placement of shared variables
that two processes use for communication. I guess it's possible that
data is moving cache-to-cache and doesn't care where the backing memory is.
Note that I only want to reduce the number of page-aligned allocations.
I'd preserve cacheline alignment. So, no worry about false sharing due
to a sender thrashing on one end of a FIFO and a receiver on the other.
