On Jan 19, 2012, at 11:58 AM, George Bosilca wrote: > This is a critical change, with a significant impact on the code base. > Basically by moving the binding later in the code after the modex was > completed, all memory allocated before (which is all memory allocated during > the registration of all OMPI modules) will endue being on the wrong NUMA node > (at least by default on Linux). This can have tremendous impact on > performance!
Yes and no. First, any process launched via mpirun is already bound (both cpu and memory) prior to being exec'd, so this change has no impact on them. Second, any process direct-launched where the RM is setting the binding is also bound prior to being exec'd, so this change has no impact there. Thus, the change -only- impacts those processes that are direct-launched but want OMPI to bind them anyway. In that one case, this change is required because the proc has no way to self-determine its binding prior to doing the modex. Jeff/Nathan are adding language to btl.h to highlight the need to avoid allocating memory during registration - we informally have always stated that any allocation should only occur during add_procs, but people may have forgotten over time. Memory allocated by other frameworks should follow similar rules. > > george. > > On Jan 19, 2012, at 13:38 , r...@osl.iu.edu wrote: > >> Author: rhc >> Date: 2012-01-19 13:38:58 EST (Thu, 19 Jan 2012) >> New Revision: 25742 >> URL: https://svn.open-mpi.org/trac/ompi/changeset/25742 >> >> Log: >> Correct ordering in MPI_Init so that we do the modex prior to attempting to >> bind ourselves in the direct launch case as the modex contains info required >> for self-binding. >> >> Text files modified: >> trunk/ompi/runtime/ompi_mpi_init.c | 238 >> ++++++++++++++++++++-------------------- >> 1 files changed, 119 insertions(+), 119 deletions(-) > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
