As a result of this patch the first decode of a peer host name might happen in the middle of a debug message (on the first call to ompi_proc_get_hostname). Such a behavior might generate deadlocks based on the level of output verbosity, and has significant potential to reintroduce the recursive behavior the new state machine was supposed to remove.
George. On Aug 17, 2013, at 02:49 , svn-commit-mai...@open-mpi.org wrote: > Author: rhc (Ralph Castain) > Date: 2013-08-16 20:49:18 EDT (Fri, 16 Aug 2013) > New Revision: 29040 > URL: https://svn.open-mpi.org/trac/ompi/changeset/29040 > > Log: > When we direct launch an application, we rely on PMI for wireup support. In > doing so, we lose the de facto data compression we get from the ORTE modex > since we no longer get all the wireup info from every proc in a single blob. > Instead, we have to iterate over all the procs, calling PMI_KVS_get for every > value we require. > > This creates a really bad scaling behavior. Users have found a nearly 20% > launch time differential between mpirun and PMI, with PMI being the slower > method. Some of the problem is attributable to poor exchange algorithms in > RM's like Slurm and Alps, but we make things worse by calling "get" so many > times. > > Nathan (with a tad advice from me) has attempted to alleviate this problem by > reducing the number of "get" calls. This required the following changes: > > * upon first request for data, have the OPAL db pmi component fetch and > decode *all* the info from a given remote proc. It turned out we weren't > caching the info, so we would continually request it and only decode the > piece we needed for the immediate request. We now decode all the info and > push it into the db hash component for local storage - and then all > subsequent retrievals are fulfilled locally > > * reduced the amount of data by eliminating the exchange of the OMPI_ARCH > value if heterogeneity is not enabled. This was used solely as a check so we > would error out if the system wasn't actually homogeneous, which was fine > when we thought there was no cost in doing the check. Unfortunately, at large > scale and with direct launch, there is a non-zero cost of making this test. > We are open to finding a compromise (perhaps turning the test off if > requested?), if people feel strongly about performing the test > > * reduced the amount of RTE data being automatically fetched, and fetched the > rest only upon request. In particular, we no longer immediately fetch the > hostname (which is only used for error reporting), but instead get it when > needed. Likewise for the RML uri as that info is only required for some (not > all) environments. In addition, we no longer fetch the locality unless > required, relying instead on the PMI clique info to tell us who is on our > local node (if additional info is required, the fetch is performed when a > modex_recv is issued). > > Again, all this only impacts direct launch - all the info is provided when > launched via mpirun as there is no added cost to getting it > > Barring objections, we may move this (plus any required other pieces) to the > 1.7 branch once it soaks for an appropriate time.
