Those are excellent questions that I have asked as well at various times :-)
Some thoughts below On Mar 7, 2010, at 1:20 PM, George Bosilca wrote: > Quick question about this. We now have an OPAL level progress thread, which > enables the machinery at the OPAL level. Unfortunately, this doesn't say > anything about what the MPI level will do? That is correct and has always been the case. The OPAL progress thread only indicates that opal_progress is being called via a separate thread. Currently, turning "on" the opal progress thread automatically turns "on" opal thread support and enables MPI thread multiple. However, the BTLs may or may not be involved (see below). With this change, you can turn "on" the opal progress thread and/or the opal thread support but not enable MPI thread multiple if you choose not to do so. Similarly, if you enable MPI thread multiple you will automatically turn "on" the opal thread support, but you will -not- turn "on" the opal progress thread. This is required behavior as some (most?) of the BTL's are not safe when opal progress thread is active. > Moreover, this is quite confusing as there are no communications layers in > OPAL so one can ask what an OPAL level --enable-progress-thread means. It means that anything involving OPAL events will be progressed. So async messages coming into ORTE, for example, can be supported without waiting for someone to call into the OMPI library. My understanding is that the design decision to have a "central" progress thread at the OPAL layer was intended to help avoid thread-lock and unnecessary overhead caused by having multiple progress threads throughout the code. I'm content to let the original design stand for now and address that question (i.e., OPAL vs ORTE progress thread) as a separate issue for the future. This RFC is solely to change the configure option names to remove the badly overloaded and confusing --enable-mpi-threads > > This raise several related questions. Do you expect to have a ORTE level > progress thread even if the MPI level do not have one? I didn't look at the > code, but I have a strong doubt about such mix-up between threads > requirements. Yep - but as the original RFC discussion explained, not inside MPI apps. The desire is to allow mpirun and orted to utilize threads without stipulating that they can -only- do so if MPI apps are thread-enabled. The two situations are completely orthogonal and should not be connected via the configure options. > > How do we know when MPI needs a progress thread? There is no option for this. > Or should we define that if MPI_THREAD_MULTIPLE is supported and > OPAL_PROGRESS_THREAD is enabled this means the BTLs can register their own > progress thread? At the moment, the BTLs already use their own progress threads and do -not- utilize the OPAL progress thread. Why the various BTL developers chose to do this is unknown to me and essentially irrelevant to this RFC. What the BTL developers may want to do is review the reasons behind this design decision. As I understand it, there was consideration of this question, and it was a made decision (as opposed to a simple oversight) to have BTL-specific progress threads instead of relying on the OPAL progress thread. > > george. > > > On Mar 4, 2010, at 16:17 , Jeff Squyres wrote: > >> WHAT: Rename the --enable-*-threads configure switches and ENABLE*THREAD* >> macros. >> (see previous RFC: >> http://www.open-mpi.org/community/lists/devel/2010/01/7366.php) >> >> WHY: The fact that thread safety in OPAL and ORTE requires a configure >> switch with "mpi" in the name is very non-intuitive. Additionally, >> MPI_THREAD_MULTIPLE support is not necessarily the same thing as OPAL thread >> support (MTM needs OPAL thread support, but not the other way around), and >> we are seeing a growing advantage/need for ORTE to utilize threads in mpirun >> and orted irrespective of the MPI layer's threading abilities. >> >> WHERE: Mostly opal/config/opal_config_threads.m4, something new in >> ompi/config/*.m4, and wherever the current ENABLE*THREAD* macros are >> currently used in the current code base. >> >> WHEN: Next Friday COB >> >> TIMEOUT: COB, Friday, Feb 5, 2010 >> >> ------------------------ >> >> More details: >> >> Cisco is starting to investigate using ORTE and OPAL in various threading >> scenarios. The fact that you need to enable thread safety in ORTE/OPAL with >> a configure switch that has the word "mpi" in it is extremely >> counter-intuitive (it bit some of our engineers very badly, and they were >> mighty annoyed!!). In addition, we ran into problems where it was >> advantageous to have threads in ORTE, but we couldn't do it without forcing >> thread support into the MPI layer because the switch is universal. >> >> Since this functionality actually has nothing to do with MPI (it's actually >> the other way around -- MPI_THREAD_MULTIPLE needs this functionality), we >> really should divorce MPI threading functionality from whether threading >> machinery is enabled in OPAL or not. >> >> These names were proposed at the end of the previous RFC and no one >> objected, so I'm sending this around as a new RFC to ensure we're all on the >> same sheet of music: >> >> --enable-opal-progress-threads: enables progress thread machinery in opal >> --> this is just a renaming from --enable-progress-threads >> --> the corresponding #define stays the same: OPAL_ENABLE_PROGRES_THREADS >> >> --enable-opal-multi-threads: enables multi threaded machinery in opal >> --> this is just a renaming from --enable-mpi-threads >> --> the corresponding #define also renames; from OPAL_ENABLE_MPI_THREADS to >> OPAL_ENABLE_MULTI_THREADS >> >> --enable-mpi-thread-multiple: enables the use of MPI_THREAD_MULTIPLE; *ONLY* >> affects the MPI layer >> --> use of this switch explicitly implies --enable-opal-multi-threads >> --> new #define: OMPI_ENABLE_THREAD_MULTIPLE >> >> We can keep and deprecate the old configure options if desired: >> >> --enable-mpi-threads: deprecated synonym for --enable-mpi-thread-multiple >> --enable-progress-threads: deprecated synonym for >> --enable-opal-progress-threads >> >> ..although I'm somewhat inclined to ditch them unless someone has strong >> feelings about keeping them. >> >> Doing the name change in OPAL and ORTE is fairly straightforward -- it's >> essentially an s/foo/bar/g kind of operation. It'll likely take a little >> more effort in the MPI layer because the places where the current #defines >> are used may need to switch to the new name or to the new >> OMPI_ENABLE_THREAD_MULTIPLE name (and maybe some new logic? I am not sure >> without looking into it closer). >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
