diff --git a/MPI_README.txt b/MPI_README.txt index 3ec9383..b6e98f5 100644 --- a/MPI_README.txt +++ b/MPI_README.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) MPI in OFED 1.2 README - May 2007 + June 2007 =============================================================================== @@ -19,7 +19,7 @@ Two MPI stacks are included in this release of OFED: - Ohio State University (OSU) MVAPICH 0.9.7 (Modified by Mellanox Technologies) -- Open MPI 1.2.1-1 +- Open MPI 1.2.2-1 Setup, compilation and run information of OSU MVAPICH and Open MPI is provided below in sections 2 and 3 respectively. @@ -39,7 +39,88 @@ and/or LD_LIBRARY_PATH so as to install the desired compiled MPI stacks. OFED includes four basic tests that can be run against each MPI stack: bandwidth (bw), latency (lt), Intel MPI Benchmark, and Presta. The tests are located under: /mpi///tests/, -where is /usr/local/ofed by default. +where is /usr by default. + +1.3 Selecting which MPI to use: mpi-selector +-------------------------------------------- +Depending on how the OFED installer was run, multiple different MPI +implementations may be installed on your system. The OFED installer +will run an MPI selector tool during the installation process, +presenting a menu-based interface to select which MPI implementation +is set as the default for all users. This MPI selector tool can be +re-run at any time by the administrator after the OFED installer +completes to modify the site-wide default MPI implementation selection +by invoking the "mpi-selector-menu" command (root access is typically +required to change the site-wide default). + +The mpi-selector-menu command can also be used by non-administrative +users to override the site-wide default MPI implementation selection +by setting a per-user default. Specifically: unless a user runs the +MPI selector tool to set a per-user default, their environment will be +setup for the site-wide default MPI implementation. + +Note that the default MPI selection does *not* affect the shell from +which the command was invoked (or any other shells that were already +running when the MPI selector tool was invoked). The default +selection is only changed for *new* shells started after the selector +tool was invoked. It is recommended that once the default MPI +implementation is changed via the selector tool, users should logout +and login again to ensure that they have a consistent view of the +default MPI implementation. Other tools can be used to change the MPI +environment in the current shell, such as the environment modules +software package (which is not included in the OFED software package; +see http://modules.sourceforge.net/ for details). + +Note that the site-wide default is set in a file that is typically not +on a networked filesystem, and is therefore specific to the host on +which it was run. As such, it is recommended to run the +mpi-selector-menu command on all hosts in a cluster, picking the same +default MPI implementation on each. It may be more convenient, +however, to use the the mpi-selector command in script-based scenarios +(such as running on every host in a cluster); mpi-selector effects all +the same functionality as mpi-selector-menu, but is intended for +automated environments. See the mpi-selector(1) manual page for more +details. + +Additionally, per-user defaults are set in a file in the user's $HOME +directory. If this directory is not on a network-shared filesystem +between all hosts that will be used for MPI applications, then it also +needs to be propagated to all relevant hosts. + +Note: The MPI selector tool typically sets the PATH and/or +LD_LIBRARY_PATH for a given MPI implementation. This step can, of +course, also be performed manually by a user or on a site-wide basis. +The MPI selector tool simply bundles up this functionality in a +convenient set of command line tools and menus. + +1.4 Updating MPI Installations +------------------------------ + +!!!MELLANOX AND OSU: PLEASE UPDATE/CORRECT THIS SECTION!!! + +Note that all of the MPI implementations included in the OFED software +package are the versions that were available when OFED v1.2 was +released. They have been QA tested with this version of OFED and are +fully supported. + +However, note that administrators can go to the web sites of each MPI +implementation and download / install newer versions after OFED has +been successfully installed. There is nothing specific about the +OFED-included MPI software packages that prohibit installing +newer/other MPI implementations. + +It should be also noted that versions of MPI released after OFED v1.2 +are not supported by OFED. But since each MPI has its own release +schedule and QA process (each of which involves testing with the OFED +stack), it may sometimes be desirable -- or even advisable, depending +on how old the MPI implementations are that are included in OFED -- to +download install a newer version of MPI. + +The web sites of each MPI implementation are listed below: + +- Open MPI: http://www.open-mpi.org/ +- MVAPICH: ... +- MVAPICH2: ... =============================================================================== 2. OSU MVAPICH MPI @@ -118,50 +199,73 @@ rsh, add to the mpirun_rsh command the parameter: -rsh =============================================================================== Open MPI is a next-generation MPI implementation from the Open MPI -Project (http://www.open-mpi.org/). Version 1.1.1-1 of Open MPI is -included in this release, which is also available directly -from the main Open MPI web site. This MPI stack is being offered in -OFED as a "technology preview," meaning that it is not officially -supported yet. It is expected that future releases of OFED will have -fully supported versions of Open MPI. +Project (http://www.open-mpi.org/). Version 1.2-2-1 of Open MPI is +included in this release, which is also available directly from the +main Open MPI web site. A working Fortran compiler is not required to build Open MPI, but some -of the included MPI tests are written in Fortran. These tests will not -compile/run if Open MPI is built without Fortran support. +of the included MPI tests are written in Fortran. These tests will +not compile/run if Open MPI is built without Fortran support. The following compilers are supported by OFED's Open MPI package: GNU, Pathscale, Intel, or Portland. The install script prompts the user -for the compiler with which to build the Open MPI RPM. Note that more +for the compiler with which to build the Open MPI RPM. Note that more than one compiler can be selected simultaneously, if desired. Users should check the main Open MPI web site for additional documentation and support. (Note: The FAQ file considers InfiniBand tuning among other issues.) -3.1 Setting up for Open MPI: ----------------------------- -The Open MPI team strongly advises users to put the Open MPI installation -directory in their PATH and LD_LIBRARY_PATH. This can be done at the -system level if all users are going to use Open MPI. Specifically: +3.1 Setting up for Open MPI +--------------------------- +Selecting to use Open MPI via the mpi-selector-mpi and mpi-selector +tools will perform all the necessary setup for users to build and run +Open MPI applications. If you use the MPI selector tools, you can +skip the rest of this section. + +If you do not wish to use the MPI selector tools, the Open MPI team +strongly advises users to put the Open MPI installation directory in +their PATH and LD_LIBRARY_PATH. This can be done at the system level +if all users are going to use Open MPI. Specifically: + - add /bin to PATH - add /lib to LD_LIBRARY_PATH - is the directory where the desired Open MPI instance was installed. -("instance" refers to the compiler used for Open MPI compilation at install -time.) + is the directory where the desired Open MPI instance was +installed ("instance" refers to the compiler used for Open MPI +compilation at install time.). -If using rsh or ssh to launch MPI jobs, you *must* set the variables described -above in your shell startup files (e.g., .bashrc, .cshrc, etc.). +If you are using a job scheduler to launch MPI jobs (e.g., SLURM, +Torque), setting the PATH and LD_LIBRARY_PATH is still required, but +it does not need to be set in your shell startup files. Procedures +describing how to add these values to PATH and LD_LIBRARY_PATH are +described in detail at: -If you are using a job scheduler to launch MPI jobs (e.g., SLURM, Torque), -setting the PATH and LD_LIBRARY_PATH is still required, but it does -not need to be set in your shell startup files. Procedures describing -how to add these values to PATH and LD_LIBRARY_PATH are described in -detail at: http://www.open-mpi.org/faq/?category=running -3.2 Compiling Open MPI Applications: ------------------------------------- +3.2 Open MPI Installation Support / Updates +------------------------------------------- +The OFED package will install Open MPI with support for TCP, shared +memory, and the OpenFabrics network stacks. No other networks are +supported by the OFED Open MPI installation. + +Open MPI supports a wide variety of run-time environments. The OFED +installer will not include support for all of them, however (e.g., +Torque and PBS-based environments are not supported by the +OFED-installed Open MPI). + +The ompi_info command can be used to see what support was installed; +look for plugins for your specific environment / network / etc. If +you do not see them, the OFED installer did not include support for +them. + +As described above, administrators or users can go to the Open MPI web +site and download / install either a newer version of Open MPI (if +available), or the same version with different configuration options +(e.g., support for Torque / PBS-based environments). + +3.3 Compiling Open MPI Applications +----------------------------------- (copied from http://www.open-mpi.org/faq/?category=mpi-apps -- see this web page for more details) @@ -205,45 +309,62 @@ specifically, if you run into a compiler or linker error, check your source code and/or back-end compiler -- it is usually not the fault of the Open MPI wrapper compiler. -3.3 Running Open MPI Applications: +3.4 Running Open MPI Applications: ---------------------------------- Open MPI uses either the "mpirun" or "mpiexec" commands to launch -applications. If your cluster uses a resource manager (such as SLURM -or Torque), providing a hostfile is not necessary: +applications. If your cluster uses a resource manager (such as +SLURM), providing a hostfile is not necessary: > mpirun -np 4 my_mpi_application If you use rsh/ssh to launch applications, they must be set up to NOT prompt for a password (see http://www.open-mpi.org/faq/?category=rsh -for more details on this topic). Moreover, you need to provide a hostfile -containing a list of hosts to run on. +for more details on this topic). Moreover, you need to provide a +hostfile containing a list of hosts to run on. Example: > cat hostfile - node1.example.com - node2.example.com - node3.example.com - node4.example.com + host1.example.com + host2.example.com + host3.example.com + host4.example.com > mpirun -np 4 -hostfile hostfile my_mpi_application - (application runs on all 4 nodes) + (application runs on all 4 hosts) -In the following examples, replace with the number of nodes to run on, -and with the filename of a valid hostfile listing the nodes +In the following examples, replace with the number of hosts to run on, +and with the filename of a valid hostfile listing the hosts to run on. Example1: Running the OSU bandwidth: - > cd /usr/local/ofed/mpi/gcc/openmpi-1.1.1-1/tests/osu_benchmarks-2.2 +!!! SOMEONE PLEASE CHECK THESE DIRECTORIES AND EXECUTABLE NAMES + > cd /usr/mpi/gcc/openmpi-1.2.2-1/tests/osu_benchmarks-2.2 > mpirun -np -hostfile osu_bw Example2: Running the Intel MPI Benchmark benchmarks: - > cd /usr/local/ofed/mpi/gcc/openmpi-1.1.1-1/tests/IMB-2.3 +!!! SOMEONE PLEASE CHECK THESE DIRECTORIES AND EXECUTABLE NAMES + > cd /usr/mpi/gcc/openmpi-1.2.2-1/tests/IMB-2.3 > mpirun -np -hostfile IMB-MPI1 Example3: Running the Presta benchmarks: - > cd /usr/local/ofed/mpi/gcc/openmpi-1.1.1-1/tests/presta-1.4.0 +!!! SOMEONE PLEASE CHECK THESE DIRECTORIES AND EXECUTABLE NAMES + > cd /usr/mpi/gcc/openmpi-1.2.2-1/tests/presta-1.4.0 > mpirun -np -hostfile com -o 100 + +3.5 More Open MPI Information +----------------------------- +Much, much more information is available about using and tuning Open +MPI (to include OpenFabrics-specific tunable parameters) on the Open +MPI web site FAQ: + + http://www.open-mpi.org/faq/ + +Users who cannot find the answers that they are looking for, or are +experiencing specific problems should consult the "how to get help" web +page for more information: + + http://www.open-mpi.org/community/help/ diff --git a/open_mpi_release_notes.txt b/open_mpi_release_notes.txt index e963e7f..09fd757 100755 --- a/open_mpi_release_notes.txt +++ b/open_mpi_release_notes.txt @@ -1,17 +1,11 @@ Open Fabrics Enterprise Distribution (OFED) - Open MPI in OFED 1.1 Copyrights, License, and Release Notes + Open MPI in OFED 1.2 Copyrights, License, and Release Notes - May 2007 + June 2007 Open MPI Copyrights ------------------- -Most files in this release are marked with the copyrights of the -organizations who have edited them. The copyrights below generally -reflect members of the Open MPI core team who have contributed code to -this release. The copyrights for code used under license from other -parties are included in the corresponding files. - Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana University Research and Technology Corporation. All rights reserved. @@ -27,6 +21,8 @@ Copyright (c) 2006 Voltaire, Inc. All rights reserved. Copyright (c) 2006 Sun Microsystems, Inc. All rights reserved. Copyright (c) 2007 Myricom, Inc. All rights reserved. +Additional copyrights may follow + Open MPI License ---------------- Redistribution and use in source and binary forms, with or without @@ -96,28 +92,54 @@ Much, much more information is also available in the Open MPI FAQ: OFED-Specific Release Notes --------------------------- -SLES 10 with non-gcc compiler support: +** SLES 10 with Pathscale compiler support: -Open MPI can be built from source to support environments not -supported by the default OFED installer. The source code for Open MPI -can be extracted from the SRPM shipped with OFED or downloaded from -the main Open MPI web site: http://www.open-mpi.org/. +Using the Pathscale compiler to build Open MPI on SLES10 may result in +a non-functional Open MPI installation (every Open MPI command fails). +If this problem occurs, please use a different compiler to compile +Open MPI. -To compile with Open MPI from source with OFED support, fully install -the rest of OFED and then configure Open MPI with the -"--with-openib=/usr" command line option. See the rest of the -documentation below for other configure command line options and -installation instructions. - -Intel compiler support: +** Intel compiler support: Some versions of the Intel 9.1 C++ compiler suite series produce incorrect code when used with the Open MPI C++ bindings. Symptoms of this problem include crashing applications (e.g., segmentation violations) and Open MPI producing errors about incorrect parameters. +Be sure to upgrade to the latest maintenance release of the Intel 9.1 +compiler to avoid these problems. + +** Installing newer versions of Open MPI after OFED is installed: + +Open MPI can be built from source after OFED is fully installed. The +source code for Open MPI can be extracted from the SRPM shipped with +OFED or downloaded from the main Open MPI web site: +http://www.open-mpi.org/. + +To compile with Open MPI from source with OFED support, fully install +the rest of OFED. If you used the default prefix for the OFED +installation (/usr), Open MPI should build with OpenFabrics support by +default. If you used a different OFED prefix, you must tell Open MPI +what it is with the "--with-openib=" switch to +configure. You can verify that Open MPI installed with OpenFabrics +support by running: + + shell$ ompi_info | grep openib + MCA btl: openib (MCA v1.0, API v1.0.1, Component v1.3) + +See the rest of the documentation below for other configure command +line options and installation instructions. + +=========================================================================== + +Much, much more information is also available in the Open MPI FAQ: + + http://www.open-mpi.org/faq/ =========================================================================== +General Release Notes +--------------------- + The following abbreviated list of release notes applies to this code base as of this writing (1 March 2007):