Hello Richard, Yes NWCHEM can be run on IB using 1.4.1. If you have built openmpi with IB support. Note: If your IB cards are qlogic you need to compile NWCHEM with MPI-SPAWN. Rangam
Settings for my Build with MPI-SPAWN: export ARMCI_NETWORK=MPI-SPAWN export IB_HOME=/usr export IB_INCLUDE=/usr/include export IB_LIB=/usr/lib64 export IB_LIB_NAME="-libverbs -libumad -lpthread " export NWCHEM_TOP=/lustre/work/apps/nwchem-5.1.1 export NWCHEM_MODULES="venus geninterface all" export LIBMPI="-lmpi" export ARMCI_DEFAULT_SHMMAX=256 export BLASLIB=goto2_penrynp-r1.00 export BLASLOC=/lustre/work/apps/goto/ export BLASOPT="-L/lustre/work/apps/goto/ -l$BLASLIB" export CC=icc export CFLG="-xP -fPIC" export CXX=icpc export F77=ifort export F90=ifort export FC=ifort export FL=ifort export LARGE_FILES=TRUE export LIB_DEFINES=-DDFLT_TOT_MEM=2147483647 export MPI_LOC=/lustre/work/apps/IB-ICC-IFORT-OPENMPI-IB/ export MPI_INCLUDE=$MPI_LOC/include export MPI_LIB=$MPI_LOC/lib export MPI_BIN=$MPI_LOC/bin export NWCHEM_TARGET=LINUX64 export TARGET=LINUX64 export USE_MPI=y Setting with OPENIB export ARMCI_NETWORK=OPENIB export IB_HOME=/usr export IB_INCLUDE=/usr/include export IB_LIB=/usr/lib64 export IBV_FORK_SAFE=1 export NWCHEM_TOP=/lustre/work/apps/nwchem-5.1.1 export NWCHEM_MODULES="all qm geninterface" #export LIBMPI="-lmpich -libumad -libverbs -lrdmacm -pthread" export LIBMPI="-lmpi -pthread -libumad -libverbs -lrdmacm -pthread" export ARMCI_DEFAULT_SHMMAX=256 export BLASLIB=goto2_penrynp-r1.00 export BLASLOC=/lustre/work/apps/goto/ export BLASOPT="-L/lustre/work/apps/goto/ -l$BLASLIB" export CC=icc export CFLG="-xP -fPIC" export CXX=icpc export F77=ifort export F90=ifort export FC=ifort export FL=ifort export GOTO_NUM_THREADS=1 export LARGE_FILES=TRUE export LIB_DEFINES=-DDFLT_TOT_MEM=2147483647 export MA_USE_ARMCI_MEM=1 export MPI_LOC=/lustre/work/apps/IB-ICC-IFORT-OPENMPI export MPI_INCLUDE=$MPI_LOC/include export MPI_LIB=$MPI_LOC/lib export MPI_BIN=$MPI_LOC/bin export NWCHEM_TARGET=LINUX64 export OMP_NUM_THREADS=1 export TARGET=LINUX64 export USE_MPI=y ________________________________________ From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] On Behalf Of Richard Walsh [richard.wa...@csi.cuny.edu] Sent: Thursday, May 06, 2010 1:06 PM To: us...@open-mpi.org Subject: [OMPI users] Can NWChem be run with OpenMPI over an InfiniBand interconnect ... ?? All, I have built NWChem successfully, and trying to run it with an Intel built version of OpenMPI 1.4.1. If I force to run over over 1 GigE maintenance interconnect it works, but when I try it over the default InfiniBand communications network it fails with: -------------------------------------------------------------------------- An MPI process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. The process that invoked fork was: Local host: gpute-2 (PID 15996) MPI_COMM_WORLD rank: 0 If you are *absolutely sure* that your application will successfully and correctly survive a call to fork(), you may disable this warning by setting the mpi_warn_on_fork MCA parameter to 0. -------------------------------------------------------------------------- This looks to be a known problem. Is there I go around? I have seen it suggested in some places that I need to use Mellanox's version of MPI, which is not an option and surprises me as they are a big OFED contributor. What are my options ... other than using GigE ... ?? Thanks, rbw Richard Walsh Parallel Applications and Systems Manager CUNY HPC Center, Staten Island, NY 718-982-3319 612-382-4620 Mighty the Wizard Who found me at sunrise Sleeping, and woke me And learn'd me Magic! Think green before you print this email. _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
