I've spent the last week or two trying to figure out exactly how to
build MVAPICH2 with ROMIO *and* PVFS2 *and* native Infiniband support,
and I finally got it working :-)
**Note: These instructions are for configuring with the openfabrics
stack, it should be conceptually the same for the other IB stacks.
In order to properly configure mvapich2 to use/support all three
features you need to do the following:
Modify the make.mvapich2.ofa file to include the necessary Libs and
Cflags, if you specify CFLAGS or LIBS as part of the command line, it
will overwrite these options and your build will fail:
1. Add -lpvfs2 to LIBS:
export LIBS=${LIBS:--L${OPEN_IB_LIB} ${BLCR_LIB} ${RDMA_CM_LIBS} -libverbs
-libumad -lpthread -lpvfs2}
2. Add necessary options to CFLAGS, specifically: -I{PVFS2-HOME/lib}
export CFLAGS=${CFLAGS:--D${ARCH} ${SMP_FLAG} -DUSE_HEADER_CACHING ${PTMALLOC}
-DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS
-D_SHMEM_COLL_ ${RDMA_CM_FLAG} ${CR_FLAG} ${BLCR_INC} -I${OPEN_IB_HOME}/include
$OPT_FLAG
-I{PVFS2-HOME/lib}}
3. Modify the configure line to include --with-file-system=pvfs2
./configure --prefix=${PREFIX} ${MULTI_THREAD} \
--with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd \
${ROMIO} ${SHARED_LIBS} --with-file-system=pvfs2 --without-mpe 2>&1 |tee
config-mine.log
Now you can configure and build everything properly :-)
`ROMIO=yes OPEN_IB_HOME=/usr/local/ LDFLAGS="-m64 -L{PVFS2-HOME/lib}"
./make.mvapich2.ofa`
Initial results have been quite impressive, using mpi-io-test, I was
able to get 15.04Gbps on a 16Gbps link during one test.
Good luck,
Kyle
--
Kyle Schochenmaier
[EMAIL PROTECTED]
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
