[EMAIL PROTECTED] wrote on Thu, 10 May 2007 17:42 -0500:
> I've spent the last week or two trying to figure out exactly how to
> build MVAPICH2 with ROMIO *and* PVFS2 *and* native Infiniband support,
> and I finally got it working :-)
>
> **Note: These instructions are for configuring with the openfabrics
> stack, it should be conceptually the same for the other IB stacks.
>
> In order to properly configure mvapich2 to use/support all three
> features you need to do the following:
>
> Modify the make.mvapich2.ofa file to include the necessary Libs and
> Cflags, if you specify CFLAGS or LIBS as part of the command line, it
> will overwrite these options and your build will fail:
>
> 1. Add -lpvfs2 to LIBS:
> export LIBS=${LIBS:--L${OPEN_IB_LIB} ${BLCR_LIB} ${RDMA_CM_LIBS} -libverbs
> -libumad -lpthread -lpvfs2}
>
> 2. Add necessary options to CFLAGS, specifically: -I{PVFS2-HOME/lib}
> export CFLAGS=${CFLAGS:--D${ARCH} ${SMP_FLAG} -DUSE_HEADER_CACHING
> ${PTMALLOC} -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS
> -D_SHMEM_COLL_ ${RDMA_CM_FLAG} ${CR_FLAG} ${BLCR_INC}
> -I${OPEN_IB_HOME}/include $OPT_FLAG
> -I{PVFS2-HOME/lib}}
>
> 3. Modify the configure line to include --with-file-system=pvfs2
>
> ./configure --prefix=${PREFIX} ${MULTI_THREAD} \
> --with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd \
> ${ROMIO} ${SHARED_LIBS} --with-file-system=pvfs2 --without-mpe 2>&1 |tee
> config-mine.log
>
>
> Now you can configure and build everything properly :-)
>
> `ROMIO=yes OPEN_IB_HOME=/usr/local/ LDFLAGS="-m64 -L{PVFS2-HOME/lib}"
> ./make.mvapich2.ofa`
>
>
> Initial results have been quite impressive, using mpi-io-test, I was
> able to get 15.04Gbps on a 16Gbps link during one test.
Great work, Kyle. Thanks for helping us all out with this.
Although perhaps the #2 (and #3) step should look more like
$PVFS2_HOME/include, and define PVFS2_HOME somewhere. Bash doesn't
like "-" in names. I'm sure we'll figure it out when we need to.
There's also the command "pvfs2-config --cflags" and "pvfs2-config
--libs" that may be of some help, but maybe not since you already
have the IB incs/libs defined as part of the mvapich build script.
-- Pete
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
