Christopher Tanner wrote:
I am setting up a 16-node (homogeneous) cluster running Ubuntu 8.04
server with Mellanox Infiniband cards. I downloaded (from the
OpenFabrics website), compiled, and installed the following IB packages
on the master node into the /usr/local/lib directory. The /usr/local
directory is being shared to all of the nodes via NFS. All packages
seemed to compile and install fine.
libibverbs
librdmacm
libibcm
libipathverbs
dapl
compat-dapl
libmlx4
libmthca
libcxgb3
libibcommon
libibumad
libibmad
opensm
infiniband-diags
I have a few questions:
a) Do I need to run 'make install' on each node or just the master node?
All of the libraries in /usr/local/lib are visible to all nodes...
Stated another way, does 'make install' put files elsewhere beside the
/usr/local/lib directory? Does it alter OS configuration files to tell
it to look for certain files in /usr/local/lib?
No, all the packages above will put their files under /usr/local
b) I know I need to load the IB kernel modules (mlx4_core, mlx4_ib,
rdma_ucm, ib_core, ib_mad, ib_mthca, ib_umad, ib_uverbs) in order for
the IB cards to work. Are these compiled and installed with the above
packages? Where does the kernel know where to look for modules? (Sorry,
this question is very similar to the first one).
The packages above are user space libraries/binaries. To install kernel
modules you should download the latest version of the ofa_1_4_kernel tgz file
from:
http://www.openfabrics.org/downloads/ofa_1_4_kernel/
To install, run:
./configure --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mthca_debug-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_debug-mod
--with-cxgb3-mod --with-ehca-mod --with-ipoib-mod --with-ipoib_debug-mod (... , see --help)
make
make install
c) The OFED software stack contains some stuff that isn't available for
source download (e.g. ib-bonding, ibsim, libsdp). Are these necessary
for the IB network to operate correctly? Since I'm running Ubuntu,
obviously the src.rpm file won't work...
All OFED tgz files that are available under:
http://www.openfabrics.org/~vlad/ofed_1_4/SOURCES/
ib-bonding source RPM can be downloaded from (you can open it to get tgz file
using cpio, if you need):
http://www.openfabrics.org/~monis/ofed_1_4/
This packages are not necessary for the IB network to operate correctly, but
it depends on what are you planning to do.
Regards,
Vladimir
Thanks to all for you help. Previous responses regarding issues with
OpenSM worked great.
-------------------------------------------
Chris Tanner
Space Systems Design Lab
Georgia Institute of Technology
[EMAIL PROTECTED]
-------------------------------------------
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
