Thanks Robert, Looking through your notes, we've followed similar steps. I should mention that I do have Infiniband working on this server, including ipoib. It's purely the nfs-rdma part I'm struggling with.
My latest attempt has been to attempt a custom install, selecting nfs-rdma manually. That worked fine on the development server, but when I attempted to run it on the live server I had an error: "/lib/modules/....../build/scripts is required to build kernel-ib RPM. Please install the corresponding kernel-source or kernel-devel RPM. tk rpm is required to install ibutils" The problem is, for XenServer, you can't get the kernel-source on the live server, instead Citrix ship a DDK (driver development kit) virtual machine, which is a pre-built system with all the kernel sources you need. OFED builds fine on the DDK machine, and for the default installation, I can compile everything there and then transfer the OFED folder (with the compiled RPM files) to the live server. At that point install.pl runs fine, without needing to re-compile things. However, it appears that when I choose some elements in the custom install, it is not using these RPM files, and is trying to compile kernel-ib again from scratch. I have attached a file listing the custom modules I have chosen in case it helps. Ross On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<[email protected]> wrote: > Hi Ross, > > I installed OFED 1.3.1 (Older I know) with other options OK on a Centos > 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is > some notes I took (I used the GUI installer) > > Hope this helps, > > Rob > > 1. Download the OFED package from > http://www.openfabrics.org/downloads/OFED/ > 2. Extract it: tar -xvzf OFED-1.3.1.tgz > 3. Check for the dependencies by running ./install.pl and choosing > documentation ("0"). > 4. Install all required dependencies using yum install package nam. > For Centos dependencies try: yum install gcc libstdc++-devel > libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build > gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel > zlib-devel You may need to run the yum command three times (It takes a > few runs for all the dependencies to sort themselves out) > 5. It is recommended you now upgrade the kernel to make sure the > sources match it. For centos do: yum upgrade kernel (Or yum upgrade > kernel-xen for Xen setups) > 6. Run ./install.pl again but this time choose to install ("2") and > choose custom install ("4"). See the appendix for a recommendation on > which modules to install. > 7. Install should now complete, if it fails check the log (Most > likely you have not installed a required software package - go back to > step 3) > 8. Say yes to configuring IPOIB (If you plan to use it) and > manually set the IP and subnet. When it asks for a network it wants the > base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 > subnet then the "Network" would be 192.168.10.0) > 9. Reboot > 10. Check the install using /etc/init.d/openibd status. This should > show one interface for each Infiniband port (First will be "ib0"). You > should also see quite a few modules loaded, the important ones are > (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core > & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - > IPOIB and ib_uverbs - Direct/Verb) > 11. Run ibstat - This should display the installed devices and their > firmware versions and hardware / GUI IDs. If you need to update the > firmware for Mellanox see > http://www.mellanox.com/support/firmware_download.php I find it easier > to update using Windows and the WinOF software (Windows equivalent of > OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Ross Smith > Sent: 26 June 2009 11:36 > To: [email protected]; [email protected] > Cc: Yehonatan Yossef > Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 > > Hello everyone, > > Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually > Citrix XenServer), with OFED-1.4.1? > > The default install of OFED doesn't include nfs-rdma, and I'm > struggling to work out which components I do and don't need for a > custom build. > > What I would like to include is: > - Core infiniband components > - Drivers for Mellanox Infinihost III Lx (MHES14) > - IPoIB > - nfs-rdma > - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, > ibhosts, etc...) > > My last attempt was following the instructions from OFED 1.4 - I > edited ofed.conf to add nfsrdma=y, and attempted a default install. > However after running install.pl, that change appears to have been > overwritten. > > I am also having problems updating mount.nfs on this machine. Both > nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: > "blkid/blkid.h: No such file or directory", and I cannot yet find a > source for this file. > > Finally, my understanding is that nfs-rdma is in beta status for > CentOS 5.2, but I couldn't see any open bugs on the tracker, are there > any issues I should be aware of? > > thanks, > > Ross > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > The SAQ Group > > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > SAQ is the trading name of SEMTEC Limited. Registered in England & Wales > Company Number: 06481952 > > http://www.saqnet.co.uk AS29219 > > SAQ Group Delivers high quality, honestly priced communication and I.T. > services to UK Business. > > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : > Backups : Managed Networks : Remote Support. > > ISPA Member > >
packages to install: Select Option [1-4]:4 Install kernel-ib? [y/N]:y Install core module? [y/N]:y Install mthca module? [y/N]:y Install mlx4 module? [y/N]:n Install mlx4_en module? [y/N]:n Install cxgb3 module? [y/N]:n Install nes module? [y/N]:n Install ipoib module? [y/N]:y Install sdp module? [y/N]:n Install srp module? [y/N]:n Install srpt module? [y/N]:n Install rds module? [y/N]:n Install qlgc_vnic module? [y/N]:n Install iser module? (open-iscsi will also be installed) [y/N]:n Install nfsrdma module? [y/N]:y Install kernel-ib-devel? [y/N]:n Install ib-bonding? [y/N]:n Install ib-bonding-debuginfo? [y/N]:n Install libibverbs? [y/N]:y Install libibverbs-devel? [y/N]:y Install libibverbs-devel-static? [y/N]:n Install libibverbs-utils? [y/N]:y Install libibverbs-debuginfo? [y/N]:n Install libmthca? [y/N]:y Install libmthca-devel-static? [y/N]:n Install libmthca-debuginfo? [y/N]:n Install libmlx4? [y/N]:n Install libmlx4-devel? [y/N]:n Install libmlx4-debuginfo? [y/N]:n Install libcxgb3? [y/N]:n Install libcxgb3-devel? [y/N]:n Install libcxgb3-debuginfo? [y/N]:n Install libnes? [y/N]:n Install libnes-devel-static? [y/N]:n Install libnes-debuginfo? [y/N]:n Install libibcm? [y/N]:n Install libibcm-devel? [y/N]:n Install libibcm-debuginfo? [y/N]:n Install libibcommon? [y/N]:y Install libibcommon-devel? [y/N]:n Install libibcommon-static? [y/N]:n Install libibcommon-debuginfo? [y/N]:n Install libibumad? [y/N]:y Install libibumad-devel? [y/N]:n Install libibumad-static? [y/N]:n Install libibumad-debuginfo? [y/N]:n Install libibmad? [y/N]:n Install libibmad-devel? [y/N]:n Install libibmad-static? [y/N]:n Install libibmad-debuginfo? [y/N]:n Install ibsim? [y/N]:n Install ibsim-debuginfo? [y/N]:n Install librdmacm? [y/N]:y Install librdmacm-utils? [y/N]:y Install librdmacm-devel? [y/N]:n Install librdmacm-debuginfo? [y/N]:n Install libsdp? [y/N]:n Install libsdp-devel? [y/N]:n Install libsdp-debuginfo? [y/N]:n Install opensm? [y/N]:n Install opensm-libs? [y/N]:n Install opensm-devel? [y/N]:n Install opensm-debuginfo? [y/N]:n Install opensm-static? [y/N]:n Install compat-dapl? [y/N]:n Install compat-dapl-devel? [y/N]:n Install dapl? [y/N]:n Install dapl-devel? [y/N]:n Install dapl-devel-static? [y/N]:n Install dapl-utils? [y/N]:n Install dapl-debuginfo? [y/N]:n Install perftest? [y/N]:n Install mstflint? [y/N]:y Install tvflash? [y/N]:n Install qlvnictools? [y/N]:n Install sdpnetstat? [y/N]:n Install srptools? [y/N]:n Install rds-tools? [y/N]:n Install rnfs-utils? [y/N]:y Install ibutils? [y/N]:y Install infiniband-diags? [y/N]:y Install qperf? [y/N]:n Install qperf-debuginfo? [y/N]:n Install ofed-docs? [y/N]:y Install ofed-scripts? [y/N]:y Install tgt-generic? [y/N]:n Install mpi-selector? [y/N]:n Install openmpi_gcc? [y/N]:n Install mpitests_openmpi_gcc? [y/N]:n Please enter the OFED installation directory: [/usr]:
_______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
