On Fri, Jun 26, 2009 at 03:31:50PM +0100, Ross Smith wrote: > Latest news: I've bodged the installation, but I'm having problems > with nfs.mount now. > > Since everything worked with nfs-rdma on the development server, I > manually copied the kerel-ib rpm file from there, and installed that > on top of the default infiniband install on the live server, using the > command: > > # rpm -iv --replacepkgs kernel-ib....... > > With that done modprobe xprtrdma now works. > > Since the kernel-ib file appears to be the only one modified as > nfsrdma support is added, I assume this is all that's needed. Can > anybody confirm if this is ok, or whether I'm going to break things > horribly by doing this? > > I also have nfs v1.1.6 installed, but it doesn't seem to understand > the rdma option. The system is still using tcp by default, and if I > try to use rdma, I get this: > > # mount -o rdma, port=20049 server:path /mnt > mount.nfs: Unsupported nfs mount option: rdma > > However, nfs is definitely the correct version: > # mount.nfs -V > mount.nfs (linux nfs-utils 1.1.6) > > Even using the mount.rnfs utility supplied with OFED doesn't help: > # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm > # mount.rnfs server:path /mnt -o rdma,port=20049 > mount.rnfs: Unsupported nfs mount option: rdma
You need the "-i" option for older kernels. In this case, you would want: mount.rnfs server:path /mnt -i -o rdma,port=20049 > > And if I use mount.rnfs without the rdma option, "cat /proc/mounts" > reports that it's just using tcp. > > I'm getting read speeds of around 400MB/s using regular NFS which > isn't bad, but I know I can achieve over 900MB/s with this server over > Infiniband, so there's still some way to go. > > Ross > > > > On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith<[email protected]> wrote: > > Ok, I've made a little more progress. > > > > The issue with nfs-utils was solved by installing efsprogs-devel, and > > mount.nfs is now reporting that it's running v1.1.6. > > > > I still can't install OFED with nfs-rdma on the live server though, it > > appears that it's unable to use the rpm package to install with nfs > > support. > > > > I've now discovered how to use the ofed.conf file, and this is the > > procedure I'm following: > > > > - On the development server, I run install.pl and perform a standard install > > - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y > > - Still on the development server I run: > > # install.pl -c ofed.conf > > - That can be seen to install most items from the rpm, and to > > recompile kernel-ib with the nfsrdma option. > > - I then package up the OFED folder with tar, and send it to the live server > > - On the live server I run the same command: > > # install.pl -c ofed.conf > > > > Despite it running with exactly the same options, I get the error: > > > > "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib > > RPM. > > Please install the corresponding kernel-source or kerenel-devel RPM." > > > > If I exclude the nfsrdma option and repeat all the above steps, the > > script installs fine from the pre-compiled RPM. > > > > It looks to me like a problem with the install script when choosing > > nfs-rdma. > > > > Does anybody have any suggestions as to how I can work around this? > > Am I able to simply install the RPM files manually? > > > > Ross > > > > > > > > On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith<[email protected]> wrote: > >> Thanks Robert, > >> > >> Looking through your notes, we've followed similar steps. I should > >> mention that I do have Infiniband working on this server, including > >> ipoib. It's purely the nfs-rdma part I'm struggling with. > >> > >> My latest attempt has been to attempt a custom install, selecting > >> nfs-rdma manually. That worked fine on the development server, but > >> when I attempted to run it on the live server I had an error: > >> > >> "/lib/modules/....../build/scripts is required to build kernel-ib RPM. > >> Please install the corresponding kernel-source or kernel-devel RPM. > >> tk rpm is required to install ibutils" > >> > >> The problem is, for XenServer, you can't get the kernel-source on the > >> live server, instead Citrix ship a DDK (driver development kit) > >> virtual machine, which is a pre-built system with all the kernel > >> sources you need. > >> > >> OFED builds fine on the DDK machine, and for the default installation, > >> I can compile everything there and then transfer the OFED folder (with > >> the compiled RPM files) to the live server. At that point install.pl > >> runs fine, without needing to re-compile things. > >> > >> However, it appears that when I choose some elements in the custom > >> install, it is not using these RPM files, and is trying to compile > >> kernel-ib again from scratch. > >> > >> I have attached a file listing the custom modules I have chosen in > >> case it helps. > >> > >> Ross > >> > >> > >> > >> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<[email protected]> wrote: > >>> Hi Ross, > >>> > >>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos > >>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is > >>> some notes I took (I used the GUI installer) > >>> > >>> Hope this helps, > >>> > >>> Rob > >>> > >>> 1. Download the OFED package from > >>> http://www.openfabrics.org/downloads/OFED/ > >>> 2. Extract it: tar -xvzf OFED-1.3.1.tgz > >>> 3. Check for the dependencies by running ./install.pl and choosing > >>> documentation ("0"). > >>> 4. Install all required dependencies using yum install package nam. > >>> For Centos dependencies try: yum install gcc libstdc++-devel > >>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build > >>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel > >>> zlib-devel You may need to run the yum command three times (It takes a > >>> few runs for all the dependencies to sort themselves out) > >>> 5. It is recommended you now upgrade the kernel to make sure the > >>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade > >>> kernel-xen for Xen setups) > >>> 6. Run ./install.pl again but this time choose to install ("2") and > >>> choose custom install ("4"). See the appendix for a recommendation on > >>> which modules to install. > >>> 7. Install should now complete, if it fails check the log (Most > >>> likely you have not installed a required software package - go back to > >>> step 3) > >>> 8. Say yes to configuring IPOIB (If you plan to use it) and > >>> manually set the IP and subnet. When it asks for a network it wants the > >>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 > >>> subnet then the "Network" would be 192.168.10.0) > >>> 9. Reboot > >>> 10. Check the install using /etc/init.d/openibd status. This should > >>> show one interface for each Infiniband port (First will be "ib0"). You > >>> should also see quite a few modules loaded, the important ones are > >>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core > >>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - > >>> IPOIB and ib_uverbs - Direct/Verb) > >>> 11. Run ibstat - This should display the installed devices and their > >>> firmware versions and hardware / GUI IDs. If you need to update the > >>> firmware for Mellanox see > >>> http://www.mellanox.com/support/firmware_download.php I find it easier > >>> to update using Windows and the WinOF software (Windows equivalent of > >>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). > >>> > >>> -----Original Message----- > >>> From: [email protected] > >>> [mailto:[email protected]] On Behalf Of Ross Smith > >>> Sent: 26 June 2009 11:36 > >>> To: [email protected]; [email protected] > >>> Cc: Yehonatan Yossef > >>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 > >>> > >>> Hello everyone, > >>> > >>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually > >>> Citrix XenServer), with OFED-1.4.1? > >>> > >>> The default install of OFED doesn't include nfs-rdma, and I'm > >>> struggling to work out which components I do and don't need for a > >>> custom build. > >>> > >>> What I would like to include is: > >>> - Core infiniband components > >>> - Drivers for Mellanox Infinihost III Lx (MHES14) > >>> - IPoIB > >>> - nfs-rdma > >>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, > >>> ibhosts, etc...) > >>> > >>> My last attempt was following the instructions from OFED 1.4 - I > >>> edited ofed.conf to add nfsrdma=y, and attempted a default install. > >>> However after running install.pl, that change appears to have been > >>> overwritten. > >>> > >>> I am also having problems updating mount.nfs on this machine. Both > >>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: > >>> "blkid/blkid.h: No such file or directory", and I cannot yet find a > >>> source for this file. > >>> > >>> Finally, my understanding is that nfs-rdma is in beta status for > >>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there > >>> any issues I should be aware of? > >>> > >>> thanks, > >>> > >>> Ross > >>> _______________________________________________ > >>> general mailing list > >>> [email protected] > >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >>> > >>> To unsubscribe, please visit > >>> http://openib.org/mailman/listinfo/openib-general > >>> > >>> The SAQ Group > >>> > >>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > >>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales > >>> Company Number: 06481952 > >>> > >>> http://www.saqnet.co.uk AS29219 > >>> > >>> SAQ Group Delivers high quality, honestly priced communication and I.T. > >>> services to UK Business. > >>> > >>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit > >>> : Backups : Managed Networks : Remote Support. > >>> > >>> ISPA Member > >>> > >>> > >> > > > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
