Krishna, If you would like to do some testing/development on NFS/RDMA, take a look at the current NFS/RDMA code. There are instructions on were to get it and how to set it up here:
http://nfs-rdma.sourceforge.net/Documents/README I'm revising the instructions for 2.6.25. I'll be posting the new version once the first 2.6.25-rc is released. We would appreciate feedback in this area as well. james On Thu, 31 Jan 2008, Krishna Kumar2 wrote: > Hi Jeff & James, > > Great. If you let me know when the bits are ready (I don't always read the > mailing list), > I should be able to get some testing done. > > Thanks, > > - KK > > Jeff Becker <[EMAIL PROTECTED]> wrote on 01/30/2008 11:02:09 PM: > > > Hi all. > > > > James Lentini wrote: > > > On Wed, 30 Jan 2008, Krishna Kumar2 wrote: > > > > > > > > >> Hi James, > > >> > > >> Since you had mentioned in an earlier email that NFS-RDMA server > > >> side will be present in OFED1.4, > > >> > > > > > > Actually, that was Tziporet. > > > > > > > > >> do you know if any port of the server code to OFED1.3 (when it comes > > >> out) will happen? Is there any effort for that, any work ongoing, > > >> any help required, etc? > > >> > > > > > > Jeff Becker had looked into this. We would definitely appreciate the > > > help. > > > > > I have set up a git tree for NFSoRDMA and succesfully merged it with, > > and built it on OFED 1.3-rcx. I'm currently doing the backports (SLES 10 > > SP1 first). All this is in preparation for OFED 1.4, as that is when > > NFSoRDMA will be included in OFED. I think I have this > > patching/backporting stuff under control. However, my testing resources > > are limited. Thus depending on your platform, I might be able to point > > you at OFED 1.3 based bits for testing if/when they are ready. Thanks. > > > > -jeff > > > > > The NFS framework has changed significantly in several areas in recent > > > kernel releases. This has made backporting the NFS/RDMA code to older > > > kernels challenging. > > > > > > If you are interested in working on OFED1.3 support, let us know. > > > > > > > > >> I couldn't find the release time lines for OFED1.4, is there any > > >> link on openfabrics homepage? > > >> > > > > > > I'm not involved with the OFED1.4 planning. Tziporet, is there > > > information on this? > > > > > > > > >> Thanks, > > >> > > >> - KK > > >> > > >> [EMAIL PROTECTED] wrote on 01/29/2008 08:23:46 PM: > > >> > > >> > > >>> On Tue, 29 Jan 2008, Pawel Dziekonski wrote: > > >>> > > >>> > > >>>> On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote: > > >>>> > > >>>>> On Sat, 26 Jan 2008, Pawel Dziekonski wrote: > > >>>>> > > >>>>> > > >>>>>> I pulled Tom's tree from new url and build a kernel. > > >>>>>> > > >>>>> If you enabled support for INFINIBAND drivers (IB and iWARP > support) > > >>>>> and NFS client/server support, the kernel should be ready to go > (run > > >>>>> "grep RDMA /your_kernel_sources/.config" to confirm that > > >>>>> CONFIG_SUNRPC_XPRT_RDMA is either m or y). > > >>>>> > > >>>>> NFS/RDMA doesn't require OFED be installed. OFED is a release of > the > > >>>>> Linux kernel sources and some userspace libraries/tools. If you are > > >>>>> > > >>>>>> then I downloaded OFED from > > >>>>>> http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz, > > >>>>>> > > >>>>> I don't know what the above URL contains. The latest code is in Tom > > >>>>> Tucker's tree (and now NFS server maintainer Bruce Fields tree). It > > >>>>> > > >> is > > >> > > >>>> hi, > > >>>> > > >>>> back to subject on a proper mailing list. > > >>>> > > >>>> I have a >3 year experience with mellanox hardware and IBGold so I > > >>>> basically know what OFED is all about. up to now i was only using > > >>>> IBGold since IB drivers appeared in kernel pretty recently. > > >>>> > > >>> You'll want to use the mainline kernel's IB drivers for NFS/RDMA. > > >>> We've been developing the NFS/RDMA software on the OpenFabrics (aka > > >>> OpenIB) code since it was merged into 2.6.10 in Dec 2004. > > >>> > > >>> > > >>>> currently I have new hardware. I'm running Tom's kernel and already > > >>>> did some MPI tests. SDP is not working, probably because sdp kernel > > >>>> modules where not build. ;) I understand that those modules are only > > >>>> available from ofa-kernel. please correct me if i'm wrong. > > >>>> > > >>> Correct. SDP has never been submitted to mainline Linux. > > >>> > > >>> > > >>>> system is Scientic Linux 4.5, which is supposed to be a fully > > >>>> compatible RH4 clone. hardware is Supermicro mobos with Mellanox > > >>>> MT25204 and Flextronisc switch. > > >>>> > > >>>> error log from ofa-kernel build: > > >>>> > > >>> Is your goal to build a kernel with an NFS/RDMA server? If so, the > > >>> kernel sources from Tom Tucker's git tree are the ones you want, not > > >>> the old OFED 1.2-based packages which are out of date. > > >>> > > >>> Did you try setting up the NFS/RDMA server on the kernel used for > your > > >>> MPI tests above? > > >>> > > >>> > > >>>>>> make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6' > > >>>>>> test -e include/linux/autoconf.h -a -e include/config/auto.conf || > > >>>>>> > > >> ( \ > > >> > > >>>>>> echo; \ > > >>>>>> echo " ERROR: Kernel configuration is invalid."; \ > > >>>>>> echo " include/linux/autoconf.h or > include/config/auto.conf > > >>>>>> > > >> are > > >> > > >>> missing."; \ > > >>> > > >>>>>> echo " Run 'make oldconfig && make prepare' on kernel src > > >>>>>> > > >> to fix it."; \ > > >> > > >>>>>> echo; \ > > >>>>>> /bin/false) > > >>>>>> > > >>>>>> obviously, doing 'make oldconfig && make prepare' does not help. > > >>>>>> anyway, above mentioned files do exist: > > >>>>>> > > >>>>>> # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h, > > >>>>>> > > >>> include/config/auto.conf} > > >>> > > >>>>>> -rw-r--r-- 1 root root 10156 Jan 25 17:42 > > >>>>>> > > >> /usr/src/ib/xprt-switch-2. > > >> > > >>> 6/include/config/auto.conf > > >>> > > >>>>>> -rw-r--r-- 1 root root 14733 Jan 25 17:42 > > >>>>>> > > >> /usr/src/ib/xprt-switch-2. > > >> > > >>> 6/include/linux/autoconf.h > > >>> > > >>>>>> despite of above, compilation continues but fails with: > > >>>>>> > > >>>>>> gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1. > > >>>>>> > > >>> 2/drivers/infiniband/core/.mad.o.d -nostdinc -isystem > > >>> > > >> /usr/lib/gcc/x86_64- > > >> > > >>> redhat-linux/3.4.6/include -D__KERNEL__ > > >>> > > >> -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1. > > >> > > >>> 2/include -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2 > > >>> > > >> /drivers/infiniband/include > > >> > > >>> -Iinclude -include include/linux/autoconf.h -include > > >>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h > -Wall > > >>> > > >> -Wundef > > >> > > >>> -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common > > >>> > > >> -Werror- > > >> > > >>> implicit-function-declaration -Os -m64 -mno-red-zone > -mcmodel=kernel > > >>> > > >> -pipe - > > >> > > >>> Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time > > >>> > > >> -mno-sse - > > >> > > >>> mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args > -DCONFIG_AS_CFI=1 > > >>> > > >> - > > >> > > >>> DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer > -Wdeclaration-after- > > >>> statement -DMODULE -D"KBUILD_STR(s)=#s" - > > >>> D"KBUILD_BASENAME=KBUILD_STR(mad)" > -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)" > > >>> > > >> -c - > > >> > > >>> o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.! > > >>> tmp > > >>> > > >>>> _mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2 > > >>>> > > >> /drivers/infiniband/core/mad.c > > >> > > >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2 > > >>>>>> > > >> /drivers/infiniband/core/mad.c: In > > >> > > >>> function `ib_mad_init_module': > > >>> > > >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2 > > >>>>>> > > >> /drivers/infiniband/core/mad.c: > > >> > > >>> 2966: error: too many arguments to function `kmem_cache_create' > > >>> > > >>>>>> make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1. > > >>>>>> > > >>> 2/drivers/infiniband/core/mad.o] Error 1 > > >>> > > >>>>>> make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1. > > >>>>>> > > >>> 2/drivers/infiniband/core] Error 2 > > >>> > > >>>>>> make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2 > > >>>>>> > > >> /drivers/infiniband] Error 2 > > >> > > >>>>>> make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error > > >>>>>> > > >> 2 > > >> > > >>>>>> make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6' > > >>>>>> make: *** [kernel] Error 2 > > >>>>>> error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install) > > >>>>>> > > >>>>>> full log: > > >>>>>> https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log > > >>>>>> > > >>>> thanks in advance for any help, P > > >>>> > > >>>> > > >>>> -- > > >>>> Pawel Dziekonski <[EMAIL PROTECTED]> > > >>>> Wroclaw Centre for Networking & Supercomputing, HPC Department > > >>>> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, > > >>>> > > >> POLAND > > >> > > >>>> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl > > >>>> > > >>>> > > >>>> > > >> > ------------------------------------------------------------------------- > > >> > > >>>> This SF.net email is sponsored by: Microsoft > > >>>> Defy all challenges. Microsoft(R) Visual Studio 2008. > > >>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > >>>> _______________________________________________ > > >>>> nfs-rdma-devel mailing list > > >>>> [EMAIL PROTECTED] > > >>>> https://lists.sourceforge.net/lists/listinfo/nfs-rdma-devel > > >>>> > > >>>> > > >>> _______________________________________________ > > >>> general mailing list > > >>> [email protected] > > >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > >>> > > >>> To unsubscribe, please visit > > >>> > > >> http://openib.org/mailman/listinfo/openib-general > > >> > > >> > > > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
