Re: [Lustre-discuss] lo2iblnd and Mellanox IB question
I've had to rebuild against the Mellanox OFED every time I change Lustre or OFED versions. It's a bit of a catch 22 situation because you have to build the Mellanox OFED against the Lustre kernel, install the Mellanox OFED, then rebuild the Lustre modules against the Mellanox OFED. The procedure I use is as follows... * install upgraded Lustre kernel and kernel-devel rpms * rebuild Mellanox OFED against Lustre kernel - mount -o loop MLNX_OFED.iso /root/mnt - /root/mnt/docs/mlnx_add_kernel_support.sh -i /root/MLNX_OFED.iso * install Mellanox OFED from rebuilt MLNX_OFED.iso * install kernel-ib-devel from rebuilt MLNX_OFED.iso Now rebuld lustre-modules RPM to get ko2iblnd.ko which is compatible with Mellanox kernel-ib drivers... * cd /usr/src/lustre-x.x.x * configure --with-o2ib=/usr/src/openib * make rpms Ron. -Original Message- From: lustre-discuss-boun...@lists.lustre.org [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Ms. Megan Larko Sent: November 20, 2012 4:21 PM To: Lustre User Discussion Mailing List Subject: [Lustre-discuss] lo2iblnd and Mellanox IB question Hello to Everyone! I have a question to which I think I know the answer, but I am seeking confirmation (re-assurance?). I have build a RHEL 6.2 system with lustre-2.1.2. I am using the rpms from the Whamcloud site for linux kernel 2.6.32_220.17.1.el6_lustre.x85_64 along with the version-matching lustre, lustre-modules, lustre-ldiskfs, and kernel-devel,I also have from the Whamcloud site kernel-ib-1.8.5-2.6.32-220.17.1.el6_lustre.x86_64 and the related kernel-ib-devel for same. The lustre file system works properly for TCP. I would like to use InfiniBand. The system has a new Mellanox card for which mlxn1 firmware and drivers were installed. After this was done (I cannot speak to before) the IB network will come up on boot and copy and ping in a traditional network fashion. Hard Part: I would like to run the lustre file system on the IB (ib0). I re-created the lustre network to use /etc/modprobe.d/lustre.conf pointing to o2ib in place of tcp0. I rebuilt the mgs/mdt and all osts to use the IB network (the mgs/mds --failnode=[new_IB_addr] and the osts point to mgs on IB net). When I modprobe lustre to start the system I receive error messages stating that there are Input/Output errors on lustre modules fld.ko, fid,ko, mdc.ko osc.ko lov.ko. The lustre.ko cannot be started. A look in /var/log/messages reveals many Unknown symbol and Disagrees about version of symbol from the ko2iblnd module. A modprobe --dump-modversions /path/to/kernel/lo2iblnd.ko shows it pointing to the Modules.symvers of the lustre kernel. Am I correct in thinking that because of the specific Mellanox IB hardware I have (with its own /usr/src/ofa_kernel/Module.symvers file), that I have to build Lustre-2.1.2 from tarball to use the configure --with-o2ib=/usr/src/ofa_kernel mandating that this system use the ofa_kernel-1.8.5 modules and not the OFED 1.8.5 from the kernel-ib rpms to which Lustre defaults in the Linux kernel? Is a rebuild of lustre from source mandartory or is there a way in which I may point to the appropriate symbols needed by the ko2iblnd.ko? Enjoy the Thanksgiving holiday for those U.S. readers.To everyone else in the world, have a great weekend! Megan Larko Hewlett-Packard ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] New Test Framework Development Group Call - Correction
Hi, Chris. I'm still confused about the time. Every Tuesday starting on the 27th November 01:00 CST, 08:00 PST, 17:00 GMT, 21:00 MSK 00:00 CST, 08:00 PST, 16:00 GMT, 20:00 MSK Both of those listings are inaccurate. Neither 01:00 or 00:00 CST == 08:00 PST. Since 8 PST == 16 GMT == 20 MSK, I'm guessing that you mean: 10:00 CST, 08:00 PST, 16:00 GMT, 20:00 MSK Is that right? Thanks, -Cory On 11/22/2012 11:47 AM, Gearing, Chris wrote: Hi All, I sent out a mail about the new test framework development group call and as Bruce spotted didn't get the timezones right! Again call is open to all and has the following time and bridge information. The correct times are Every Tuesday starting on the 27th November: 00:00 CST, 08:00 PST, 16:00 GMT, 20:00 MSK Bridge Info: +1 916-356-2663, Bridge: 1, Passcode: 1146033 Note: We have a whole bunch of world wide local numbers so if your not US based (and I'm not) please email me your location and I'll reply with with your local number. Many thanks Chris - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] [lustre-devel] New Test Framework Development Group Call - Correction
On Mon, Nov 26, 2012 at 11:56 AM, Cory Spitz spitz...@cray.com wrote: Hi, Chris. I'm still confused about the time. Every Tuesday starting on the 27th November 01:00 CST, 08:00 PST, 17:00 GMT, 21:00 MSK 00:00 CST, 08:00 PST, 16:00 GMT, 20:00 MSK Both of those listings are inaccurate. Neither 01:00 or 00:00 CST == 08:00 PST. Since 8 PST == 16 GMT == 20 MSK, I'm guessing that you mean: 10:00 CST, 08:00 PST, 16:00 GMT, 20:00 MSK Is that right? For myself, I'd as soon see strictly UTC for international times (aka GMT or Zulu). I know my offset and, obviously, it is often confusing to get the time zone correct for other people in other time zones. I had also presumed 16:00 Zulu was correct. :) Cheers - Bruce ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
