What does dmesg tell you? Normally it will report some module has incorrect symbols, which means you compiled against a different version of the kernel source. OFED/MOFED libraries, etc.
> On Sep 25, 2018, at 05:14, Tung-Han Hsieh <[email protected]> > wrote: > > Dear All, > > I found that my lustre-2.10.5 with ZFS (either 0.7.9 or 0.7.11) > cannot load the "lustre" modules because it cannot load the > "obdclass.ko" module. The error message is the following: > > # modprobe -v -v obdclass > insmod /lib/modules/3.12.72/updates/fs/lustre/obdclass.ko > libkmod: INFO ../libkmod/libkmod-module.c:829 kmod_module_insert_module: > Failed to insert module '/lib/modules/3.12.72/updates/fs/lustre/obdclass.ko': > No such device > ERROR: could not insert 'obdclass': No such device > libkmod: INFO ../libkmod/libkmod.c:319 kmod_unref: context 0x7fb945d321e0 > released > > Could anyone suggest how to debug ? > > Thanks very much. > > > T.H.Hsieh > > > On Tue, Sep 25, 2018 at 12:14:00AM +0800, Tung-Han Hsieh wrote: >> Dear Nathaniel, >> >> Thank you very much for your kindly reply. Indeed I modified the >> lustre-2.10.5 codes: >> >> lustre/osd-zfs/osd_object.c >> lustre/osd-zfs/osd_xattr.c >> >> for the declaration: >> >> inode_timespec_t now; >> >> Similar to what you have done in your patch. So I can compile >> lustre-2.10.5 cleanly with zfs-0.7.11. Sorry I forgot to mention. >> >> But my problem is still there. Actually I just tried: >> >> 1. Applying your patch to the original lustre-2.10.5 code, and >> recompile with spl-0.7.11 and zfs-0.7.11. But loading "lustre" >> module still gives "no such device" error. >> >> 2. I recompile the original lustre-2.10.5 with spl-0.7.9 and >> zfs-0.7.9. They can be compiled cleanly. But again I got the >> "no such device" error when loading "lustre" module. >> >> I am wondering that I must overlooked a trivial step, something >> like one (or some) of the utilities in /opt/lustre/sbin/* should >> be linked to /sbin/ or /usr/sbin/ .... >> >> Any suggestions are very appreciated. >> >> Thank you very much. >> >> >> T.H.Hsieh >> >> >> On Mon, Sep 24, 2018 at 01:21:19PM +0000, Nathaniel Clark wrote: >>> Hello Tung-Han, >>> >>> ZFS 0.7.11 doesn’t compile cleanly with Lustre, yet. >>> >>> There’s a ticket for adding ZFS 0.7.11 support to lustre: >>> https://jira.whamcloud.com/browse/LU-11393 >>> >>> It has patches for master (pre-2.12) and a separate patch for 2.10. >>> >>> — >>> Nathaniel Clark <[email protected]<mailto:[email protected]>> >>> Senior Engineer >>> Whamcloud / DDN >>> >>> On Sep 24, 2018, at 2:15 PM, Tung-Han Hsieh >>> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Dear All, >>> >>> I am trying to install Lustre version 2.10.5 with ZFS-0.7.11 >>> from source code. After compilation and installation, I tried >>> to load the "lustre" module, but encountered the following >>> error: >>> >>> # modprobe lustre >>> could not load module 'lustre': no such device >>> >>> My procedure of installation is the following: >>> >>> 1. Compile vanilla kernel 3.12.72 downloaded from: >>> https://mirrors.edge.kernel.org/pub/linux/kernel/v3.x/linux-3.12.72.tar.gz >>> >>> 2. Compile spl-0.7.11 downloaded from: >>> >>> https://github.com/zfsonlinux/zfs/releases/download/zfs-0.7.11/spl-0.7.11.tar.gz >>> >>> with the following steps: >>> # ./configure --prefix=/opt/lustre --with-linux=/path/to/linux-3.12.72 >>> # make >>> # make install >>> >>> 3. Compile zfs-0.7.11 downloaded from: >>> >>> https://github.com/zfsonlinux/zfs/releases/download/zfs-0.7.11/zfs-0.7.11.tar.gz >>> >>> with the following steps: >>> # ./configure --prefix=/opt/lustre \ >>> --with-linux=/path/to/linux-3.12.72 \ >>> --with-spl=/path/to/spl-0.7.11 >>> # make >>> # make install >>> >>> 4. Compile lustre downloaded from: >>> >>> https://downloads.whamcloud.com/public/lustre/lustre-2.10.5/sles12sp3/client/SRPMS/lustre-2.10.5-1.src.rpm >>> >>> Then I unpack the SRPM by the command: >>> # rpm2cpio lustre-2.10.5-1.src.rpm | cpio --extract --make-directories >>> >>> and compile it by the following: >>> # ./configure --prefix=/opt/lustre \ >>> --with-linux=/path/to/linux-3.12.72 \ >>> --with-spl=/path/to/spl-0.7.11 \ >>> --with-zfs=/path/to/zfs-0.7.11 \ >>> --with-o2ib=no \ >>> --disable-ldiskfs >>> # make >>> # make install >>> >>> 5. I have made sure the following settings and utilities are correct: >>> - PATH contains /opt/lustre/bin and /opt/lustre/sbin >>> - /sbin/mount.lustre exists. >>> - /sbin/mount.zfs exists. >>> - /usr/sbin/l_getidentity exists. >>> - /usr/sbin/ko2iblnd-probe exists. >>> - /etc/modprobe.d/lustre.conf contains: >>> options lnet networks=tcp >>> - /etc/modprobe.d/ko2iblnd.conf contains: >>> alias ko2iblnd-opa ko2iblnd >>> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 >>> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 >>> fmr_flush_trigger=512 fmr_cache=1 >>> install ko2iblnd /usr/sbin/ko2iblnd-probe >>> >>> Then I tried to run "modprobe lustre", it says "no such device" error. >>> >>> I tried to replace Lustre-2.10.5 by Lustre-2.9 downloaded from: >>> >>> https://downloads.whamcloud.com/public/lustre/lustre-2.9.0/sles12sp1/client/SRPMS/lustre-2.9.0-1.src.rpm >>> >>> and proceed exactly the same installation steps. Everything works fine. >>> >>> Could anyone suggest me what have I missed for lustre-2.10.5 ? Or suggest >>> me how to debug. >>> >>> Thanks very much. >>> >>> >>> T.H.Hsieh >>> _______________________________________________ >>> lustre-discuss mailing list >>> [email protected] >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>> >> _______________________________________________ >> lustre-discuss mailing list >> [email protected] >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > _______________________________________________ > lustre-discuss mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Cheers, Andreas --- Andreas Dilger CTO Whamcloud
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
