Re: [lustre-discuss] Centos 7.7 upgrade
On Tue, 2 Jun 2020, Alastair Basden wrote: Thanks - the problem turned out to be the kernel-devel module. Just what was wrong with kernel-devel? I am having the same problem, and there is nothing amiss that I can spot. Thanks. Steve ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Centos 7.7 upgrade
Hi all, Thanks - the problem turned out to be the kernel-devel module. Cheers, Alastair. On Tue, 2 Jun 2020, Pascal Suter wrote: Hi where you using the rpms from the whamcloud repo? if so, check if you have installed the kmod-lustre-osd-zfs and lustre-osd-zfs-mount packages. IIRC i had the same errors when the kmod-lustre-osd-zfs package was missing on my system. cheers Pascal On 6/2/20 1:20 AM, Alastair Basden wrote: Hi, We have just upgraded Lustre servers from 2.12.2 on centos 7.6 to 2.12.3 on centos 7.7. The OSSs are on top of zfs (0.7.13 as recommended), and we are using 3.10.0-1062.1.1.el7_lustre.x86_64 After the update, Lustre will no longer mount - and messages such as: Jun 2 00:02:44 hostname kernel: LustreError: 158-c: Can't load module 'osd-zfs' Jun 2 00:02:44 hostname kernel: LustreError: Skipped 875 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) OBD: unknown type: osd-zfs Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Cannot create device lustfs-OST0006-osd of type osd-zfs : -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) lustfs-OST0006-osd attach error -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Unable to start osd on lustfs-ost6/ost6: -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-19) Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) Skipped 887 previous similar messages Does anyone have any ideas? Thanks, Alastair. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Centos 7.7 upgrade
Hi where you using the rpms from the whamcloud repo? if so, check if you have installed the kmod-lustre-osd-zfs and lustre-osd-zfs-mount packages. IIRC i had the same errors when the kmod-lustre-osd-zfs package was missing on my system. cheers Pascal On 6/2/20 1:20 AM, Alastair Basden wrote: Hi, We have just upgraded Lustre servers from 2.12.2 on centos 7.6 to 2.12.3 on centos 7.7. The OSSs are on top of zfs (0.7.13 as recommended), and we are using 3.10.0-1062.1.1.el7_lustre.x86_64 After the update, Lustre will no longer mount - and messages such as: Jun 2 00:02:44 hostname kernel: LustreError: 158-c: Can't load module 'osd-zfs' Jun 2 00:02:44 hostname kernel: LustreError: Skipped 875 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) OBD: unknown type: osd-zfs Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Cannot create device lustfs-OST0006-osd of type osd-zfs : -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) lustfs-OST0006-osd attach error -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Unable to start osd on lustfs-ost6/ost6: -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-19) Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) Skipped 887 previous similar messages Does anyone have any ideas? Thanks, Alastair. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Centos 7.7 upgrade
Hi Jeff, Yes zfs modules are there - at least, I can see the zpools okay. But, dkms status reports: lustre-zfs, 2.12.3, 3.10.0-957.10.1.el7_lustre.x86_64, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) spl, 0.7.13, 3.10.0-1062.9.1.el7.x86_64, x86_64: installed spl, 0.7.13, 3.10.0-957.10.1.el7_lustre.x86_64, x86_64: installed zfs, 0.7.13, 3.10.0-1062.9.1.el7.x86_64, x86_64: installed zfs, 0.7.13, 3.10.0-957.10.1.el7_lustre.x86_64, x86_64: installed So it would seem there might be problems somehwere (though I've just checked on an older 7.6 system which is working, and that also gives the same warnings). Seems to be a problem with the osd-zfs module not loading. Cheers, Alastair. On Mon, 1 Jun 2020, Jeff Johnson wrote: Alastair, Are you sure you have functioning ZFS modules for that kernel, and that they are loaded? Are you able to see your zpools? Did you use DKMS for either ZFS, Lustre or both? If so, what does `dkms status` report? --Jeff On Mon, Jun 1, 2020 at 4:21 PM Alastair Basden wrote: Hi, We have just upgraded Lustre servers from 2.12.2 on centos 7.6 to 2.12.3 on centos 7.7. The OSSs are on top of zfs (0.7.13 as recommended), and we are using 3.10.0-1062.1.1.el7_lustre.x86_64 After the update, Lustre will no longer mount - and messages such as: Jun 2 00:02:44 hostname kernel: LustreError: 158-c: Can't load module 'osd-zfs' Jun 2 00:02:44 hostname kernel: LustreError: Skipped 875 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) OBD: unknown type: osd-zfs Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Cannot create device lustfs-OST0006-osd of type osd-zfs : -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) lustfs-OST0006-osd attach error -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Unable to start osd on lustfs-ost6/ost6: -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-19) Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) Skipped 887 previous similar messages Does anyone have any ideas? Thanks, Alastair. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Centos 7.7 upgrade
Alastair, Are you sure you have functioning ZFS modules for that kernel, and that they are loaded? Are you able to see your zpools? Did you use DKMS for either ZFS, Lustre or both? If so, what does `dkms status` report? --Jeff On Mon, Jun 1, 2020 at 4:21 PM Alastair Basden wrote: > Hi, > > We have just upgraded Lustre servers from 2.12.2 on centos 7.6 to 2.12.3 > on centos 7.7. > > The OSSs are on top of zfs (0.7.13 as recommended), and we are using > 3.10.0-1062.1.1.el7_lustre.x86_64 > > After the update, Lustre will no longer mount - and messages such as: > Jun 2 00:02:44 hostname kernel: LustreError: 158-c: Can't load module > 'osd-zfs' > Jun 2 00:02:44 hostname kernel: LustreError: Skipped 875 previous similar > messages > Jun 2 00:02:44 hostname kernel: LustreError: > 226253:0:(genops.c:397:class_newdev()) OBD: unknown type: osd-zfs > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_config.c:403:class_attach()) Cannot create device > lustfs-OST0006-osd of type osd-zfs : -19 > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_config.c:403:class_attach()) Skipped 881 previous similar > messages > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount.c:197:lustre_start_simple()) lustfs-OST0006-osd attach > error -19 > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount.c:197:lustre_start_simple()) Skipped 881 previous > similar messages > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount_server.c:1947:server_fill_super()) Unable to start osd > on lustfs-ost6/ost6: -19 > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount_server.c:1947:server_fill_super()) Skipped 881 previous > similar messages > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-19) > Jun 2 00:02:44 hostname kernel: LustreError: > 226265:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 881 previous > similar messages > Jun 2 00:02:44 hostname kernel: LustreError: > 226253:0:(genops.c:397:class_newdev()) Skipped 887 previous similar messages > > Does anyone have any ideas? > > Thanks, > Alastair. > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Centos 7.7 upgrade
Hi, We have just upgraded Lustre servers from 2.12.2 on centos 7.6 to 2.12.3 on centos 7.7. The OSSs are on top of zfs (0.7.13 as recommended), and we are using 3.10.0-1062.1.1.el7_lustre.x86_64 After the update, Lustre will no longer mount - and messages such as: Jun 2 00:02:44 hostname kernel: LustreError: 158-c: Can't load module 'osd-zfs' Jun 2 00:02:44 hostname kernel: LustreError: Skipped 875 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) OBD: unknown type: osd-zfs Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Cannot create device lustfs-OST0006-osd of type osd-zfs : -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_config.c:403:class_attach()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) lustfs-OST0006-osd attach error -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:197:lustre_start_simple()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Unable to start osd on lustfs-ost6/ost6: -19 Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount_server.c:1947:server_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-19) Jun 2 00:02:44 hostname kernel: LustreError: 226265:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 881 previous similar messages Jun 2 00:02:44 hostname kernel: LustreError: 226253:0:(genops.c:397:class_newdev()) Skipped 887 previous similar messages Does anyone have any ideas? Thanks, Alastair. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
