Re: [lustre-discuss] Repeated ZFS panics on MDT

2023-03-15 Thread Mountford, Christopher J. (Dr.) via lustre-discuss
The ZFS scrub completed without any errors/corrections. Following the scrub (and clearing all users from our cluster login nodes) I remounted the MDT and it appears to be running fine (just running the remaining batch jobs). I'm now able to get onto our monitoring system - hopefully a look at

Re: [lustre-discuss] Node Failure in Lustre

2023-03-15 Thread Laura Hild via lustre-discuss
Hi Nick- If there is no MDS/MGS/OSS currently hosting a particular MDT/MGT/OST, then what is stored there will not be accessible. I suggest looking at https://doc.lustre.org/lustre_manual.xhtml#lustrerecovery -Laura ___ lustre-discuss mailing

Re: [lustre-discuss] Configuring LustreFS Over DRBD

2023-03-15 Thread Laura Hild via lustre-discuss
Hi Shambhu- I believe neither the ldiskfs nor ZFS OSDs support an active-active configuration (except in the sense that one can have multiple targets, some of which are active on one host and others on another). It seems reasonable to me, only having used DRBD and Lustre independently of each

Re: [lustre-discuss] Repeated ZFS panics on MDT

2023-03-15 Thread Mountford, Christopher J. (Dr.) via lustre-discuss
Hi Colin, Not yet, we last scrubbed the pool ~2 weeks ago when we first saw this problem. I've got a few additional tests to run now to see if we can track the cause to a particular job/process, but kicking off a scrub is my next thing to do (It should only take ~40 minutes, it's a fairly

Re: [lustre-discuss] Repeated ZFS panics on MDT

2023-03-15 Thread Colin Faber via lustre-discuss
Have you tried resilvering the pool? On Wed, Mar 15, 2023, 11:57 AM Mountford, Christopher J. (Dr.) via lustre-discuss wrote: > I'm hoping someone offer some suggestions. > > We have a problem on our production Lustre/ZFS filesystem (CentOS 7, ZFS > 0.7.13, Lustre 2.12.9), so far I've drawn a

[lustre-discuss] Repeated ZFS panics on MDT

2023-03-15 Thread Mountford, Christopher J. (Dr.) via lustre-discuss
I'm hoping someone offer some suggestions. We have a problem on our production Lustre/ZFS filesystem (CentOS 7, ZFS 0.7.13, Lustre 2.12.9), so far I've drawn a blank trying to track down the cause of this. We see the following zfs panic message in the logs (in every case the VERIFY3/panic

[lustre-discuss] Configuring LustreFS Over DRBD

2023-03-15 Thread Shambhu Raje via lustre-discuss
I am trying to configure a clustered file system over DRBD software so that if we mount a file system just like LustreFS over DRBD set -up in dual primary mode it can provide us with the real time replication of data . Can I configure lustre file system over DRBD in redhat 8.7 ... If yes, how

Re: [lustre-discuss] Node Failure in Lustre

2023-03-15 Thread Nick dan via lustre-discuss
Hi Okay. Thank you for the information Can you tell if the MDS/MGS or the OSS server goes down, how will the failure be handled on Lustre level? On Wed, 15 Mar 2023 at 13:45, Andreas Dilger wrote: > No, because the remote-attached SSDs are part of the ZFS pool and any > drive failures a t that

Re: [lustre-discuss] Node Failure in Lustre

2023-03-15 Thread Andreas Dilger via lustre-discuss
No, because the remote-attached SSDs are part of the ZFS pool and any drive failures a t that level are the responsibility of ZFS in that case to manage the failed drives (eg. with RAID) and for you to have system monitors in place to detect this case and alert you to the drive failures. This

[lustre-discuss] Node Failure in Lustre

2023-03-15 Thread Nick dan via lustre-discuss
Hi There is a situation where disks from multiple servers are sent to a main server.(Lustre storage) Zpool is created from the SSDs and mkfs.lustre is done using zfs as a backend file system. Lustre client is also connected. If one of the nodes from where the SSDs are sent goes down, will the