Could you provide the meta data contents of the affected resource ("drbdadm dump-md") along with resource configuration contents? This might help in giving some answers...
On Mon, 20 Sep 2021 at 07:03, Michael Labriola <michael.d.labri...@gmail.com> wrote: > I've just had a resource come up Diskless after an accidental > shutdown. It wasn't a power loss situation, I literally executed > poweroff on the wrong machine via ssh. Trying to raise this one > particular resource (the other 11 resources were fine) spits out a ton > of this: > > extent 4765184 beyond end of bitmap! > extent 5236736 beyond end of bitmap! > extent 5246976 beyond end of bitmap! > extent 5247485 beyond end of bitmap! > extent 5249536 beyond end of bitmap! > extent 5249664 beyond end of bitmap! > extent 5249665 beyond end of bitmap! > extent 5249666 beyond end of bitmap! > extent 5249667 beyond end of bitmap! > extent 5249668 beyond end of bitmap! > extent 5249669 beyond end of bitmap! > extent 5249670 beyond end of bitmap! > extent 5249671 beyond end of bitmap! > extent 5249672 beyond end of bitmap! > extent 5249673 beyond end of bitmap! > ... > extent 6781952 beyond end of bitmap! > extent 6782464 beyond end of bitmap! > extent 6782976 beyond end of bitmap! > ../shared/drbdmeta.c:1996:apply_al: ASSERT(bm_pos - bm_on_disk_pos <= > chunk - this_extent_size) failed. > > The hardware is all good, the underlying block device is mountable, > and I can still get all my data. I just can't attach the disk to the > resource anymore. > > My question is twofold: > > 1. Is there any way to recover this short of re-creating the drbd > metadata and doing a full sync? That's the most obvious course of > action, but it's a 50T resource and I don't want to wait 4 days for it > to finish if I don't have to. > > 2. Help me understand what happened, so I can make sure it doesn't > happen again. My resource's metadata got corrupted somehow, right? > Did I have data being written to my resource when it shut down that > didn't make it all the way to disk, specifically in the meta data? > Even though it was a "safe shutdown"? > > Both my servers are running Ubuntu 20.04.2 w/ DRBD9 (module 9.0.29, > utils 9.18.0) > > Thanks! > > -- > Michael D Labriola > 21 Rip Van Winkle Cir > Warwick, RI 02886 > 401-316-9844 (cell) > _______________________________________________ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > drbd-user@lists.linbit.com > https://lists.linbit.com/mailman/listinfo/drbd-user >
_______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user