Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On Wed, Aug 14, 2019 at 09:53:21AM +0200, Jamie wrote: > Hm, I'm taking a wild guess here and say that I will also have to upgrade to > drbd 9 on the server where my secondary is placed? Or would that not be a > problem, just upgrading on my computer and then not upgrading the server. > The problem here ist, that we're five people running our primary virtual > machines on our computers and have only one server where the secondary vms > are kept. Would we all need to upgrade, or is it just enough that I upgrade > the drbd on my computer now and wait with upgrading the server until > everyone has upgraded their drbd? Do you guys have any experience with that? I would switch all at the same time. 8.4 to 9.0 was anticipated to be a thing, but it never was. Customers switching to 9 started from fresh or created a 9 resource and copied the data. Connecting 8.4 to 9.0 should be possible, but I'd say it is pretty untested. I'd try to avoid it. Best, rck ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On Tue, Aug 13, 2019 at 06:34:44PM +0100, Eddie Chapman wrote: > On 13/08/2019 16:22, Jamie wrote: > > On Mon, Aug 13, 2019 at 17:17:31, Jamie wrote: > > > > > Sorry, I tried posting this message this morning and something > > > apparently didn't work out correctly... :( > > > > Okay, consider me stupid...? Somehow, I couldn't see my post from this > > morning until just a moment ago. I apologize for the double posting. > > *heads desk* > > I think it was just queued waiting for moderator approval, as I also didn't > get your original message from the until about 4.5 hrs after you sent it. Correct. That depends a bit on who of us does the moderation. I usually clear the moderation flag after some time when the name feels familiar and I know the person isn't a spam bot... Regards, rck ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On 13/08/2019 16:22, Jamie wrote: On Mon, Aug 13, 2019 at 17:17:31, Jamie wrote: Sorry, I tried posting this message this morning and something apparently didn't work out correctly... :( Okay, consider me stupid...? Somehow, I couldn't see my post from this morning until just a moment ago. I apologize for the double posting. *heads desk* I think it was just queued waiting for moderator approval, as I also didn't get your original message from the until about 4.5 hrs after you sent it. Anyway, I believe the issue Roland was referring to is being discussed on the Linux kernel mailing list at the moment, I spotted it the other day when reading the stable releases list. https://www.spinics.net/lists/stable/msg319730.html Christoph Böhmwalder from LINBIT has posted a proposed fix for the issue, which is basically a regression in drbd mainline which was accidentally introduced by a tree-wide code change, and which you and others appear to have been bitten by. The thing is, Christoph and Roland are just now in the process of getting their proposed fix (which isn't a simple revert as it was a tree-wide change) accepted by the block subsytem maintainer, and then it will end up in Linus Torvald's tree once the maintainer submits his next batch of fixes to Linus. Once that happens, you have to wait for it to eventually be backported by someone and accepted into one of the frequent upstream kernel stable releases (the rule is all stable patches must exist in Linus' tree first), and then hopefully Fedora will pick it up and apply it to their kernel. Once Fedora has released an updated kernel you would then finally see the fix when you install that. Personally I would just upgrade to drbd 9 and then you don't have to wait for the above process :-) (as the issue only affects drbd 8) ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On 13/08/2019 13:16, Roland Kammerer wrote: On Tue, Aug 13, 2019 at 12:40:56PM +0100, Eddie Chapman wrote: I interpreted the package "drbd.x86_64 - version 9.5.0-2.fc30" as being the kernel module version 9.5, since what else is left after you've packages the utilities and udev scripts? There's only the kernel component. Without reading the rest: no. There is no such kmod version. And most likely - I never looked at the Fedora packages - it is meta package that pulls in the user space. We (LINBIT) also provide a meta package named like this :-/ I don't like it, it confuses people. And here we have one more prove. I still assume the kmod comes from your kernel. Again, cat /proc/drbd. If that shows 9.5, then allow me to join your next time travel... Regards, rck Ha ha ha, goodness me, yes I completely forgot the kernel component versions numbers are 9.0.x not 9.x. The *utilities* are 9.x. For some reason my mind has almost completely stopped seeing the 0 in the middle for the kernel module after all these years. :-) A couple mails back I even referred to the kernel module as both 9.x and 9.0.x in the same sentence, I really am losing it! Apologies if I might have led anyone else into insanity with me as well :-) ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On 13/08/2019 07:12, Roland Kammerer wrote: On Mon, Aug 12, 2019 at 12:00:40PM +0200, Jamie wrote: Hi all, I've encountered quite a problem after updating my Fedora 30 and I hope someone might've come across this problem as well because I couldn't really find a lot of information online. I'm using the following: * drbd.x86_64 - version 9.5.0-2.fc30 * drbd-udev.x86_64 - version 9.5.0-2.fc30 * drbd-utils.x86_64 - version 9.5.0-2.fc30 After updating Fedora 30 from kernel version *5.1.19-300*.fc30.x86_64 to kernel version *5.2.5-200*.fc30.x86_64 the drbd service won't even start anymore, giving me the following output (names are anonymized): This is just utils and udev and stuff, but not the actual kmod, right? The kmod comes from upstream, is that correct? (cat /proc/drbd). If so, this would explain it. Unfortunately in kernel DRBD broke between 5.1 and 5.2. There are already discussions how to fix it. Regards, rck Hi Roland, I interpreted the package "drbd.x86_64 - version 9.5.0-2.fc30" as being the kernel module version 9.5, since what else is left after you've packages the utilities and udev scripts? There's only the kernel component. But my curiosity got the better of me and, in fact, drbd.x86_64 isn't anything at all. It's just a metapackage containing just 2 files (COPYING and ChangeLog): https://fedora.pkgs.org/30/fedora-x86_64/drbd-9.5.0-2.fc30.x86_64.rpm.html The kernel module drbd.ko is in the kernel-core-5.0.9-301.fc30.x86_64.rpm package: https://fedora.pkgs.org/30/fedora-x86_64/kernel-core-5.0.9-301.fc30.x86_64.rpm.html So as the drbd kernel module is in the main kernel package then, yes, as you suspect it has to be drbd8, certainly, as drbd9 is not upstream yet. And yes I think you're right the OPs problem is likely the 5.0 to 5.1 bug being discussed on the kernel.org stable list at the moment. Surprises me, as Fedora is considered a fairly "bleeding edge" distro, that ironically as a user the default position is you end up running ancient drbd8. "Yes please, give me the very latest GCC 9, glibc 2.29, kernel 5.x, bash 5.0 .. and .. ok I'll take drbd8" :-) I know, joking aside, Fedora is just sticking with what comes with the upstream kernel, whatever that maybe. But it is a strange position when you think about it. They're taking the utilities from Linbit and packaging them, so why not just take the kernel component directly from Linbit too? Then you get the current version 9 as well, which isn't even remotely "bleeding edge" by Fedora's standards. To the OP, my personal suggestion: you've chosen Fedora with all its shiny new software versions (a good choice I think), so blacklist the bundled and ancient drbd.ko and install drbd9, go on, do it :-) Seriously, drbd9 is awesome, you'll never look back. It's very easy to install from source. In fact Linbit probably has rpm packages, so go ahead and uninstall completely the (also ancient by the looks of things) drbd utilities rpms, and install the utilities from Linbit too. As a bonus you will also resolve your current problem, which is a drbd8 only bug. Don't worry, you don't need to "re-do" all your data, you can use the existing metadata. Just follow whatever is in Linbit's docs about upgrading from 8 to 9. Regards, Eddie ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
On Mon, Aug 12, 2019 at 12:00:40PM +0200, Jamie wrote: > Hi all, > > I've encountered quite a problem after updating my Fedora 30 and I hope > someone might've come across this problem as well because I couldn't really > find a lot of information online. > > I'm using the following: > > * drbd.x86_64 - version 9.5.0-2.fc30 > * drbd-udev.x86_64 - version 9.5.0-2.fc30 > * drbd-utils.x86_64 - version 9.5.0-2.fc30 > > After updating Fedora 30 from kernel version *5.1.19-300*.fc30.x86_64 to > kernel version *5.2.5-200*.fc30.x86_64 the drbd service won't even start > anymore, giving me the following output (names are anonymized): This is just utils and udev and stuff, but not the actual kmod, right? The kmod comes from upstream, is that correct? (cat /proc/drbd). If so, this would explain it. Unfortunately in kernel DRBD broke between 5.1 and 5.2. There are already discussions how to fix it. Regards, rck ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Failed starting drbd after updating Fedora 30
Hello Jamie, On 12/08/2019 11:00, Jamie wrote: //Aug 12 11:28:34 ///serverjm/ drbdadm[8142]: Command 'drbdsetup-84 new-resource VMNAME' terminated with exit code 20// I'm not familiar in the slightest with how Fedora packages drbd in your installation. However the above line strongly suggests to me that the init script that is being run by systemd is a drbd8 script, whereas the packages you have installed now appear to be drbd9 (9.5.0-2.fc30). So evidently upgrading Fedora for you has involved also upgrading from drbd8 to drbd9. (By the way, do those Fedora package version numbers correspond to drbd upstream release numbers? If so drbd kernel version 9.5 is quite old and lots has been fixed since then. Again I know nothing about how the distros package drbd these days as I install from source, but you should be running the latest 9.0.19-1 kernel version if you care about your data at all, and especially since Fedora ships a very recent kernel version.) There are some changes needed to your drbd configuration files when upgrading from 8 to 9. So my suggestion would be (if you really are now running drbd9 as you appear to be doing) that you go though those configuration files and make the those changes (check the drbd9 docs for what is needed), and then perhaps remove the existing v8 style init script from systemd and re-add the new v9 one that comes with your 9.5.0-2.fc30 packages. regards, Eddie ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
