Hi Steffen, If you just see the leftover pending-action on one node it would be interesting if restarting of pacemaker on one of the other nodes does sync it to all of the nodes.
Regards, Klaus On 1/7/21 9:54 AM, renayama19661...@ybb.ne.jp wrote: > Hi Steffen, > >> Unfortunately not sure about the exact scenario. But I have been doing >> some recent experiments with node standby/unstandby stop/start. This >> to get procedures right for updating node rpms etc. >> >> Later I noticed the uncomforting "pending fencing actions" status msg. > Okay! > > Repeat the standby and unstandby steps in the same way to check. > We will start checking after tomorrow, so I think it will take some time > until next week. > > > Many thanks, > Hideo Yamauchi. > > > > ----- Original Message ----- >> From: "renayama19661...@ybb.ne.jp" <renayama19661...@ybb.ne.jp> >> To: Reid Wahl <nw...@redhat.com>; Cluster Labs - All topics related to >> open-source clustering welcomed <users@clusterlabs.org> >> Cc: >> Date: 2021/1/7, Thu 17:51 >> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status >> >> Hi Steffen, >> Hi Reid, >> >> The fencing history is kept inside stonith-ng and is not written to cib. >> However, getting the entire cib and getting it sent will help you to >> reproduce >> the problem. >> >> Best Regards, >> Hideo Yamauchi. >> >> >> ----- Original Message ----- >>> From: Reid Wahl <nw...@redhat.com> >>> To: renayama19661...@ybb.ne.jp; Cluster Labs - All topics related to >> open-source clustering welcomed <users@clusterlabs.org> >>> Date: 2021/1/7, Thu 17:39 >>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status >>> >>> >>> Hi, Steffen. Those attachments don't contain the CIB. They contain the >> `pcs config` output. You can get the cib with `pcs cluster cib > >> $(hostname).cib.xml`. >>> >>> Granted, it's possible that this fence action information wouldn't >> be in the CIB at all. It might be stored in fencer memory. >>> >>> On Thu, Jan 7, 2021 at 12:26 AM <renayama19661...@ybb.ne.jp> wrote: >>> >>> Hi Steffen, >>>>> Here CIB settings attached (pcs config show) for all 3 of my nodes >>>>> (all 3 seems 100% identical), node03 is the DC. >>>> >>>> Thank you for the attachment. >>>> >>>> What is the scenario when this situation occurs? >>>> In what steps did the problem appear when fencing was performed (or >> failed)? >>>> >>>> Best Regards, >>>> Hideo Yamauchi. >>>> >>>> >>>> ----- Original Message ----- >>>>> From: Steffen Vinther Sørensen <svint...@gmail.com> >>>>> To: renayama19661...@ybb.ne.jp; Cluster Labs - All topics related >> to open-source clustering welcomed <users@clusterlabs.org> >>>>> Cc: >>>>> Date: 2021/1/7, Thu 17:05 >>>>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs >> status >>>>> Hi Hideo, >>>>> >>>>> Here CIB settings attached (pcs config show) for all 3 of my nodes >>>>> (all 3 seems 100% identical), node03 is the DC. >>>>> >>>>> Regards >>>>> Steffen >>>>> >>>>> On Thu, Jan 7, 2021 at 8:06 AM <renayama19661...@ybb.ne.jp> >> wrote: >>>>>> Hi Steffen, >>>>>> Hi Reid, >>>>>> >>>>>> I also checked the Centos source rpm and it seems to include a >> fix for the >>>>> problem. >>>>>> As Steffen suggested, if you share your CIB settings, I might >> know >>>>> something. >>>>>> If this issue is the same as the fix, the display will only be >> displayed on >>>>> the DC node and will not affect the operation. >>>>>> The pending actions shown will remain for a long time, but >> will not have a >>>>> negative impact on the cluster. >>>>>> Best Regards, >>>>>> Hideo Yamauchi. >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> > From: Reid Wahl <nw...@redhat.com> >>>>>> > To: Cluster Labs - All topics related to open-source >> clustering >>>>> welcomed <users@clusterlabs.org> >>>>>> > Cc: >>>>>> > Date: 2021/1/7, Thu 15:58 >>>>>> > Subject: Re: [ClusterLabs] Pending Fencing Actions shown >> in pcs status >>>>>> > >>>>>> > It's supposedly fixed in that version. >>>>>> > - https://bugzilla.redhat.com/show_bug.cgi?id=1787749 >>>>>> > - https://access.redhat.com/solutions/4713471 >>>>>> > >>>>>> > So you may be hitting a different issue (unless >> there's a bug in >>>>> the >>>>>> > pcmk 1.1 backport of the fix). >>>>>> > >>>>>> > I may be a little bit out of my area of knowledge here, >> but can you >>>>>> > share the CIBs from nodes 1 and 3? Maybe Hideo, Klaus, or >> Ken has some >>>>>> > insight. >>>>>> > >>>>>> > On Wed, Jan 6, 2021 at 10:53 PM Steffen Vinther Sørensen >>>>>> > <svint...@gmail.com> wrote: >>>>>> >> >>>>>> >> Hi Hideo, >>>>>> >> >>>>>> >> If the fix is not going to make it into the CentOS7 >> pacemaker >>>>> version, >>>>>> >> I guess the stable approach to take advantage of it >> is to build >>>>> the >>>>>> >> cluster on another OS than CentOS7 ? A little late >> for that in >>>>> this >>>>>> >> case though :) >>>>>> >> >>>>>> >> Regards >>>>>> >> Steffen >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Thu, Jan 7, 2021 at 7:27 AM >> <renayama19661...@ybb.ne.jp> >>>>> wrote: >>>>>> >> > >>>>>> >> > Hi Steffen, >>>>>> >> > >>>>>> >> > The fix pointed out by Reid is affecting it. >>>>>> >> > >>>>>> >> > Since the fencing action requested by the DC >> node exists >>>>> only in the >>>>>> > DC node, such an event occurs. >>>>>> >> > You will need to take advantage of the modified >> pacemaker to >>>>> resolve >>>>>> > the issue. >>>>>> >> > >>>>>> >> > Best Regards, >>>>>> >> > Hideo Yamauchi. >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > ----- Original Message ----- >>>>>> >> > > From: Reid Wahl <nw...@redhat.com> >>>>>> >> > > To: Cluster Labs - All topics related to >> open-source >>>>> clustering >>>>>> > welcomed <users@clusterlabs.org> >>>>>> >> > > Cc: >>>>>> >> > > Date: 2021/1/7, Thu 15:07 >>>>>> >> > > Subject: Re: [ClusterLabs] Pending Fencing >> Actions >>>>> shown in pcs >>>>>> > status >>>>>> >> > > >>>>>> >> > > Hi, Steffen. Are your cluster nodes all >> running the >>>>> same >>>>>> > Pacemaker >>>>>> >> > > versions? This looks like Bug 5401[1], >> which is fixed >>>>> by upstream >>>>>> >> > > commit df71a07[2]. I'm a little bit >> confused about >>>>> why it >>>>>> > only shows >>>>>> >> > > up on one out of three nodes though. >>>>>> >> > > >>>>>> >> > > [1] >> https://bugs.clusterlabs.org/show_bug.cgi?id=5401 >>>>>> >> > > [2] >>>>> https://github.com/ClusterLabs/pacemaker/commit/df71a07 >>>>>> >> > > >>>>>> >> > > On Tue, Jan 5, 2021 at 8:31 AM Steffen >> Vinther Sørensen >>>>>> >> > > <svint...@gmail.com> wrote: >>>>>> >> > >> >>>>>> >> > >> Hello >>>>>> >> > >> >>>>>> >> > >> node 1 is showing this in 'pcs >> status' >>>>>> >> > >> >>>>>> >> > >> Pending Fencing Actions: >>>>>> >> > >> * reboot of >> kvm03-node02.avigol-gcs.dk pending: >>>>>> > client=crmd.37819, >>>>>> >> > >> origin=kvm03-node03.avigol-gcs.dk >>>>>> >> > >> >>>>>> >> > >> node 2 and node 3 outputs no such >> thing (node 3 is >>>>> DC) >>>>>> >> > >> >>>>>> >> > >> Google is not much help, how to >> investigate this >>>>> further and >>>>>> > get rid >>>>>> >> > >> of such terrifying status message ? >>>>>> >> > >> >>>>>> >> > >> Regards >>>>>> >> > >> Steffen >>>>>> >> > >> >> _______________________________________________ >>>>>> >> > >> Manage your subscription: >>>>>> >> > >> >>>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >> > >> >>>>>> >> > >> ClusterLabs home: >> https://www.clusterlabs.org/ >>>>>> >> > >> >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > -- >>>>>> >> > > Regards, >>>>>> >> > > >>>>>> >> > > Reid Wahl, RHCA >>>>>> >> > > Senior Software Maintenance Engineer, Red >> Hat >>>>>> >> > > CEE - Platform Support Delivery - >> ClusterHA >>>>>> >> > > >>>>>> >> > > >> _______________________________________________ >>>>>> >> > > Manage your subscription: >>>>>> >> > > >> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >> > > >>>>>> >> > > ClusterLabs home: >> https://www.clusterlabs.org/ >>>>>> >> > > >>>>>> >> > >>>>>> >> > _______________________________________________ >>>>>> >> > Manage your subscription: >>>>>> >> > >> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >> > >>>>>> >> > ClusterLabs home: https://www.clusterlabs.org/ >>>>>> >> _______________________________________________ >>>>>> >> Manage your subscription: >>>>>> >> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >> >>>>>> >> ClusterLabs home: https://www.clusterlabs.org/ >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Regards, >>>>>> > >>>>>> > Reid Wahl, RHCA >>>>>> > Senior Software Maintenance Engineer, Red Hat >>>>>> > CEE - Platform Support Delivery - ClusterHA >>>>>> > >>>>>> > _______________________________________________ >>>>>> > Manage your subscription: >>>>>> > https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> > >>>>>> > ClusterLabs home: https://www.clusterlabs.org/ >>>>>> > >>>>>> >>>>>> _______________________________________________ >>>>>> Manage your subscription: >>>>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >>>>>> ClusterLabs home: https://www.clusterlabs.org/ >>>> _______________________________________________ >>>> Manage your subscription: >>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>> >>>> ClusterLabs home: https://www.clusterlabs.org/ >>>> >>> -- >>> >>> Regards, >>> >>> Reid Wahl, RHCA >>> >>> Senior Software Maintenance Engineer, Red Hat >>> CEE - Platform Support Delivery - ClusterHA >>> >>> >> _______________________________________________ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ >> > _______________________________________________ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
