Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 06/25/2018 12:01 PM, 井上 和徳 wrote:
>> -Original Message-
>> From: Klaus Wenninger [mailto:kwenn...@redhat.com]
>> Sent: Wednesday, June 13, 2018 6:40 PM
>> To: Cluster Labs - All topics related to open-source clustering welcomed; 井上 
>> 和
>> 徳
>> Subject: Re: [ClusterLabs] Questions about SBD behavior
>>
>> On 06/13/2018 10:58 AM, 井上 和徳 wrote:
>>> Thanks for the response.
>>>
>>> As of v1.3.1 and later, I recognized that real quorum is necessary.
>>> I also read this:
>>>
>> https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker#Watchdog-based_self
>> -fencing_with_resource_recovery
>>> As related to this specification, in order to use pacemaker-2.0,
>>> we are confirming the following known issue.
>>>
>>> * When SIGSTOP is sent to the pacemaker process, no failure of the
>>>   resource will be detected.
>>>   https://lists.clusterlabs.org/pipermail/users/2016-September/011146.html
>>>   https://lists.clusterlabs.org/pipermail/users/2016-October/011429.html
>>>
>>>   I expected that it was being handled by SBD, but no one detected
>>>   that the following process was frozen. Therefore, no failure of
>>>   the resource was detected either.
>>>   - pacemaker-based
>>>   - pacemaker-execd
>>>   - pacemaker-attrd
>>>   - pacemaker-schedulerd
>>>   - pacemaker-controld
>>>
>>>   I confirmed this, but I couldn't read about the correspondence
>>>   situation.
>>>
>> https://wiki.clusterlabs.org/w/images/1/1a/Recent_Work_and_Future_Plans_for_SB
>> D_1.1.pdf
>> You are right. The issue was known as when I created these slides.
>> So a plan for improving the observation of the pacemaker-daemons
>> should have gone into that probably.
>>
> It's good news that there is a plan to improve.
> So I registered it as a memorandum in CLBZ:
> https://bugs.clusterlabs.org/show_bug.cgi?id=5356
>
> Best Regards
Wasn't there a bug filed before?

Klaus

>
>> Thanks for bringing this to the table.
>> Guess the issue got a little bit neglected recently.
>>
>>> As a result of our discussion, we want SBD to detect it and reset the
>>> machine.
>> Implementation wise I would go for some kind of a split
>> solution between pacemaker & SBD. Thinking of Pacemaker
>> observing the sub-daemons by itself while there would be
>> some kind of a heartbeat (implicitly via corosync or explicitly)
>> between pacemaker & SBD that assures this internal
>> observation is doing it's job properly.
>>
>>> Also, for users who do not have shared disk or qdevice,
>>> we need an option to work even without real quorum.
>>> (fence races are going to avoid with delay attribute:
>>>  https://access.redhat.com/solutions/91653
>>>  https://access.redhat.com/solutions/1293523)
>> I'm not sure if I get your point here.
>> Watchdog-fencing on a 2-node-cluster without
>> additional qdevice or shared disk is like denying
>> the laws of physics in my mind.
>> At the moment I don't see why auto_tie_breaker
>> wouldn't work on a 4-node and up cluster here.
>>
>> Regards,
>> Klaus
>>> Best Regards,
>>> Kazunori INOUE
>>>
>>>> -Original Message-
>>>> From: Users [mailto:users-boun...@clusterlabs.org] On Behalf Of Klaus 
>>>> Wenninger
>>>> Sent: Friday, May 25, 2018 4:08 PM
>>>> To: users@clusterlabs.org
>>>> Subject: Re: [ClusterLabs] Questions about SBD behavior
>>>>
>>>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>>>>> Hi,
>>>>>
>>>>> I am checking the watchdog function of SBD (without shared block-device).
>>>>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
>>>>> on the
>>>> remaining node.
>>>>> Is this the designed behavior?
>>>> SBD without a shared block-device doesn't really make sense on
>>>> a two-node cluster.
>>>> The basic idea is - e.g. in a case of a networking problem -
>>>> that a cluster splits up in a quorate and a non-quorate partition.
>>>> The quorate partition stays over while SBD guarantees a
>>>> reliable watchdog-based self-fencing of the non-quorate partition
>>>> within a defined timeout.
>>>> This idea of course doesn't work with just 2 nodes.
>>>> Taking quorum info from the 2-node feature of corosync (automatically
>>>> switching on wai

Re: [ClusterLabs] Questions about SBD behavior

[井上 和徳]

> -Original Message-
> From: Klaus Wenninger [mailto:kwenn...@redhat.com]
> Sent: Wednesday, June 13, 2018 6:40 PM
> To: Cluster Labs - All topics related to open-source clustering welcomed; 井上 和
> 徳
> Subject: Re: [ClusterLabs] Questions about SBD behavior
> 
> On 06/13/2018 10:58 AM, 井上 和徳 wrote:
> > Thanks for the response.
> >
> > As of v1.3.1 and later, I recognized that real quorum is necessary.
> > I also read this:
> >
> https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker#Watchdog-based_self
> -fencing_with_resource_recovery
> >
> > As related to this specification, in order to use pacemaker-2.0,
> > we are confirming the following known issue.
> >
> > * When SIGSTOP is sent to the pacemaker process, no failure of the
> >   resource will be detected.
> >   https://lists.clusterlabs.org/pipermail/users/2016-September/011146.html
> >   https://lists.clusterlabs.org/pipermail/users/2016-October/011429.html
> >
> >   I expected that it was being handled by SBD, but no one detected
> >   that the following process was frozen. Therefore, no failure of
> >   the resource was detected either.
> >   - pacemaker-based
> >   - pacemaker-execd
> >   - pacemaker-attrd
> >   - pacemaker-schedulerd
> >   - pacemaker-controld
> >
> >   I confirmed this, but I couldn't read about the correspondence
> >   situation.
> >
> https://wiki.clusterlabs.org/w/images/1/1a/Recent_Work_and_Future_Plans_for_SB
> D_1.1.pdf
> You are right. The issue was known as when I created these slides.
> So a plan for improving the observation of the pacemaker-daemons
> should have gone into that probably.
> 

It's good news that there is a plan to improve.
So I registered it as a memorandum in CLBZ:
https://bugs.clusterlabs.org/show_bug.cgi?id=5356

Best Regards

> Thanks for bringing this to the table.
> Guess the issue got a little bit neglected recently.
> 
> >
> > As a result of our discussion, we want SBD to detect it and reset the
> > machine.
> 
> Implementation wise I would go for some kind of a split
> solution between pacemaker & SBD. Thinking of Pacemaker
> observing the sub-daemons by itself while there would be
> some kind of a heartbeat (implicitly via corosync or explicitly)
> between pacemaker & SBD that assures this internal
> observation is doing it's job properly.
> 
> >
> > Also, for users who do not have shared disk or qdevice,
> > we need an option to work even without real quorum.
> > (fence races are going to avoid with delay attribute:
> >  https://access.redhat.com/solutions/91653
> >  https://access.redhat.com/solutions/1293523)
> I'm not sure if I get your point here.
> Watchdog-fencing on a 2-node-cluster without
> additional qdevice or shared disk is like denying
> the laws of physics in my mind.
> At the moment I don't see why auto_tie_breaker
> wouldn't work on a 4-node and up cluster here.
> 
> Regards,
> Klaus
> >
> > Best Regards,
> > Kazunori INOUE
> >
> >> -Original Message-
> >> From: Users [mailto:users-boun...@clusterlabs.org] On Behalf Of Klaus 
> >> Wenninger
> >> Sent: Friday, May 25, 2018 4:08 PM
> >> To: users@clusterlabs.org
> >> Subject: Re: [ClusterLabs] Questions about SBD behavior
> >>
> >> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
> >>> Hi,
> >>>
> >>> I am checking the watchdog function of SBD (without shared block-device).
> >>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
> >>> on the
> >> remaining node.
> >>> Is this the designed behavior?
> >> SBD without a shared block-device doesn't really make sense on
> >> a two-node cluster.
> >> The basic idea is - e.g. in a case of a networking problem -
> >> that a cluster splits up in a quorate and a non-quorate partition.
> >> The quorate partition stays over while SBD guarantees a
> >> reliable watchdog-based self-fencing of the non-quorate partition
> >> within a defined timeout.
> >> This idea of course doesn't work with just 2 nodes.
> >> Taking quorum info from the 2-node feature of corosync (automatically
> >> switching on wait-for-all) doesn't help in this case but instead
> >> would lead to split-brain.
> >> What you can do - and what e.g. pcs does automatically - is enable
> >> the auto-tie-breaker instead of two-node in corosync. But that
> >> still doesn't give you a higher availability than the one of the
> >> winner of auto-tie-breaker. (Maybe in

Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 06/13/2018 10:58 AM, 井上 和徳 wrote:
> Thanks for the response.
>
> As of v1.3.1 and later, I recognized that real quorum is necessary.
> I also read this:
> https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker#Watchdog-based_self-fencing_with_resource_recovery
>
> As related to this specification, in order to use pacemaker-2.0,
> we are confirming the following known issue.
>
> * When SIGSTOP is sent to the pacemaker process, no failure of the
>   resource will be detected.
>   https://lists.clusterlabs.org/pipermail/users/2016-September/011146.html
>   https://lists.clusterlabs.org/pipermail/users/2016-October/011429.html
>
>   I expected that it was being handled by SBD, but no one detected
>   that the following process was frozen. Therefore, no failure of
>   the resource was detected either.
>   - pacemaker-based
>   - pacemaker-execd
>   - pacemaker-attrd
>   - pacemaker-schedulerd
>   - pacemaker-controld
>
>   I confirmed this, but I couldn't read about the correspondence
>   situation.
>   
> https://wiki.clusterlabs.org/w/images/1/1a/Recent_Work_and_Future_Plans_for_SBD_1.1.pdf
You are right. The issue was known as when I created these slides.
So a plan for improving the observation of the pacemaker-daemons
should have gone into that probably.

Thanks for bringing this to the table.
Guess the issue got a little bit neglected recently.

>
> As a result of our discussion, we want SBD to detect it and reset the
> machine.

Implementation wise I would go for some kind of a split
solution between pacemaker & SBD. Thinking of Pacemaker
observing the sub-daemons by itself while there would be
some kind of a heartbeat (implicitly via corosync or explicitly)
between pacemaker & SBD that assures this internal
observation is doing it's job properly.

>
> Also, for users who do not have shared disk or qdevice,
> we need an option to work even without real quorum.
> (fence races are going to avoid with delay attribute:
>  https://access.redhat.com/solutions/91653
>  https://access.redhat.com/solutions/1293523)
I'm not sure if I get your point here.
Watchdog-fencing on a 2-node-cluster without
additional qdevice or shared disk is like denying
the laws of physics in my mind.
At the moment I don't see why auto_tie_breaker
wouldn't work on a 4-node and up cluster here.

Regards,
Klaus
>
> Best Regards,
> Kazunori INOUE
>
>> -Original Message-
>> From: Users [mailto:users-boun...@clusterlabs.org] On Behalf Of Klaus 
>> Wenninger
>> Sent: Friday, May 25, 2018 4:08 PM
>> To: users@clusterlabs.org
>> Subject: Re: [ClusterLabs] Questions about SBD behavior
>>
>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>>> Hi,
>>>
>>> I am checking the watchdog function of SBD (without shared block-device).
>>> In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
>>> the
>> remaining node.
>>> Is this the designed behavior?
>> SBD without a shared block-device doesn't really make sense on
>> a two-node cluster.
>> The basic idea is - e.g. in a case of a networking problem -
>> that a cluster splits up in a quorate and a non-quorate partition.
>> The quorate partition stays over while SBD guarantees a
>> reliable watchdog-based self-fencing of the non-quorate partition
>> within a defined timeout.
>> This idea of course doesn't work with just 2 nodes.
>> Taking quorum info from the 2-node feature of corosync (automatically
>> switching on wait-for-all) doesn't help in this case but instead
>> would lead to split-brain.
>> What you can do - and what e.g. pcs does automatically - is enable
>> the auto-tie-breaker instead of two-node in corosync. But that
>> still doesn't give you a higher availability than the one of the
>> winner of auto-tie-breaker. (Maybe interesting if you are going
>> for a load-balancing-scenario that doesn't affect availability or
>> for a transient state while setting up a cluste node-by-node ...)
>> What you can do though is using qdevice to still have 'real-quorum'
>> info with just 2 full cluster-nodes.
>>
>> There was quite a lot of discussion round this topic on this
>> thread previously if you search the history.
>>
>> Regards,
>> Klaus
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


 

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[井上 和徳]

Thanks for the response.

As of v1.3.1 and later, I recognized that real quorum is necessary.
I also read this:
https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker#Watchdog-based_self-fencing_with_resource_recovery

As related to this specification, in order to use pacemaker-2.0,
we are confirming the following known issue.

* When SIGSTOP is sent to the pacemaker process, no failure of the
  resource will be detected.
  https://lists.clusterlabs.org/pipermail/users/2016-September/011146.html
  https://lists.clusterlabs.org/pipermail/users/2016-October/011429.html

  I expected that it was being handled by SBD, but no one detected
  that the following process was frozen. Therefore, no failure of
  the resource was detected either.
  - pacemaker-based
  - pacemaker-execd
  - pacemaker-attrd
  - pacemaker-schedulerd
  - pacemaker-controld

  I confirmed this, but I couldn't read about the correspondence
  situation.
  
https://wiki.clusterlabs.org/w/images/1/1a/Recent_Work_and_Future_Plans_for_SBD_1.1.pdf

As a result of our discussion, we want SBD to detect it and reset the
machine.

Also, for users who do not have shared disk or qdevice,
we need an option to work even without real quorum.
(fence races are going to avoid with delay attribute:
 https://access.redhat.com/solutions/91653
 https://access.redhat.com/solutions/1293523)

Best Regards,
Kazunori INOUE

> -Original Message-
> From: Users [mailto:users-boun...@clusterlabs.org] On Behalf Of Klaus 
> Wenninger
> Sent: Friday, May 25, 2018 4:08 PM
> To: users@clusterlabs.org
> Subject: Re: [ClusterLabs] Questions about SBD behavior
> 
> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
> > Hi,
> >
> > I am checking the watchdog function of SBD (without shared block-device).
> > In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
> > the
> remaining node.
> > Is this the designed behavior?
> 
> SBD without a shared block-device doesn't really make sense on
> a two-node cluster.
> The basic idea is - e.g. in a case of a networking problem -
> that a cluster splits up in a quorate and a non-quorate partition.
> The quorate partition stays over while SBD guarantees a
> reliable watchdog-based self-fencing of the non-quorate partition
> within a defined timeout.
> This idea of course doesn't work with just 2 nodes.
> Taking quorum info from the 2-node feature of corosync (automatically
> switching on wait-for-all) doesn't help in this case but instead
> would lead to split-brain.
> What you can do - and what e.g. pcs does automatically - is enable
> the auto-tie-breaker instead of two-node in corosync. But that
> still doesn't give you a higher availability than the one of the
> winner of auto-tie-breaker. (Maybe interesting if you are going
> for a load-balancing-scenario that doesn't affect availability or
> for a transient state while setting up a cluste node-by-node ...)
> What you can do though is using qdevice to still have 'real-quorum'
> info with just 2 full cluster-nodes.
> 
> There was quite a lot of discussion round this topic on this
> thread previously if you search the history.
> 
> Regards,
> Klaus
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[Andrei Borzenkov]

On Mon, May 28, 2018 at 10:47 AM, Klaus Wenninger  wrote:
> On 05/28/2018 09:43 AM, Klaus Wenninger wrote:
>> On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
>>> 25.05.2018 14:44, Klaus Wenninger пишет:
 On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
> wrote:
>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>>> Hi,
>>>
>>> I am checking the watchdog function of SBD (without shared 
>>> block-device).
>>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
>>> on the remaining node.
>>> Is this the designed behavior?
>> SBD without a shared block-device doesn't really make sense on
>> a two-node cluster.
>> The basic idea is - e.g. in a case of a networking problem -
>> that a cluster splits up in a quorate and a non-quorate partition.
>> The quorate partition stays over while SBD guarantees a
>> reliable watchdog-based self-fencing of the non-quorate partition
>> within a defined timeout.
> Does it require no-quorum-policy=suicide or it decides completely
> independently? I.e. would it fire also with no-quorum-policy=ignore?
 Finally it will in any case. But no-quorum-policy decides how
 long this will take. In case of suicide the inquisitor will immediately
 stop tickling the watchdog. In all other cases the pacemaker-servant
 will stop pinging the inquisitor which will makes the servant
 timeout after a default of 4 seconds and then the inquisitor will
 stop tickling the watchdog.
 But that is just relevant if Corosync doesn't have 2-node enabled.
 See the comment below for that case.

>> This idea of course doesn't work with just 2 nodes.
>> Taking quorum info from the 2-node feature of corosync (automatically
>> switching on wait-for-all) doesn't help in this case but instead
>> would lead to split-brain.
> So what you are saying is that SBD ignores quorum information from
> corosync and takes its own decisions based on pure count of nodes. Do
> I understand it correctly?
 Yes, but that is just true for this case where Corosync has 2-node
 enabled.
> In all other cases (might it be clusters with more than 2 nodes
 or clusters with just 2 nodes but without 2-node enabled in
 Corosync) pacemaker-servant takes quorum-info from
 pacemaker, which will probably come directly from Corosync
 nowadays.
 But as said if 2-node is configured with Corosync everything
 is different: The node-counting is then actually done
 by the cluster-servant and this one will stop pinging the
 inquisitor (instead of the pacemaker-servant) if it doesn't
 count more than 1 node.

>>> Is it conditional on having no shared device or it just checks two_node
>>> value? If it always behaves this way, even with real shared device
>>> present, it means sbd is fundamentally incompatible with two_node and it
>>> better be mentioned in documentation.
>> If you are referring to counting the nodes instead of taking
>> quorum-info from pacemaker in case of 2-node configured
>> with corosync, that is universal.
>>
>> And actually the reason why it is there is to be able to use
>> sbd with a single disk on 2-nodes having 2-node enabled.
>>
>> Imagine quorum-info from corosync/pacemaker being used
>> in that case:
>> Image a cluster (node-a & node-b). node-a looses connection
>> to the network and to the shared storage. node-a will still
>> receive positive quorum from corosync as it has seen the other
>> node already (since it is up). This will make it ignore the
>> loss of the disk (survive on pacemaker).
>> node-b is quoruate as well, sees the disk, uses the disk to
>> fence node-a and will after a timeout assume node-a to be
>> down -> split-brain.
> Seeing the disk will prevent the reboot if that is what
> was missing for you.

Yes, this was not exactly clear. Thank you!

>>
 That all said I've just realized that setting 2-node in Corosync
 shouldn't really be dangerous anymore although it doesn't make
 the cluster especially useful either in case of SBD without disk(s).

 Regards,
 Klaus
>> ___
>> Users mailing list: Users@clusterlabs.org
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 05/28/2018 09:43 AM, Klaus Wenninger wrote:
> On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
>> 25.05.2018 14:44, Klaus Wenninger пишет:
>>> On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
 On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
 wrote:
> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>> Hi,
>>
>> I am checking the watchdog function of SBD (without shared block-device).
>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
>> on the remaining node.
>> Is this the designed behavior?
> SBD without a shared block-device doesn't really make sense on
> a two-node cluster.
> The basic idea is - e.g. in a case of a networking problem -
> that a cluster splits up in a quorate and a non-quorate partition.
> The quorate partition stays over while SBD guarantees a
> reliable watchdog-based self-fencing of the non-quorate partition
> within a defined timeout.
 Does it require no-quorum-policy=suicide or it decides completely
 independently? I.e. would it fire also with no-quorum-policy=ignore?
>>> Finally it will in any case. But no-quorum-policy decides how
>>> long this will take. In case of suicide the inquisitor will immediately
>>> stop tickling the watchdog. In all other cases the pacemaker-servant
>>> will stop pinging the inquisitor which will makes the servant
>>> timeout after a default of 4 seconds and then the inquisitor will
>>> stop tickling the watchdog.
>>> But that is just relevant if Corosync doesn't have 2-node enabled.
>>> See the comment below for that case.
>>>
> This idea of course doesn't work with just 2 nodes.
> Taking quorum info from the 2-node feature of corosync (automatically
> switching on wait-for-all) doesn't help in this case but instead
> would lead to split-brain.
 So what you are saying is that SBD ignores quorum information from
 corosync and takes its own decisions based on pure count of nodes. Do
 I understand it correctly?
>>> Yes, but that is just true for this case where Corosync has 2-node
>>> enabled.
 In all other cases (might it be clusters with more than 2 nodes
>>> or clusters with just 2 nodes but without 2-node enabled in
>>> Corosync) pacemaker-servant takes quorum-info from
>>> pacemaker, which will probably come directly from Corosync
>>> nowadays.
>>> But as said if 2-node is configured with Corosync everything
>>> is different: The node-counting is then actually done
>>> by the cluster-servant and this one will stop pinging the
>>> inquisitor (instead of the pacemaker-servant) if it doesn't
>>> count more than 1 node.
>>>
>> Is it conditional on having no shared device or it just checks two_node
>> value? If it always behaves this way, even with real shared device
>> present, it means sbd is fundamentally incompatible with two_node and it
>> better be mentioned in documentation.
> If you are referring to counting the nodes instead of taking
> quorum-info from pacemaker in case of 2-node configured
> with corosync, that is universal.
>
> And actually the reason why it is there is to be able to use
> sbd with a single disk on 2-nodes having 2-node enabled.
>
> Imagine quorum-info from corosync/pacemaker being used
> in that case:
> Image a cluster (node-a & node-b). node-a looses connection
> to the network and to the shared storage. node-a will still
> receive positive quorum from corosync as it has seen the other
> node already (since it is up). This will make it ignore the
> loss of the disk (survive on pacemaker).
> node-b is quoruate as well, sees the disk, uses the disk to
> fence node-a and will after a timeout assume node-a to be
> down -> split-brain.
Seeing the disk will prevent the reboot if that is what
was missing for you.
>
>>> That all said I've just realized that setting 2-node in Corosync
>>> shouldn't really be dangerous anymore although it doesn't make
>>> the cluster especially useful either in case of SBD without disk(s).
>>>
>>> Regards,
>>> Klaus
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
> 25.05.2018 14:44, Klaus Wenninger пишет:
>> On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
>>> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
>>> wrote:
 On 05/25/2018 07:31 AM, 井上 和徳 wrote:
> Hi,
>
> I am checking the watchdog function of SBD (without shared block-device).
> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
> on the remaining node.
> Is this the designed behavior?
 SBD without a shared block-device doesn't really make sense on
 a two-node cluster.
 The basic idea is - e.g. in a case of a networking problem -
 that a cluster splits up in a quorate and a non-quorate partition.
 The quorate partition stays over while SBD guarantees a
 reliable watchdog-based self-fencing of the non-quorate partition
 within a defined timeout.
>>> Does it require no-quorum-policy=suicide or it decides completely
>>> independently? I.e. would it fire also with no-quorum-policy=ignore?
>> Finally it will in any case. But no-quorum-policy decides how
>> long this will take. In case of suicide the inquisitor will immediately
>> stop tickling the watchdog. In all other cases the pacemaker-servant
>> will stop pinging the inquisitor which will makes the servant
>> timeout after a default of 4 seconds and then the inquisitor will
>> stop tickling the watchdog.
>> But that is just relevant if Corosync doesn't have 2-node enabled.
>> See the comment below for that case.
>>
 This idea of course doesn't work with just 2 nodes.
 Taking quorum info from the 2-node feature of corosync (automatically
 switching on wait-for-all) doesn't help in this case but instead
 would lead to split-brain.
>>> So what you are saying is that SBD ignores quorum information from
>>> corosync and takes its own decisions based on pure count of nodes. Do
>>> I understand it correctly?
>> Yes, but that is just true for this case where Corosync has 2-node
>> enabled.
>>> In all other cases (might it be clusters with more than 2 nodes
>> or clusters with just 2 nodes but without 2-node enabled in
>> Corosync) pacemaker-servant takes quorum-info from
>> pacemaker, which will probably come directly from Corosync
>> nowadays.
>> But as said if 2-node is configured with Corosync everything
>> is different: The node-counting is then actually done
>> by the cluster-servant and this one will stop pinging the
>> inquisitor (instead of the pacemaker-servant) if it doesn't
>> count more than 1 node.
>>
> Is it conditional on having no shared device or it just checks two_node
> value? If it always behaves this way, even with real shared device
> present, it means sbd is fundamentally incompatible with two_node and it
> better be mentioned in documentation.

If you are referring to counting the nodes instead of taking
quorum-info from pacemaker in case of 2-node configured
with corosync, that is universal.

And actually the reason why it is there is to be able to use
sbd with a single disk on 2-nodes having 2-node enabled.

Imagine quorum-info from corosync/pacemaker being used
in that case:
Image a cluster (node-a & node-b). node-a looses connection
to the network and to the shared storage. node-a will still
receive positive quorum from corosync as it has seen the other
node already (since it is up). This will make it ignore the
loss of the disk (survive on pacemaker).
node-b is quoruate as well, sees the disk, uses the disk to
fence node-a and will after a timeout assume node-a to be
down -> split-brain.

>
>> That all said I've just realized that setting 2-node in Corosync
>> shouldn't really be dangerous anymore although it doesn't make
>> the cluster especially useful either in case of SBD without disk(s).
>>
>> Regards,
>> Klaus

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[Andrei Borzenkov]

25.05.2018 14:44, Klaus Wenninger пишет:
> On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
>> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
>> wrote:
>>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
 Hi,

 I am checking the watchdog function of SBD (without shared block-device).
 In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
 the remaining node.
 Is this the designed behavior?
>>> SBD without a shared block-device doesn't really make sense on
>>> a two-node cluster.
>>> The basic idea is - e.g. in a case of a networking problem -
>>> that a cluster splits up in a quorate and a non-quorate partition.
>>> The quorate partition stays over while SBD guarantees a
>>> reliable watchdog-based self-fencing of the non-quorate partition
>>> within a defined timeout.
>> Does it require no-quorum-policy=suicide or it decides completely
>> independently? I.e. would it fire also with no-quorum-policy=ignore?
> 
> Finally it will in any case. But no-quorum-policy decides how
> long this will take. In case of suicide the inquisitor will immediately
> stop tickling the watchdog. In all other cases the pacemaker-servant
> will stop pinging the inquisitor which will makes the servant
> timeout after a default of 4 seconds and then the inquisitor will
> stop tickling the watchdog.
> But that is just relevant if Corosync doesn't have 2-node enabled.
> See the comment below for that case.
> 
>>
>>> This idea of course doesn't work with just 2 nodes.
>>> Taking quorum info from the 2-node feature of corosync (automatically
>>> switching on wait-for-all) doesn't help in this case but instead
>>> would lead to split-brain.
>> So what you are saying is that SBD ignores quorum information from
>> corosync and takes its own decisions based on pure count of nodes. Do
>> I understand it correctly?
> 
> Yes, but that is just true for this case where Corosync has 2-node
> enabled.
> > In all other cases (might it be clusters with more than 2 nodes
> or clusters with just 2 nodes but without 2-node enabled in
> Corosync) pacemaker-servant takes quorum-info from
> pacemaker, which will probably come directly from Corosync
> nowadays.
> But as said if 2-node is configured with Corosync everything
> is different: The node-counting is then actually done
> by the cluster-servant and this one will stop pinging the
> inquisitor (instead of the pacemaker-servant) if it doesn't
> count more than 1 node.
> 

Is it conditional on having no shared device or it just checks two_node
value? If it always behaves this way, even with real shared device
present, it means sbd is fundamentally incompatible with two_node and it
better be mentioned in documentation.

> That all said I've just realized that setting 2-node in Corosync
> shouldn't really be dangerous anymore although it doesn't make
> the cluster especially useful either in case of SBD without disk(s).
> 
> Regards,
> Klaus
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  wrote:
>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>>> Hi,
>>>
>>> I am checking the watchdog function of SBD (without shared block-device).
>>> In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
>>> the remaining node.
>>> Is this the designed behavior?
>> SBD without a shared block-device doesn't really make sense on
>> a two-node cluster.
>> The basic idea is - e.g. in a case of a networking problem -
>> that a cluster splits up in a quorate and a non-quorate partition.
>> The quorate partition stays over while SBD guarantees a
>> reliable watchdog-based self-fencing of the non-quorate partition
>> within a defined timeout.
> Does it require no-quorum-policy=suicide or it decides completely
> independently? I.e. would it fire also with no-quorum-policy=ignore?

Finally it will in any case. But no-quorum-policy decides how
long this will take. In case of suicide the inquisitor will immediately
stop tickling the watchdog. In all other cases the pacemaker-servant
will stop pinging the inquisitor which will makes the servant
timeout after a default of 4 seconds and then the inquisitor will
stop tickling the watchdog.
But that is just relevant if Corosync doesn't have 2-node enabled.
See the comment below for that case.

>
>> This idea of course doesn't work with just 2 nodes.
>> Taking quorum info from the 2-node feature of corosync (automatically
>> switching on wait-for-all) doesn't help in this case but instead
>> would lead to split-brain.
> So what you are saying is that SBD ignores quorum information from
> corosync and takes its own decisions based on pure count of nodes. Do
> I understand it correctly?

Yes, but that is just true for this case where Corosync has 2-node
enabled.

In all other cases (might it be clusters with more than 2 nodes
or clusters with just 2 nodes but without 2-node enabled in
Corosync) pacemaker-servant takes quorum-info from
pacemaker, which will probably come directly from Corosync
nowadays.
But as said if 2-node is configured with Corosync everything
is different: The node-counting is then actually done
by the cluster-servant and this one will stop pinging the
inquisitor (instead of the pacemaker-servant) if it doesn't
count more than 1 node.

That all said I've just realized that setting 2-node in Corosync
shouldn't really be dangerous anymore although it doesn't make
the cluster especially useful either in case of SBD without disk(s).

Regards,
Klaus
>
>> What you can do - and what e.g. pcs does automatically - is enable
>> the auto-tie-breaker instead of two-node in corosync. But that
>> still doesn't give you a higher availability than the one of the
>> winner of auto-tie-breaker. (Maybe interesting if you are going
>> for a load-balancing-scenario that doesn't affect availability or
>> for a transient state while setting up a cluste node-by-node ...)
>> What you can do though is using qdevice to still have 'real-quorum'
>> info with just 2 full cluster-nodes.
>>
>> There was quite a lot of discussion round this topic on this
>> thread previously if you search the history.
>>
>> Regards,
>> Klaus
>>
>>> [vmrh75b]# cat /etc/corosync/corosync.conf
>>> (snip)
>>> quorum {
>>> provider: corosync_votequorum
>>> two_node: 1
>>> }
>>>
>>> [vmrh75b]# cat /etc/sysconfig/sbd
>>> # This file has been generated by pcs.
>>> SBD_DELAY_START=no
>>> ## SBD_DEVICE="/dev/vdb1"
>>> SBD_OPTS="-vvv"
>>> SBD_PACEMAKER=yes
>>> SBD_STARTMODE=always
>>> SBD_WATCHDOG_DEV=/dev/watchdog
>>> SBD_WATCHDOG_TIMEOUT=5
>>>
>>> [vmrh75b]# crm_mon -r1
>>> Stack: corosync
>>> Current DC: vmrh75a (version 2.0.0-0.1.rc4.el7-2.0.0-rc4) - partition with 
>>> quorum
>>> Last updated: Fri May 25 13:36:07 2018
>>> Last change: Fri May 25 13:35:22 2018 by root via cibadmin on vmrh75a
>>>
>>> 2 nodes configured
>>> 0 resources configured
>>>
>>> Online: [ vmrh75a vmrh75b ]
>>>
>>> No resources
>>>
>>> [vmrh75b]# pcs property show
>>> Cluster Properties:
>>>  cluster-infrastructure: corosync
>>>  cluster-name: my_cluster
>>>  dc-version: 2.0.0-0.1.rc4.el7-2.0.0-rc4
>>>  have-watchdog: true
>>>  stonith-enabled: false
>>>
>>> [vmrh75b]# ps -ef | egrep "sbd|coro|pace"
>>> root  2169 1  0 13:34 ?00:00:00 sbd: inquisitor
>>> root  2170  2169  0 13:34 ?00:00:00 sbd: watcher: Pacemaker
>>> root  2171  2169  0 13:34 ?00:00:00 sbd: watcher: Cluster
>>> root  2172 1  0 13:34 ?00:00:00 corosync
>>> root  2179 1  0 13:34 ?00:00:00 /usr/sbin/pacemakerd -f
>>> haclust+  2180  2179  0 13:34 ?00:00:00 
>>> /usr/libexec/pacemaker/pacemaker-based
>>> root  2181  2179  0 13:34 ?00:00:00 
>>> /usr/libexec/pacemaker/pacemaker-fenced
>>> root  2182  2179  0 13:34 ?00:00:00 
>>> /usr/libexec/pacemaker/pacemaker-execd
>>> haclust+  2183  2179  0 13:34 ?00:00:00 
>>> 

Re: [ClusterLabs] Questions about SBD behavior

[Andrei Borzenkov]

On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  wrote:
> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>> Hi,
>>
>> I am checking the watchdog function of SBD (without shared block-device).
>> In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
>> the remaining node.
>> Is this the designed behavior?
>
> SBD without a shared block-device doesn't really make sense on
> a two-node cluster.
> The basic idea is - e.g. in a case of a networking problem -
> that a cluster splits up in a quorate and a non-quorate partition.
> The quorate partition stays over while SBD guarantees a
> reliable watchdog-based self-fencing of the non-quorate partition
> within a defined timeout.

Does it require no-quorum-policy=suicide or it decides completely
independently? I.e. would it fire also with no-quorum-policy=ignore?

> This idea of course doesn't work with just 2 nodes.
> Taking quorum info from the 2-node feature of corosync (automatically
> switching on wait-for-all) doesn't help in this case but instead
> would lead to split-brain.

So what you are saying is that SBD ignores quorum information from
corosync and takes its own decisions based on pure count of nodes. Do
I understand it correctly?

> What you can do - and what e.g. pcs does automatically - is enable
> the auto-tie-breaker instead of two-node in corosync. But that
> still doesn't give you a higher availability than the one of the
> winner of auto-tie-breaker. (Maybe interesting if you are going
> for a load-balancing-scenario that doesn't affect availability or
> for a transient state while setting up a cluste node-by-node ...)
> What you can do though is using qdevice to still have 'real-quorum'
> info with just 2 full cluster-nodes.
>
> There was quite a lot of discussion round this topic on this
> thread previously if you search the history.
>
> Regards,
> Klaus
>
>>
>> [vmrh75b]# cat /etc/corosync/corosync.conf
>> (snip)
>> quorum {
>> provider: corosync_votequorum
>> two_node: 1
>> }
>>
>> [vmrh75b]# cat /etc/sysconfig/sbd
>> # This file has been generated by pcs.
>> SBD_DELAY_START=no
>> ## SBD_DEVICE="/dev/vdb1"
>> SBD_OPTS="-vvv"
>> SBD_PACEMAKER=yes
>> SBD_STARTMODE=always
>> SBD_WATCHDOG_DEV=/dev/watchdog
>> SBD_WATCHDOG_TIMEOUT=5
>>
>> [vmrh75b]# crm_mon -r1
>> Stack: corosync
>> Current DC: vmrh75a (version 2.0.0-0.1.rc4.el7-2.0.0-rc4) - partition with 
>> quorum
>> Last updated: Fri May 25 13:36:07 2018
>> Last change: Fri May 25 13:35:22 2018 by root via cibadmin on vmrh75a
>>
>> 2 nodes configured
>> 0 resources configured
>>
>> Online: [ vmrh75a vmrh75b ]
>>
>> No resources
>>
>> [vmrh75b]# pcs property show
>> Cluster Properties:
>>  cluster-infrastructure: corosync
>>  cluster-name: my_cluster
>>  dc-version: 2.0.0-0.1.rc4.el7-2.0.0-rc4
>>  have-watchdog: true
>>  stonith-enabled: false
>>
>> [vmrh75b]# ps -ef | egrep "sbd|coro|pace"
>> root  2169 1  0 13:34 ?00:00:00 sbd: inquisitor
>> root  2170  2169  0 13:34 ?00:00:00 sbd: watcher: Pacemaker
>> root  2171  2169  0 13:34 ?00:00:00 sbd: watcher: Cluster
>> root  2172 1  0 13:34 ?00:00:00 corosync
>> root  2179 1  0 13:34 ?00:00:00 /usr/sbin/pacemakerd -f
>> haclust+  2180  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-based
>> root  2181  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-fenced
>> root  2182  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-execd
>> haclust+  2183  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-attrd
>> haclust+  2184  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-schedulerd
>> haclust+  2185  2179  0 13:34 ?00:00:00 
>> /usr/libexec/pacemaker/pacemaker-controld
>>
>> [vmrh75b]# pcs cluster stop vmrh75a
>> vmrh75a: Stopping Cluster (pacemaker)...
>> vmrh75a: Stopping Cluster (corosync)...
>>
>> [vmrh75b]# tail -F /var/log/messages
>> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: Our peer on the DC 
>> (vmrh75a) is dead
>> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
>> S_NOT_DC -> S_ELECTION
>> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
>> S_ELECTION -> S_INTEGRATION
>> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Node vmrh75a state is 
>> now lost
>> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Removing all vmrh75a 
>> attributes for peer loss
>> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Lost attribute writer 
>> vmrh75a
>> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Purged 1 peer with 
>> id=1 and/or uname=vmrh75a from the membership cache
>> May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Node vmrh75a state 
>> is now lost
>> May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Purged 1 peer with 
>> id=1 and/or uname=vmrh75a from the membership cache
>> May 25 13:37:00 vmrh75b 

Re: [ClusterLabs] Questions about SBD behavior

[Klaus Wenninger]

On 05/25/2018 07:31 AM, 井上 和徳 wrote:
> Hi,
>
> I am checking the watchdog function of SBD (without shared block-device).
> In a two-node cluster, if one cluster is stopped, watchdog is triggered on 
> the remaining node.
> Is this the designed behavior?

SBD without a shared block-device doesn't really make sense on
a two-node cluster.
The basic idea is - e.g. in a case of a networking problem -
that a cluster splits up in a quorate and a non-quorate partition.
The quorate partition stays over while SBD guarantees a
reliable watchdog-based self-fencing of the non-quorate partition
within a defined timeout.
This idea of course doesn't work with just 2 nodes.
Taking quorum info from the 2-node feature of corosync (automatically
switching on wait-for-all) doesn't help in this case but instead
would lead to split-brain.
What you can do - and what e.g. pcs does automatically - is enable
the auto-tie-breaker instead of two-node in corosync. But that
still doesn't give you a higher availability than the one of the
winner of auto-tie-breaker. (Maybe interesting if you are going
for a load-balancing-scenario that doesn't affect availability or
for a transient state while setting up a cluste node-by-node ...)
What you can do though is using qdevice to still have 'real-quorum'
info with just 2 full cluster-nodes.

There was quite a lot of discussion round this topic on this
thread previously if you search the history.

Regards,
Klaus

>
> [vmrh75b]# cat /etc/corosync/corosync.conf
> (snip)
> quorum {
> provider: corosync_votequorum
> two_node: 1
> }
>
> [vmrh75b]# cat /etc/sysconfig/sbd
> # This file has been generated by pcs.
> SBD_DELAY_START=no
> ## SBD_DEVICE="/dev/vdb1"
> SBD_OPTS="-vvv"
> SBD_PACEMAKER=yes
> SBD_STARTMODE=always
> SBD_WATCHDOG_DEV=/dev/watchdog
> SBD_WATCHDOG_TIMEOUT=5
>
> [vmrh75b]# crm_mon -r1
> Stack: corosync
> Current DC: vmrh75a (version 2.0.0-0.1.rc4.el7-2.0.0-rc4) - partition with 
> quorum
> Last updated: Fri May 25 13:36:07 2018
> Last change: Fri May 25 13:35:22 2018 by root via cibadmin on vmrh75a
>
> 2 nodes configured
> 0 resources configured
>
> Online: [ vmrh75a vmrh75b ]
>
> No resources
>
> [vmrh75b]# pcs property show
> Cluster Properties:
>  cluster-infrastructure: corosync
>  cluster-name: my_cluster
>  dc-version: 2.0.0-0.1.rc4.el7-2.0.0-rc4
>  have-watchdog: true
>  stonith-enabled: false
>
> [vmrh75b]# ps -ef | egrep "sbd|coro|pace"
> root  2169 1  0 13:34 ?00:00:00 sbd: inquisitor
> root  2170  2169  0 13:34 ?00:00:00 sbd: watcher: Pacemaker
> root  2171  2169  0 13:34 ?00:00:00 sbd: watcher: Cluster
> root  2172 1  0 13:34 ?00:00:00 corosync
> root  2179 1  0 13:34 ?00:00:00 /usr/sbin/pacemakerd -f
> haclust+  2180  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-based
> root  2181  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-fenced
> root  2182  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-execd
> haclust+  2183  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-attrd
> haclust+  2184  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-schedulerd
> haclust+  2185  2179  0 13:34 ?00:00:00 
> /usr/libexec/pacemaker/pacemaker-controld
>
> [vmrh75b]# pcs cluster stop vmrh75a
> vmrh75a: Stopping Cluster (pacemaker)...
> vmrh75a: Stopping Cluster (corosync)...
>
> [vmrh75b]# tail -F /var/log/messages
> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: Our peer on the DC 
> (vmrh75a) is dead
> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
> S_NOT_DC -> S_ELECTION
> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
> S_ELECTION -> S_INTEGRATION
> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Node vmrh75a state is 
> now lost
> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Removing all vmrh75a 
> attributes for peer loss
> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Lost attribute writer 
> vmrh75a
> May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Purged 1 peer with 
> id=1 and/or uname=vmrh75a from the membership cache
> May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Node vmrh75a state is 
> now lost
> May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Purged 1 peer with 
> id=1 and/or uname=vmrh75a from the membership cache
> May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Node vmrh75a state is 
> now lost
> May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Purged 1 peer with 
> id=1 and/or uname=vmrh75a from the membership cache
> May 25 13:37:00 vmrh75b pacemaker-controld[2185]: warning: Input 
> I_ELECTION_DC received in state S_INTEGRATION from do_election_check
> May 25 13:37:01 vmrh75b sbd[2171]:   cluster:  warning: set_servant_health: 
> Connected to corosync but requires both nodes present
> May 25 13:37:01 vmrh75b sbd[2171]:   cluster:  warning: 

[ClusterLabs] Questions about SBD behavior

[井上 和徳]

Hi,

I am checking the watchdog function of SBD (without shared block-device).
In a two-node cluster, if one cluster is stopped, watchdog is triggered on the 
remaining node.
Is this the designed behavior?


[vmrh75b]# cat /etc/corosync/corosync.conf
(snip)
quorum {
provider: corosync_votequorum
two_node: 1
}

[vmrh75b]# cat /etc/sysconfig/sbd
# This file has been generated by pcs.
SBD_DELAY_START=no
## SBD_DEVICE="/dev/vdb1"
SBD_OPTS="-vvv"
SBD_PACEMAKER=yes
SBD_STARTMODE=always
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=5

[vmrh75b]# crm_mon -r1
Stack: corosync
Current DC: vmrh75a (version 2.0.0-0.1.rc4.el7-2.0.0-rc4) - partition with 
quorum
Last updated: Fri May 25 13:36:07 2018
Last change: Fri May 25 13:35:22 2018 by root via cibadmin on vmrh75a

2 nodes configured
0 resources configured

Online: [ vmrh75a vmrh75b ]

No resources

[vmrh75b]# pcs property show
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: my_cluster
 dc-version: 2.0.0-0.1.rc4.el7-2.0.0-rc4
 have-watchdog: true
 stonith-enabled: false

[vmrh75b]# ps -ef | egrep "sbd|coro|pace"
root  2169 1  0 13:34 ?00:00:00 sbd: inquisitor
root  2170  2169  0 13:34 ?00:00:00 sbd: watcher: Pacemaker
root  2171  2169  0 13:34 ?00:00:00 sbd: watcher: Cluster
root  2172 1  0 13:34 ?00:00:00 corosync
root  2179 1  0 13:34 ?00:00:00 /usr/sbin/pacemakerd -f
haclust+  2180  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-based
root  2181  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-fenced
root  2182  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-execd
haclust+  2183  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-attrd
haclust+  2184  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-schedulerd
haclust+  2185  2179  0 13:34 ?00:00:00 
/usr/libexec/pacemaker/pacemaker-controld

[vmrh75b]# pcs cluster stop vmrh75a
vmrh75a: Stopping Cluster (pacemaker)...
vmrh75a: Stopping Cluster (corosync)...

[vmrh75b]# tail -F /var/log/messages
May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: Our peer on the DC 
(vmrh75a) is dead
May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
S_NOT_DC -> S_ELECTION
May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition 
S_ELECTION -> S_INTEGRATION
May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Node vmrh75a state is 
now lost
May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Removing all vmrh75a 
attributes for peer loss
May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Lost attribute writer 
vmrh75a
May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Purged 1 peer with id=1 
and/or uname=vmrh75a from the membership cache
May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Node vmrh75a state is 
now lost
May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Purged 1 peer with id=1 
and/or uname=vmrh75a from the membership cache
May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Node vmrh75a state is 
now lost
May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Purged 1 peer with id=1 
and/or uname=vmrh75a from the membership cache
May 25 13:37:00 vmrh75b pacemaker-controld[2185]: warning: Input I_ELECTION_DC 
received in state S_INTEGRATION from do_election_check
May 25 13:37:01 vmrh75b sbd[2171]:   cluster:  warning: set_servant_health: 
Connected to corosync but requires both nodes present
May 25 13:37:01 vmrh75b sbd[2171]:   cluster:  warning: notify_parent: 
Notifying parent: UNHEALTHY (6)
May 25 13:37:01 vmrh75b sbd[2169]: warning: inquisitor_child: cluster health 
check: UNHEALTHY
May 25 13:37:01 vmrh75b sbd[2169]: warning: inquisitor_child: Servant cluster 
is outdated (age: 226)
May 25 13:37:01 vmrh75b sbd[2170]:  pcmk:   notice: unpack_config: Watchdog 
will be used via SBD if fencing is required
May 25 13:37:01 vmrh75b sbd[2170]:  pcmk: info: 
determine_online_status: Node vmrh75b is online
May 25 13:37:01 vmrh75b sbd[2170]:  pcmk: info: unpack_node_loop: Node 
2 is already processed
May 25 13:37:01 vmrh75b sbd[2170]:  pcmk: info: unpack_node_loop: Node 
2 is already processed
May 25 13:37:01 vmrh75b sbd[2171]:   cluster:  warning: notify_parent: 
Notifying parent: UNHEALTHY (6)
May 25 13:37:01 vmrh75b corosync[2172]: [TOTEM ] A new membership 
(192.168.28.132:5712) was formed. Members left: 1
May 25 13:37:01 vmrh75b corosync[2172]: [QUORUM] Members[1]: 2
May 25 13:37:01 vmrh75b corosync[2172]: [MAIN  ] Completed service 
synchronization, ready to provide service.
May 25 13:37:01 vmrh75b pacemakerd[2179]: notice: Node vmrh75a state is now lost
May 25 13:37:01 vmrh75b pacemaker-controld[2185]: notice: Node vmrh75a state is 
now lost
May 25 13:37:01 vmrh75b pacemaker-controld[2185]: warning: Stonith/shutdown of 
node vmrh75a was not expected
May 25 13:37:02 vmrh75b sbd[2171]:   cluster:  warning: