Re: [ClusterLabs] [Problem] In RHEL8.4beta, pgsql resource control fails.
Hi Klaus,
Hi Ken,
We have confirmed that the operation is improved by the test.
Thank you for your prompt response.
We look forward to including this fix in the release version of RHEL 8.4.
Best Regards,
Hideo Yamauchi.
- Original Message -
> From: "renayama19661...@ybb.ne.jp"
> To: "kwenn...@redhat.com" ; Cluster Labs - All topics
> related to open-source clustering welcomed ; Cluster
> Labs - All topics related to open-source clustering welcomed
>
> Cc:
> Date: 2021/4/13, Tue 07:08
> Subject: Re: [ClusterLabs] [Problem] In RHEL8.4beta, pgsql resource control
> fails.
>
> Hi Klaus,
> Hi Ken,
>
>> I've opened https://github.com/ClusterLabs/pacemaker/pull/2342 with
>
>> I guess the simplest possible solution to the immediate issue so
>> that we can discuss it.
>
>
> Thank you for the fix.
>
>
> I have confirmed that the fixes have been merged.
>
> I'll test this fix today just in case.
>
> Many thanks,
> Hideo Yamauchi.
>
>
> - Original Message -
>> From: Klaus Wenninger
>> To: renayama19661...@ybb.ne.jp; Cluster Labs - All topics related to
> open-source clustering welcomed
>> Cc:
>> Date: 2021/4/12, Mon 22:22
>> Subject: Re: [ClusterLabs] [Problem] In RHEL8.4beta, pgsql resource control
> fails.
>>
>> On 4/9/21 5:13 PM, Klaus Wenninger wrote:
>>> On 4/9/21 4:04 PM, Klaus Wenninger wrote:
On 4/9/21 3:45 PM, Klaus Wenninger wrote:
> On 4/9/21 3:36 PM, Klaus Wenninger wrote:
>> On 4/9/21 2:37 PM, renayama19661...@ybb.ne.jp wrote:
>>> Hi Klaus,
>>>
>>> Thanks for your comment.
>>>
Hmm ... is that with selinux enabled?
Respectively do you see any related avc messages?
>>>
>>> Selinux is not enabled.
>>> Isn't crm_mon caused by not returning a response
> when
>> pacemakerd
>>> prepares to stop?
> yep ... that doesn't look good.
> While in pcmk_shutdown_worker ipc isn't handled.
Stop ... that should actually work as pcmk_shutdown_worker
should exit quite quickly and proceed after mainloop
dispatching when called again.
Don't see anything atm that might be blocking for longer ...
but let me dig into it further ...
>>> What happens is clear (thanks Ken for the hint ;-) ).
>>> When pacemakerd is shutting down - already when it
>>> shuts down the resources and not just when it starts to
>>> reap the subdaemons - crm_mon reads that state and
>>> doesn't try to connect to the cib anymore.
>> I've opened https://github.com/ClusterLabs/pacemaker/pull/2342 with
>> I guess the simplest possible solution to the immediate issue so
>> that we can discuss it.
> Question is why that didn't create issue earlier.
> Probably I didn't test with resources that had crm_mon in
> their stop/monitor-actions but sbd should have run into
> issues.
>
> Klaus
>> But when shutting down a node the resources should be
>> shutdown before pacemakerd goes down.
>> But let me have a look if it can happen that pacemakerd
>> doesn't react to the ipc-pings before. That btw. might
> be
>> lethal for sbd-scenarios (if the phase is too long and it
>> migh actually not be defined).
>>
>> My idea with selinux would have been that it might block
>> the ipc if crm_mon is issued by execd. But well forget
>> about it as it is not enabled ;-)
>>
>>
>> Klaus
>>>
>>> pgsql needs the result of crm_mon in demote processing
> and
>> stop
>>> processing.
>>> crm_mon should return a response even after pacemakerd
> goes
>> into a
>>> stop operation.
>>>
>>> Best Regards,
>>> Hideo Yamauchi.
>>>
>>>
>>> - Original Message -
From: Klaus Wenninger
To: renayama19661...@ybb.ne.jp; Cluster Labs - All
>
>> topics related
to open-source clustering welcomed
>>
Cc:
Date: 2021/4/9, Fri 21:12
Subject: Re: [ClusterLabs] [Problem] In
> RHEL8.4beta,
>> pgsql
resource control fails.
On 4/8/21 11:21 PM, renayama19661...@ybb.ne.jp
> wrote:
> Hi Ken,
> Hi All,
>
> In the pgsql resource, crm_mon is executed
> in the
>> process of
> demote and
stop, and the result is processed.
> However, pacemaker included in RHEL8.4beta
> fails
>> to execute
> this crm_mon.
> - The problem also occurs on github
master(c40e18f085fad9ef1d9d79f671ed8a69eb3e753f).
> The problem can be easily reproduced in the
>> following ways.
>
> Step1. Modify to execute crm_mon in the stop
>
>> process of the
> Dummy resource.
>
>
> dummy_stop() {
> mon=$(crm_mon -1)
> ret=$?
Re: [ClusterLabs] Single-node automated startup question
If it's a VM or container - it should be on a third location. Using a VM hosted on one of the nodes is like giving that node more votes in a two-node cluster. Cheap 3rd node for quorum makes more sense to me. Best Regards,Strahil Nikolov On Wed, Apr 14, 2021 at 21:19, Antony Stone wrote: On Wednesday 14 April 2021 at 19:33:39, Strahil Nikolov wrote: > What about a small form factor device to serve as a quorum maker ? > Best Regards,Strahil Nikolov If you're going to take that approach, why not a virtual machine or two, hosted inside the physical machine which is your single real node? I'm not necessarily advocating this method of achieving quorum, but it's probably an idea worth considering for specific situations. Antony. -- Someone has stolen all the toilets from New Scotland Yard. Police say they have absolutely nothing to go on. Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
What about a small form factor device to serve as a quorum maker ? Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
On Wednesday 14 April 2021 at 19:33:39, Strahil Nikolov wrote: > What about a small form factor device to serve as a quorum maker ? > Best Regards,Strahil Nikolov If you're going to take that approach, why not a virtual machine or two, hosted inside the physical machine which is your single real node? I'm not necessarily advocating this method of achieving quorum, but it's probably an idea worth considering for specific situations. Antony. -- Someone has stolen all the toilets from New Scotland Yard. Police say they have absolutely nothing to go on. Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Coming in Pacemaker 2.0.1: build-time default for resource-stickiness
Hello all, I hope to have the first Pacemaker 2.0.1 release candidate ready next week! A recently added feature is a new build-time option to change the resource-stickiness default for new CIBs. Currently, resource-stickiness defaults to 0, meaning the cluster is free to move resources around to balance them across nodes and so forth. Many new users are surprised by this behavior and expect sticky behavior by default. Now, building Pacemaker using ./configure --with-resource-stickiness- default= tells Pacemaker to add a rsc_defaults section to empty CIBs with a resource-stickiness of . Distributions and users who build from source can set this if they're tried of explaining stickiness to surprised users and expect fewer users to be surprised by stickiness. :) Adding a resource default to all new CIBs is an unusual way of changing a default. We can't simply leave it to higher-level tools, because when creating a cluster, the cluster may not be started immediately and thus there is no way to set the property. Also, the user has a variety of ways to create or start a cluster, so no tool can assume it has full control. We leave the implicit default stickiness at 0, and instead set the configured default via a rsc_defaults entry in new CIBs, so that it won't affect existing clusters or rolling upgrades (current users won't see behavior change), and unlike implicit defaults, users can query and remove resource defaults. -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
On Wed, 2021-04-14 at 18:00 +0300, Andrei Borzenkov wrote: > On 14.04.2021 17:50, Digimer wrote: > > Hi all, > > > > As we get close to finish our Anvil! switch to pacemaker, I'm > > trying > > to tie up loose ends. One that I want feedback on is the pacemaker > > version of cman's old 'post_join_delay' feature. > > > > Use case example; > > > > A common use for the Anvil! is remote deployments where there is > > no > > (IT) humans available. Think cargo ships, field data collection, > > etc. So > > it's entirely possible that a node could fail and not be repaired > > for > > weeks or even months. With this in mind, it's also feasible that a > > solo > > node later loses power, and then reboots. In such a case, 'pcs > > cluster > > start' would never go quorate as the peer is dead. > > > > In cman, during startup, if there was no reply from the peer > > after > > post_join_delay seconds, the peer would get fenced and then the > > cluster > > would finish coming up. Being two_node, it would also become > > quorate and > > start hosting services. Of course, this opens the risk of a fence > > loop, > > but we have other protections in place to prevent that, so a fence > > loop > > is not a concern. > > > > My question then is two-fold; > > > > 1. Is there a pacemaker equivalent to 'post_join_delay'? (Fence the > > peer > > and, if successful, become quorate)? > > > > Startup fencing is pacemaker default (startup-fencing cluster > option). Start-up fencing will have the desired effect in >2 node cluster, but in 2-node cluster the corosync wait_for_all option is key. If wait_for_all is true (which is the default when two_node is set), then a node that comes up alone will wait until it sees the other node at least once before becoming quorate. This prevents an isolated node from coming up and fencing a node that's happily running. Setting wait_for_all to false will make an isolated node immediately become quorate. It will do what you want, which is fence the other node and take over resources, but the danger is that this node is the one that's having trouble (e.g. can't see the other node due to a network card issue). The healthy node could fence the unhealthy node, which might then reboot and come up and shoot the healthy node. There's no direct equivalent of a delay before becoming quorate, but I don't think that helps -- the boot time acts as a sort of random delay, and a delay doesn't help the issue of an unhealthy node shooting a healthy one. My recommendation would be to set wait_for_all to true as long as both nodes are known to be healthy. Once an unhealthy node is down and expected to stay down, set wait_for_all to false on the healthy node so it can reboot and bring the cluster up. (The unhealthy node will still have wait_for_all=true, so it won't cause any trouble even if it comes up.) > > > 2. If not, was this a conscious decision not to add it for some > > reason, > > or was it simply never added? If it was consciously decided to not > > have > > it, what was the reasoning behind it? > > > > I can replicate this behaviour in our code, but I don't want to > > do > > that if there is a compelling reason that I am not aware of. > > > > So, > > > > A) is there a pacemaker version of post_join_delay? > > B) is there a compelling argument NOT to use post_join_delay > > behaviour > > in pacemaker I am not seeing? > > > > Thanks! > > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
On 14.04.2021 17:50, Digimer wrote: > Hi all, > > As we get close to finish our Anvil! switch to pacemaker, I'm trying > to tie up loose ends. One that I want feedback on is the pacemaker > version of cman's old 'post_join_delay' feature. > > Use case example; > > A common use for the Anvil! is remote deployments where there is no > (IT) humans available. Think cargo ships, field data collection, etc. So > it's entirely possible that a node could fail and not be repaired for > weeks or even months. With this in mind, it's also feasible that a solo > node later loses power, and then reboots. In such a case, 'pcs cluster > start' would never go quorate as the peer is dead. > > In cman, during startup, if there was no reply from the peer after > post_join_delay seconds, the peer would get fenced and then the cluster > would finish coming up. Being two_node, it would also become quorate and > start hosting services. Of course, this opens the risk of a fence loop, > but we have other protections in place to prevent that, so a fence loop > is not a concern. > > My question then is two-fold; > > 1. Is there a pacemaker equivalent to 'post_join_delay'? (Fence the peer > and, if successful, become quorate)? > Startup fencing is pacemaker default (startup-fencing cluster option). > 2. If not, was this a conscious decision not to add it for some reason, > or was it simply never added? If it was consciously decided to not have > it, what was the reasoning behind it? > > I can replicate this behaviour in our code, but I don't want to do > that if there is a compelling reason that I am not aware of. > > So, > > A) is there a pacemaker version of post_join_delay? > B) is there a compelling argument NOT to use post_join_delay behaviour > in pacemaker I am not seeing? > > Thanks! > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Single-node automated startup question
Hi all, As we get close to finish our Anvil! switch to pacemaker, I'm trying to tie up loose ends. One that I want feedback on is the pacemaker version of cman's old 'post_join_delay' feature. Use case example; A common use for the Anvil! is remote deployments where there is no (IT) humans available. Think cargo ships, field data collection, etc. So it's entirely possible that a node could fail and not be repaired for weeks or even months. With this in mind, it's also feasible that a solo node later loses power, and then reboots. In such a case, 'pcs cluster start' would never go quorate as the peer is dead. In cman, during startup, if there was no reply from the peer after post_join_delay seconds, the peer would get fenced and then the cluster would finish coming up. Being two_node, it would also become quorate and start hosting services. Of course, this opens the risk of a fence loop, but we have other protections in place to prevent that, so a fence loop is not a concern. My question then is two-fold; 1. Is there a pacemaker equivalent to 'post_join_delay'? (Fence the peer and, if successful, become quorate)? 2. If not, was this a conscious decision not to add it for some reason, or was it simply never added? If it was consciously decided to not have it, what was the reasoning behind it? I can replicate this behaviour in our code, but I don't want to do that if there is a compelling reason that I am not aware of. So, A) is there a pacemaker version of post_join_delay? B) is there a compelling argument NOT to use post_join_delay behaviour in pacemaker I am not seeing? Thanks! -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
