[ClusterLabs] Pacemaker 2.1.6-rc1 now available

[Ken Gaillot]

Hi all,

The first release candidate for Pacemaker 2.1.6 is now available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-2.1.6-rc1

Highlights include new features for managing node attributes, and
easier use of resource descroptions.

Utilization node attributes may now be transient. attrd_updater now
supports a --wait parameter to return only once the new value is
effective locally or cluster-wide. Both crm_attribute and attrd_updater
support --pattern in more usages.

Resource descriptions may now be managed using the crm_resource --
element option, and will be displayed in crm_mon --show-detail output.

In addition, alerts and alert recipients may now be temporarily
disabled by setting the new "enabled" meta-attribute to false.

This release also includes a number of bug fixes. For details, see the
above link.

Everyone is encouraged to download, compile and test the new release.
We do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated.

Many thanks to all contributors of source code and language
translations to this release, including binlingyu, Chris Lumens, Fabio
M. Di Nitto, Gao,Yan, Grace Chin, Ken Gaillot, Klaus Wenninger,
lihaipeng, liupei, liutong, Reid Wahl, Tahlia Richardson, wanglujun,
WangMengabc, xuezhixin, and zhanghuanhuan.
-- 
Ken Gaillot 

___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

Re: [ClusterLabs] HA problem: No live migration when setting node on standby

[Andrei Borzenkov]

On Mon, Apr 17, 2023 at 10:48 AM Philip Schiller
 wrote:
>
> Hello Andrei,
>
> you wrote:
>
> >>As a workaround you could add dummy clone resource colocated with and
> >>ordered after your DRBD masters and order VM after this clone.
>
> Thanks for the idea. This looks like a good option to solve my problem.
>
> I have also researched a little more and came up with an option which seems 
> to work for my case.
> Would you be so kind to evaluate if i understand it correctly?
>
> As mentioned in the original thread
> >> Wed Apr 12 05:28:48 EDT 2023
>
> My system looks like this:
>
> >>I am using a simple two-nodes cluster with Zvol -> DRBD -> Virsh in
> >>primary/primary mode (necessary for live migration).
>
>
> Where drbd-resources and zvol are clones.
> So it is basically a chain of resources, first zvol then drbd then vm.
>
> From documentation i read that in those cases order constraints are not even 
> necessary. This can be done with colocations constraints only
> -> 
> https://access.redhat.com/documentation/de-de/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-orderconstraints-haar#s2-resourceorderlist-HAAR
>
> There is stated:
> >> A common situation is for an administrator to create a chain of ordered
> >> resources, where, for example, resource A starts before resource B which
> >> starts before resource C. If your configuration requires that you
> >> create a set of resources that is colocated and started in order, you
> >> can configure a resource group that contains those resources, as
> >> described in Section 6.5, “Resource Groups”.
>
> I can't create a Resource Group because apparently clone-resources are not 
> supported. So i have the following setup now:
>
> >> colocation 
> >> colocation-mas-drbd-alarmanlage-clo-pri-zfs-drbd_storage-INFINITY inf: 
> >> mas-drbd-alarmanlage clo-pri-zfs-drbd_storage
> >> colocation colocation-pri-vm-alarmanlage-mas-drbd-alarmanlage-INFINITY 
> >> inf: pri-vm-alarmanlage:Started mas-drbd-alarmanlage:Master
> >> location location-pri-vm-alarmanlage-s0-200 pri-vm-alarmanlage 200: s0
>
> Migration works flawless and also the startup is correct: zvol -> drbd -> vm
>

To my best knowledge there is no implied ordering for colocated
resources. So it may work in your case simply due to specific timings.
I would not rely on it. Any software or hardware change may change
timings.

> I am little bit concerned though. Does corosync work like an interpeter and 
> knows the correct order when i do  before  drbd/vm>?
>

Colocation and ordering are entirely orthogonal. Colocating defines
where pacemaker will attempt to start resources while ordering defines
in which order it does it. It is a bit more complicated in case of
promotable clones, because master is not static and is determined at
run time based on resource agent behavior. So pacemaker may delay
placement of dependent resources until masters are known which may
look like ordering.

> Another thing is the Multistate Constraint which i implanted -> 
> pri-vm-alarmanlage:Started mas-drbd-alarmanlage:Master
> Is this equivalent to the  order-mas-drbd-alarmanlage-pri-vm-alarmanlage-mandatory 
> mas-drbd-alarmanlage:promote pri-vm-alarmanlage:start> which i was trying to 
> achieve?
>
> Basically i just want to have zvol started then drbd stared and promoted to 
> master state and then finally vm started. All on the same node.
> Can you confirm that my cluster does this behavior permanently with this 
> configuration.
>

No, I cannot (but I am happy to be proved wrong).

> Note that I would like to avoid any order constraints and dummy resources if 
> possible. But if it is unavoidable let me know.
>
> Thanks for the replies.
>
> With kind regards
> Philip.
>
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

Re: [ClusterLabs] VirtualDomain - node map - ?

[lejeczek via Users]

On 17/04/2023 06:17, Andrei Borzenkov wrote:

On 16.04.2023 16:29, lejeczek via Users wrote:



On 16/04/2023 12:54, Andrei Borzenkov wrote:

On 16.04.2023 13:40, lejeczek via Users wrote:

Hi guys

Some agents do employ that concept of node/host map 
which I

do not see in any manual/docs that this agent does - would
you suggest some technique or tips on how to achieve
similar?
I'm thinking specifically of 'migrate' here, as I 
understand

'migration' just uses OS' own resolver to call migrate_to
node.



No, pacemaker decides where to migrate the resource and
calls agents on the current source and then on the
intended target passing this information.



Yes pacemaker does that but - as I mentioned - some agents
do employ that "internal" nodes map "technique".
I see no mention of that/similar in the manual for
VirtualDomain so I asked if perhaps somebody had an idea of
how to archive such result by some other means.



What about showing example of these "some agents" or 
better describing what you want to achieve?



'ocf_heartbeat_galera' is one such agent.
But after more fiddling with 'VirtualDomain' I understand 
this agent does offer this functionality tough not in that 
clear way nor to that full extent, but it's there.

So, what I needed was possible to achieve.
thanks, L.
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

Re: [ClusterLabs] Help with tweaking an active/passive NFS cluster

[Ronny Adsetts]

Andrei Borzenkov wrote on 05/04/2023 08:36:
> On Fri, Mar 31, 2023 at 12:42 AM Ronny Adsetts
>  wrote:
>>
>> Hi,
>>
>> I wonder if someone more familiar with the workings of pacemaker/corosync 
>> would be able to assist in solving an issue.
>>
>> I have a 3-node NFS cluster which exports several iSCSI LUNs. The LUNs are 
>> presented to the nodes via multipathd.
>>
>> This all works fine except that I can't stop just one export. Sometimes I 
>> need to take a single filesystem offline for maintenance for example. Or if 
>> there's an issue and a filesystem goes offline and can't come back.
>>
>> There's a trimmed down config below but essentially I want all the NFS 
>> exports on one node but I don't want any of the exports to block. So it's OK 
>> to stop (or fail) a single export.
>>
>> My config has a group for each export and filesystem and another group for 
>> the NFS server and VIP. I then co-locate them together.
>>
>> Cut-down config to limit the number of exports:
>>
>> node 1: nfs-01
>> node 2: nfs-02
>> node 3: nfs-03
>> primitive NFSExportAdminHomes exportfs \
>> params clientspec="172.16.40.0/24" options="rw,async,no_root_squash" 
>> directory="/srv/adminhomes" fsid=dcfd1bbb-c026-4d6d-8541-7fc29d6fef1a \
>> op monitor timeout=20 interval=10 \
>> op_params interval=10
>> primitive NFSExportArchive exportfs \
>> params clientspec="172.16.40.0/24" options="rw,async,no_root_squash" 
>> directory="/srv/archive" fsid=3abb6e34-bff2-4896-b8ff-fc1123517359 \
>> op monitor timeout=20 interval=10 \
>> op_params interval=10 \
>> meta target-role=Started
>> primitive NFSExportDBBackups exportfs \
>> params clientspec="172.16.40.0/24" options="rw,async,no_root_squash" 
>> directory="/srv/dbbackups" fsid=df58b9c0-593b-45c0-9923-155b3d7d9483 \
>> op monitor timeout=20 interval=10 \
>> op_params interval=10
>> primitive NFSFSAdminHomes Filesystem \
>> params device="/dev/mapper/adminhomes-part1" 
>> directory="/srv/adminhomes" fstype=xfs \
>> op start interval=0 timeout=120 \
>> op monitor interval=60 timeout=60 \
>> op_params OCF_CHECK_LEVEL=20 \
>> op stop interval=0 timeout=240
>> primitive NFSFSArchive Filesystem \
>> params device="/dev/mapper/archive-part1" directory="/srv/archive" 
>> fstype=xfs \
>> op start interval=0 timeout=120 \
>> op monitor interval=60 timeout=60 \
>> op_params OCF_CHECK_LEVEL=20 \
>> op stop interval=0 timeout=240 \
>> meta target-role=Started
>> primitive NFSFSDBBackups Filesystem \
>> params device="/dev/mapper/dbbackups-part1" 
>> directory="/srv/dbbackups" fstype=xfs \
>> op start timeout=60 interval=0 \
>> op monitor interval=20 timeout=40 \
>> op stop timeout=60 interval=0 \
>> op_params OCF_CHECK_LEVEL=20
>> primitive NFSIP-01 IPaddr2 \
>> params ip=172.16.40.17 cidr_netmask=24 nic=ens14 \
>> op monitor interval=30s
>> group AdminHomes NFSFSAdminHomes NFSExportAdminHomes \
>> meta target-role=Started
>> group Archive NFSFSArchive NFSExportArchive \
>> meta target-role=Started
>> group DBBackups NFSFSDBBackups NFSExportDBBackups \
>> meta target-role=Started
>> group NFSServerIP NFSIP-01 NFSServer \
>> meta target-role=Started
>> colocation NFSMaster inf: NFSServerIP AdminHomes Archive DBBackups
> 
> This is entirely equivalent to defining a group and says that
> resources must be started in strict order on the same node. Like with
> a group, if an earlier resource cannot be started, all following
> resources are not started either.
> 
>> property cib-bootstrap-options: \
>> have-watchdog=false \
>> dc-version=2.0.1-9e909a5bdd \
>> cluster-infrastructure=corosync \
>> cluster-name=nfs-cluster \
>> stonith-enabled=false \
>> last-lrm-refresh=1675344768
>> rsc_defaults rsc-options: \
>> resource-stickiness=200
>>
>>
>> The problem is that if one export fails, none of the following exports will 
>> be attempted. Reading the docs, that's to be expected as each item in the 
>> colocation needs the preceding item to succeed.
>>
>> I tried changing the colocation line like so to remove the dependency:
>>
>> colocation NFSMaster inf: NFSServerIP ( AdminHomes Archive DBBackups )
>>
> 
> 1. The ( AdminHomes Archive DBBackups ) creates a set with
> sequential=false. Now, the documentation for "sequential" is one of
> the most obscure I have seen, but judging by "the individual members
> within any one set may or may not be colocated relative to each other
> (determined by the set’s sequential property)" and "A colocated set
> with sequential="false" makes sense only if there is another set in
> the constraint. Otherwise, the constraint has no effect" members of a
> set with sequential=false are not colocated on the same node.
> 
> 2. The condition is backward. You colocate 

Re: [ClusterLabs] VirtualDomain - node map - ?

[Klaus Wenninger]

On Mon, Apr 17, 2023 at 6:17 AM Andrei Borzenkov 
wrote:

> On 16.04.2023 16:29, lejeczek via Users wrote:
> >
> >
> > On 16/04/2023 12:54, Andrei Borzenkov wrote:
> >> On 16.04.2023 13:40, lejeczek via Users wrote:
> >>> Hi guys
> >>>
> >>> Some agents do employ that concept of node/host map which I
> >>> do not see in any manual/docs that this agent does - would
> >>> you suggest some technique or tips on how to achieve
> >>> similar?
> >>> I'm thinking specifically of 'migrate' here, as I understand
> >>> 'migration' just uses OS' own resolver to call migrate_to
> >>> node.
> >>>
> >>
> >> No, pacemaker decides where to migrate the resource and
> >> calls agents on the current source and then on the
> >> intended target passing this information.
> >>
> >
> > Yes pacemaker does that but - as I mentioned - some agents
> > do employ that "internal" nodes map "technique".
> > I see no mention of that/similar in the manual for
> > VirtualDomain so I asked if perhaps somebody had an idea of
> > how to archive such result by some other means.
> >
>
> What about showing example of these "some agents" or better describing
> what you want to achieve?
>

yep more context would be useful.
Or are you referring to the host-mapping used for fencing?

Klaus

> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
>
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

Re: [ClusterLabs] resource going to blocked status while we restart service via systemctl twice

[Klaus Wenninger]

On Mon, Apr 17, 2023 at 9:25 AM S Sathish S via Users 
wrote:

> Hi Team,
>
>
>
> TEST_node1 resource going to blocked status while we restart service via
> systemctl twice in less time/before completion of 1st systemctl command.
>
> In older pacemaker version 2.0.2 we don’t see this issue, only observing
> this issue on latest pacemaker version 2.1.15.
>

I'm not sure which change in particular with 2.1.5 would have created the
behavioral
change in your configuration. (rember discussion about reacting to
systemd-events
in pacemaker but didn't find anything already implemented on a quick check
of the
sources)
But basically afaik you are not expected to interfere with resources that
are under
pacemaker-control via anything else than pacemaker administration tooling
(high
or low level like e.g. pcs, crmsh, crm_resource, ...).
Otherwise you will see unexpected behavior. If you manage to do a restart
within
a monitoring-interval from the pacemaker-side you may get away without any
impact on the pacemaker-side though.

Klaus


>
> [root@node1 ~]# pcs resource status TEST_node1
>
>   * TEST_node1  (ocf::provider:TEST_RA):  Started node1
>
> [root@node1 ~]# systemctl restart TESTec
>
> [root@node1 ~]# cat /var/pid/TEST.pid
>
> 271466
>
> [root@node1 ~]# systemctl restart TESTec
>
> [root@node1 ~]# cat /var/pid/TEST.pid
>
> 271466
>
> [root@node1 ~]# pcs resource status TEST_node1
>
>   * TEST_node1  (ocf::provider:TEST_RA):  FAILED node1 (blocked)
>
> [root@node1 ~]#
>
>
>
>
>
> [root@node1 ~]# pcs resource config TEST_node1
>
> Resource: TEST_node1 (class=ocf provider=provider type=TEST_RA)
>
>   Meta Attributes: TEST_node1-meta_attributes
>
> failure-timeout=120s
>
> migration-threshold=5
>
> priority=60
>
>   Operations:
>
> migrate_from: TEST_node1-migrate_from-interval-0s
>
>   interval=0s
>
>   timeout=20
>
> migrate_to: TEST_node1-migrate_to-interval-0s
>
>   interval=0s
>
>   timeout=20
>
> monitor: TEST_node1-monitor-interval-10s
>
>   interval=10s
>
>   timeout=120s
>
>   on-fail=restart
>
> reload: TEST_node1-reload-interval-0s
>
>   interval=0s
>
>   timeout=20
>
> start: TEST_node1-start-interval-0s
>
>   interval=0s
>
>   timeout=120s
>
>   on-fail=restart
>
> stop: TEST_node1-stop-interval-0s
>
>   interval=0s
>
>   timeout=120s
>
>   on-fail=block
>
> [root@node1 ~]#
>
>
>
> Thanks and Regards,
>
> S Sathish S
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] HA problem: No live migration when setting node on standby

[Philip Schiller]

Hello Andrei,

you wrote:

As a workaround you could add dummy clone resource colocated with and 
ordered after your DRBD masters and order VM after this clone.


Thanks for the idea. This looks like a good option to solve my problem.

I have also researched a little more and came up with an option which seems to 
work for my case.
Would you be so kind to evaluate if i understand it correctly?

As mentioned in the original thread

/Wed Apr 12 05:28:48 EDT 2023/


My system looks like this:


I am using a simple two-nodes cluster with Zvol -> DRBD -> Virsh in
primary/primary mode (necessary for live migration).



Where drbd-resources and zvol are clones.
So it is basically a chain of resources, first zvol then drbd then vm.

From documentation i read that in those cases order constraints are not even 
necessary. This can be done with colocations constraints only
->https://access.redhat.com/documentation/de-de/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-orderconstraints-haar#s2-resourceorderlist-HAAR

There is stated:
A common situation is for an administrator to create a chain of ordered 
resources, where, for example, resource A starts before resource B which
starts before resource C. If your configuration requires that you 
create a set of resources that is colocated and started in order, you 
can configure a resource group that contains those resources, as 
described in Section 6.5, “Resource Groups”  .


I can't create a Resource Group because apparently clone-resources are not 
supported. So i have the following setup now:


colocation colocation-mas-drbd-alarmanlage-clo-pri-zfs-drbd_storage-INFINITY 
inf: mas-drbd-alarmanlage clo-pri-zfs-drbd_storage
colocation colocation-pri-vm-alarmanlage-mas-drbd-alarmanlage-INFINITY inf: 
pri-vm-alarmanlage:Started mas-drbd-alarmanlage:Master
location location-pri-vm-alarmanlage-s0-200 pri-vm-alarmanlage 200: s0


Migration works flawless and also the startup is correct: zvol -> drbd -> vm

I am little bit concerned though. Does corosync work like an interpeter and knows the 
correct order when i do  before ?

Another thing is the Multistate Constraint which i implanted -> 
pri-vm-alarmanlage:Started mas-drbd-alarmanlage:Master
Is this equivalent to the  which i was trying to achieve?

Basically i just want to have zvol started then drbd stared and promoted to 
master state and then finally vm started. All on the same node.
Can you confirm that my cluster does this behavior permanently with this 
configuration.

Note that I would like to avoid any order constraints and dummy resources if 
possible. But if it is unavoidable let me know.

Thanks for the replies.

With kind regards
Philip.
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] resource going to blocked status while we restart service via systemctl twice

[S Sathish S via Users]

Hi Team,

TEST_node1 resource going to blocked status while we restart service via 
systemctl twice in less time/before completion of 1st systemctl command.
In older pacemaker version 2.0.2 we don't see this issue, only observing this 
issue on latest pacemaker version 2.1.15.

[root@node1 ~]# pcs resource status TEST_node1
  * TEST_node1  (ocf::provider:TEST_RA):  Started node1
[root@node1 ~]# systemctl restart TESTec
[root@node1 ~]# cat /var/pid/TEST.pid
271466
[root@node1 ~]# systemctl restart TESTec
[root@node1 ~]# cat /var/pid/TEST.pid
271466
[root@node1 ~]# pcs resource status TEST_node1
  * TEST_node1  (ocf::provider:TEST_RA):  FAILED node1 (blocked)
[root@node1 ~]#


[root@node1 ~]# pcs resource config TEST_node1
Resource: TEST_node1 (class=ocf provider=provider type=TEST_RA)
  Meta Attributes: TEST_node1-meta_attributes
failure-timeout=120s
migration-threshold=5
priority=60
  Operations:
migrate_from: TEST_node1-migrate_from-interval-0s
  interval=0s
  timeout=20
migrate_to: TEST_node1-migrate_to-interval-0s
  interval=0s
  timeout=20
monitor: TEST_node1-monitor-interval-10s
  interval=10s
  timeout=120s
  on-fail=restart
reload: TEST_node1-reload-interval-0s
  interval=0s
  timeout=20
start: TEST_node1-start-interval-0s
  interval=0s
  timeout=120s
  on-fail=restart
stop: TEST_node1-stop-interval-0s
  interval=0s
  timeout=120s
  on-fail=block
[root@node1 ~]#

Thanks and Regards,
S Sathish S
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/