[ClusterLabs] Antw: [EXT] [Problem] crm_attirbute fails to expand run options.

[Ulrich Windl]

>>>  schrieb am 07.03.2023 um 08:41 in Nachricht
<215883790.380746.1678174910897.javamail.ya...@mail.yahoo.co.jp>:
> Hi All,
> 
> The crm_attribute command expands the contents of options from the 
> OCF_RESOURCE_INSTANCE environment variable if the p option is not specified.
> 
> However, if -INFINITY is specified as the value of the v option at this 
> time, crm_attributes will incorrectly expand -INFINITY as an option and 
> processing will fail.
> 
> 
> [root@rh91-01dev tools]# crm_attribute -p pgsql  -v 100
> [root@rh91-01dev tools]# crm_attribute -p pgsql  -v -INFINITY

Would "crm_attribute -p pgsql  -v -- -INFINITY" work?

> 
> [root@rh91-01dev tools]# OCF_RESOURCE_INSTANCE=pgsql crm_attribute -p  -v 
> 100
> [root@rh91-01dev tools]# OCF_RESOURCE_INSTANCE=pgsql crm_attribute -p  -v 
> -INFINITY
> crm_attribute: Could not map name=FINITY to a UUID
> 
> 
> 
> This problem occurs with the latest resource agent running on RHEL9.1, but 
> also with the development version of pacemaker.
> 
> 
> Due to this issue, some resource agents such as pgsql will fail to configure 
> a cluster after version 4.12.
> 
> It's a very serious problem.
> 
> RAs such as pgsql should be run with the p option(OCF_RESOURCE_INSTANCE) 
> provisionally when paired with a pacemaker version that does not resolve this 
> issue.
> 
> (snip)
> ocf_promotion_score() {
> ocf_version_cmp "$OCF_RESKEY_crm_feature_set" "3.10.0"
> res=$?
> if [ $res -eq 2 ] || [ $res -eq 1 ] || ! have_binary "crm_master"; 
> then
> ${HA_SBIN_DIR}/crm_attribute -p $OCF_RESOURCE_INSTANCE $@
> else
> ${HA_SBIN_DIR}/crm_master -l reboot $@
> fi
> (snip)
> 
> 
> This content has also been registered in the following Bugzilla:
> https://bugs.clusterlabs.org/show_bug.cgi?id=5509 
> 
> Best Regards,
> Hideo Yamauchi.
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Release crmsh 4.4.1

[Ulrich Windl]

>>> Xin Liang via Users  schrieb am 03.03.2023 um 08:00 
>>> in
Nachricht


> Hello everyone!
> 
> I'm happy to announce the release crmsh 4.4.1 now is available!
> 
> Changes since tag 4.4.0
> 
> Features:
> 
>   *   Enable "crm configure show related:" to show the objects by 
> given ra type (PR#978)
>   *   Parametrize the ra trace log dir 
> (PR#939)
>   *   Enable -N option setup the current node and peers all together 
> (PR#961)
> 
> Major fixes:
> 
>   *   Fix: utils: use -o and -n to compare files instead of strings for 
> crm_diff (bsc#1201312)
>   *   Fix: crm report: use sudo when under non root and hacluster user 
> (bsc#1199634)
>   *   Fix: utils: wait4dc: Make change since output of 'crmadmin -S' 
> changed(bsc#1199412)
>   *   Fix: bootstrap: stop and disable csync2.socket on removed node 
> (bsc#1199325)
>   *   Fix: crm report: Read data in a save way, to avoid 

/save/safe/ # I guess

> UnicodeDecodeError(bsc#1198180)
>   *   Fix: qdevice: Add lock to protect init_db_on_qnetd function 
> (bsc#1197323)
>   *   Fix: utils: Only raise exception when return code of systemctl command 
> over ssh larger than 4 (bsc#1196726)
> 
> Thanks to everyone who contributed to this release!
> More changes details please see 
> https://github.com/ClusterLabs/crmsh/blob/master/ChangeLog 
> Any feedback and suggestions are big welcome!
> 
> 
> 
> Regards,
> xin




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: Re: Antw: [EXT] resource cloned group colocations

[Ulrich Windl]

>>> Gerald Vogt  schrieb am 02.03.2023 um 17:27 in Nachricht
:
> On 02.03.23 14:51, Ulrich Windl wrote:
>>>>> Gerald Vogt  schrieb am 02.03.2023 um 14:43 in Nachricht
>> <9ba5cd78-7b3d-32ef-38cf-5c5632c46...@spamcop.net>:
>>> On 02.03.23 14:30, Ulrich Windl wrote:
>>>>>>> Gerald Vogt  schrieb am 02.03.2023 um 08:41 in 
>>>>>>> Nachricht
>>>> <624d0b70-5983-4d21-6777-55be91688...@spamcop.net>:
>>>>> Hi,
>>>>>
>>>>> I am setting up a mail relay cluster which main purpose is to maintain
>>>>> the service ips via IPaddr2 and move them between cluster nodes when
>>>>> necessary.
>>>>>
>>>>> The service ips should only be active on nodes which are running all
>>>>> necessary mail (systemd) services.
>>>>>
>>>>> So I have set up a resource for each of those services, put them into a
>>>>> group in order they should start, cloned the group as they are normally
>>>>> supposed to run on the nodes at all times.
>>>>>
>>>>> Then I added an order constraint
>>>>>  start mail-services-clone then start mail1-ip
>>>>>  start mail-services-clone then start mail2-ip
>>>>>
>>>>> and colocations to prefer running the ips on different nodes but only
>>>>> with the clone running:
>>>>>
>>>>>  colocation add mail2-ip with mail1-ip -1000
>>>>>  colocation mail1-ip with mail-services-clone
>>>>>  colocation mail2-ip with mail-services-clone
>>>>>
>>>>> as well as a location constraint to prefer running the first ip on the
>>>>> first node and the second on the second
>>>>>
>>>>>  location mail1-ip prefers ha1=2000
>>>>>  location mail2-ip prefers ha2=2000
>>>>>
>>>>> Now if I stop pacemaker on one of those nodes, e.g. on node ha2, it's
>>>>> fine. mail2-ip will be moved immediately to ha3. Good.
>>>>>
>>>>> However, if pacemaker on ha2 starts up again, it will immediately remove
>>>>> mail2-ip from ha3 and keep it offline, while the services in the group are
>>>>> starting on ha2. As the services unfortunately take some time to come
>>>>> up, mail2-ip is offline for more than a minute.
>>>>
>>>> That is because you wanted "mail2-ip prefers ha2=2000", so if the cluster
>>> _can_ run it there, then it will, even if it's running elsewhere.
>>>>
>>>> Maybe explain what you really want.
>>>
>>> As I wrote before: (and I have "fixed" my copy error above to use
>>> consistent resource names now)
>>>
>>> 1. I want to run all required services on all running nodes at all times.
>>>
>>> 2. I want two service IPs mail1-ip (ip1) and mail2-ip (ip2) running on
>>> the cluster but only on nodes where all required services are already
>>> running (and not just starting)
>>>
>>> 3. Both IPs should be running on two different nodes if possible.
>>>
>>> 4. Preferably mail1-ip should be on node ha1 if ha1 is running with all
>>> required services.
>>>
>>> 5. Preferably mail2-ip should be on node ha2 if ha1 is running with all
>>> required services.
>>>
>>> So most importantly: I want ip resources mail1-ip and mail2-ip only be
>>> active on nodes which are already running all services. They should only
>>> be moved to nodes on which all services are already running.
>> 
>> Hi!
>> 
>> Usually I prefer simple solutions over hightly complex ones.
>> WOuld it work to use a negative colocation for both IPs, as well as a 
> stickiness of maybe 500, then reducing the "prefer" value to something small 
> as 5 or 10.
>> Then the IP will stay elsewhere as long as the "basement services" run 
> there.
>> 
>> This approach does not change the order of resource operations; instead it 
> kind of minimizes them.
>> In my experience most people overspecify what the cluster should do.
> 
> Well, I guess it's not possible using a group. A group resource seems to 
> be good for a colocation the moment the first resource in the group has 
> been started (or even: is starting?). For a group which takes longer 
> time to completely start, that just doesn't work.
> 
> So I suppose the only two options would be to

[ClusterLabs] Antw: Re: Antw: [EXT] resource cloned group colocations

[Ulrich Windl]

>>> Gerald Vogt  schrieb am 02.03.2023 um 14:43 in Nachricht
<9ba5cd78-7b3d-32ef-38cf-5c5632c46...@spamcop.net>:
> On 02.03.23 14:30, Ulrich Windl wrote:
>>>>> Gerald Vogt  schrieb am 02.03.2023 um 08:41 in Nachricht
>> <624d0b70-5983-4d21-6777-55be91688...@spamcop.net>:
>>> Hi,
>>>
>>> I am setting up a mail relay cluster which main purpose is to maintain
>>> the service ips via IPaddr2 and move them between cluster nodes when
>>> necessary.
>>>
>>> The service ips should only be active on nodes which are running all
>>> necessary mail (systemd) services.
>>>
>>> So I have set up a resource for each of those services, put them into a
>>> group in order they should start, cloned the group as they are normally
>>> supposed to run on the nodes at all times.
>>>
>>> Then I added an order constraint
>>> start mail-services-clone then start mail1-ip
>>> start mail-services-clone then start mail2-ip
>>>
>>> and colocations to prefer running the ips on different nodes but only
>>> with the clone running:
>>>
>>> colocation add mail2-ip with mail1-ip -1000
>>> colocation mail1-ip with mail-services-clone
>>> colocation mail2-ip with mail-services-clone
>>>
>>> as well as a location constraint to prefer running the first ip on the
>>> first node and the second on the second
>>>
>>> location mail1-ip prefers ha1=2000
>>> location mail2-ip prefers ha2=2000
>>>
>>> Now if I stop pacemaker on one of those nodes, e.g. on node ha2, it's
>>> fine. mail2-ip will be moved immediately to ha3. Good.
>>>
>>> However, if pacemaker on ha2 starts up again, it will immediately remove
>>> mail2-ip from ha3 and keep it offline, while the services in the group are
>>> starting on ha2. As the services unfortunately take some time to come
>>> up, mail2-ip is offline for more than a minute.
>> 
>> That is because you wanted "mail2-ip prefers ha2=2000", so if the cluster 
> _can_ run it there, then it will, even if it's running elsewhere.
>> 
>> Maybe explain what you really want.
> 
> As I wrote before: (and I have "fixed" my copy error above to use 
> consistent resource names now)
> 
> 1. I want to run all required services on all running nodes at all times.
> 
> 2. I want two service IPs mail1-ip (ip1) and mail2-ip (ip2) running on 
> the cluster but only on nodes where all required services are already 
> running (and not just starting)
> 
> 3. Both IPs should be running on two different nodes if possible.
> 
> 4. Preferably mail1-ip should be on node ha1 if ha1 is running with all 
> required services.
> 
> 5. Preferably mail2-ip should be on node ha2 if ha1 is running with all 
> required services.
> 
> So most importantly: I want ip resources mail1-ip and mail2-ip only be 
> active on nodes which are already running all services. They should only 
> be moved to nodes on which all services are already running.

Hi!

Usually I prefer simple solutions over hightly complex ones.
WOuld it work to use a negative colocation for both IPs, as well as a 
stickiness of maybe 500, then reducing the "prefer" value to something small as 
5 or 10.
Then the IP will stay elsewhere as long as the "basement services" run there.

This approach does not change the order of resource operations; instead it kind 
of minimizes them.
In my experience most people overspecify what the cluster should do.

Kimnd regards,
Ulrich Windl

> 
> Thanks,
> 
> Gerald
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] resource cloned group colocations

[Ulrich Windl]

>>> Gerald Vogt  schrieb am 02.03.2023 um 08:41 in Nachricht
<624d0b70-5983-4d21-6777-55be91688...@spamcop.net>:
> Hi,
> 
> I am setting up a mail relay cluster which main purpose is to maintain 
> the service ips via IPaddr2 and move them between cluster nodes when 
> necessary.
> 
> The service ips should only be active on nodes which are running all 
> necessary mail (systemd) services.
> 
> So I have set up a resource for each of those services, put them into a 
> group in order they should start, cloned the group as they are normally 
> supposed to run on the nodes at all times.
> 
> Then I added an order constraint
>start mail-services-clone then start mail1-ip
>start mail-services-clone then start mail2-ip
> 
> and colocations to prefer running the ips on different nodes but only 
> with the clone running:
> 
>colocation add mail2-ip with mail1-ip -1000
>colocation ip1 with mail-services-clone
>colocation ip2 with mail-services-clone
> 
> as well as a location constraint to prefer running the first ip on the 
> first node and the second on the second
> 
>location ip1 prefers ha1=2000
>location ip2 prefers ha2=2000
> 
> Now if I stop pacemaker on one of those nodes, e.g. on node ha2, it's 
> fine. ip2 will be moved immediately to ha3. Good.
> 
> However, if pacemaker on ha2 starts up again, it will immediately remove 
> ip2 from ha3 and keep it offline, while the services in the group are 
> starting on ha2. As the services unfortunately take some time to come 
> up, ip2 is offline for more than a minute.

That is because you wanted "ip2 prefers ha2=2000", so if the cluster _can_ run 
it there, then it will, even if it's running elsewhere.

Maybe explain what you really want.

> 
> It seems the colocations with the clone are already good once the clone 
> group begins to start services and thus allows the ip to be removed from 
> the current node.
> 
> I was wondering how can I define the colocation to be accepted only if 
> all services in the clone have been started? And not once the first 
> service in the clone is starting?
> 
> Thanks,
> 
> Gerald
> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Systemd resource started on node after reboot before cluster is stable ?

[Ulrich Windl]

>>> Adam Cecile  schrieb am 16.02.2023 um 11:13 in 
>>> Nachricht
<4f0c1203-fd4b-a1cc-4e2f-44384c720...@le-vert.net>:

> On 2/16/23 07:57, Ulrich Windl wrote:
>>>>> Adam Cecile  schrieb am 15.02.2023 um 10:49 in
>> Nachricht
>> :

[...]
>>> Any idea why this happened ?
>> Your journal (syslog) should tell you!
> 
> Indeed, I overlooked yesterday... But it says it's pacemaker that 
> decided to start it:
> 
> 
[...]
> Feb 15 09:43:26 server3 pacemaker-controld[862]:  warning: Another DC 
> detected: server2 (op=noop)

This does not look OK! Does it happen all the time?


> Feb 15 09:43:26 server3 pacemaker-fenced[858]:  notice: Node server2 
> state is now member
> Feb 15 09:43:26 server3 pacemaker-controld[862]:  notice: State 
> transition S_ELECTION -> S_RELEASE_DC
> Feb 15 09:43:26 server3 pacemaker-controld[862]:  warning: Cancelling 
> timer for action 12 (src=67)
> Feb 15 09:43:26 server3 pacemaker-controld[862]:  notice: No need to 
> invoke the TE (A_TE_HALT) in state S_RELEASE_DC
> Feb 15 09:43:26 server3 pacemaker-attrd[860]:  notice: Node server2 
> state is now member
> Feb 15 09:43:26 server3 pacemaker-controld[862]:  notice: State 
> transition S_PENDING -> S_NOT_DC
> Feb 15 09:43:27 server3 pacemaker-attrd[860]:  notice: Setting 
> #attrd-protocol[server1]: (unset) -> 2
> Feb 15 09:43:27 server3 pacemaker-attrd[860]:  notice: Detected another 
> attribute writer (server2), starting new election
> Feb 15 09:43:27 server3 pacemaker-attrd[860]:  notice: Setting 
> #attrd-protocol[server2]: (unset) -> 2
> Feb 15 09:43:27 server3 IPaddr2(Shared-IPv4)[1258]: INFO:
> Feb 15 09:43:27 server3 ntpd[602]: Listen normally on 8 eth0 10.13.68.12:123
> Feb 15 09:43:27 server3 ntpd[602]: new interface(s) found: waking up 
> resolver
> => Feb 15 09:43:28 server3 pacemaker-controld[862]:  notice: Result of 
> start operation for tomcat9 on server3: ok
> Feb 15 09:43:29 server3 corosync[568]:   [KNET  ] pmtud: PMTUD link 
> change for host: 2 link: 0 from 485 to 1397
> Feb 15 09:43:29 server3 corosync[568]:   [KNET  ] pmtud: PMTUD link 
> change for host: 1 link: 0 from 485 to 1397
> Feb 15 09:43:29 server3 corosync[568]:   [KNET  ] pmtud: Global data MTU 
> changed to: 1397
> => Feb 15 09:43:29 server3 pacemaker-controld[862]:  notice: Requesting 
> local execution of stop operation for tomcat9 on server3
> 
> 
> Any idea ?




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Systemd resource started on node after reboot before cluster is stable ?

[Ulrich Windl]

>>> Adam Cecile  schrieb am 15.02.2023 um 10:49 in
Nachricht
:
> Hello,
> 
> Just had some issue with unexpected server behavior after reboot. This 
> node was powered off, so cluster was running fine with this tomcat9 
> resource running on a different machine.
> 
> After powering on this node again, it briefly started tomcat before 
> joining the cluster and decided to stop it again. I'm not sure why.
> 
> 
> Here is the systemctl status tomcat9 on this host:
> 
> tomcat9.service - Apache Tomcat 9 Web Application Server
>   Loaded: loaded (/lib/systemd/system/tomcat9.service; disabled; 
> vendor preset: enabled)
>  Drop-In: /etc/systemd/system/tomcat9.service.d
>   └─override.conf
>   Active: inactive (dead)
> Docs: https://tomcat.apache.org/tomcat-9.0-doc/index.html 
> 
> Feb 15 09:43:27 server tomcat9[1398]: Starting service [Catalina]
> Feb 15 09:43:27 server tomcat9[1398]: Starting Servlet engine: [Apache 
> Tomcat/9.0.43 (Debian)]
> Feb 15 09:43:27 server tomcat9[1398]: [...]
> Feb 15 09:43:29 server systemd[1]: Stopping Apache Tomcat 9 Web 
> Application Server...
> Feb 15 09:43:29 server systemd[1]: tomcat9.service: Succeeded.
> Feb 15 09:43:29 server systemd[1]: Stopped Apache Tomcat 9 Web 
> Application Server.
> Feb 15 09:43:29 server systemd[1]: tomcat9.service: Consumed 8.017s CPU 
> time.
> 
> You can see it is disabled and should NOT be started with the same, 
> start/stop is under Corosync control
> 
> 
> The systemd resource is defined like this:
> 
> primitive tomcat9 systemd:tomcat9.service \
>  op start interval=0 timeout=120 \
>  op stop interval=0 timeout=120 \
>  op monitor interval=60 timeout=100
> 
> 
> Any idea why this happened ?

Your journal (syslog) should tell you!

> 
> Best regards, Adam.



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Coming in Pacemaker 2.1.6: node attribute enhancements

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 06.02.2023 um 16:29 in 
>>> Nachricht
<1fc864736b788762d00fbc0b78da1b34fc1137d3.ca...@redhat.com>:
> Hi all,
> 
> Node attributes will receive several enhacements in the Pacemaker 2.1.6
> release, expected in late May. They are available now in the main
> branch.
> 
> A longstanding request to support transient utilization attributes has
> been implemented. Utilization attributes are used to assign resources
> to nodes based on local metrics such as CPU cores, RAM size, etc.
> Previously, these metrics could only be specified as permanent node
> attributes. As of 2.1.6, they can be set as transient node attributes
> as well (which last only until the cluster is restarted on the node).

Can you explain the advantages of that change?

> 
> Separately, attrd_updater will now support the -P/--pattern option to
> modify all attributes whose name matches a given pattern. Previously,
> only crm_attribute had the option.
> 
> attrd_updater and crm_attribute also both get a new -W/--wait option
> that can be set to "no", "local", or "cluster" when changing an
> attribute value.
> 
> With --wait=no (the default), the current behavior is preserved: the
> command returns as soon as the request has been submitted to the
> attribute manager.
> 
> With --wait=local, the tool will not return until the new value has
> been set locally. That is, a query run on the same node would return
> the new value.
> 
> With --wait=cluster, the tool will not return until the new value has
> been set on all nodes. That is, a query on any node would return the
> new value.
> 
> The --wait option will benefit scripts and resource agents that need
> specific guarantees. The feature set can be compared against 3.17.4 to
> determine support.
> -- 
> Ken Gaillot 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Problem with MariaDB cluster

[Ulrich Windl]

>>> Reid Wahl  schrieb am 27.01.2023 um 09:32 in Nachricht
:
> On Fri, Jan 27, 2023 at 12:23 AM Thomas CAS  wrote:
> 
>> Hello Reid,
>>
>>
>>
>> Thank you so much for your answer and bug report.
>>
>> If it is a bug, I do not understand why the problem is present in
>> production but not on my lab which is identical?
>>
> 
> That's a good question. I'm not sure, and I haven't worked much with this
> resource agent. Does the lab show interesting logs at all from the resource

Well, it's not an incredibly clever proposal (just obvious), but anyway:

Did you compare the RA files for each node? Did you compare the configs for 
each node?

> agent during startup? I wonder if it's hitting the same "No MySQL master
> present" issue but *not* the error that follows. That error is coming from
> mysql itself, not from the resource agent.
> 
> After the agent hits the "No MySQL master present" issue, it calls the
> unset_master() function. You can take a look at the agent script (in
> /usr/lib/ocf/resource.d/heartbeat/mysql by default) to see all the things
> that unset_master() is doing, and try to determine what's behaving
> differently in production vs. lab.
> 
[...]

Regards,
Ulrich




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Load balancing, of a sort

[Ulrich Windl]

>>> Antony Stone  schrieb am 25.01.2023 um
13:48 in
Nachricht <202301251348.58372.antony.st...@ha.open.source.it>:
> Hi.
> 
> I have a corosync / pacemaker 3-node cluster with a resource group which can

> 
> run on any node in the cluster.
> 
> Every night a cron job on the node which is running the resources performs 
> "crm_standby -v on" followed a short while later by "crm_standby -v off" in

> order to force the resources to migrate to another node member.

Hi!

As Ken pointed out, there are more elegant ways to do that (not to talk about
more elegant ways to rotate the logs, but admittedly there are bad applications
out in the wild).

Hard to say what your cluster status exactly is, but my favourite command is
"crm_mon -1Arfj" to check 

Regards,
Ulrich


> 
> We do this partly to verify that all nodes are capable of running the 
> resources, and partly because some of those resources generate significant 
> log 
> files, and if one machine just keeps running them day after day, we run out

> of 
> disk space (which effectively means we just need to add more capacity to the

> 
> machines, which can be done, but at a cost).
> 
> So long as a machine gets a day when it's not running the resources, a 
> combination of migrating the log files to a central server, plus standard 
> logfile rotation, takes care of managing the disk space.
> 
> What I notice, though, is that two of the machines tend to swap the 
> resources 
> between them, and the third machine hardly ever becomes the active node.
> 
> Is there some way of influencing the node selection mechanism when resources

> 
> need to move away from the currently active node, so that, for example, the

> least recently used node could be favoured over the rest?
> 
> 
> Thanks,
> 
> 
> Antony.
> 
> -- 
> I want to build a machine that will be proud of me.
> 
>  - Danny Hillis, creator of The Connection Machine
> 
>Please reply to the
list;
>  please *don't* CC 
> me.
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] resource-agents v4.12.0

[Ulrich Windl]

>>> Oyvind Albrigtsen  schrieb am 25.01.2023 um 10:50 in
Nachricht <20230125095007.bxalry76nfo7m...@redhat.com>:
> ClusterLabs is happy to announce resource-agents v4.12.0.
> 
> Source code is available at:
> https://github.com/ClusterLabs/resource-agents/releases/tag/v4.12.0 
> 
> The most significant enhancements in this release are:
> - new resource agents:
>   - azure-events-az
>   - iface-macvlan
> 
> - bugfixes and enhancements:
>   - configure/spec: remove google-api-client BuildReq as the gcp-agents 
> build fine without it
>   - configure: fix bashisms
>   - dev-guide: update default branch name
>   - ocf-shellfuncs: add ocf_promotion_score() to use crm_attribute 
> -p/crm_master depending on Pacemaker version
>   - spec: remove redhat-lsb-core dependency (lsb_release)
>   - spec: use cluster-glue-devel for opensuse > 15
>   - all agents: update to promotable terms
>   - CTDB: change public_addresses validate-all warning to info
>   - CTDB: move process to root cgroup if realtime scheduling is enabled
>   - Filesystem: improve logic for UUID/label and add note that 
> /dev/disk/by-{uuid,label}/ are preferred on Linux

>   - Filesystem: remove validate-all mountpoint warning as it is auto-created 
> during start-action if it doesnt exist

Actually I had always considered that feature to be undesirable: If the cluster 
should mount a filesystem where the mountpoint does not exist, it's a local 
configuration error IMHO. Most likely the admin forgot several more steps in 
the setup...

...
Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Re: corosync 2.4.4 version provide secure the communication by default

[Ulrich Windl]

>>> Jan Friesse  schrieb am 23.01.2023 um 15:54 in 
>>> Nachricht
:
> On 23/01/2023 12:51, Ulrich Windl wrote:
>>>>> Jan Friesse  schrieb am 23.01.2023 um 10:20 in 
>>>>> Nachricht
>> :
>>> Hi,
>>>
>>> On 23/01/2023 01:37, S Sathish S via Users wrote:
>>>> Hi Team,
>>>>
>>>> corosync 2.4.4 version provide mechanism to secure the communication path
>>> between nodes of a cluster by default? bcoz in our configuration secauth is
>>> turned off but still communication occur is encrypted.
>>>>
>>>> Note : Capture tcpdump for port 5405 and I can see that the data is already
>>> garbled and not in the clear.
>>>
>>> It's binary protocol so don't expect some really readable format (like
>>> xml/json/...). But with your config it should be unencrypted. You can
>>> check message "notice  [TOTEM ] Initializing transmit/receive security
>>> (NSS) crypto: none hash: none" during start of corosync.
>> 
>> Probably a good example for "a false feeling of security" (you think the 
> comminication is encrypted, while in fact it is not).
> 
> Yeah, "none" and "none" is definitively "false feeling of security" and 
> definitively suggest communication is encrypted. Sigh...

I meant "looking at the bytes on the network", not at the tool's output...

> 
> 
>> 
>>>
>>> Regards,
>>> Honza
>>>
>>>
>>>>
>>>> [root@node1 ~]# cat /etc/corosync/corosync.conf
>>>> totem {
>>>>   version: 2
>>>>   cluster_name: OCC
>>>>  secauth: off
>>>>   transport: udpu
>>>> }
>>>>
>>>> nodelist {
>>>>   node {
>>>>   ring0_addr: node1
>>>>   nodeid: 1
>>>>   }
>>>>
>>>>   node {
>>>>   ring0_addr: node2
>>>>   nodeid: 2
>>>>   }
>>>>
>>>>   node {
>>>>   ring0_addr: node3
>>>>   nodeid: 3
>>>>   }
>>>> }
>>>>
>>>> quorum {
>>>>   provider: corosync_votequorum
>>>> }
>>>>
>>>> logging {
>>>>   to_logfile: yes
>>>>   logfile: /var/log/cluster/corosync.log
>>>>   to_syslog: no
>>>>   timestamp: on
>>>> }
>>>>
>>>> Thanks and Regards,
>>>> S Sathish S
>>>>
>>>>
>>>> ___
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>>
>>>
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
>> 
>> 
>> 
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: corosync 2.4.4 version provide secure the communication by default

[Ulrich Windl]

>>> Jan Friesse  schrieb am 23.01.2023 um 10:20 in 
>>> Nachricht
:
> Hi,
> 
> On 23/01/2023 01:37, S Sathish S via Users wrote:
>> Hi Team,
>> 
>> corosync 2.4.4 version provide mechanism to secure the communication path 
> between nodes of a cluster by default? bcoz in our configuration secauth is 
> turned off but still communication occur is encrypted.
>> 
>> Note : Capture tcpdump for port 5405 and I can see that the data is already 
> garbled and not in the clear.
> 
> It's binary protocol so don't expect some really readable format (like 
> xml/json/...). But with your config it should be unencrypted. You can 
> check message "notice  [TOTEM ] Initializing transmit/receive security 
> (NSS) crypto: none hash: none" during start of corosync.

Probably a good example for "a false feeling of security" (you think the 
comminication is encrypted, while in fact it is not).

> 
> Regards,
>Honza
> 
> 
>> 
>> [root@node1 ~]# cat /etc/corosync/corosync.conf
>> totem {
>>  version: 2
>>  cluster_name: OCC
>> secauth: off
>>  transport: udpu
>> }
>> 
>> nodelist {
>>  node {
>>  ring0_addr: node1
>>  nodeid: 1
>>  }
>> 
>>  node {
>>  ring0_addr: node2
>>  nodeid: 2
>>  }
>> 
>>  node {
>>  ring0_addr: node3
>>  nodeid: 3
>>  }
>> }
>> 
>> quorum {
>>  provider: corosync_votequorum
>> }
>> 
>> logging {
>>  to_logfile: yes
>>  logfile: /var/log/cluster/corosync.log
>>  to_syslog: no
>>  timestamp: on
>> }
>> 
>> Thanks and Regards,
>> S Sathish S
>> 
>> 
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: cibadmin response unexpected

[Ulrich Windl]

>>> Reid Wahl  schrieb am 18.01.2023 um 05:46 in Nachricht
:
> On Tue, Jan 17, 2023 at 7:17 PM d tbsky  wrote:
>>
>> Hi:
>>I am using RHEL 9.1 with pacemaker-cli-2.1.4-5. I tried command below:
>>
>> > cibadmin -Q -o xxx
>>
>> I expect the result tell me that "xxx" scope is not exist, but the
>> command return the whole configuration. is this the normal behavior?

It seems to be bad design: There should not have been output and there should
have been an exit code != 0.
However as wrong usages seem to be out in the wild, so it cannot be fixed
even. Very unfortunate.
Building on such parts to make a reliable system is really a challenge!
So for the future it's best to avoid releasing "quick & dirty" solutions IMHO


>>
>> thanks a lot for help.
> 
> Yes -- we recently added a warning for invalid scope, but we can't
> throw an error until a compatibility break. We have to preserve the
> existing behavior until then.
> 
> New warning:
> Low: tools: Validate scope in cibadmin.c
> (https://github.com/ClusterLabs/pacemaker/commit/0b5f8a24)

IMHO trying to retrieve something that does not exist should output nothing
AND prefrably an error exit code, too (see above).

Regards,
Ulrich


> 
>>
>> Regards,
>> tbskyd
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
> 
> 
> -- 
> Regards,
> 
> Reid Wahl (He/Him)
> Senior Software Engineer, Red Hat
> RHEL High Availability - Pacemaker
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Failed 'virsh' call when test RA run by crm_resource (con't) - SOLVED!

[Ulrich Windl]

>>> Reid Wahl  schrieb am 13.01.2023 um 07:55 in Nachricht
:
> On Thursday, January 12, 2023, Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
>>>>> Reid Wahl  schrieb am 12.01.2023 um 18:00 in
> Nachricht
>> :
>>> On Thu, Jan 12, 2023 at 6:24 AM Madison Kelly  wrote:
>>
>> ...
>>
>>> Hooray! I'm really glad someone figured this out.
>>>
>>> Based on a link that was shared in another thread, maybe it worked
>>> fine on my machine due to a newer polkit version.
>>> - https://gitlab.com/libvirt/libvirt/-/issues/366#note_1102131966 
>>>
>>> The RHEL 8 BZ that's linked in the ServerFault thread was auto-closed
>>> by a bot. Not sure if the fix will find its way in some other way or
>>> not.
>>> - https://bugzilla.redhat.com/show_bug.cgi?id=1726714 
>>
>> In a standard configuration you can still add comments even if the bug is
> closed.
> 
> Yep, I don't think I said otherwise

Sorry, it sounded like the bug being closed prevents any further action.

> 
>>
>>>
>>>>
>>>> --
>>>> Madison Kelly
>>>> Alteeve's Niche!
>>>> Chief Technical Officer
>>>> c: +1-647-471-0951
>>>> https://alteeve.com/ 
>>>>
>>>> ___
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Reid Wahl (He/Him)
>>> Senior Software Engineer, Red Hat
>>> RHEL High Availability - Pacemaker
>>>
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
>>
>>
>>
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
>>
> 
> -- 
> Regards,
> 
> Reid Wahl (He/Him)
> Senior Software Engineer, Red Hat
> RHEL High Availability - Pacemaker




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Failed 'virsh' call when test RA run by crm_resource (con't) - SOLVED!

[Ulrich Windl]

>>> Reid Wahl  schrieb am 12.01.2023 um 18:00 in Nachricht
:
> On Thu, Jan 12, 2023 at 6:24 AM Madison Kelly  wrote:

...

> Hooray! I'm really glad someone figured this out.
> 
> Based on a link that was shared in another thread, maybe it worked
> fine on my machine due to a newer polkit version.
> - https://gitlab.com/libvirt/libvirt/-/issues/366#note_1102131966 
> 
> The RHEL 8 BZ that's linked in the ServerFault thread was auto-closed
> by a bot. Not sure if the fix will find its way in some other way or
> not.
> - https://bugzilla.redhat.com/show_bug.cgi?id=1726714 

In a standard configuration you can still add comments even if the bug is 
closed.

> 
>>
>> --
>> Madison Kelly
>> Alteeve's Niche!
>> Chief Technical Officer
>> c: +1-647-471-0951
>> https://alteeve.com/ 
>>
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
> -- 
> Regards,
> 
> Reid Wahl (He/Him)
> Senior Software Engineer, Red Hat
> RHEL High Availability - Pacemaker
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Failed 'virsh' call when test RA run by crm_resource (con't)

[Ulrich Windl]

>>> Madison Kelly  schrieb am 12.01.2023 um 07:36 in
Nachricht
<2e68d19b-90e2-98e2-47ed-17c4f69df...@alteeve.com>:
> On 2023-01-12 01:26, Reid Wahl wrote:

...

> Appreciate the stab, didn't stop the hang though :(

Well virsh has debugging options, too 

> 
> -- 
> Madison Kelly
> Alteeve's Niche!
> Chief Technical Officer
> c: +1-647-471-0951
> https://alteeve.com/ 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Failed 'virsh' call when test RA run by crm_resource (con't)

[Ulrich Windl]

>>> Madison Kelly  schrieb am 12.01.2023 um 05:10 in 
>>> Nachricht
:
> Hi all,
> 
>There was a lot of sub-threads, so I figured it's helpful to start a 
> new thread with a summary so far. For context; I have a super simple 
> perl script that pretends to be an RA for the sake of debugging.
> 
> https://pastebin.com/9z314TaB 

One thing I noticed: When "opening" a pipe, checking the "close" status is 
sometimes even more important than checking the open status; you don't check 
the close status...


> 
>I've had variations log environment variables and confirmed that all 
> the variables in the direct call that work are in the crm_resource 
> triggered call. There are no selinux issues logged in audit.log and 
> selinux is permissive. The script logs the real and effective UID and 
> GID and it's the same in both instances. Calling other shell programs 
> (tested with 'hostname') run fine, this is specifically crm_resource -> 
> test RA -> virsh call.
> 
>I ran strace on the virsh call from inside my test script (changing 
> 'virsh.good' to 'virsh.bad' between running directly and via 
> crm_resource. The strace runs made six files each time. Below are 
> pastebin links with the outputs of the six runs in one paste, but each 
> file's output is in it's own block (search for file: to see the 
> different file outputs)
> 
> Good/direct run of the test RA:
> - https://pastebin.com/xtqe9NSG 
> 
> Bad/crm_resource triggered run of the test RA:
> - https://pastebin.com/vBiLVejW 
> 
> Still absolutely stumped.
> 
> -- 
> Madison Kelly
> Alteeve's Niche!
> Chief Technical Officer
> c: +1-647-471-0951
> https://alteeve.com/ 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: RA hangs when called by crm_resource (resending text format)

[Ulrich Windl]

>>> Madison Kelly  schrieb am 11.01.2023 um 22:06 in
Nachricht
<8a2f2d45-0419-8e97-1805-2998a9b83...@alteeve.com>:
> On 2023-01-11 01:13, Vladislav Bogdanov wrote:
>> I suspect that valudate action is run as a non-root user.
> 
> I modified the script to log the real and effective UIDs and it's 
> running as root in both instances.

I'm not running Redhat, but could it be one of the additional security
features (selinux)?
If possible maybe try to disable those for the test, or test your RA on
non-Redhat (just for testing).

Regards,
Ulrich

> 
>> Madison Kelly  11 января 2023 г. 07:06:55 написал:
>> 
>>> On 2023-01-11 00:21, Madison Kelly wrote:
 On 2023-01-11 00:14, Madison Kelly wrote:
> Hi all,
>
> Edit: Last message was in HTML format, sorry about that.
>
>I've got a hell of a weird problem, and I am absolutely stumped on
> what's going on.
>
>The short of it is; if my RA is called from the command line, it's
> fine. If a resource exists, monitor, enable, disable, all that stuff
> works just fine. If I try to create a resource, it hangs on the
> validate stage. Specifically, it hangs when 'pcs' calls:
>
> crm_resource --validate --output-as xml --class ocf --agent server
> --provider alteeve --option name=
>
>Specifically, it hangs when it tries to make a shell call (to
> virsh, specifically, but that doesn't matter). So to debug, I started
> stripping down my RA simpler and simpler until I was left with the
> very most basic of programs;
>
> https://pastebin.com/VtSpkwMr 
>
>That is literally the simplest program I could write that made the
> shell call. The 'open()' call is where it hangs.
>
> When I call directly;
>
> time /usr/lib/ocf/resource.d/alteeve/server --validate-all --server
> srv04-test; echo rc:$?
>
> 
> real0m0.061s
> user0m0.037s
> sys0m0.014s
> rc:0
> 
>
> It's just fine. I can see in the log the output from the 'virsh' call
> as well. However, when I call from crm_resource;
>
> time crm_resource --validate --output-as xml --class ocf --agent
> server --provider alteeve --option name=srv04-test; echo rc:$?
>
> 
> 
> provider="alteeve">
>  
>   execution_message="Timed Out" reason="Resource agent did not exit
> within specified timeout"/>
>
>
>  
>crm_resource: Error performing operation: Error
> occurred
>  
>
> 
>
> real0m20.521s
> user0m0.022s
> sys0m0.010s
> rc:1
> 
>
> In the log file, I see (from line 20 of the super-simple-test-script):
>
> 
> Calling: [/usr/bin/virsh dumpxml --inactive srv04-test 2>&1;
> /usr/bin/echo return_code:0 |]
> 
>
> Then nothing else.
>
> The strace output is: https://pastebin.com/raw/UCEUdBeP 
>
> Environment;
>
> * selinux is permissive
> * Pacemaker 2.1.5-4.el8
> * pcs 0.10.15
> * 4.18.0-408.el8.x86_64
> * CentOS Stream release 8
>
> Any help is appreciated, I am stumped. :/

 After sending this, I tried having my "RA" call 'hostname', and that
 worked fine. I switched back to 'virsh list --all', and that hangs. So
 it seems to somehow be related to call 'virsh' specifically.

>>>
>>> OK, so more info... Knowing now that it's a problem with the virsh call
>>> specifically (but only when validating, existing VMs monitor, enable,
>>> disable fine, all which repeatedly call virsh), I noticed a few things.
>>>
>>> First, I see in the logs:
>>>
>>> 
>>> Jan 11 00:30:43 mk-a07n02.digimer.ca libvirtd[2937]: Cannot recv data:
>>> Connection reset by peer
>>> 
>>>
>>> So with this, I further simplified my test script to this:
>>>
>>> https://pastebin.com/Ey8FdL1t 
>>>
>>> Then when I ran my test script directly, the strace output is:
>>>
>>> Good: https://pastebin.com/Trbq67ub 
>>>
>>> When my script is called via crm_resource, the strace is this:
>>>
>>> Bad: https://pastebin.com/jtbzHrUM 
>>>
>>> The first difference I can see happens around line 929 in the good
>>> paste, the line "futex(0x7f48b0001ca0, FUTEX_WAKE_PRIVATE, 1) = 0"
>>> exists, which doesn't in the bad paste. Shortly after, I start seeing:
>>>
>>> 
>>> line: [write(4, "\1\0\0\0\0\0\0\0", 8) = 8]
>>> line: [brk(NULL)   = 0x562b7877d000]
>>> line: [brk(0x562b787aa000) = 0x562b787aa000]
>>> line: [write(4, "\1\0\0\0\0\0\0\0", 8) = 8]
>>> 
>>>
>>> Around line 959 in the bad paste. There are more brk() lines, and not
>>> long after the output stops.
>>>
>>> -- 
>>> Madison Kelly
>>> Alteeve's Niche!
>>> Chief Technical Officer
>>> c: +1-647-471-0951
>>> https://alteeve.com/ 
>>>
>>> ___
>>> Manage your 

[ClusterLabs] Antw: [EXT] pacemaker user question

[Ulrich Windl]

>>> "Jelen, Piotr"  schrieb am 10.01.2023 um 14:51 
>>> in
Nachricht


> HI ,
> 
> I would like to ask you if the hacluster and haclinet group is hardcoded 
> into the pacemaker or we can use other uid/gid than the standard 189/189?

I'd consider any software that hard-codes UIDs or GID into binaries (or even 
shell scripts) to be broken.
While user names might be acceptable, it woul still be better of those are 
configurable.
So I guess it should work...

> 
> I was able to create pacemaker  cluster with different uid/gid in my home 
> cluster lab  by running
> 
> esg4bel37# groupadd -r haclient -g 2655
> esg4bel37# useradd -r -g haclient -u 2655-s /sbin/nologin -c "cluster user" 
> hacluster
> esg4bel39# groupadd -r haclient -g 2655
> esg4bel39# useradd -r -g haclient -u 2655-s /sbin/nologin -c "cluster user" 
> hacluster
> before installing pacemaker cluster
> 
> Can you please tell me if this type of installation might cause any issue?
> [image.png]
> 
> Regards
> Piotr Jelen
> Senior Systems Platform Engineer
> 
> Mastercard
> Mountain View, Central Park  | Leopard
> [cid:image001.png@01D924F7.81DDF770]
> 
> 
> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for 
> the use of the intended recipient and may contain information that is 
> privileged, confidential or exempt from disclosure under applicable law. If 
> you are not the intended recipient, any disclosure, distribution or other use 
> of this e-mail message or attachments is prohibited. If you have received 
> this e-mail message in error, please delete and notify the sender 
> immediately. Thank you.




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: RA hangs when called by crm_resource (resending text format)

[Ulrich Windl]

>>> Madison Kelly  schrieb am 11.01.2023 um 06:21 in 
>>> Nachricht
<74df2c8e-1cff-ba07-7f4a-070be296b...@alteeve.com>:
> On 2023-01-11 00:14, Madison Kelly wrote:
>> Hi all,
>> 
>> Edit: Last message was in HTML format, sorry about that.
>> 
>>I've got a hell of a weird problem, and I am absolutely stumped on 
>> what's going on.
>> 
>>The short of it is; if my RA is called from the command line, it's 
>> fine. If a resource exists, monitor, enable, disable, all that stuff 
>> works just fine. If I try to create a resource, it hangs on the validate 
>> stage. Specifically, it hangs when 'pcs' calls:
>> 
>> crm_resource --validate --output-as xml --class ocf --agent server 
>> --provider alteeve --option name=
>> 
>>Specifically, it hangs when it tries to make a shell call (to virsh, 
>> specifically, but that doesn't matter). So to debug, I started stripping 
>> down my RA simpler and simpler until I was left with the very most basic 
>> of programs;
>> 
>> https://pastebin.com/VtSpkwMr 
>> 
>>That is literally the simplest program I could write that made the 
>> shell call. The 'open()' call is where it hangs.
>> 
>> When I call directly;
>> 
>> time /usr/lib/ocf/resource.d/alteeve/server --validate-all --server 
>> srv04-test; echo rc:$?
>> 
>> 
>> real0m0.061s
>> user0m0.037s
>> sys0m0.014s
>> rc:0
>> 
>> 
>> It's just fine. I can see in the log the output from the 'virsh' call as 
>> well. However, when I call from crm_resource;
>> 
>> time crm_resource --validate --output-as xml --class ocf --agent server 
>> --provider alteeve --option name=srv04-test; echo rc:$?
>> 
>> 
>> 
>>> provider="alteeve">
>>  
>>  > execution_message="Timed Out" reason="Resource agent did not exit within 
>> specified timeout"/>
>>
>>
>>  
>>crm_resource: Error performing operation: Error 
>> occurred
>>  
>>
>> 
>> 
>> real0m20.521s
>> user0m0.022s
>> sys0m0.010s
>> rc:1
>> 
>> 
>> In the log file, I see (from line 20 of the super-simple-test-script):
>> 
>> 
>> Calling: [/usr/bin/virsh dumpxml --inactive srv04-test 2>&1; 
>> /usr/bin/echo return_code:0 |]
>> 

In VirtualDomain RA I found a similar command (assuming that works):
 virsh $VIRSH_OPTIONS dumpxml --inactive --security-info ${DOMAIN_NAME} >
 ${CFGTMP}

virsh is somewhat strange; libvirtd is running, right?

>> 
>> Then nothing else.
>> 
>> The strace output is: https://pastebin.com/raw/UCEUdBeP 
>> 
>> Environment;
>> 
>> * selinux is permissive
>> * Pacemaker 2.1.5-4.el8
>> * pcs 0.10.15
>> * 4.18.0-408.el8.x86_64
>> * CentOS Stream release 8
>> 
>> Any help is appreciated, I am stumped. :/
> 
> After sending this, I tried having my "RA" call 'hostname', and that 
> worked fine. I switched back to 'virsh list --all', and that hangs. So 
> it seems to somehow be related to call 'virsh' specifically.
> 
> -- 
> Madison Kelly
> Alteeve's Niche!
> Chief Technical Officer
> c: +1-647-471-0951
> https://alteeve.com/ 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] RA hangs when called by crm_resource

[Ulrich Windl]

Depending on the langiuage your RA is written in, you could debug it, or try 
ocf-tester to debug your RA.
For shell scripts you could add some "ocf_log debug ..." statements.

>>> Madison Kelly  schrieb am 11.01.2023 um 06:11 in 
>>> Nachricht
<06935f6a-a858-c8fe-7b81-168157e5c...@alteeve.com>:
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] clear_failcount operation times out, makes it impossible to use the cluster

[Ulrich Windl]

>>> Ulrich Windl schrieb am 05.01.2023 um 08:22 in Nachricht <63B67A9F.36B :
161 :
60728>:
>>>> Krzysztof Bodora  schrieb am 04.01.2023 um
09:51
> in Nachricht :
> > It's an old installation, the error started appearing when one of the 
> > nodes was disconnected and the OS was re-installed, after which the 
> > pacemaker configuration was restored from a backup (pcs cluster 
> > cib-push) and the node rejoined the cluster. The failcount itself was at 
> > 8 for some time before, though. The configuration looks like this:
> 
> My guess is that it was "not enough" to restore the pacemaker configuration

> only.
> What was the reason to re-install the OS? In the last 30 years I never had 
> to reinstall an OS, even if support suggested to do so .
> I suggest to repeat and validate al lthe steps needed to set up a new 
> cluster.
> What does "corosync-cfgtool -s" say?

I forgot to ask: And the media used to reinstall was the same used to install
the OS initially, meaning: do all the nodes run the same software versions
now?

> 
> Regards,
> Ulrich
> 
> 
> > 
> > pcs config
> > 
> > Cluster Name:
> > Corosync Nodes:
> >   10.151.50.43 10.151.50.42
> > Pacemaker Nodes:
> >   swdal1-ISCSI01 swdal1-ISCSI02
> > 
> > Resources:
> >   Clone: ping_resource-clone
> >Resource: ping_resource (class=ocf provider=pacemaker type=ping)
> > Attributes: multiplier=1000 dampen=15s attempts=4 timeout=3 
> > host_list="10.151.16.60 10.151.16.50 10.151.17.50 10.151.17.60"
> > Operations: start interval=0s timeout=60 
> > (ping_resource-start-interval-0s)
> > stop interval=0s timeout=20
(ping_resource-stop-interval-0s)
> > monitor interval=5s timeout=15s 
> > (ping_resource-monitor-interval-5s)
> >   Resource: Pool-1 (class=ocf provider=oe type=zfs)
> >Attributes: pool_name=Pool-1 pool_id=9200090953161398950 
> > encryption_password_hash=None
> >Meta Attrs: failure-timeout=30 is-managed=True 
> > encryption_password_hash=None
> >Operations: start interval=0s timeout=300 (Pool-1-start-interval-0s)
> >stop interval=0s timeout=300 (Pool-1-stop-interval-0s)
> >monitor interval=10 timeout=60
(Pool-1-monitor-interval-10)
> >   Resource: Pool-0 (class=ocf provider=oe type=zfs)
> >Attributes: pool_name=Pool-0 pool_id=4165732781319344895 
> > encryption_password_hash=None
> >Meta Attrs: failure-timeout=30 is-managed=True 
> > encryption_password_hash=None
> >Operations: start interval=0s timeout=300 (Pool-0-start-interval-0s)
> >stop interval=0s timeout=300 (Pool-0-stop-interval-0s)
> >monitor interval=10 timeout=60
(Pool-0-monitor-interval-10)
> > 
> > Stonith Devices:
> > Fencing Levels:
> > 
> > Location Constraints:
> >Resource: Pool-0
> >  Enabled on: swdal1-ISCSI01 (score:1) 
> > (id:location-Pool-0-swdal1-ISCSI01-1)
> >  Constraint: location-Pool-0
> >Rule: score=-INFINITY boolean-op=or (id:location-Pool-0-rule)
> >  Expression: pingd lt 1  (id:location-Pool-0-rule-expr)
> >  Expression: not_defined pingd (id:location-Pool-0-rule-expr-1)
> >Resource: Pool-1
> >  Enabled on: swdal1-ISCSI01 (score:1) 
> > (id:location-Pool-1-swdal1-ISCSI01-1)
> >  Constraint: location-Pool-1
> >Rule: score=-INFINITY boolean-op=or (id:location-Pool-1-rule)
> >  Expression: pingd lt 1  (id:location-Pool-1-rule-expr)
> >  Expression: not_defined pingd (id:location-Pool-1-rule-expr-1)
> > Ordering Constraints:
> > Colocation Constraints:
> > 
> > Resources Defaults:
> >   resource-stickiness: 10
> > Operations Defaults:
> >   record-pending: true
> > 
> > Cluster Properties:
> >   batch-limit: 1
> >   cluster-infrastructure: corosync
> >   cluster-recheck-interval: 180
> >   dc-version: 1.1.12-1.1.12+git+561c4cf
> >   no-quorum-policy: ignore
> >   stonith-enabled: false
> >   stop-orphan-resources: false
> > 
> > W dniu 02.01.2023 o 13:12, Ulrich Windl pisze:
> >> Hi!
> >>
> >> I wonder: Is this a new installation, or is it a new bug in an old 
> > installation?
> >> For the first case I'd recommend to start with current software, and for
the 
> 
> > second case please describe what had changed or what had triggered the 
> > situation.
> >> Also provide basic configuration data, please.
> >>
> >> Regards,

[ClusterLabs] Antw: Re: Antw: [EXT] clear_failcount operation times out, makes it impossible to use the cluster

[Ulrich Windl]

>>> Krzysztof Bodora  schrieb am 04.01.2023 um
09:51
in Nachricht :
> It's an old installation, the error started appearing when one of the 
> nodes was disconnected and the OS was re-installed, after which the 
> pacemaker configuration was restored from a backup (pcs cluster 
> cib-push) and the node rejoined the cluster. The failcount itself was at 
> 8 for some time before, though. The configuration looks like this:

My guess is that it was "not enough" to restore the pacemaker configuration
only.
What was the reason to re-install the OS? In the last 30 years I never had to
reinstall an OS, even if support suggested to do so .
I suggest to repeat and validate al lthe steps needed to set up a new
cluster.
What does "corosync-cfgtool -s" say?

Regards,
Ulrich


> 
> pcs config
> 
> Cluster Name:
> Corosync Nodes:
>   10.151.50.43 10.151.50.42
> Pacemaker Nodes:
>   swdal1-ISCSI01 swdal1-ISCSI02
> 
> Resources:
>   Clone: ping_resource-clone
>Resource: ping_resource (class=ocf provider=pacemaker type=ping)
> Attributes: multiplier=1000 dampen=15s attempts=4 timeout=3 
> host_list="10.151.16.60 10.151.16.50 10.151.17.50 10.151.17.60"
> Operations: start interval=0s timeout=60 
> (ping_resource-start-interval-0s)
> stop interval=0s timeout=20
(ping_resource-stop-interval-0s)
> monitor interval=5s timeout=15s 
> (ping_resource-monitor-interval-5s)
>   Resource: Pool-1 (class=ocf provider=oe type=zfs)
>Attributes: pool_name=Pool-1 pool_id=9200090953161398950 
> encryption_password_hash=None
>Meta Attrs: failure-timeout=30 is-managed=True 
> encryption_password_hash=None
>Operations: start interval=0s timeout=300 (Pool-1-start-interval-0s)
>stop interval=0s timeout=300 (Pool-1-stop-interval-0s)
>monitor interval=10 timeout=60 (Pool-1-monitor-interval-10)
>   Resource: Pool-0 (class=ocf provider=oe type=zfs)
>Attributes: pool_name=Pool-0 pool_id=4165732781319344895 
> encryption_password_hash=None
>Meta Attrs: failure-timeout=30 is-managed=True 
> encryption_password_hash=None
>Operations: start interval=0s timeout=300 (Pool-0-start-interval-0s)
>stop interval=0s timeout=300 (Pool-0-stop-interval-0s)
>monitor interval=10 timeout=60 (Pool-0-monitor-interval-10)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
>Resource: Pool-0
>  Enabled on: swdal1-ISCSI01 (score:1) 
> (id:location-Pool-0-swdal1-ISCSI01-1)
>  Constraint: location-Pool-0
>Rule: score=-INFINITY boolean-op=or (id:location-Pool-0-rule)
>  Expression: pingd lt 1  (id:location-Pool-0-rule-expr)
>  Expression: not_defined pingd (id:location-Pool-0-rule-expr-1)
>Resource: Pool-1
>  Enabled on: swdal1-ISCSI01 (score:1) 
> (id:location-Pool-1-swdal1-ISCSI01-1)
>  Constraint: location-Pool-1
>Rule: score=-INFINITY boolean-op=or (id:location-Pool-1-rule)
>  Expression: pingd lt 1  (id:location-Pool-1-rule-expr)
>  Expression: not_defined pingd (id:location-Pool-1-rule-expr-1)
> Ordering Constraints:
> Colocation Constraints:
> 
> Resources Defaults:
>   resource-stickiness: 10
> Operations Defaults:
>   record-pending: true
> 
> Cluster Properties:
>   batch-limit: 1
>   cluster-infrastructure: corosync
>   cluster-recheck-interval: 180
>   dc-version: 1.1.12-1.1.12+git+561c4cf
>   no-quorum-policy: ignore
>   stonith-enabled: false
>   stop-orphan-resources: false
> 
> W dniu 02.01.2023 o 13:12, Ulrich Windl pisze:
>> Hi!
>>
>> I wonder: Is this a new installation, or is it a new bug in an old 
> installation?
>> For the first case I'd recommend to start with current software, and for
the 
> second case please describe what had changed or what had triggered the 
> situation.
>> Also provide basic configuration data, please.
>>
>> Regards,
>> Ulrich
>>
>>
>>>>> Krzysztof Bodora  schrieb am 02.01.2023 um
12:16
>> in Nachricht <37d86b8d-c59f-5fe3-cba1-41d2c84fc...@open-e.com>:
>>> Hello Clusterlabs,
>>>
>>> I'm getting this error in the logs:
>>>
>>> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:error:
>>> print_synapse: [Action7]: In-flight crm op
>>> ping_resource_clear_failcount_0   on swdal1-ISCSI01 (priority: 0,
>>> waiting: none)
>>>
>>> My specfications:
>>>
>>> OS: Debian 8
>>> Pacemaker version: 1.1.12
>>> Kernel version: 4.19.190
>>>
>>> I'd like to know what can cause this error to happen and how to prev

[ClusterLabs] Antw: [EXT] clear_failcount operation times out, makes it impossible to use the cluster

[Ulrich Windl]

Hi!

I wonder: Is this a new installation, or is it a new bug in an old installation?
For the first case I'd recommend to start with current software, and for the 
second case please describe what had changed or what had triggered the 
situation.
Also provide basic configuration data, please.

Regards,
Ulrich


>>> Krzysztof Bodora  schrieb am 02.01.2023 um 
>>> 12:16
in Nachricht <37d86b8d-c59f-5fe3-cba1-41d2c84fc...@open-e.com>:
> Hello Clusterlabs,
> 
> I'm getting this error in the logs:
> 
> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:error: 
> print_synapse: [Action7]: In-flight crm op 
> ping_resource_clear_failcount_0   on swdal1-ISCSI01 (priority: 0, 
> waiting: none)
> 
> My specfications:
> 
> OS: Debian 8
> Pacemaker version: 1.1.12
> Kernel version: 4.19.190
> 
> I'd like to know what can cause this error to happen and how to prevent 
> it in the future. I'm also currently unable to update to a newer version 
> of pacemaker.
> 
> Here is some context for when it happens. It seems that the 
> ping_resource resources are in 'Restart' state:
> 
> Dec 20 09:24:23 [57851] swdal1-ISCSI01pengine:   notice: 
> LogActions:Restart ping_resource:0 (Started swdal1-ISCSI01)
> Dec 20 09:24:23 [57851] swdal1-ISCSI01pengine:   notice: 
> LogActions:Restart ping_resource:1 (Started swdal1-ISCSI02)
> 
> which causes pacemaker to try to clear the failcounts on those resources:
> 
> Dec 20 09:24:23 [57862] swdal1-ISCSI01   crmd: info: 
> do_state_transition:   State transition S_POLICY_ENGINE -> 
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE 
> origin=handle_response ]
> Dec 20 09:24:23 [57862] swdal1-ISCSI01   crmd: info: 
> do_te_invoke:  Processing graph 11 (ref=pe_calc-dc-1671528262-59) 
> derived from /var/lib/pacemaker/pengine/pe-input-518.bz2
> Dec 20 09:24:23 [57862] swdal1-ISCSI01   crmd: info: 
> te_crm_command:Executing crm-event (7): clear_failcount on 
> swdal1-ISCSI01
> Dec 20 09:24:23 [57862] swdal1-ISCSI01   crmd: info: 
> handle_failcount_op:   Removing failcount for ping_resource
> Dec 20 09:24:23 [57841] swdal1-ISCSI01cib: info: 
> cib_process_request:   Forwarding cib_delete operation for section 
> //node_state[@uname='swdal1-ISCSI01']//lrm_resource[@id='ping_resource']/lrm
> _rsc_op[@id='ping_resource_last_failure_0'] 
> to master (origin=local/crmd/118)
> Dec 20 09:24:23 [57841] swdal1-ISCSI01cib: info: 
> cib_process_request:   Completed cib_delete operation for section 
> //node_state[@uname='swdal1-ISCSI01']//lrm_resource[@id='ping_resource']/lrm
> _rsc_op[@id='ping_resource_last_failure_0']: 
> OK (rc=0, origin=swdal1-ISCSI01/crmd/118, version=0.60.0)
> Dec 20 09:24:28 [57841] swdal1-ISCSI01cib: info: 
> cib_process_ping:  Reporting our current digest to swdal1-ISCSI01: 
> ccf71244504d3deb02d0da64fa72cedc for 0.60.0 (0x55788a83c4b0 0)
> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:  warning: 
> action_timer_callback: Timer popped (timeout=2, abort_level=0, 
> complete=false)
> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:error: 
> print_synapse: [Action7]: In-flight crm op 
> ping_resource_clear_failcount_0   on swdal1-ISCSI01 (priority: 0, 
> waiting: none)
> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:   notice: 
> abort_transition_graph:Transition aborted: Action lost 
> (source=action_timer_callback:772, 0)
> Dec 20 09:25:43 [57862] swdal1-ISCSI01   crmd:   notice: 
> run_graph: Transition 11 (Complete=1, Pending=0, Fired=0, 
> Skipped=9, Incomplete=2, 
> Source=/var/lib/pacemaker/pengine/pe-input-518.bz2): Stopped
> 
> Clearing the failcount fails, so the whole transition is aborted. This 
> make it impossible to do anything in the cluster, for example move 
> Pool-0 resource, as it also trigger the clear_failcount operation which 
> fails and aborts the transition, for example:
> 
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine: info: 
> RecurringOp:Start recurring monitor (5s) for ping_resource:0 on 
> swdal1-ISCSI01
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine: info: 
> RecurringOp:Start recurring monitor (5s) for ping_resource:1 on 
> swdal1-ISCSI02
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine: info: 
> RecurringOp:Start recurring monitor (10s) for Pool-0 on 
> swdal1-ISCSI02
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine:   notice: 
> LogActions:Restart ping_resource:0 (Started swdal1-ISCSI01)
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine:   notice: 
> LogActions:Restart ping_resource:1 (Started swdal1-ISCSI02)
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine: info: 
> LogActions:Leave   Pool-1  (Started swdal1-ISCSI01)
> Dec 20 09:35:04 [57851] swdal1-ISCSI01pengine:   notice: 
> LogActions:MovePool-0  (Started swdal1-ISCSI01 -> 
> swdal1-ISCSI02)
> Dec 20 09:35:04 

[ClusterLabs] Antw: [EXT] Re: Stonith external/ssh "device"?

[Ulrich Windl]

>>> Antony Stone  schrieb am 21.12.2022 um
17:19 in
Nachricht <202212211719.34369.antony.st...@ha.open.source.it>:
> On Wednesday 21 December 2022 at 16:59:16, Antony Stone wrote:
> 
>> Hi.
>> 
>> I'm implementing fencing on a 7‑node cluster as described recently:
>> https://lists.clusterlabs.org/pipermail/users/2022‑December/030714.html 
>> 
>> I'm using external/ssh for the time being, and it works if I test it
using:
>> 
>> stonith ‑t external/ssh ‑p "nodeA nodeB nodeC" ‑T reset nodeB
>> 
>> 
>> However, when it's supposed to be invoked because a node has got stuck, I
>> simply find syslog full of the following (one from each of the other six
>> nodes in the cluster):
>> 
>> pacemaker‑fenced[3262]:   notice: Operation reboot of nodeB by 
for
>> pacemaker‑controld.26852@nodeA.93b391b2: No such device
>> 
>> I have defined seven stonith resources, one for rebooting each machine,
and
>> I can see from "crm status" that they have been assigned randomly amongst
>> the other servers, usually one per server, so that looks good.
>> 
>> 
>> The main things that puzzle me about the log message are:
>> 
>> a) why does it say ""?  Is this more like "anyone", meaning that
>> no‑ one in particular is required to do this task, provided that at least
>> someone does it?  Does this indicate a configuration problem?
> 
> PS: I've just noticed that I'm also getting log entries immediately 
> afterwards:
> 
> pacemaker‑controld[3264]:   notice: Peer nodeB was not terminated (reboot)
by 
> 
>  on behalf of pacemaker‑controld.26852: No such device
> 
>> b) what is this "device" referred to?  I'm using "external/ssh" so there
is
>> no actual Stonith device for power‑cycling hardware machines ‑ am I
>> supposed to define some sort of dummy device somewhere?
>> 
>> For clarity, this is what I have added to my cluster configuration to set
>> this up:
>> 
>> primitive reboot_nodeA   stonith:external/sshparams
hostlist="nodeA"
>> location only_nodeA  reboot_nodeA‑inf: nodeA

"location only_nodeA" meaning "location not_nodeA"? ;-)


>> 
>> ...repeated for all seven nodes.
>> 
>> I also have "stonith‑enabled=yes" in the cib‑bootstrap‑options.
>> 
>> 
>> Ideas, anyone?
>> 
>> Thanks,
>> 
>> 
>> Antony.
> 
> ‑‑ 
> Normal people think "If it ain't broke, don't fix it".
> Engineers think "If it ain't broke, it doesn't have enough features yet".
> 
>Please reply to the
list;
>  please *don't* CC 
> me.
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Bug pacemaker with multiple IP

[Ulrich Windl]

You could also try something like "watch fuser $(which ip)" or (if you can)
write a program using inotify and IN_OPEN to see which procrees are opening the
binary.

>>> Thomas CAS  schrieb am 21.12.2022 um 09:24 in Nachricht
:

> Ken,
> 
> Antivirus (sophos‑av) is running but not in "real time access scanning", the

> scheduled scan is however at 9pm every day.
> 7 minutes later, we got these alerts. 
> The anti virus may indeed be the cause.
> 
> I had the case on December 13 (with systemctl here):
> 
> pacemaker.log‑20221217.gz:Dec 13 21:07:53 wd‑websqlng01 pacemaker‑controld 

> [5082] (process_lrm_event)  notice: wd‑websqlng01‑NGINX_monitor_15000:454 [

> /etc/init.d/nginx: 33: /lib/lsb/init‑functions.d/40‑systemd: systemctl: Text

> file busy\n/etc/init.d/nginx: 82: /lib/lsb/init‑functions.d/40‑systemd: 
> /bin/systemctl: Text file busy\n ]
> pacemaker.log‑20221217.gz:Dec 13 21:07:53 wd‑websqlng01 pacemaker‑controld 

> [5082] (process_lrm_event)  notice: wd‑websqlng01‑NGINX_monitor_15000:454 [

> /etc/init.d/nginx: 33: /lib/lsb/init‑functions.d/40‑systemd: systemctl: Text

> file busy\n/etc/init.d/nginx: 82: /lib/lsb/init‑functions.d/40‑systemd: 
> /bin/systemctl: Text file busy\n ]
> 
> After, this happens rarely, we had the case in August:
> 
> pacemaker.log‑20220826.gz:Aug 25 21:06:31 wd‑websqlng01 pacemaker‑controld 

> [3718] (process_lrm_event)  notice: 
> wd‑websqlng01‑NGINX‑VIP‑232_monitor_1:2877 [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file 
> busy\nocf‑exit‑reason:IPaddr2 only supported Linux.\n ]
> pacemaker.log‑20220826.gz:Aug 25 21:06:31 wd‑websqlng01 pacemaker‑controld 

> [3718] (process_lrm_event)  notice: 
> wd‑websqlng01‑NGINX‑VIP‑231_monitor_1:2880 [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file 
> busy\nocf‑exit‑reason:IPaddr2 only supported Linux.\n ]
> 
> It's always around 9:00‑9:07 pm, 
> I'll move the virus scan to 10pm and see.
> 
> Thanks,
> Best regards,
> 
> Thomas Cas  |  Technicien du support infogérance
> PHONE : +33 3 51 25 23 26   WEB : www.ikoula.com/en 
> IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims ‑ FRANCE
> Before printing this letter, think about the impact on the environment!
> 
> ‑Message d'origine‑
> De : Reid Wahl  
> Envoyé : mardi 20 décembre 2022 20:34
> À : Cluster Labs ‑ All topics related to open‑source clustering welcomed 
> 
> Cc : Ken Gaillot ; Service Infogérance 
> 
> Objet : Re: [ClusterLabs] Bug pacemaker with multiple IP
> 
> [Vous ne recevez pas souvent de courriers de nw...@redhat.com. Découvrez 
> pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification
]
> 
> On Tue, Dec 20, 2022 at 6:25 AM Thomas CAS  wrote:
>>
>> Hello Ken,
>>
>> Thanks for your answer.
>> There was no update running at the time of the bug, which is why I thought

> that having too many IPs caused this type of error.
>> The /usr/sbin/ip executable was not being modified either.
>>
>> We have many clusters, and only this one has so many IPs and this problem.
> 
> How often does this happen, and is it reliably reproducible under any 
> circumstances? Any antivirus software running? It'd be nice to check 
> something like lsof or strace while it's happening, but that may not be 
> feasible if it's sporadic; running those at every monitor would generate
lots 
> of logs.
> 
> AFAICT, having multiple processes execute (or read) the `ip` binary 
> simultaneously *shouldn't* cause problems, as long as nothing opens it for 
> write.
> 
>>
>> Best regards,
>>
>> Thomas Cas  |  Technicien du support infogérance
>> PHONE : +33 3 51 25 23 26   WEB : 
>
https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ikoula.c

>
om%2Fen=05%7C01%7Ctcas%40ikoula.com%7C9aab91944bd6454a773808dae2c13ae4%7C
>
cb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638071616800939086%7CUnknown%7CTWF
>
pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
>
C3000%7C%7C%7C=oYe7ws2%2BPx3sMblOFBgkuXuSHTdguzB%2Flk83O5W2MjE%3D
> d=0
>> IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims ‑ FRANCE Before 
>> printing this letter, think about the impact on the environment!
>>
>> ‑Message d'origine‑
>> De : Ken Gaillot  Envoyé : lundi 19 décembre 2022 
>> 22:08 À : Cluster Labs ‑ All topics related to open‑source clustering 
>> welcomed  Cc : Service Infogérance 
>>  Objet : Re: [ClusterLabs] Bug pacemaker with 
>> multiple IP
>>
>> [Vous ne recevez pas souvent de courriers de kgail...@redhat.com. 
>> Découvrez pourquoi ceci est important à 
>> https://aka.ms/LearnAboutSenderIdentification ]
>>
>> On Mon, 2022‑12‑19 at 09:48 +, Thomas CAS wrote:
>> > Hello Clusterlabs,
>> >
>> > I would like to report a bug on Pacemaker with the "IPaddr2"
>> > resource:
>> >
>> > OS: Debian 10
>> > Kernel: Linux wd‑websqlng01 4.19.0‑18‑amd64 #1 SMP Debian 4.19.208‑1
>> > (2021‑09‑29) x86_64 GNU/Linux
>> > 

[ClusterLabs] Antw: [EXT] Re: Bug pacemaker with multiple IP

[Ulrich Windl]

Hi!

I wonder: Could the error message be triggered by adding an exclusive manatory
lock in the ip binary?
If that triggers the bug, I'm rather sure that the error message is bad.
Shouldn't that be EWOULDBLOCK then?
(I have no idea how Sophos AV works, though. If they open the files to check
in write-mode, it's really stupid then IMHO)

Regards,
Ulrich


>>> Reid Wahl  schrieb am 21.12.2022 um 10:19 in Nachricht
:
> On Wed, Dec 21, 2022 at 12:24 AM Thomas CAS  wrote:
>>
>> Ken,
>>
>> Antivirus (sophos-av) is running but not in "real time access scanning",
the 
> scheduled scan is however at 9pm every day.
>> 7 minutes later, we got these alerts.
>> The anti virus may indeed be the cause.
> 
> I see. That does seem fairly likely. At least, there's no other
> obvious candidate for the cause.
> 
> I used to work on a customer-facing support team for the ClusterLabs
> suite, and we received a fair number of cases where bizarre issues
> (such as hangs and access errors) were apparently caused by an
> antivirus. In those cases, all other usual lines of investigation were
> exhausted, and when we asked the customer to disable their AV, the
> issue disappeared. This happened with several different AV products.
> 
> I can't say with any certainty that the AV is causing your issue, and
> I know it's frustrating that you won't know whether any given
> intervention worked, since this only happens once every few months.
> 
> You may want to either exclude certain files from the scan, or write a
> short script to place the cluster in maintenance mode before the scan
> and take it out of maintenance after the scan is complete.
> 
>>
>> I had the case on December 13 (with systemctl here):
>>
>> pacemaker.log-20221217.gz:Dec 13 21:07:53 wd-websqlng01 pacemaker-controld 

> [5082] (process_lrm_event)  notice: wd-websqlng01-NGINX_monitor_15000:454 [

> /etc/init.d/nginx: 33: /lib/lsb/init-functions.d/40-systemd: systemctl: Text

> file busy\n/etc/init.d/nginx: 82: /lib/lsb/init-functions.d/40-systemd: 
> /bin/systemctl: Text file busy\n ]
>> pacemaker.log-20221217.gz:Dec 13 21:07:53 wd-websqlng01 pacemaker-controld 

> [5082] (process_lrm_event)  notice: wd-websqlng01-NGINX_monitor_15000:454 [

> /etc/init.d/nginx: 33: /lib/lsb/init-functions.d/40-systemd: systemctl: Text

> file busy\n/etc/init.d/nginx: 82: /lib/lsb/init-functions.d/40-systemd: 
> /bin/systemctl: Text file busy\n ]
>>
>> After, this happens rarely, we had the case in August:
>>
>> pacemaker.log-20220826.gz:Aug 25 21:06:31 wd-websqlng01 pacemaker-controld 

> [3718] (process_lrm_event)  notice: 
> wd-websqlng01-NGINX-VIP-232_monitor_1:2877 [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file 
> busy\nocf-exit-reason:IPaddr2 only supported Linux.\n ]
>> pacemaker.log-20220826.gz:Aug 25 21:06:31 wd-websqlng01 pacemaker-controld 

> [3718] (process_lrm_event)  notice: 
> wd-websqlng01-NGINX-VIP-231_monitor_1:2880 [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file 
> busy\nocf-exit-reason:IPaddr2 only supported Linux.\n ]
>>
>> It's always around 9:00-9:07 pm,
>> I'll move the virus scan to 10pm and see.
> 
> That also sounds like a good plan to confirm the cause :) It might
> take a while to find out though.
> 
>>
>> Thanks,
>> Best regards,
>>
>> Thomas Cas  |  Technicien du support infogérance
>> PHONE : +33 3 51 25 23 26   WEB : www.ikoula.com/en 
>> IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE
>> Before printing this letter, think about the impact on the environment!
>>
>> -Message d'origine-
>> De : Reid Wahl 
>> Envoyé : mardi 20 décembre 2022 20:34
>> À : Cluster Labs - All topics related to open-source clustering welcomed 
> 
>> Cc : Ken Gaillot ; Service Infogérance 
> 
>> Objet : Re: [ClusterLabs] Bug pacemaker with multiple IP
>>
>> [Vous ne recevez pas souvent de courriers de nw...@redhat.com. Découvrez 
> pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification
]
>>
>> On Tue, Dec 20, 2022 at 6:25 AM Thomas CAS  wrote:
>> >
>> > Hello Ken,
>> >
>> > Thanks for your answer.
>> > There was no update running at the time of the bug, which is why I
thought 
> that having too many IPs caused this type of error.
>> > The /usr/sbin/ip executable was not being modified either.
>> >
>> > We have many clusters, and only this one has so many IPs and this
problem.
>>
>> How often does this happen, and is it reliably reproducible under any 
> circumstances? Any antivirus software running? It'd be nice to check 
> something like lsof or strace while it's happening, but that may not be 
> feasible if it's sporadic; running those at every monitor would generate
lots 
> of logs.
>>
>> AFAICT, having multiple processes execute (or read) the `ip` binary 
> simultaneously *shouldn't* cause problems, as long as nothing opens it for 
> write.
>>
>> >
>> > Best regards,
>> >
>> > Thomas Cas  |  

[ClusterLabs] Antw: Re: Antw: [EXT] Re: Stonith

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 20.12.2022 um 16:21 in
Nachricht
<3a5960c2331f97496119720f6b5a760b3fe3bbcf.ca...@redhat.com>:
> On Tue, 2022‑12‑20 at 11:33 +0300, Andrei Borzenkov wrote:
>> On Tue, Dec 20, 2022 at 10:07 AM Ulrich Windl
>>  wrote:
>> > > But keep in mind that if the whole site is down (or unaccessible)
>> > > you
>> > > will not have access to IPMI/PDU/whatever on this site so your
>> > > stonith
>> > > agents will fail ...
>> > 
>> > But, considering the design, such site won't have a quorum and
>> > should commit suicide, right?
>> > 
>> 
>> Not by default.
> 
> And even if it does, the rest of the cluster can't assume that it did,
> so resources can't be recovered. It could work with sbd, but the poster
> said that the physical hosts aren't accessible.

Why? Assuming fencing is configured, the nodes part of the quorum should wait
for fencing delay, assuming fencing (or suicide) was done.
Then they can manage resources. OK, a non-working fencing or suicide mechanism
is a different story...

Regards,
Ulrich


> ‑‑ 
> Ken Gaillot 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Bug pacemaker with multiple IP

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 19.12.2022 um 22:07 in 
>>> Nachricht
<68e568cdcd675f8e4de8b01cd3ed27e2eb4ed28c.ca...@redhat.com>:
...
> If simultaneous monitors is somehow causing the problem, you should be
> able to work around it by using different intervals for different
> monitors.

I disagree: If simultaneous monitors trigger such behavior, ther is a critical 
bug somewhere!

...

Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Bug pacemaker with multiple IP

[Ulrich Windl]

>>> Thomas CAS  schrieb am 19.12.2022 um 10:48 in Nachricht
:

> Hello Clusterlabs,
> 
> I would like to report a bug on Pacemaker with the "IPaddr2" resource:
> 
> OS: Debian 10
> Kernel: Linux wd‑websqlng01 4.19.0‑18‑amd64 #1 SMP Debian 4.19.208‑1
(2021‑09‑29) 
> x86_64 GNU/Linux
> Pacemaker version: 2.0.1‑5+deb10u2
> 
> You will find the configuration of our cluster with 2 nodes attached.
> 
> Bug :
> 
> We have several IP configured in the cluster configuration (12)
> Sometimes the cluster is unstable with the following errors in the pacemaker

> logs:
> 
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑232_monitor_1:28835:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]

As Ken pointed out already,

"text file busy" in Linux means that some process tried to write to the binary
while it is being executed (at least once). However you can update a running
binary by "removing" (i.e. unlinking) it and then place a new binary. Running
processes will continue to use the old binary while new invocations will use
the new binary.

I suggest to tracxe what's going on. It does not sound good.
Still, I'm not claiming there is no bug in the RA ;-)

Regards,
Ulrich


> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑239_monitor_1:28877:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑239_monitor_1:28877:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑234_monitor_1:28830:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑231_monitor_1:28900:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑231_monitor_1:28900:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑235_monitor_1:28905:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑235_monitor_1:28905:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑237_monitor_1:28890:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑237_monitor_1:28890:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑238_monitor_1:28876:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP‑238_monitor_1:28876:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP_monitor_1:28880:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] 
> (operation_finished)   notice: NGINX‑VIP_monitor_1:28880:stderr [ 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: 
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> 
> The reason is that there are a lot of IPs configured and if the monitors 
> take place at the same time it causes this type of error.
> 
> Best regards,
> 
> Thomas Cas  |  Technicien du support infogérance
> PHONE : +33 3 51 25 23 26   WEB : 
> www.ikoula.com/en
> IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims 

[ClusterLabs] Antw: [EXT] Re: Stonith

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 19.12.2022 um 14:17 in
Nachricht
:
> On Mon, Dec 19, 2022 at 4:01 PM Antony Stone
>  wrote:
>>
>> On Monday 19 December 2022 at 13:55:45, Andrei Borzenkov wrote:
>>
>> > On Mon, Dec 19, 2022 at 3:44 PM Antony Stone
>> >
>> >  wrote:
>> > > So, do I simply create one stonith resource for each server, and rely on
>> > > some other random server to invoke it when needed?
>> >
>> > Yes, this is the most simple approach. You need to restrict this
>> > stonith resource to only one cluster node (set pcmk_host_list).
>>
>> So, just to be clear, I create one stonith resource for each machine which
>> needs to be able to be shut down by some other server?
>>
> 
> Correct.
> 
>> I ask simply because the acronym stonith refers to "the other node", so it
>> sounds to me more like something I need to define so that a working machine 
> can
>> kill another one.
>>
> 
> Yes, you define a stonith resource that can kill node A and nodes B,
> C, D, ... will use this resource to kill A when needed. As long as
> your stonith resource can actually work on any node it does not matter
> which one will do the killing. You can restrict which nodes can use
> this stonith agent using usual location constraints if necessary.
> 
> But keep in mind that if the whole site is down (or unaccessible) you
> will not have access to IPMI/PDU/whatever on this site so your stonith
> agents will fail ...

But, considering the design, such site won't have a quorum and should commit 
suicide, right?

> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] RFQ: Clusterlabs pacemaker administration

[Ulrich Windl]

Hi!

First I think your e-mail disclaimer makes little sense when sending to a
public list.
Next I think it would be better to ask your question specific for the
distribution you use (assuming you do use some common distribution).
I'm sure Redhat and SUSE at least have some training offerings.
Maybe you can get if from some third party, too.

Regards,
Ulrich

>>> "Genessi C. Crisolo via Users"  schrieb am
14.12.2022 um
03:06 in Nachricht
:
> Dear Clusterlabs team,
> 
> Good day!
> 
> Would like to know if you can deliver a virtual training for Clusterlabs 
> pacemaker administration?
> 
> If yes, please share with us the costing and class schedule.
> 
> Hope to hear from one of your team soonest! Thank you.
> 
> Regards,
> 
> Genessi C. Crisolo
> Product Manager ‑ Customer Solutions and Delivery
> Trends Academy
> [image003]
> M   +63 917 874 2936
> T+63 2 8811 8181 | +63 2 8863 2123 loc 1655
> Egccris...@trends.com.ph
> 
> Trends & Technologies, Inc.
> 16th Floor Trafalgar Plaza
> 105 H.V. Dela Costa St., Salcedo Village,
> Makati City 1227 Philippines
> 
>
www.trends.com.ph
%2F%2Fwww.trends.com.ph%2F=04%7C01%7C%7Cd9ea69a980a24097919308d90df75ea7%
>
7Cc13441a455b94ea9af761aedaaf25d48%7C0%7C0%7C637556178373490716%7CUnknown%7CT
>
WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C1000=ZWDckQhk3rr1ovrZ1Y1H71kp8bba%2BJHBUoKI30CqYUM%3D=0> |

>
https://www.trendsnet.com.ph/
url=https%3A%2F%2Fwww.trendsnet.com.ph%2F=04%7C01%7C%7Cd9ea69a980a2409791
>
9308d90df75ea7%7Cc13441a455b94ea9af761aedaaf25d48%7C0%7C0%7C63755617837350071
>
2%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
>
iLCJXVCI6Mn0%3D%7C1000=VzZ1snqOGtw8M6h5nxpklt%2BuSc2X6obcMTSXnm9IzDQ%3D
> eserved=0>
> 
> [cid:image002.gif@01D90FA3.636EEC40]
> 
> This email (including any attachment) is intended for the sole use of the 
> intended recipients and may contain confidential and/or privileged 
> information. Any unauthorized use, disclosure, dissemination, distribution,

> duplication of and reliance on this communication is strictly prohibited. If

> you are not the intended recipient, please notify the sender immediately by

> replying to this email and delete or destroy the original email
communication 
> and any copy. There are no guarantees that email transmissions are
error‑free 
> or secure, as information could be intercepted, corrupted, lost, destroyed,

> arrive late, be incomplete, or contain viruses. The sender does not accept 
> liability for any errors or omissions in the content of this message or 
> damages caused by any virus. The company cannot be held responsible for the

> content of this email, nor can it be responsible for the consequences of the

> actions taken based on the information provided in this mail. No employee or

> agent is authorized to conclude any binding agreement on behalf of the 
> company by email without express written confirmation from a company's 
> authorized officer. Any views or opinions presented in this email are solely

> those of the author and do not necessarily represent those of the company.



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: mdraid - pacemaker resource agent

[Ulrich Windl]

>>> Roger Zhou via Users  schrieb am 09.12.2022 um 11:26
in
Nachricht :

> On 12/9/22 17:36, Jelen, Piotr wrote:
>> Hi Roger,
>> 
>> Thank you for your quick reply,
>> The mdraid resource agent works very well for us,
>> Can you please tell me if there is any resource agent or tool build in 
> pacemaker  for sync configuration file(s) such as /etc/exports for nfs 
> service between cluster nodes without mounting shares and creating symbolic

> link(s)  ?

I wonder (also see below Roger's response): Don't you add exportfs resources
to your NFS server? Then you don't need to sync the exports file.
primitive prm_nfs_XYZ_exp exportfs \
params clientspec="@XYZ_hosts" directory="/exports/XYZ" fsid=...
options="rw,..."

OK, we don't sync exportfs, but use netgroups from LDAP to define host access.
That way we don't have to adjust the cluster config when new NFS clients are
added...

> 
> 
> I incline to ask you describe your use case a little bit more. Without 
> knowing
> it well, my view might be not fit you.
> 
> csync2 is often used by ClusterLabs community. But it is an individual
tool,
> and is not part of pacemaker. https://github.com/linbit/csync2 
> 
> However, you refers to /etc/exports. I guess your use case is High
Available
> NFS? If so, you could leverage `man ocf_heartbeat_exportfs`. But this 
> resource
> agent doesn't use /etc/exports at all.
> 
> 
> BR,
> Roger
> 
> 
> 
>> 
>> Thank you
>> Piotr Jelen
>> Senior Systems Platform Engineer
>> 
>> Mastercard
>> Mountain View, Central Park  | Leopard
>> 
>> 
>> -Original Message-
>> From: Roger Zhou 
>> Sent: Thursday 8 December 2022 05:56
>> To: Cluster Labs - All topics related to open-source clustering welcomed 
> ; Jelen, Piotr 
>> Cc: Nielsen, Laust 
>> Subject: {EXTERNAL} Re: [ClusterLabs] mdraid - pacemaker resource agent
>> 
>> CAUTION: The message originated from an EXTERNAL SOURCE. Please use caution

> when opening attachments, clicking links or responding to this email.
>> 
>> 
>> 
>> On 12/7/22 18:44, Jelen, Piotr wrote:
>>> Hi ClusterLabs team ,
>>>
>>> I would like to ask if this resource agent was tested and if it can be
>>> use in production?
>>> resource-agents/mdraid at main · ClusterLabs/resource-agents · GitHub
>>> >> erLabs_resource-2Dagents_blob_main_heartbeat_mdraid=DwIDaQ=uc5ZRXl
>>> 8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY=UzPNWvRnAChMyFZDGSUcEqtZElcu_Xn
>>> gP1GiT3HLUmI=OHY6Rfbo326vRgEACkHp-D4nbNFyihVuQak3w9YeTXJQ1SeBV2P-c7c
>>> GumwQuS51=pOg6p7HopMV1E14QJmLReC7z_2_IBPhFWTWst5YO1FA= >
>>>
>> 
>> Yes. Why not ;). We don't see big missing piece, though might have some 
> improvement for the certain scenario. Anyway, you could report the issue, or

> provide improvement code, if any.
>> 
>> Cheers,
>> Roger
>> 
>>>
>>> Thank you
>>>
>>> *Piotr Jelen*
>>>
>>> Senior Systems Platform Engineer
>>>
>>> Mastercard
>>>
>>> Mountain View, Central Park  | Leopard
>>>
>>>[mastercard.com] 
>>>
>>>
>>> CONFIDENTIALITY NOTICE This e-mail message and any attachments are
>>> only for the use of the intended recipient and may contain information
>>> that is privileged, confidential or exempt from disclosure under
>>> applicable law. If you are not the intended recipient, any disclosure,
>>> distribution or other use of this e-mail message or attachments is
>>> prohibited. If you have received this e-mail message in error, please
delete 
> and notify the sender immediately. Thank you.
>>>
>>> ___
>>> Manage your subscription:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.clusterlabs 
>>> .org_mailman_listinfo_users=DwIDaQ=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmC
>>> F6SP0bDlmMmY=UzPNWvRnAChMyFZDGSUcEqtZElcu_XngP1GiT3HLUmI=OHY6Rfbo3
>>> 26vRgEACkHp-D4nbNFyihVuQak3w9YeTXJQ1SeBV2P-c7cGumwQuS51=ffs3MlOD3Dhd
>>> GPsQ5ikhdnYpI1fwmHosCj3lcGclCC8=
>>>
>>> ClusterLabs home:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.clusterlabs.o 
>>> rg_=DwIDaQ=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY=UzPNWvRnA
>>> ChMyFZDGSUcEqtZElcu_XngP1GiT3HLUmI=OHY6Rfbo326vRgEACkHp-D4nbNFyihVuQ
>>> ak3w9YeTXJQ1SeBV2P-c7cGumwQuS51=z_dYDvqUIF8kA-K2k0kgO_YirTwL_MFT9jED
>>> iJ8rJrw=
>> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for

> the use of the intended recipient and may contain information that is 
> privileged, confidential or exempt from disclosure under applicable law. If

> you are not the intended recipient, any disclosure, distribution or other
use 
> of this e-mail message or attachments is prohibited. If you have received 
> this e-mail message in error, please delete and notify the sender 
> immediately. Thank you.
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___

[ClusterLabs] Antw: [EXT] Samba failover and Windows access

[Ulrich Windl]

>>> Dave Withheld  schrieb am 08.12.2022 um 08:03 in
Nachricht
:

> In our production factory, we run a 2‑node cluster on CentOS 8 with
pacemaker, 
> a virtual IP, and drbd for shared storage with samba (among other services)

> running as a resource on the active node.  Everything works great except
when 

Did you consider using CTDB Samba and dual-primary DRBD instead? CTDB samba
replicates connection information to the nodes, so if one goes down, the other
knows about the clients...

> we fail over.  All resources are moved to the other node and start just
fine, 
> but Windows hosts that have connections to the samba shares all have to be 
> rebooted before they can reconnect.  Clients that were not connected can 
> connect.  We have samba configured for only SMB1 protocol and all Windows 
> clients are configured to allow it.
> 
> Maybe this is a question for the samba folks, but thought I'd try here first

> since it's only a problem when the other node takes over the samba resource.
 
> Anyone seen this problem and solved it?



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] mdraid ‑ pacemaker resource agent

[Ulrich Windl]

>>> Ulrich Windl schrieb am 08.12.2022 um 07:55 in Nachricht <63918A51.502 :
161 :
60728>:
>>>> "Jelen, Piotr"  schrieb am 07.12.2022 um
11:44 in
> Nachricht
>
 om>
> 
> > Hi ClusterLabs team ,
> > 
> > I would like to ask if this resource agent was tested and if it can be use

> > in production?
> 
> Hi!
> 
> We use it in production for more than 10 years now. Of course it requires a

> little it of thinking when configuring it cluster-wide.

s/ it / bit / # the first one

> We also had used the clustered-RAID variant for a year or so.
> 
> What made you think it does not work?
> 
> Regards,
> Ulrich
> 
> > resource‑agents/mdraid at main * ClusterLabs/resource‑agents * 
> >
GitHub<https://github.com/ClusterLabs/resource‑agents/blob/main/heartbeat/md

> ra 
> >id>
> > 
> > Thank you
> > Piotr Jelen
> > Senior Systems Platform Engineer
> > 
> > Mastercard
> > Mountain View, Central Park  | Leopard
> > [cid:image001.png@01D90A28.C8BE8900]
> > 
> > 
> > CONFIDENTIALITY NOTICE This e‑mail message and any attachments are only
for 
> > the use of the intended recipient and may contain information that is 
> > privileged, confidential or exempt from disclosure under applicable law.
If 
> > you are not the intended recipient, any disclosure, distribution or other

> use 
> > of this e‑mail message or attachments is prohibited. If you have received

> this 
> > e‑mail message in error, please delete and notify the sender immediately.

> > Thank you.
> 
> 
> 
> 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] mdraid ‑ pacemaker resource agent

[Ulrich Windl]

>>> "Jelen, Piotr"  schrieb am 07.12.2022 um 11:44
in
Nachricht


> Hi ClusterLabs team ,
> 
> I would like to ask if this resource agent was tested and if it can be use 
> in production?

Hi!

We use it in production for more than 10 years now. Of course it requires a
little it of thinking when configuring it cluster-wide.
We also had used the clustered-RAID variant for a year or so.

What made you think it does not work?

Regards,
Ulrich

> resource‑agents/mdraid at main * ClusterLabs/resource‑agents * 
>
GitHubid>
> 
> Thank you
> Piotr Jelen
> Senior Systems Platform Engineer
> 
> Mastercard
> Mountain View, Central Park  | Leopard
> [cid:image001.png@01D90A28.C8BE8900]
> 
> 
> CONFIDENTIALITY NOTICE This e‑mail message and any attachments are only for

> the use of the intended recipient and may contain information that is 
> privileged, confidential or exempt from disclosure under applicable law. If

> you are not the intended recipient, any disclosure, distribution or other
use 
> of this e‑mail message or attachments is prohibited. If you have received
this 
> e‑mail message in error, please delete and notify the sender immediately. 
> Thank you.



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Preventing a resource from migrating to / starting on a node

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 29.11.2022 um 20:41 in
Nachricht
<23873b16397171f0880ee691a0aef5a6e3ef39a4.ca...@redhat.com>:
> The use case here is for external code (DRBD) to ban a resource from a
> node. While DRBD could add/remove location constraints, it's better to
> have permanent (rule‑based) location constraints in the configuration,
> and let DRBD just set or unset a node attribute to trigger the
> constraint.
> 
> As a general rule, it's best to avoid automated configuration changes
> (via resource agents, crons, DRBD, etc.) other than node attributes.
> There's just more room for something to go wrong.

I think setting the wrong node attributes (combined with the "right" rules)
can be just as desastous as adding bad location constraints.

...

Regards,
Ulrich

___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Preventing a resource from migrating to / starting on a node

[Ulrich Windl]

Why can't you use a plain location constraint?

>>> Madison Kelly  schrieb am 29.11.2022 um 05:21 in 
>>> Nachricht
<19cbecab-a7a0-5c3a-d074-efd3e8374...@alteeve.com>:
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] DRBD Dual Primary Write Speed Extremely Slow

[Ulrich Windl]

>>> Tyler Phillippe via Users  schrieb am 14.11.2022 um
15:48 in Nachricht :
> Hi Vladislav,
> 
> If I don't use the Scale-Out File Server, I don't have any issues with iSCSI 
> speeds: if I directly connect the LUN(s) to the individual servers, I get 
> 'full' speed - it just seems the Scale-Out File Server is causing the issue. 
> Very strange of Microsoft (though not totally unexpected!). I can't find too 
> much online about Scale-Out File Server, other than generic setup 
> information.

Well, it's off-topic, but when I install Windows updates it needs at least on 
hour on a fast machine with lot of RAM, but spinning disks; when I install 
Linux updates it takes about 10 minutes. Windows is not fast with SSDs, it's 
unbearable slow without! ;-)


[...]


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] DRBD Dual Primary Write Speed Extremely Slow

[Ulrich Windl]

Hi!

If you have planty of RAM you could configure an iSCSI disk using a ram disk 
and try how much I/O you get from there.
Maybe you issue is not-su-much DRBD related. However when my local MD-RAID1 
resyncs with about 120MB/s (spinning disks), the system also is hardly usable.

Regards,
Ulrich

>>> Tyler Phillippe via Users  schrieb am 13.11.2022 um
19:26 in Nachricht :
> Hello all,
> 
> I have setup a Linux cluster on 2x CentOS 8 Stream machines - it has 
> resources to manage a dual primary, GFS2 DRBD setup. DRBD and the cluster 
> have a diskless witness. Everything works fine - I have the dual primary DRBD 
> working and it is able to present an iSCSI LUN out to my LAN. However, the 
> DRBD write speed is terrible. The backing DRBD disks (HDD) are RAID10 using 
> mdadm and they (re)sync at around 150MB/s. DRBD verify has been limited to 
> 100MB/s, but left untethered, it will get to around 140MB/s. If I write data 
> to the iSCSI LUN, I only get about 10-15MB/s. Here's the DRBD 
> global_common.conf - these are exactly the same on both machines:
> 
> global {
> usage-count no;
> udev-always-use-vnr;
> }
> 
> common {
> handlers {
> }
> 
> startup {
> wfc-timeout 5;
> degr-wfc-timeout 5;
> }
> 
> options {
> auto-promote yes;
> quorum 1;
> on-no-data-accessible suspend-io;
> on-no-quorum suspend-io;
> }
> 
> disk {
> al-extents 4096;
> al-updates yes;
> no-disk-barrier;
> disk-flushes;
> on-io-error detach;
> c-plan-ahead 0;
> resync-rate 100M;
> }
> 
> net {
> protocol C;
> allow-two-primaries yes;
> cram-hmac-alg "sha256";
> csums-alg "sha256";
> verify-alg "sha256";
> shared-secret "secret123";
> max-buffers 36864;
> rcvbuf-size 5242880;
> sndbuf-size 5242880;
> }
> }
> 
> Respectfully,
>  Tyler




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: [External] : Re: Fence Agent tests

[Ulrich Windl]

>>> Jehan-Guillaume de Rorthais via Users  schrieb am
05.11.2022 um 22:17 in Nachricht <20221105221756.20ea6761@karst>:

...

> In Pacemaker mode, SBD is watching the two most important part of the 
> cluster:
> Pacemaker and Corosync:
> 
> * the "Pacemaker watcher" of SBD connects to the CIB and check it's still
>   updated on a regular basis and the self-node is marked online.
> * the "Cluster watchers" all connect with each others using a dedicated
>   communication group in corosync ring(s).
> 
> Both watchers can report a failure to SBD that would self-stop the node.
> 
> If the network if down, I suppose the cluster watcher should complain. But I
> suspect Pacemaker somehow keeps reporting as quorate, thus, forbidding SBD 
> to
> kill the whole node...

Wow! I think that should have gone into sbd's manual page; I never read such 
before.
Very helpful.

...

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: [External] : Re: Fence Agent tests

[Ulrich Windl]

Hi!

Maybe see "test-watchdog" in sbd's manual page ;-)

Regards,
Ulrich

>>> Robert Hayden  schrieb am 05.11.2022 um 19:47
in
Nachricht


>>  ‑Original Message‑
>> From: Users  On Behalf Of Valentin Vidic
>> via Users
>> Sent: Saturday, November 5, 2022 1:07 PM
>> To: users@clusterlabs.org 
>> Cc: Valentin Vidić 
>> Subject: Re: [ClusterLabs] [External] : Re: Fence Agent tests
>> 
>> On Sat, Nov 05, 2022 at 05:20:47PM +, Robert Hayden wrote:
>> > The OCI compute instances don't have a hardware watchdog, only the
>> software watchdog.
>> > So, when the network goes completely hung (e.g. firewall‑cmd panic‑on),
>> all network
>> > traffic stops which implies that IO to the SBD device also stops.  I do
not 
> see
>> the software
>> > watchdog take any action in response to the network hang.
>> 
>> It seems like the watchdog is not working or is not configured with a
>> correct timeout here. sbd will not refresh the watchdog if it fails to
>> read from the disk, so the watchdog should eventually expire and reset
>> the node.
> 
> That was my impression as well...so I may have something wrong.  My 
> expectation was that SBD daemon
> should be writing to the /dev/watchdog within 20 seconds and the kernel 
> watchdog would self fence.
> 
> Here is my setup
> root:dh2vgmprepap02:ablgmprep:/root:# grep ^SBD /etc/sysconfig/sbd
> SBD_DEVICE=/dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1
> SBD_PACEMAKER=yes
> SBD_STARTMODE=always
> SBD_DELAY_START=no
> SBD_WATCHDOG_DEV=/dev/watchdog
> SBD_WATCHDOG_TIMEOUT=5
> SBD_TIMEOUT_ACTION=flush,reboot
> SBD_MOVE_TO_ROOT_CGROUP=auto
> SBD_OPTS=
> 
> root:dh2vgmprepap02:ablgmprep:/root:# sbd ‑d 
> /dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1 dump
> ==Dumping header on disk 
> /dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1
> Header version : 2.1
> UUID   : 04096cc5‑1fb8‑44da‑9c4f‑4b6034a0fe06
> Number of slots: 255
> Sector size: 512
> Timeout (watchdog) : 20
> Timeout (allocate) : 2
> Timeout (loop) : 1
> Timeout (msgwait)  : 40
> ==Header on disk
/dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1 
> is dumped
> 
> root:dh2vgmprepap02:ablgmprep:/root:# pcs stonith sbd status  ‑‑full
> SBD STATUS
> :  |  | 
> dh2vgmprepap03: YES | YES | YES
> dh2vgmprepap02: YES | YES | YES
> 
> Messages list on device 
> '/dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1':
> 0   dh2vgmprepap03  clear
> 1   dh2vgmprepap02  clear
> 
> 
> SBD header on device 
> '/dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1':
> ==Dumping header on disk 
> /dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1
> Header version : 2.1
> UUID   : 04096cc5‑1fb8‑44da‑9c4f‑4b6034a0fe06
> Number of slots: 255
> Sector size: 512
> Timeout (watchdog) : 20
> Timeout (allocate) : 2
> Timeout (loop) : 1
> Timeout (msgwait)  : 40
> ==Header on disk
/dev/disk/by‑id/scsi‑360e59ebc0f414569bcc7a5e4a6d58ccb‑part1 
> is dumped
> 
> 
>> 
>> ‑‑
>> Valentin
>> ___
>> Manage your subscription:
>>
https://urldefense.com/v3/__https://lists.clusterlabs.org/mailman/listinfo/u

>> sers__;!!ACWV5N9M2RV99hQ!LPMOKgky02sAjkujkuJM8HLR5G5hAfCaQGPF
>> Zymg81e8rf3Z1klCgoi4HAicoJr6wBEhEvnYaLZ6G1vRBDTKyw$
>> 
>> ClusterLabs home:
>> https://urldefense.com/v3/__https://www.clusterlabs.org/__;!!ACWV5N9 
>> M2RV99hQ!LPMOKgky02sAjkujkuJM8HLR5G5hAfCaQGPFZymg81e8rf3Z1klCg
>> oi4HAicoJr6wBEhEvnYaLZ6G1tNVtP0BA$
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] VirtualDomain did not stop although "crm resource stop"

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 02.11.2022
um
18:05 in Nachricht
<973456304.11152064.1667408710062.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> i think i found the reason, but i want to be sure.
> I wanted to stop a VirtualDomain and did a "crm resource stop ..."
> But it didn't shut down. After waiting several minutes i stopped with 
> libvirt, circumventing the cluster software.
> First i wondered "why didn't it shutdown ?", but then i realized that also a

> Livemigration of another VirtualDomain was running and was started before
the 
> stop of the resource.
> Am i correct that that it didn't shutdown because of the running 
> Live-Migration and would have start the shutdown when the Live-Migration is

> finished ?

"Use the force^H^H^H^H^Hsource^H^H^H^H^H^Hlogs, Luke!" ;-)

I think the logs should tell you the details you are asking for.

Regards,
Ulrich


> 
> Bernd
> 
> -- 
> Bernd Lentes 
> System Administrator 
> Institute for Metabolism and Cell Death (MCD) 
> Building 25 - office 122 
> HelmholtzZentrum München 
> bernd.len...@helmholtz-muenchen.de 
> phone: +49 89 3187 1241
>+49 89 3187 49123 
> fax:   +49 89 3187 2294 
> https://www.helmholtz-munich.de/en/mcd 
> 
> Public key: 
> 30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 
> 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82
fc 
> cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3
a7 
> 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92
67 
> 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53
89 
> 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2
e3 
> 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2
fa 
> 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2
67 
> 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85
08 
> d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac
58 
> f1 38 43 0e 72 af 02 03 01 00 01



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: crm resource trace

[Ulrich Windl]

Hi!

I didn't read all the logs, mostly because I think those are very hard to read 
for humans.
But I noticed: config="/mnt/share/vm_genetrap.xml"

I think using /mnt as permanent path for anything is a bad idea.

Regards,
Ulrich

>>> "Lentes, Bernd"  schrieb am 17.10.2022 
>>> um
12:43 in Nachricht
<47865548.10389842.1666003412419.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> i try to find out why there is sometimes a restart of the resource and 
> sometimes not.
> Unpredictable behaviour is someting i expect from Windows, not from Linux.
> Here you see two "crm resource trace "resource"".
> In the first case the resource is restarted , in the second not.
> The command i used is identical in both cases.
> 
> ha-idg-2:~/trace-untrace # date; crm resource trace vm-genetrap
> Fri Oct 14 19:05:51 CEST 2022
> INFO: Trace for vm-genetrap is written to /var/lib/heartbeat/trace_ra/
> INFO: Trace set, restart vm-genetrap to trace non-monitor operations
> 
> 
> ==
> 
> 1st try:
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  
> Diff: --- 7.28974.3 2
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  
> Diff: +++ 7.28975.0 299af44e1c8a3867f9e7a4b25f2c3d6a
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  +  
> /cib:  @epoch=28975, @num_updates=0
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++ 
> /cib/configuration/resources/primitive[@id='vm-genetrap']/operations/op[@id='
> vm-genetrap-monitor-30']:   id="vm-genetrap-monitor-30-instance_a
> ttributes"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>id="vm-genetrap-monito
> r-30-instance_attributes-trace_ra"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
> 
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++ 
> /cib/configuration/resources/primitive[@id='vm-genetrap']/operations/op[@id='
> vm-genetrap-stop-0']:   id="vm-genetrap-stop-0-instance_attribute
> s"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>id="vm-genetrap-stop-0-ins
> tance_attributes-trace_ra"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
> 
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++ 
> /cib/configuration/resources/primitive[@id='vm-genetrap']/operations/op[@id='
> vm-genetrap-start-0']:   id="vm-genetrap-start-0-instance_attribu
> tes"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
> id="vm-genetrap-start-0-i
> nstance_attributes-trace_ra"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>  
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++ 
> /cib/configuration/resources/primitive[@id='vm-genetrap']/operations/op[@id='
> vm-genetrap-migrate_from-0']:   id="vm-genetrap-migrate_from-0-in
> stance_attributes"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>id="vm-genetrap-mi
> grate_from-0-instance_attributes-trace_ra"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
> 
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++ 
> /cib/configuration/resources/primitive[@id='vm-genetrap']/operations/op[@id='
> vm-genetrap-migrate_to-0']:   id="vm-genetrap-migrate_to-0-instan
> ce_attributes"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>  id="vm-genetrap-migr
> ate_to-0-instance_attributes-trace_ra"/>
> Oct 14 19:05:52 [25996] ha-idg-1cib: info: cib_perform_op:  ++   
>   
>   
> Oct 14 19:05:52 [26001] ha-idg-1   crmd: info: 
> abort_transition_graph:  Transition 791 aborted by 
> instance_attributes.vm-genetrap-monitor-30-instance_attributes 

[ClusterLabs] Antw: [EXT] Re: trace of resource ‑ sometimes restart, sometimes not

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 07.10.2022 um 01:08 in
Nachricht
:

[...]
> 
> trace_ra is unusual in that it's supported automatically by the OCF
> shell functions, rather than by the agents directly. That means it's
> not advertised in metadata. Otherwise agents could mark it as
> reloadable, and reload would be a quick no‑op.

That reads like "Dirty hack, dirty consequences" ;-)

[...]

Regards,
Ulrich

___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] trace of resource ‑ sometimes restart, sometimes not

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 06.10.2022
um
21:05 in Nachricht
<280206366.19581344.1665083124300.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> i have some problems with our DLM, so i wanted to trace it. Yesterday i just

> set a trace for "monitor". No restart of DLM afterwards. It went fine as 
> expected.
> I got logs in /var/lib/heartbeat/trace_ra. After some monitor i stopped 
> tracing.
> 
> Today i set a trace for all operations.
> Now resource DLM restarted:
> * Restartdlm:0   ( ha-idg-1 )   due to 
> resource definition change
> I didn't expect that so i had some trouble.

# crm configure
edit ...
verify
ptest nograph #!!!
commit

--
If you used that, you would have seen the restart.
Despite of that I wonder why enabling tracing to start/stop must induce a
resource restart.

Bernd, are you sure that was the only thing changed? Do you have a record of
commands issued?

> Is the difference in this behaviour intentional ? If yes, why ? Is there a 
> rule ?
> 
> Furthermore i'd like to ask where i can find more information about DLM, 
> because it is a mystery for me.
> Sometimes the DLM does not respond to the "monitor", so it needs to be 
> restarted, and therefore all depending resources (which is a lot).
> This happens under some load (although not completely overwhelmed).

Regards,
Ulrich

> 
> Thanks.
> 
> Bernd
> 
> -- 
> Bernd Lentes 
> System Administrator 
> Institute for Metabolism and Cell Death (MCD) 
> Building 25 - office 122 
> HelmholtzZentrum München 
> bernd.len...@helmholtz-muenchen.de 
> phone: +49 89 3187 1241
>+49 89 3187 49123 
> fax:   +49 89 3187 2294 
> https://www.helmholtz-munich.de/en/mcd 
> 
> Public key: 
> 30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 
> 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82
fc 
> cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3
a7 
> 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92
67 
> 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53
89 
> 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2
e3 
> 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2
fa 
> 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2
67 
> 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85
08 
> d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac
58 
> f1 38 43 0e 72 af 02 03 01 00 01



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: DRBD and SQL Server

[Ulrich Windl]

>>> Brian  schrieb am 26.09.2022 um 12:10 in
Nachricht
<5cb7ca0c-c502-44a9-8c74-c14d87f5e...@tenethor.ddns.net>:
> Eric
> 
> Up until recently I was running mariaDB on my cluster with a DRBD storage. 
> As for the server it ran perfectly fine in my use. Just a home server with 
> not too much demand. 
> 
> About I week ago I decided to move from DRBD to a NAS tho because I kept 
> having issues with the DRBD syncing and split brain. I believe it was 
> probably a configuration problem on my part but I couldn’t figure it out and

> I had the NAS already setup so the switch was easy. To DRBDs credit, when it

> worked it worked perfectly fine and I did like the idea of having the data 
> multiply redundant and not having a single point of failure like I do with 
> the NAS. 

We had a scenario that was like using two NAS in different buildigs, mirrored
using clustered mdraid. Even earlier we were using LVM mirroring instead. At
the moment we are using two NAS systems, too, but they are mirroring
transparently to the host...

> 
> My next project maybe to setup a glusterFS instead and see how that fairs. 
> 
> Brian
> 
> 
> Sent from my iPhone
> 
>> On 26. Sep 2022, at 10:22 AM, Reid Wahl  wrote:
>> 
>> On Sun, Sep 25, 2022 at 8:49 PM Eric Robinson  
> wrote:
>>> 
>>> Hey list,
>>> 
>>> 
>>> 
>>> Anybody have experience running SQL Server on DRBD? I’d ask this in the
DRBD 
> list but that one is like a ghost town. This list is the next best option.
>> 
>> I think the DRBD Slack channel is marginally more active than their
>> list, for what it's worth.
>>> 
>>> 
>>> 
>>> -Eric
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Disclaimer : This email and any files transmitted with it are confidential

> and intended solely for intended recipients. If you are not the named 
> addressee you should not disseminate, distribute, copy or alter this email.

> Any views or opinions presented in this email are solely those of the author

> and might not represent those of Physician Select Management. Warning: 
> Although Physician Select Management has taken reasonable precautions to 
> ensure no viruses are present in this email, the company cannot accept 
> responsibility for any loss or damage arising from the use of this email or

> attachments.
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>> 
>>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
>> 
>> 
>> -- 
>> Regards,
>> 
>> Reid Wahl (He/Him)
>> Senior Software Engineer, Red Hat
>> RHEL High Availability - Pacemaker
>> 
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] RFE: sdb clone

[Ulrich Windl]

Hi!

I have a proposal (request) for enhancing sbd:
(I'm not suggesting a complete rewrite with reasonable options, as I had don 
that before already ;-))
When configuring an additional disk device, it would be quite handy to be able 
to "clone" the configuration from an existing device.
As I understand it, sbd uses the physical block size for disk layout, so a 
simple dd won't do the job when old and new device use different physical block 
sizes, right?
I think manually entering the parameters for the sbd header (the way it has to 
be done now) is quite error-prone, and it's easy to configure devices using 
different timing parameters.

Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] DC marks itself as OFFLINE, continues orchestrating the other nodes

[Ulrich Windl]

>>> Lars Ellenberg  schrieb am 08.09.2022 um 15:01
in
Nachricht :

> Scenario:
> three nodes, no fencing (I know)
> break network, isolating nodes
> unbreak network, see how cluster partitions rejoin and resume service
> 
> 
> Funny outcome:
> /usr/sbin/crm_mon  ‑x pe‑input‑689.bz2
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: mqhavm24 (version 1.1.24.linbit‑2.0.el7‑8f22be2ae) ‑
partition 
> with quorum
>   * Last updated: Thu Sep  8 14:39:54 2022
>   * Last change:  Thu Aug 11 12:33:02 2022 by root via crm_resource on 
> mqhavm24
>   * 3 nodes configured
>   * 16 resource instances configured (2 DISABLED)
> 
> Node List:
>   * Online: [ mqhavm34 mqhavm37 ]
>   * OFFLINE: [ mqhavm24 ]
> 
> 
> Note how the current DC considers itself as OFFLINE!
> 
> It accepted an apparently outdated cib replaceament from one of the non‑DCs
> from a previous membership while already authoritative itself,
> overwriting its own "join" status in the cib.
> 
> I have full crm_reports and some context knowledge about the setup.
> 
> For now I'd like to know: has anyone seen this before,
> is that a known bug in corner cases/races during re‑join,
> has it even been fixed meanwhile?

I think the order ov events is important here. Maybe provide some logs?

> 
> Thanks,
> Lars
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] (no subject)

[Ulrich Windl]

>>> ???  schrieb am 07.09.2022 um 12:12 in Nachricht
:
> Hello.
> I am a student who wants to implement a redundancy system with raspberry pi.
> Last time, I posted about how to proceed with installation on raspberry pi
> and received a lot of comments.
> Among them, I searched a lot after looking at the comments saying that
> fencing stonith should not be false.
> (ex -> sudo pcs property set stonith-enabled=false)
> However, I saw a lot of posts saying that there is no choice but to do
> false because there is no ipmi in raspberry pi, and I wonder how we can
> solve it in this situation.

I think I had explained it: Depending on the details (e.g. distance between 
nodes) you might want to implement a custom fencing mechanism, like triggering 
the other node's reset input. If distance is short, some I/O port of the other 
node might be used for that. Just be creative!

Regards,
Ulrich




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Cluster does not start resources

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 24.08.2022 
>>> um
18:18 in Nachricht
<949048121.170067950.1661357898189.javamail.zim...@helmholtz-muenchen.de>:

> - On 24 Aug, 2022, at 16:26, kwenning kwenn...@redhat.com wrote:
> 
>>>
>>> if I get Ulrich right - and my fading memory of when I really used crmsh the
>>> last time is telling me the same thing ...
>>>
> 
> I get the impression many people prefer pcs to crm. Is there any reason for 
> that ?

I think the "red people" (those with the colored hat) use pcs and the "green 
people" (those with the reptile) use crm ;-)

> And can i use pcs on Suse ? If yes, how ?
> 
> Bernd




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Cluster does not start resources

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 24.08.2022 
>>> um
18:15 in Nachricht
<518321906.170067472.1661357732466.javamail.zim...@helmholtz-muenchen.de>:

> - On 24 Aug, 2022, at 16:26, kwenning kwenn...@redhat.com wrote:
> 
> 
> 
>> 
>> Guess the resources running now are those you tried to enable before
>> while they were globally stopped 
>> 
> 
> No. First i set stop-all-resources to false. Then SOME resources started.
> Then i tried several times to start some VirtualDomains using "crm resource 
> start"
> which didn't succeed. Some time later i tried it again and it succeeded ...

Logs, logs, logs... otherwise it's all just speculation.

> 
> Bernd




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Cluster does not start resources

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 24.08.2022 
>>> um
15:14 in Nachricht
<2097315299.169939002.1661346840678.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> 
> Now with "crm resource start" all resources started. I didn't change 
> anything !?!

I guess that command set the roles of all resources to "started", so you 
changed something ;-)

> 
> Bernd




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Cluster does not start resources

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 24.08.2022 
>>> um
07:13 in Nachricht
<718962332.169073979.1661318038326.javamail.zim...@helmholtz-muenchen.de>:

> 
> - On 24 Aug, 2022, at 07:03, arvidjaar arvidj...@gmail.com wrote:
> 
>> On 24.08.2022 07:34, Lentes, Bernd wrote:
>>> 
>>> 
>>> - On 24 Aug, 2022, at 05:33, Reid Wahl nw...@redhat.com wrote:
>>> 
>>> 
 The stop-all-resources cluster property is set to true. Is that 
 intentional?
>>> OMG. Thanks Reid !
>>> 
>>> But unfortunately not all virtual domains are running:
>>> 
>> 
>> what exactly is not clear in this output? All these resources are
>> explicitly disabled (target-role=stopped) and so will not be started.
>> 
> That's clear. But a manual "crm resource start virtual_domain" should start 
> them,
> but it doesn't.

Bernd,

that command would simply set the role to "started", but I guess it already is. 
Obviously to be effective stop-all-resources must habve precedence. You see?

Regards,
Ulrich


> 
> Bernd




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Cluster does not start resources

[Ulrich Windl]

>>> Reid Wahl  schrieb am 24.08.2022 um 05:33 in Nachricht
:
> On Tue, Aug 23, 2022 at 7:10 PM Lentes, Bernd
>  wrote:

...
> The stop‑all‑resources cluster property is set to true. Is that
intentional?
...

Independent of the problem, I suggest that a better log message is created.
Maybe like this:
"(all resources stopped due to stop-all-resources being true)"

It could repeat every cluster check interval ;-)

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Start resource only if another resource is stopped

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 18.08.2022 um 20:26 in
Nachricht :
...
> It is almost always wrong to have multiple independent pacemaker
> resources managing the same underlying physical resource.

It's not the cluster bible that says: "No man can have two masters" ;-)
But it applies to clusters quite nicely.

> 
> It looks like you attempt to reimplement high available NFS server on
> client side. If you insist on this, I see as the only solution separate
> resource agent that monitors state of export/data resources and sets
> attribute accordingly. But effectively you will be duplicating pacemaker
> logic.

I never did it, but didn't allow NFS to provide multiple sources, thus 
specifying redundancy (maybe never in Linux)?
The only thing is: It was designed for read-only servers as data consistency is 
quite a challenge then.

> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Start resource only if another resource is stopped

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 11.08.2022 um 21:40 in
Nachricht :
> On 11.08.2022 17:34, Miro Igov wrote:
>> Hello,
>> 
>> I am trying to create failover resource that would start if another resource
>> is stopped and stop when the resource is started back.

I wonder: Could it be implemented using a M/S (formerly master/slave) resource?

...

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] node1 and node2 communication time question

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 10.08.2022 um 10:13 in
Nachricht <2c124efb-bc28-224b-031c-3ac2c111a...@gmail.com>:
> On 10.08.2022 09:37, Ulrich Windl wrote:
>> Unfortunately the documentation for fencing agents leaves verymuch to be 
> desired:
>> When I tried to write one myself, I just stopped due to lack of details.
>> 
> 
> It is not about writing own agent but about using existing ones. There
> are enough fencing agents for most common use cases, no need to write
> anything from scratch.

I didn't find one that sets the host's LCD display to "Fencing node..." before 
actually fencing it ;-)

> 
> Do not confuse newbies even more ...
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] node1 and node2 communication time question

[Ulrich Windl]

>>> ???  schrieb am 10.08.2022 um 03:35 in Nachricht
:
> Thank you for your reply.
> Then, could you explain how to activate and set the stonith?

There is no universal solution; instead it depends on the hardware you have.
(If your devices are close to each other, one device's output port could even 
trigger the reset on the other board)
Unfortunately the documentation for fencing agents leaves verymuch to be 
desired:
When I tried to write one myself, I just stopped due to lack of details.

> Or let me know the blog or site you know.

Try Google on "pacemaker fencing stonith"

> I looked up the site I found and proceeded with the setting, and almost all
> the sites explained it with the setting I set.

Acually you could "start to play", but for real-life you would want fencing.

Regards,
Ulrich





___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Heads up for ldirectord in SLES12 SP5 "Use of uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord line 1830"

[Ulrich Windl]

Hi!

Digging further in ldirectord, I found that the utility functions do no make a 
difference between a name that is not known, and a name that is (probably) 
known, but cannot be resolved at the moment.

I hacked the corresponding functions to observe and return the error code 
(errno) as a negative number.
Short demo:

  DB<2> x ld_gethostbyname('x',AF_INET)
0  '-2' # ENOENT
  DB<3> x ld_gethostbyname('localhost',AF_INET)
0  '127.0.0.1'
  DB<4> x ld_gethostbyname('localhost',AF_INET6)
0  '[::1]'
### hacked /etc/resolv.conf to make namservers unreachable (add IPs that are no 
nameservers or don't exist), but host exists
  DB<7> x ld_gethostbyname('mail-1',AF_INET)
0  '-3' # ESRCH

Returning the error message string is a bit trickier, so I just used the error 
code.

However it's not clear what to do when the resolver fails (i.e.: name would be 
known if resolver worked). In any case it takes quite a while until an error 
result is returned.

For example (using the hacked functions):
if (($fallback->{port} =
 _getservbyname($fallback->{port}, $protocol)) =~ /^-/) {
_error($line, "invalid port for fallback server");
}

One could check for "== '-2'" instead, but still in the other case there is no 
valid port value.

Ideas?

Regards,
Ulrich

>>> Ulrich Windl schrieb am 08.08.2022 um 11:19 in Nachricht <62F0D518.3F8 : 
>>> 161 :
60728>:
> Hi!
> 
> The bug is still under investigation, but digging in the ldirectord code I 
> found this part called when stopping:
> 
> } elsif ($CMD eq "stop") {
> kill 15, $oldpid;
> ld_exit(0, "Exiting from ldirectord $CMD");
> 
> As ldirectord uses a SIGTERM handler that sets a flag only and then (at some 
> later time) the termination code will be started.
> Doesn't that mean the cluster will see a bad exit code (success while parts 
> of ldirectord are still running)?
> 
> Regards,
> Ulrich
> 
> 
> 
> >>> Ulrich Windl schrieb am 03.08.2022 um 11:13 in Nachricht <62EA3C2C.E8D : 
> >>> 161 
> :
> 60728>:
> > Hi!
> > 
> > I wanted to inform you of an unpleasant bug in ldirectord of SLES12 SP5:
> > We had a short network problem while some redundancy paths reconfigured in 
> > the infrastructure, effectively causing that some network services could 
> not 
> > be reached.
> > Unfortunately ldirectord controlled by the cluster reported a failure (the 
> > director, not the services being directed to):
> > 
> > h11 crmd[28930]:   notice: h11-prm_lvs_mail_monitor_30:369 [ Use of 
> > uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
> 
> > line 1830,  line 21. Error [33159] reading file 
> > /etc/ldirectord/mail.conf at line 10: invalid address for virtual service\n 
> ]
> > h11 ldirectord[33266]: Exiting with exit_status 2: config_error: 
> > Configuration Error
> > 
> > You can guess wat happened:
> > Pacemaker tried to recover (stop, then start), but the stop failed, too:
> > h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Use of 
> > uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
> 
> > line 1830,  line 21. ]
> > h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Error [36293] 
> 
> > reading file /etc/ldirectord/mail.conf at line 10: invalid address for 
> > virtual service ]
> > h11 crmd[28930]:   notice: Result of stop operation for prm_lvs_mail on 
> h11: 
> > 1 (unknown error)
> > 
> > A stop failure meant that the node was fenced, interrupting all the other 
> > services.
> > 
> > Examining the logs I also found this interesting type of error:
> > h11 attrd[28928]:   notice: Cannot update 
> > fail-count-prm_lvs_rksapds5#monitor_30[monitor]=(null) because peer 
> UUID 
> > not known (will retry if learned)
> > 
> > Eventually, here's the code that caused the error:
> > 
> > sub _ld_read_config_virtual_resolve
> > {
> > my($line, $vsrv, $ip_port, $af)=(@_);
> > 
> > if($ip_port){
> > $ip_port=_gethostservbyname($ip_port, $vsrv->{protocol}, 
> > $af);
> > if ($ip_port =~ /(\[[0-9A-Fa-f:]+\]):(\d+)/) {
> > $vsrv->{server} = $1;
> > $vsrv->{port} = $2;
> > } elsif($ip_port){
> > ($vsrv->{server}, $vsrv->{port}) = split /:/, 
> > $ip_port;
> > }
> > else {
> >  

[ClusterLabs] Antw: [EXT] Re: 2‑Node Cluster ‑ fencing with just one node running ?

[Ulrich Windl]

>>> Reid Wahl  schrieb am 07.08.2022 um 04:01 in Nachricht
:
> On Saturday, August 6, 2022, Strahil Nikolov via Users <
> users@clusterlabs.org> wrote:
>> By the way I remember a lot of problems with fence_ilo & fence_ilo_ssh
> (due to ILO).
>> If you receive timeouts use fence_ipmi (you have to enable IPMI in ILO).
>> Best Regards,
>> Strahil Nikolov
> 
> Of the two, fence_ipmilan/fence_ilo is much more reliable than the
> SSH-based agents in my experience.

Trying to fence remotely via in-band (cluster network) in inherently broken 
IMHO.

> 
>>
>> On Thu, Aug 4, 2022 at 23:34, Reid Wahl
>>  wrote:
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
> 
> -- 
> Regards,
> 
> Reid Wahl (He/Him)
> Senior Software Engineer, Red Hat
> RHEL High Availability - Pacemaker




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Heads up for ldirectord in SLES12 SP5 "Use of uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord line 1830"

[Ulrich Windl]

Hi!

The bug is still under investigation, but digging in the ldirectord code I 
found this part called when stopping:

} elsif ($CMD eq "stop") {
kill 15, $oldpid;
ld_exit(0, "Exiting from ldirectord $CMD");

As ldirectord uses a SIGTERM handler that sets a flag only and then (at some 
later time) the termination code will be started.
Doesn't that mean the cluster will see a bad exit code (success while parts of 
ldirectord are still running)?

Regards,
Ulrich



>>> Ulrich Windl schrieb am 03.08.2022 um 11:13 in Nachricht <62EA3C2C.E8D : 
>>> 161 :
60728>:
> Hi!
> 
> I wanted to inform you of an unpleasant bug in ldirectord of SLES12 SP5:
> We had a short network problem while some redundancy paths reconfigured in 
> the infrastructure, effectively causing that some network services could not 
> be reached.
> Unfortunately ldirectord controlled by the cluster reported a failure (the 
> director, not the services being directed to):
> 
> h11 crmd[28930]:   notice: h11-prm_lvs_mail_monitor_30:369 [ Use of 
> uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
> line 1830,  line 21. Error [33159] reading file 
> /etc/ldirectord/mail.conf at line 10: invalid address for virtual service\n ]
> h11 ldirectord[33266]: Exiting with exit_status 2: config_error: 
> Configuration Error
> 
> You can guess wat happened:
> Pacemaker tried to recover (stop, then start), but the stop failed, too:
> h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Use of 
> uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
> line 1830,  line 21. ]
> h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Error [36293] 
> reading file /etc/ldirectord/mail.conf at line 10: invalid address for 
> virtual service ]
> h11 crmd[28930]:   notice: Result of stop operation for prm_lvs_mail on h11: 
> 1 (unknown error)
> 
> A stop failure meant that the node was fenced, interrupting all the other 
> services.
> 
> Examining the logs I also found this interesting type of error:
> h11 attrd[28928]:   notice: Cannot update 
> fail-count-prm_lvs_rksapds5#monitor_30[monitor]=(null) because peer UUID 
> not known (will retry if learned)
> 
> Eventually, here's the code that caused the error:
> 
> sub _ld_read_config_virtual_resolve
> {
> my($line, $vsrv, $ip_port, $af)=(@_);
> 
> if($ip_port){
> $ip_port=_gethostservbyname($ip_port, $vsrv->{protocol}, 
> $af);
> if ($ip_port =~ /(\[[0-9A-Fa-f:]+\]):(\d+)/) {
> $vsrv->{server} = $1;
> $vsrv->{port} = $2;
> } elsif($ip_port){
> ($vsrv->{server}, $vsrv->{port}) = split /:/, 
> $ip_port;
> }
> else {
> _error($line,
> "invalid address for virtual service");
> }
> ...
> 
> The value returned by ld_gethostservbyname is undefined. I also wonder what 
> the program logic is:
> If the host looks like an hex address in square brackets, host and port are 
> split at the colon; otherwise host and port are split at the colon.
> Why not split simply at the last colon if the value is defined, AND THEN 
> check if the components look OK?
> 
> So the "invalid address for virtual service" is only invalid when the 
> resolver service (e.g. via LDAP) is unavailable.
> I used host and service names for readability.
> 
> (I reported the issue to SLES support)
> 
> Regards,
> Ulrich
> 
> 
> 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Heads up for ldirectord in SLES12 SP5 "Use of uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord line 1830"

[Ulrich Windl]

Hi!

FYI, here is a copy what I had sent to SUSE support (stating "Because of the
very same DNS resolution problem, stopping also failed"; should a temporary DNS
resolving problem cause a resource stop to fail and cause node fencing in turn?
I don't think so!):
---
The problem is the Perl code that most likely was never tested to handle a
failure of or in ld_gethostservbyname():
FIRST it should be checked whether a value was returned at all; if not there
is a failure in resolution.
In turn a failure in resolution could mean two tings:
1) The names in the configuration are not correct and will never resolve.
2) A temporary failure of some kind caused a failure and the configuration IS
CORRECT.

Clearly the bad case here was 2).

Also looking at the code I wonder why it does not handle things like this:
$ip_port=_gethostservbyname($ip_port, $vsrv->{protocol},
$af);
if ($ip_port) {
if ($ip_port =~ /^(.+):([^:]+)$/) { # replacing the split
($vsrv->{server}, $vsrv->{port}) = ($1, $2);
# this should also handle the case "$ip_port =~
/(\[[0-9A-Fa-f:]+\]):(\d+)/"
} else {
# error "unexpected return from ld_gethostservbyname"
}
} else {
# error "failed to resolve ..."
# here it's unfortunate that the original $ip_port is
lost,
# so it cannot be part of the error message
}

Despite of that is that the critical part was that the "stop" operation SEEMED
to have failed, causing fencing.
Regardless of the success of resolving the names ldirector should be able to
stop!
---
Opinions?

Regards,
Ulrich

>>> "Ulrich Windl"  schrieb am 03.08.2022
um
11:13 in Nachricht <62ea3c2c02a10004c...@gwsmtp.uni-regensburg.de>:
> Hi!
> 
> I wanted to inform you of an unpleasant bug in ldirectord of SLES12 SP5:
> We had a short network problem while some redundancy paths reconfigured in 
> the infrastructure, effectively causing that some network services could not

> be reached.
> Unfortunately ldirectord controlled by the cluster reported a failure (the 
> director, not the services being directed to):
> 
> h11 crmd[28930]:   notice: h11‑prm_lvs_mail_monitor_30:369 [ Use of 
> uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord

> line 1830,  line 21. Error [33159] reading file 
> /etc/ldirectord/mail.conf at line 10: invalid address for virtual service\n
]
> h11 ldirectord[33266]: Exiting with exit_status 2: config_error: 
> Configuration Error
> 
> You can guess wat happened:
> Pacemaker tried to recover (stop, then start), but the stop failed, too:
> h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Use of 
> uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord

> line 1830,  line 21. ]
> h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Error [36293]

> reading file /etc/ldirectord/mail.conf at line 10: invalid address for 
> virtual service ]
> h11 crmd[28930]:   notice: Result of stop operation for prm_lvs_mail on h11:

> 1 (unknown error)
> 
> A stop failure meant that the node was fenced, interrupting all the other 
> services.
> 
> Examining the logs I also found this interesting type of error:
> h11 attrd[28928]:   notice: Cannot update 
> fail‑count‑prm_lvs_rksapds5#monitor_30[monitor]=(null) because peer 
> UUID not known (will retry if learned)
> 
> Eventually, here's the code that caused the error:
> 
> sub _ld_read_config_virtual_resolve
> {
> my($line, $vsrv, $ip_port, $af)=(@_);
> 
> if($ip_port){
> $ip_port=_gethostservbyname($ip_port, $vsrv‑>{protocol},

> $af);
> if ($ip_port =~ /(\[[0‑9A‑Fa‑f:]+\]):(\d+)/) {
> $vsrv‑>{server} = $1;
> $vsrv‑>{port} = $2;
> } elsif($ip_port){
> ($vsrv‑>{server}, $vsrv‑>{port}) = split /:/,
$ip_port;
> }
> else {
> _error($line,
> "invalid address for virtual service");
> }
> ...
> 
> The value returned by ld_gethostservbyname is undefined. I also wonder what

> the program logic is:
> If the host looks like an hex address in square brackets, host and port are

> split at the colon; otherwise host and port are split at the colon.
> Why not split simply at the last colon if the value is defined, AND THEN 
> check if the components look OK?
> 
> So the "invalid address for virtual service" is only invalid when th

[ClusterLabs] Antw: [EXT] cluster log not unambiguous about state of VirtualDomains

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 03.08.2022
um
17:01 in Nachricht
<987208047.150503130.1659538882681.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> i have a strange behaviour found in the cluster log 
> (/var/log/cluster/corosync.log).
> I KNOW that i put one node (ha-idg-2) in standby mode and stopped the 
> pacemaker service on that node:
> The history of the shell says:
> 993  2022-08-02 18:28:25 crm node standby ha-idg-2
> 994  2022-08-02 18:28:58 systemctl stop pacemaker.service

I think the logs of ha-idg-2 around these commands would be interesting, as
wel las any pacemaker messages (if any) in between.

BTW, a different, but also confusing message:
Aug 03 15:16:58 h12 pengine[14727]:  warning: Processing failed start of
prm_cwd_w49_sap on h12: unknown error
Aug 03 15:16:58 h12 pengine[14727]:  warning: Forcing prm_cwd_w49_sap away
from h12 after 100 failures (max=3)

The point here is that h12 is the only active node in a two-node cluster, so
"forcing away" probably means "stop":
# crm_mon -1Arfj |grep w49
 prm_cwd_w49_sap(ocf::heartbeat:SAPInstance):   Stopped

Sometimes these messages can be confusing. You could have provided the crm_mon
output too, BTW.

Regards,
Ulrich

> 
> Later on i had some trouble with high load.
> I found contradictory entries in the log on the DC (ha-idg-1):
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-documents-oo active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-documents-oo active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-mausdb active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-mausdb active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-photoshop active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-photoshop active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-encore active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-encore active on ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource dlm:1 active on ha-idg-2
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-seneca active on ha-idg-2<===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-pathway active on ha-idg-2   <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-dietrich active on ha-idg-2  <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-sim active on ha-idg-2   <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-ssh active on ha-idg-2   <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-nextcloud active on ha-idg-2 <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource fs_ocfs2:1 active on ha-idg-2
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource gfs2_share:1 active on ha-idg-2
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-geneious active on ha-idg-2  <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource gfs2_snap:1 active on ha-idg-2  <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource vm-geneious-license-mcd active on 
> ha-idg-2 <===
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: determine_op_status: 

>Operation monitor found resource clvmd:1 active on ha-idg-2
> 
> The log says some VirtualDomains are running on ha-idg-2 !?!
> 
> But just a few lines later the log says all VirtualDomains are running on 
> ha-idg-1:
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: common_print:
> vm-mausdb   (ocf::lentes:VirtualDomain):Started ha-idg-1
> Aug 03 00:14:04 [19367] ha-idg-1pengine: info: common_print:
> vm-sim  (ocf::lentes:VirtualDomain):Started ha-idg-1  <===
> Aug 03 00:14:04 [19367] 

[ClusterLabs] Antw: [EXT] Re: Q: About a false negative of storage_mon

[Ulrich Windl]

>>> Klaus Wenninger  schrieb am 03.08.2022 um 15:51 in
Nachricht
:
> On Tue, Aug 2, 2022 at 4:10 PM Ken Gaillot  wrote:
>>
>> On Tue, 2022-08-02 at 19:13 +0900, 井上和徳 wrote:
>> > Hi,
>> >
>> > Since O_DIRECT is not specified in open() [1], it reads the buffer
>> > cache and
>> > may result in a false negative. I fear that this possibility
>> > increases
>> > in environments with large buffer cache and running disk-reading
>> > applications
>> > such as database.
>> >
>> > So, I think it's better to specify O_RDONLY|O_DIRECT, but what about
>> > it?
>> > (in this case, lseek() processing is unnecessary.)
>> >
>> > # I am ready to create a patch that works with O_DIRECT. Also, I
>> > wouldn't mind
>> > # a "change to add a new mode of inspection with O_DIRECT
>> > # (add a option to storage_mon) while keeping the current inspection
>> > process".
>> >
>> > [1]
>> > 
>
https://github.com/ClusterLabs/resource-agents/blob/main/tools/storage_mon.c#

> L47-L90
>> >
>> > Best Regards,
>> > Kazunori INOUE
>>
>> I agree, it makes sense to use O_DIRECT when available. I don't think
>> an option is necessary.
> 
> Might as well be interesting to adjust block-size/alignment to the
> device.
> Another consideration could be to on top directly access the block-layer
> using aio.

Again AIO is POSIX; it depends on the implementation what it really does.

> Both is being done in sbd (storage-based-death) and yes it as well
> adds Linux specific stuff that might have to be conditional for other OSs.
> 
> Klaus
> 
>>
>> However, O_DIRECT is not available on all OSes, so the configure script
>> should detect support. Also, it is not supported by all filesystems, so
>> if the open fails, we should retry without O_DIRECT.
>> --
>> Ken Gaillot 
>>
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Heads up for ldirectord in SLES12 SP5 "Use of uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord line 1830"

[Ulrich Windl]

Hi!

I wanted to inform you of an unpleasant bug in ldirectord of SLES12 SP5:
We had a short network problem while some redundancy paths reconfigured in the 
infrastructure, effectively causing that some network services could not be 
reached.
Unfortunately ldirectord controlled by the cluster reported a failure (the 
director, not the services being directed to):

h11 crmd[28930]:   notice: h11-prm_lvs_mail_monitor_30:369 [ Use of 
uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
line 1830,  line 21. Error [33159] reading file 
/etc/ldirectord/mail.conf at line 10: invalid address for virtual service\n ]
h11 ldirectord[33266]: Exiting with exit_status 2: config_error: Configuration 
Error

You can guess wat happened:
Pacemaker tried to recover (stop, then start), but the stop failed, too:
h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Use of 
uninitialized value $ip_port in pattern match (m//) at /usr/sbin/ldirectord 
line 1830,  line 21. ]
h11 lrmd[28927]:   notice: prm_lvs_mail_stop_0:35047:stderr [ Error [36293] 
reading file /etc/ldirectord/mail.conf at line 10: invalid address for virtual 
service ]
h11 crmd[28930]:   notice: Result of stop operation for prm_lvs_mail on h11: 1 
(unknown error)

A stop failure meant that the node was fenced, interrupting all the other 
services.

Examining the logs I also found this interesting type of error:
h11 attrd[28928]:   notice: Cannot update 
fail-count-prm_lvs_rksapds5#monitor_30[monitor]=(null) because peer UUID 
not known (will retry if learned)

Eventually, here's the code that caused the error:

sub _ld_read_config_virtual_resolve
{
my($line, $vsrv, $ip_port, $af)=(@_);

if($ip_port){
$ip_port=_gethostservbyname($ip_port, $vsrv->{protocol}, 
$af);
if ($ip_port =~ /(\[[0-9A-Fa-f:]+\]):(\d+)/) {
$vsrv->{server} = $1;
$vsrv->{port} = $2;
} elsif($ip_port){
($vsrv->{server}, $vsrv->{port}) = split /:/, $ip_port;
}
else {
_error($line,
"invalid address for virtual service");
}
...

The value returned by ld_gethostservbyname is undefined. I also wonder what the 
program logic is:
If the host looks like an hex address in square brackets, host and port are 
split at the colon; otherwise host and port are split at the colon.
Why not split simply at the last colon if the value is defined, AND THEN check 
if the components look OK?

So the "invalid address for virtual service" is only invalid when the resolver 
service (e.g. via LDAP) is unavailable.
I used host and service names for readability.

(I reported the issue to SLES support)

Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Re: Q: About a false negative of storage_mon

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 03.08.2022 um 08:58 in
Nachricht :
> On 03.08.2022 09:02, Ulrich Windl wrote:
>>>>> Ken Gaillot  schrieb am 02.08.2022 um 16:09 in
>> Nachricht
>> <0a2125a43bbfc09d2ca5bad1a693710f00e33731.ca...@redhat.com>:
>>> On Tue, 2022-08-02 at 19:13 +0900, 井上和徳 wrote:
>>>> Hi,
>>>>
>>>> Since O_DIRECT is not specified in open() [1], it reads the buffer
>>>> cache and
>>>> may result in a false negative. I fear that this possibility
>>>> increases
>>>> in environments with large buffer cache and running disk-reading
>>>> applications
>>>> such as database.
>>>>
>>>> So, I think it's better to specify O_RDONLY|O_DIRECT, but what about
>>>> it?
>>>> (in this case, lseek() processing is unnecessary.)
>>>>
>>>> # I am ready to create a patch that works with O_DIRECT. Also, I
>>>> wouldn't mind
>>>> # a "change to add a new mode of inspection with O_DIRECT
>>>> # (add a option to storage_mon) while keeping the current inspection
>>>> process".
>>>>
>>>> [1] 
>>>>
>>>
>> 
>
https://github.com/ClusterLabs/resource-agents/blob/main/tools/storage_mon.c#

>> 
>>> L47-L90
>>>>
>>>> Best Regards,
>>>> Kazunori INOUE
>>>
>>> I agree, it makes sense to use O_DIRECT when available. I don't think
>>> an option is necessary.
>>>
>>> However, O_DIRECT is not available on all OSes, so the configure script
>>> should detect support. Also, it is not supported by all filesystems, so
>>> if the open fails, we should retry without O_DIRECT.
>> 
>> I just looked it up: It seems POSIX has O_RSYNC and O_SYNC and O_DSYNC)
>> instead.
> 
> That is something entirely different. O_SYNC etc are about *file system
> level*, while O_DIRECT is about *device* level. O_DIRECT makes process
> to talk directly to device. It is unclear whether this is side effect of
> implementation or intentional.

Well the process still uses the filesystem API to open the device, so I wonder
whether it makes a big difference if O_SYNC operate on the filesystem level or
the device level.
Also if the filesystem level does not pass on O_DIRECT flags to the device,
it'll be kind of useless. OTOH, it's discussable whether an external SAN disk
system should bypass ist internal cache and read the data anew from some RAID
disks if O_DIRECT is in effect.

> 
>> The buffer cache handling may be different though.
>> 
> 
> Synchronous operation does not actually imply media access.

I'm unsure: I tought of I write with O_DSYNC and then the system has a power
failure it's guaranteed that the data is on permanent storage (otherwise it
would be kind of useless).

> 
> O_RSYNC: "the operation has been completed or diagnosed if unsuccessful.
> The read is complete only when an image of the data has been
> successfully transferred to the requesting process". Returning buffered
> data satisfies this definition. Besides, Linux does not support O_RSYNC.

OK, that's a weak definition, specifically as all normal reads are synchronous
anyway.
> 
> O_DSYNC: "the operation has been completed or diagnosed if unsuccessful.
> The write is complete only when the data specified in the write request
> is successfully transferred and all file system information required to
> retrieve the data is successfully transferred". Writing to journal
> located on external device seems to comply with this definition.
> 
> O_SYNC simply adds filesystem metadata update completion.
> 
> So no, O_SYNC & Co cannot replace O_DIRECT.

I rather meant to say: "It's better than nothing if O_DIRECT is missing"

Regards,
Ulrich

> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Antw: [EXT] Re: Q: About a false negative of storage_mon

[Ulrich Windl]

>>> "Ulrich Windl"  schrieb am 03.08.2022
um
08:02 in Nachricht <62ea0f6202a10004c...@gwsmtp.uni-regensburg.de>:
>>>> Ken Gaillot  schrieb am 02.08.2022 um 16:09 in
> Nachricht
> <0a2125a43bbfc09d2ca5bad1a693710f00e33731.ca...@redhat.com>:
>> On Tue, 2022-08-02 at 19:13 +0900, 井上和徳 wrote:
>>> Hi,
>>> 
>>> Since O_DIRECT is not specified in open() [1], it reads the buffer
>>> cache and
>>> may result in a false negative. I fear that this possibility
>>> increases
>>> in environments with large buffer cache and running disk-reading
>>> applications
>>> such as database.
>>> 
>>> So, I think it's better to specify O_RDONLY|O_DIRECT, but what about
>>> it?
>>> (in this case, lseek() processing is unnecessary.)
>>> 
>>> # I am ready to create a patch that works with O_DIRECT. Also, I
>>> wouldn't mind
>>> # a "change to add a new mode of inspection with O_DIRECT
>>> # (add a option to storage_mon) while keeping the current inspection
>>> process".
>>> 
>>> [1] 
>>> 
>>
> https://github.com/ClusterLabs/resource-agents/blob/main/tools/storage_mon.c

> #
> 
>> L47-L90
>>> 
>>> Best Regards,
>>> Kazunori INOUE
>> 
>> I agree, it makes sense to use O_DIRECT when available. I don't think
>> an option is necessary.
>> 
>> However, O_DIRECT is not available on all OSes, so the configure script
>> should detect support. Also, it is not supported by all filesystems, so
>> if the open fails, we should retry without O_DIRECT.
> 
> I just looked it up: It seems POSIX has O_RSYNC and O_SYNC and O_DSYNC)
> instead.

I missed an opening paren: "I just looked it up: It seems POSIX has O_RSYNC
(and O_SYNC and O_DSYNC) ..."

> The buffer cache handling may be different though.
> 
> Regards,
> Ulrich
> 
>> -- 
>> Ken Gaillot 
>> 
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Q: About a false negative of storage_mon

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 02.08.2022 um 16:09 in
Nachricht
<0a2125a43bbfc09d2ca5bad1a693710f00e33731.ca...@redhat.com>:
> On Tue, 2022-08-02 at 19:13 +0900, 井上和徳 wrote:
>> Hi,
>> 
>> Since O_DIRECT is not specified in open() [1], it reads the buffer
>> cache and
>> may result in a false negative. I fear that this possibility
>> increases
>> in environments with large buffer cache and running disk-reading
>> applications
>> such as database.
>> 
>> So, I think it's better to specify O_RDONLY|O_DIRECT, but what about
>> it?
>> (in this case, lseek() processing is unnecessary.)
>> 
>> # I am ready to create a patch that works with O_DIRECT. Also, I
>> wouldn't mind
>> # a "change to add a new mode of inspection with O_DIRECT
>> # (add a option to storage_mon) while keeping the current inspection
>> process".
>> 
>> [1] 
>> 
>
https://github.com/ClusterLabs/resource-agents/blob/main/tools/storage_mon.c#

> L47-L90
>> 
>> Best Regards,
>> Kazunori INOUE
> 
> I agree, it makes sense to use O_DIRECT when available. I don't think
> an option is necessary.
> 
> However, O_DIRECT is not available on all OSes, so the configure script
> should detect support. Also, it is not supported by all filesystems, so
> if the open fails, we should retry without O_DIRECT.

I just looked it up: It seems POSIX has O_RSYNC and O_SYNC and O_DSYNC)
instead.
The buffer cache handling may be different though.

Regards,
Ulrich

> -- 
> Ken Gaillot 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Q: About a false negative of storage_mon

[Ulrich Windl]

>>> "Fabio M. Di Nitto"  schrieb am 02.08.2022 um 14:30
in
Nachricht <0b26c097-1e21-3945-24ba-355cd0ccf...@fabbione.net>:
> Hello Kazunori-san,
> 
> On 02/08/2022 12.13, 井上和徳 wrote:
>> Hi,
>> 
>> Since O_DIRECT is not specified in open() [1], it reads the buffer cache
and
>> may result in a false negative. I fear that this possibility increases
>> in environments with large buffer cache and running disk-reading 
> applications
>> such as database.
>> 
>> So, I think it's better to specify O_RDONLY|O_DIRECT, but what about it?
>> (in this case, lseek() processing is unnecessary.)
> 
> I will have to defer to Christine (in CC) to this email on why we didn´t 
> use O_DIRECT.
> 
> I have a vague recollection that some storage devices didn´t like the 
> O_DIRECT option causing other issues later, but it´s been a while since 
> I touched the code.

The thing with O_DIRECT is that a "software block" has to be a multiple of a
"hardware block"; i.e: you cannot read a partial block, and the buffer has some
alignment requirements.

> 
>> 
>> # I am ready to create a patch that works with O_DIRECT. Also, I wouldn't 
> mind
>> # a "change to add a new mode of inspection with O_DIRECT
>> # (add a option to storage_mon) while keeping the current inspection 
> process".
>> 
>> [1] 
>
https://github.com/ClusterLabs/resource-agents/blob/main/tools/storage_mon.c#

> L47-L90
> 
> It might be a very reasonable solution, tho let´s wait for an answer 
> from Chrissie.
> 
> Cheers
> Fabio
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: QDevice not found after reboot but appears after cluster restart

[Ulrich Windl]

>>> "john tillman"  schrieb am 29.07.2022 um 22:51 in
Nachricht
:
>> > On Thursday 28 July 2022 at 22:17:01, john tillman wrote:
>>>
 I have a two cluster setup with a qdevice. 'pcs quorum status' from a
 cluster node shows the qdevice casting a vote.  On the qdevice node
 'corosync‑qnetd‑tool ‑s' says I have 2 connected clients and 1 cluster.
 The vote count looks correct when I shutdown either one of the cluster
 nodes or the qdevice.  So the voting seems to be working at this point.
>>>
>>> Indeed ‑ shutting down 1 of 3 nodes leaves quorum intact, therefore
>>> everything
>>> still awake knows what's going on.
>>>
 From this state, if I reboot both my cluster nodes at the same time
>>>
>>> Ugh!
>>>
 but leave the qdevice node running, the cluster will not see the
 qdevice
 when the nodes come back up: 'pcs quorum status' show 3 votes expected
 but
 only 2 votes cast (from the cluster nodes).
>>>
>>> I would think this is to be expected, since if you reboot 2 out of 3
>>> nodes,
>>> you completely lose quorum, so the single node left has no idea what to
>>> trust
>>> when the other nodes return.
>>
>> No, no.  I do have quorum after the reboots.  It is courtesy of the 2
>> cluster nodes casting their quorum votes.  However, the qdevice is not
>> casting a vote so I am down to 2 out of 3 nodes.
>>
>> And the qdevice is not part of the cluster.  It will never have any
>> resources running on it.  Its job is just to vote.
>>
>> ‑John
>>
> 
> I thought maybe the problem was that the network wasn't ready when
> corosync.service started so I forced a "ExecStartPre=/usr/bin/sleep 10"
> into it but that didn't change anything.

This type of fix is broken anyway: You are not delaying, you are waiting for
an event (network up).
Basically the OS distribution should have configured it correctly already.

In SLES15 there is:
Requires=network-online.target
After=network-online.target

> 
> I could still use some advice with debugging this oddity.  Or have I used
> up my quota of questions this year :‑)
> 
> ‑John
> 
>>>
>>> Starting from a situation such as this, your only hope is to rebuilt the
>>> cluster from scratch, IMHO.
>>>
>>>
>>> Antony.
>>>
>>> ‑‑
>>> Police have found a cartoonist dead in his house.  They say that details
>>> are
>>> currently sketchy.
>>>
>>>Please reply to the
>>> list;
>>>  please *don't*
>>> CC
>>> me.
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>
>>>
>>
>>
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
>>
> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] IPaddr2 resource times out and cant be killed

[Ulrich Windl]

>>> Ross Sponholtz  schrieb am 29.07.2022 um 21:51 in
Nachricht


> I’m running a RHEL pacemaker cluster on Azure, and I’ve gotten a failure & 
> fencing where I get these messages in the log file:
> 
> warning: vip_ABC_30_monitor_1 process (PID 1779737) timed out
> crit: vip_ABC_30_monitor_1 process (PID 1779737) will not die!
> 
> This resource uses the IPAddr2 resource agent.  I’ve looked at the agent 
> code, and I can’t pinpoint any reason it would hang up, and since the node 
> gets fenced, I can’t tell why this happens – any ideas on what kinds of 
> failures could cause this problem?

Maybe see the proposals that I had sent for Bernd Lentes on a similar issue
(Problem with DLM) last week.

> 
> Thanks,
> Ross



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: [EXT] Problem with DLM

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 26.07.2022 
>>> um
21:36 in Nachricht
<1994685463.141245271.1658864186207.javamail.zim...@helmholtz-muenchen.de>:

> 
> - On 26 Jul, 2022, at 20:06, Ulrich Windl 
> ulrich.wi...@rz.uni-regensburg.de wrote:
> 
>> Hi Bernd!
>> 
>> I think the answer may be some time before the timeout was reported; maybe a
>> network issue? Or a very high load. It's hard to say from the logs...
> 
> Yes, i had a high load before:
> Jul 20 00:17:42 [32512] ha-idg-1   crmd:   notice: 
> throttle_check_thresholds:   High CPU load detected: 90.080002
> Jul 20 00:18:12 [32512] ha-idg-1   crmd:   notice: 
> throttle_check_thresholds:   High CPU load detected: 76.169998
> Jul 20 00:18:42 [32512] ha-idg-1   crmd:   notice: 
> throttle_check_thresholds:   High CPU load detected: 85.629997
> Jul 20 00:19:12 [32512] ha-idg-1   crmd:   notice: 
> throttle_check_thresholds:   High CPU load detected: 70.660004
> Jul 20 00:19:42 [32512] ha-idg-1   crmd:   notice: 
> throttle_check_thresholds:   High CPU load detected: 58.34
> Jul 20 00:20:12 [32512] ha-idg-1   crmd: info: 
> throttle_check_thresholds:   Moderate CPU load detected: 48.740002
> Jul 20 00:20:12 [32512] ha-idg-1   crmd: info: 
> throttle_send_command:   New throttle mode: 0010 (was 0100)
> Jul 20 00:20:42 [32512] ha-idg-1   crmd: info: 
> throttle_check_thresholds:   Moderate CPU load detected: 41.88
> Jul 20 00:21:12 [32512] ha-idg-1   crmd: info: 
> throttle_send_command:   New throttle mode: 0001 (was 0010)
> Jul 20 00:21:56 [12204] ha-idg-1   lrmd:  warning: 
> child_timeout_callback:  dlm_monitor_3 process (PID 11816) timed out
> Jul 20 00:21:56 [12204] ha-idg-1   lrmd:  warning: operation_finished:   
>dlm_monitor_3:11816 - timed out after 2ms
> Jul 20 00:21:56 [32512] ha-idg-1   crmd:error: process_lrm_event:
>Result of monitor operation for dlm on ha-idg-1: Timed Out | call=1255 
> key=dlm_monitor_3 timeout=2ms
> Jul 20 00:21:56 [32512] ha-idg-1   crmd: info: exec_alert_list: 
> Sending resource alert via smtp_alert to informatic@helmholtz-muenchen.de 
> Jul 20 00:21:56 [12204] ha-idg-1   lrmd: info: 
> process_lrmd_alert_exec: Executing alert smtp_alert for 
> 8f934e90-12f5-4bad-b4f4-55ac933f01c6
> 
> Can that interfere with DLM ?

It depends ;-)
If the CPU load is mostly user load, then (also depending on the number of CPUs 
you have) proably not, but if the load is I/O or system load, it could affect 
any pacemaker process in a bad way. I think you'll have to analyze your load; 
maybe adjusting timeouts.

You could use monit to examine your system load (this is just some idle VM):
  status   OK
  monitoring statusMonitored
  monitoring mode  active
  on rebootstart
  load average [0.00] [0.00] [0.00]
  cpu  0.2%usr 0.1%sys 0.0%nice 0.0%iowait 0.0%hardirq 
0.0%softirq 0.0%steal 0.0%guest 0.0%guestnice
  memory usage 442.1 MB [22.3%]
  swap usage   20.5 MB [1.0%]
  uptime   13d 17h 41m
  boot timeThu, 14 Jul 2022 17:40:58
  filedescriptors  1376 [0.7% of 198048 limit]
  data collected   Thu, 28 Jul 2022 11:20:41

You could configurer action scripts like this:
if loadavg (1min) per core > 4 then exec "/var/lib/monit/log-top.sh"
if loadavg (5min) per core > 2 then exec "/var/lib/monit/log-top.sh"
if loadavg (15min) per core > 1 then exec "/var/lib/monit/log-top.sh"
if memory usage > 90% for 2 cycles then exec "/var/lib/monit/log-top.sh"
if swap usage > 25% for 2 cycles then exec "/var/lib/monit/log-top.sh"
if swap usage > 50% then exec "/var/lib/monit/log-top.sh"
if cpu usage (system) > 20% for 3 cycles then exec 
"/var/lib/monit/log-top.sh"
if cpu usage (wait) > 80% then exec "/var/lib/monit/log-top.sh"

A possible script could be (this mess created by< myself):
#!/bin/sh
sect()
{
echo "--- $1 ---"
shift
eval "$@"
}

{
echo "== $(/bin/date) =="
sect 'MONIT env' 'env | grep ^MONIT_'
sect 'mpstat' /usr/bin/mpstat
sect 'vmstat' /usr/bin/vmstat
sect 'top' /usr/bin/top -b -n 1 -Hi
} >> /var/log/monit/top.log


Regards,
Ulrich

> 
> Bernd



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

Re: [ClusterLabs] [EXT] Problem with DLM

[Ulrich Windl]

Hi Bernd!

I think the answer may be some time before the timeout was reported; maybe a
network issue? Or a very high load. It's hard to say from the logs...


>>> Am 26.07.2022 um 15:32, in Nachricht <6ABA7762.4E4 : 205 : 62692>, "Lentes,
Bernd" 
 schrieb:
Hi,

it seems my DLM went grazy:

/var/log/cluster/corosync.log:
Jul 20 00:21:56 [12204] ha-idg-1 lrmd: warning: child_timeout_callback:
dlm_monitor_3 process (PID 11816) timed out
Jul 20 00:21:56 [12204] ha-idg-1 lrmd: warning: operation_finished:
dlm_monitor_3:11816 - timed out after 2ms
Jul 20 00:21:56 [32512] ha-idg-1 crmd: error: process_lrm_event: Result of
monitor operation for dlm on ha-idg-1: Timed Out | call=1255
key=dlm_monitor_3 timeout=2ms
Jul 20 00:21:56 [32512] ha-idg-1 crmd: info: exec_alert_list: Sending resource
alert via smtp_alert to informatic@helmholtz-muenchen.de

/var/log/messages:
2022-07-20T00:21:56.644677+02:00 ha-idg-1 Cluster: alert_smtp.sh
2022-07-20T00:22:16.076936+02:00 ha-idg-1 kernel: [2366794.757496] dlm:
FD5D3C7CE9104CF5916A84DA0DBED302: leaving the lockspace group...
2022-07-20T00:22:16.364971+02:00 ha-idg-1 kernel: [2366795.045657] dlm:
FD5D3C7CE9104CF5916A84DA0DBED302: group event done 0 0
2022-07-20T00:22:16.364982+02:00 ha-idg-1 kernel: [2366795.045777] dlm:
FD5D3C7CE9104CF5916A84DA0DBED302: release_lockspace final free
2022-07-20T00:22:15.533571+02:00 ha-idg-1 Cluster: message repeated 22 times: [
alert_smtp.sh]
2022-07-20T00:22:17.164442+02:00 ha-idg-1 ocfs2_hb_ctl[19106]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -K -u FD5D3C7CE9104CF5916A84DA0DBED302
2022-07-20T00:22:18.904936+02:00 ha-idg-1 kernel: [2366797.586278] ocfs2:
Unmounting device (254,24) on (node 1084777482)
2022-07-20T00:22:19.116701+02:00 ha-idg-1 Cluster: alert_smtp.sh

What do these kernel messages mean ? Why stopped DLM ? I think this is the
second time this happened. It is really a show stopper because node is fenced
some minutes later:
00:34:40.709002 ha-idg: Fencing Operation Off of ha-idg-1 by ha-idg-2 for
crmd.28253@ha-idg-2: OK (ref=9710f0e2-a9a9-42c3-a294-ed0bd78bba1a)

What can i do ? Is there an alternative DLM ?
System is SLES 12 SP5. Update to SLES 15 SP3 ?

Bernd



--
Bernd Lentes
System Administrator
Institute for Metabolism and Cell Death (MCD)
Building 25 - office 122
HelmholtzZentrum München
bernd.len...@helmholtz-muenchen.de
phone: +49 89 3187 1241
+49 89 3187 49123
fax: +49 89 3187 2294
http://www.helmholtz-muenchen.de/mcd

Public key:
30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 6c
3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82 fc cc
96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3 a7 48
f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92 67 9e
ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53 89 70
51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2 e3 22
f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2 fa 27
ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2 67 53
e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85 08 d0
6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac 58 f1
38 43 0e 72 af 02 03 01 00 01

___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Cannot add a node with pcs

[Ulrich Windl]

>>> Piotr Szafarczyk  schrieb am 12.07.2022 um 12:34 in
Nachricht <38ccc24a-7b01-561c-20f8-ec2273a18...@netexpert.pl>:
> Hi,
> 
> I used to have a working cluster with 3 nodes (and stonith disabled). 

THE SLES guide says:
Important: No Support Without STONITH
You must have a node fencing mechanism for your cluster.
The global cluster options stonith-enabled and startup-fencing must be
set to true . When you change them, you lose support.

Maybe that helps.

> After an unexpected restart of one node, the cluster split. The node #2 
> started to see the others as unclean. Nodes 1 and 2 were cooperating 
> with each other, showing #2 as offline. There were no network connection 
> problems.
> 
> I removed #2 (operating from #1) with
> pcs cluster node remove n2
> 
> I verified that it had removed all configuration from #2, both for 
> corosync and for pacemaker. The cluster looks like working correctly 
> with two nodes (and no traces of #2).
> 
> Now I am trying to add the third node back.
> pcs cluster node add n2
> Disabling SBD service...
> n2: sbd disabled
> Sending 'corosync authkey', 'pacemaker authkey' to 'n2'
> n2: successful distribution of the file 'corosync authkey'
> n2: successful distribution of the file 'pacemaker authkey'
> Sending updated corosync.conf to nodes...
> n3: Succeeded
> n2: Succeeded
> n1: Succeeded
> n3: Corosync configuration reloaded
> 
> I am able to start #2 operating from #1
> 
> pcs cluster pcsd-status
>n2: Online
>n3: Online
>n1: Online
> 
> pcs cluster enable n2
> pcs cluster start n2
> 
> I can see that corosync's configuration has been updated, but 
> pacemaker's not.
> 
> _Checking from #1:_
> 
> pcs config
> Cluster Name: n
> Corosync Nodes:
>   n1 n3 n2
> Pacemaker Nodes:
>   n1 n3
> [...]
> 
> pcs status
>* 2 nodes configured
> Node List:
>* Online: [ n1 n3 ]
> [...]
> 
> pcs cluster cib scope=nodes
> 
>
>
> 
> 
> _#2 is seeing the state differently:_
> 
> pcs config
> Cluster Name: n
> Corosync Nodes:
>   n1 n3 n2
> Pacemaker Nodes:
>   n1 n2 n3
> 
> pcs status
>* 3 nodes configured
> Node List:
>* Online: [ n2 ]
>* OFFLINE: [ n1 n3 ]
> Full List of Resources:
>* No resources
> [...]
> (there are resources configured on #1 and #3)
> 
> pcs cluster cib scope=nodes
> 
>
>
>
> 
> 
> Help me diagnose it please. Where should I look for the problem? (I have 
> already tried a few things more - I see nothing helpful in log files, 
> pcs --debug shows nothing suspicious, tried even editing the CIB manually)
> 
> Best regards,
> 
> Piotr Szafarczyk




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] nfs mount resource won't stop - lazy option ?

[Ulrich Windl]

>>> Patrick Vranckx  schrieb am 08.07.2022 um 
>>> 11:25
in Nachricht <69422771-c57d-d910-8e92-8a2345847...@uclouvain.be>:
> Hi,
> 
> I have a cluster configures as such:
> 
> Online: [ mbacktm1 mbacktm2 ]
> 
> Full list of resources:
> 
>   stonith-mbt(stonith:fence_scsi):Started mbacktm2
>   Resource Group: group-mbt-stor
>   nfs-daemon(ocf::heartbeat:nfsserver):Started mbacktm2
>   mount_admnfs(ocf::heartbeat:Filesystem):Started mbacktm2
>   Resource Group: group-secondary-gateway
>   mount_admnfs_1(ocf::heartbeat:Filesystem):Started mbacktm1
> 
> 
> A daemon NFS is started on one node, a filesystem is exported and 
> mounted on both node.
> 
> If the node running the nfs server crashes, reboots or is fenced, the 
> resource "mount_admnfs_1" on the other node won't stop because the nfs 
> server will not respond (unable to unmount the filesystem anymore)
> 
> In such a situation without pacemaker, one can do a "umount -l" to solve 
> the problem.
> 
> Is there a solution  for Pacemaker ?

What is your specific problem? NFS design is that clients wait for the server 
to re-appear.

> 
> Best regards,
> 
> Patrick
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Questions related to failover time

[Ulrich Windl]

>>> ???  schrieb am 08.07.2022 um 04:30 in Nachricht
:
> Hello.
> I am a college student who wants to apply facemaker and corosync to
> raspberry pi.
> What I'm going to do is reduce the failover time of two raspberry pi.

Reduce from what to what?

> So I have a question.
> Is there a way to reduce the failover time of two raspberry pi?

See above.

> I searched and found out that I can change the monitor interval of the
> virtual ip, is that right?

You are talking about a specific resource failing?

> If it is correct, I wonder what the minimum value of monitor interval is.

In general a bit more what the longest estimated execution time will be.

> And I wonder what is the way to reduce the failover time if you say no.
> Please give us your answers.

Also give more details on the configuration.

> Thank you.
> Oh! And you can email me your answer because I might not be able to see it.

If you ask a list you are expected to read that list!

> My email is kwonos.It's k...@gmail.com.




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] is there a way to cancel a running live migration or a "resource stop" ?

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 07.07.2022
um
11:28 in Nachricht
<2146740519.119706769.1657186111711.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> is there a way to cancel a running live migration or a "resource stop" ?

And the effect of canceling would be what?
Well, the operation to cancel any operation is called fencing, but I guess
that is not what you have in mind...

Regards,
Ulrich

> 
> Bernd
> 
> -- 
> Bernd Lentes 
> System Administrator 
> Institute for Metabolism and Cell Death (MCD) 
> Building 25 - office 122 
> HelmholtzZentrum München 
> bernd.len...@helmholtz-muenchen.de 
> phone: +49 89 3187 1241
>+49 89 3187 49123 
> fax:   +49 89 3187 2294 
> http://www.helmholtz-muenchen.de/mcd 
> 
> Public key: 
> 30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 
> 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82
fc 
> cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3
a7 
> 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92
67 
> 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53
89 
> 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2
e3 
> 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2
fa 
> 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2
67 
> 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85
08 
> d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac
58 
> f1 38 43 0e 72 af 02 03 01 00 01



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] FYI: one more regression introduced in Pacemaker 2.1.3

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 27.06.2022 um 23:07 in
Nachricht
:
> Hi all,
> 
> Another regression was found that was introduced in Pacemaker 2.1.3.
> 
> As part of trying to make the output of various commands more
> consistent, as of 2.1.3, "crm_attribute ‑‑quiet ‑‑query" prints
> "(null)" for an attribute that does not exist. Previously, it would not
> print anything.

I'd consider "(null)" being the result of a NULL pointer to be f-printed with
"%s" to be a programming error always ;-)

> 
> Unfortunately, certain resource agents, at least including redis,
> depended on the previous behavior. The agents most likely to be
> affected are those for promotable resources.
> 
> The original behavior is restored as of commit 9853f4d05 in the main
> branch. Users who use promotable resources are advised to wait until
> the fix is released (expected in 2.1.5 at the end of this year) or
> ensure that their OS packages include the fix if using 2.1.3 or 2.1.4.
> ‑‑ 
> Ken Gaillot 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] modified RA can't be used

[Ulrich Windl]

>>> "Lentes, Bernd"  schrieb am 27.06.2022 
>>> um
14:54 in Nachricht
<273609196.108116045.1656334469407.javamail.zim...@helmholtz-muenchen.de>:
> Hi,
> 
> i adapted the RA ocf/heartbeat/VirtualDomain to my needs and renamed it to 
> VirtualDomain.ssh
> When i try to use it now, i get an error message.
> I start e.g. "crm configure edit vm-idcc-devel" to modify an existing 
> VirtualDomain that it uses the new RA
> and want to save it i get the following error:
> ERROR: ocf:heartbeat:VirtualDomain.ssh: got no meta-data, does this RA 
> exist?
> ERROR: ocf:heartbeat:VirtualDomain.ssh: got no meta-data, does this RA 
> exist?
> ERROR: ocf:heartbeat:VirtualDomain.ssh: no such resource agent

I'm aware that this question was answered and fixed, but still let me remark:
Did you try ocf-tester to test your RA before using it? ;-)

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Question regarding the security of corosync

[Ulrich Windl]

>>> Mario Freytag  schrieb am 17.06.2022 um 11:39 in
Nachricht
:

> Dear sirs, or madams,
> 
> I’d like to ask about the security of corosync. We’re using a Proxmox HA 
> setup in our testing environment and need to confirm it’s compliance with
PCI 
> guidelines.
> 
> We have a few questions:
> 
> Is the communication encrypted?
> What method of encryption is used?
> What method of authentication is used?
> What is the recommended way of separation for the corosync network? VLAN?

The answer you probably do not expect is this:
Assume the communication is safely encrypted, but a third party can interrupt
communication, is the system safe then?
In that light probably any system using the Internet as transport is not safe,
so you can set up your private connections (which should be private per
definition ;-))

Regards,
Ulrich


> 
> Best regards
> 
> Mario Freytag
> Systemadministrator | WEBINC GmbH & Co. KG
> 
> ​Unter den Eichen 5 Geb. F | 65195 Wiesbaden | T +49 611 541075 0
> Amtsgericht Wiesbaden | HRA 9610 | Geschäftsführung: Marina Maurer, Monika 
> Brandes



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] related to fencing in general , docker containers

[Ulrich Windl]

I had pointed out a log time ago that the existing documentation on fencing 
agents is practically unusable.

>>> Sridhar K  schrieb am 17.06.2022 um 15:53 in Nachricht
:
> Hi Team,
> 
> Please share any pointers, references, example usage's w.r.t fencing in
> general and its use w.r.t docker containers.
> 
> referring as of now
> https://clusterlabs.org/pacemaker/doc/crm_fencing.html 
> 
> need to check the feasibility of fencing w.r.t docker containers
> 
> Thank you
> 
> Regards
> Sridhar




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: Re: Antw: [EXT] Re: Why not retry a monitor (pacemaker‑execd) that got a segmentation fault?

[Ulrich Windl]

>>> Klaus Wenninger  schrieb am 15.06.2022 um 13:22 in
Nachricht
:
> On Wed, Jun 15, 2022 at 10:33 AM Ulrich Windl
>  wrote:
>>

...

>> (As said above it may be some RAM corruption where SMI (system management
>> interrupts, or so) play a role, but Dell says the hardware is OK, and using
>> SLES we don't have software support with Dell, so they won't even consider 
> that
>> fact.)
> 
> That happens inside of VMs right? I mean nodes being VMs.

No, it happens on the hypervisor nodes that are part of the cluster.

> A couple of years back I had an issue running protected mode inside
> of kvm-virtual machines on Lenovo laptops.
> That was really an SMI issue (obviously issues when an SMI interrupt
> was invoked during the CPU being in protected mode) that went away
> disabling SMI interrupts.
> I have no idea if that is still possible with current chipsets. And I'm not
> telling you to do that in production but it might be interesting to narrow
> the issue down still. One might run into thermal issues and such
> SMI is taking care of on that hardware.

Well, as I have no better idea, I'd probably even give "kick it hard with the 
foot" a chance ;-)

Regards,
Ulrich

> 
> Klaus
>>
>> But actually I start believing such a system is a good playground for any HA
>> solution ;-)
>> Unfortunately here it's much more production than playground...
>>
>> Regards,
>> Ulrich
>>
>>
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Re: Why not retry a monitor (pacemaker‑execd) that got a segmentation fault?

[Ulrich Windl]

>>> Klaus Wenninger  schrieb am 15.06.2022 um 10:00 in
Nachricht
:
> On Wed, Jun 15, 2022 at 8:32 AM Ulrich Windl
>  wrote:
>>
>> >>> Ulrich Windl schrieb am 14.06.2022 um 15:53 in Nachricht <62A892F0.174
: 161 
> :
>> 60728>:
>>
>> ...
>> > Yes it's odd, but isn't the cluster just to protect us from odd
situations?
>> > ;‑)
>>
>> I have more odd stuff:
>> Jun 14 20:40:09 rksaph18 pacemaker‑execd[7020]:  warning: 
> prm_lockspace_ocfs2_monitor_12 process (PID 30234) timed out
>> ...
>> Jun 14 20:40:14 h18 pacemaker‑execd[7020]:  crit: 
> prm_lockspace_ocfs2_monitor_12 process (PID 30234) will not die!
>> ...
>> Jun 14 20:40:53 h18 pacemaker‑controld[7026]:  warning: lrmd IPC request
525 
> failed: Connection timed out after 5000ms
>> Jun 14 20:40:53 h18 pacemaker‑controld[7026]:  error: Couldn't perform 
> lrmd_rsc_cancel operation (timeout=0): ‑110: Connection timed out (110)
>> ...
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  error: Couldn't perform 
> lrmd_rsc_exec operation (timeout=9): ‑114: Connection timed out (110)
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  error: Operation stop on 
> prm_lockspace_ocfs2 failed: ‑70
>> ...
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  warning: Input I_FAIL
received 
> in state S_NOT_DC from do_lrm_rsc_op
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  notice: State transition 
> S_NOT_DC ‑> S_RECOVERY
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  warning: Fast‑tracking
shutdown 
> in response to errors
>> Jun 14 20:42:23 h18 pacemaker‑controld[7026]:  error: Input I_TERMINATE 
> received in state S_RECOVERY from do_recover
>> Jun 14 20:42:28 h18 pacemaker‑controld[7026]:  warning: Sending IPC to lrmd

> disabled until pending reply received
>> Jun 14 20:42:28 h18 pacemaker‑controld[7026]:  error: Couldn't perform 
> lrmd_rsc_cancel operation (timeout=0): ‑114: Connection timed out (110)
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  warning: Sending IPC to lrmd

> disabled until pending reply received
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  error: Couldn't perform 
> lrmd_rsc_cancel operation (timeout=0): ‑114: Connection timed out (110)
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  notice: Stopped 2 recurring

> operations at shutdown (0 remaining)
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  error: 3 resources were
active 
> at shutdown
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  notice: Disconnected from
the 
> executor
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  notice: Disconnected from 
> Corosync
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  notice: Disconnected from
the 
> CIB manager
>> Jun 14 20:42:33 h18 pacemaker‑controld[7026]:  error: Could not recover
from 
> internal error
>> Jun 14 20:42:33 h18 pacemakerd[7003]:  error: pacemaker‑controld[7026]
exited 
> with status 1 (Error occurred)
>> Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping
pacemaker‑schedulerd
>> Jun 14 20:42:33 h18 pacemaker‑schedulerd[7024]:  notice: Caught
'Terminated' 
> signal
>> Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping pacemaker‑attrd
>> Jun 14 20:42:33 h18 pacemaker‑attrd[7022]:  notice: Caught 'Terminated' 
> signal
>> Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping pacemaker‑execd
>> Jun 14 20:42:34 h18 sbd[6856]:  warning: inquisitor_child: pcmk health 
> check: UNHEALTHY
>> Jun 14 20:42:34 h18 sbd[6856]:  warning: inquisitor_child: Servant pcmk is

> outdated (age: 41877)
>> (SBD Fencing)
>>
> 
> Rolling it up from the back I guess the reaction to self‑fence in case 
> pacemaker
> is telling it doesn't know ‑ and isn't able to find out ‑ about the
> state of the resources
> is basically correct.
> 
> Seeing the issue with the fake‑age being printed ‑ possibly causing
> confusion ‑ it reminds
> me that this should be addressed. Thought we had already but obviously
> a false memory.

Hi Klaus and others!

Well that is the current update state of SLES15 SP3; maybe upstream updates
did not make it into SLES yet; I don't know.

> 
> Would be interesting if pacemaker would recover the sub‑processes
> without sbd around
> and other ways of fencing ‑ that should kick in in a similar way ‑
> would need a significant
> time.
> As pacemakerd recently started to ping the sub‑daemons via ipc ‑
> instead of just listening
> for signals ‑ it would be interesting if logs we are seeing are
> already from that code.

The "code" probably is:
pacemaker-2.0.5+20201202.ba59be712-150300.4.21.1.x86_64

> 
> That what is happening with the monitor‑proce

[ClusterLabs] Antw: [EXT] Re: Why not retry a monitor (pacemaker‑execd) that got a segmentation fault?

[Ulrich Windl]

>>> Ulrich Windl schrieb am 14.06.2022 um 15:53 in Nachricht <62A892F0.174 : 
>>> 161 :
60728>:

...
> Yes it's odd, but isn't the cluster just to protect us from odd situations? 
> ;-)

I have more odd stuff:
Jun 14 20:40:09 rksaph18 pacemaker-execd[7020]:  warning: 
prm_lockspace_ocfs2_monitor_12 process (PID 30234) timed out
...
Jun 14 20:40:14 h18 pacemaker-execd[7020]:  crit: 
prm_lockspace_ocfs2_monitor_12 process (PID 30234) will not die!
...
Jun 14 20:40:53 h18 pacemaker-controld[7026]:  warning: lrmd IPC request 525 
failed: Connection timed out after 5000ms
Jun 14 20:40:53 h18 pacemaker-controld[7026]:  error: Couldn't perform 
lrmd_rsc_cancel operation (timeout=0): -110: Connection timed out (110)
...
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  error: Couldn't perform 
lrmd_rsc_exec operation (timeout=9): -114: Connection timed out (110)
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  error: Operation stop on 
prm_lockspace_ocfs2 failed: -70
...
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  warning: Input I_FAIL received 
in state S_NOT_DC from do_lrm_rsc_op
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  notice: State transition 
S_NOT_DC -> S_RECOVERY
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  warning: Fast-tracking shutdown 
in response to errors
Jun 14 20:42:23 h18 pacemaker-controld[7026]:  error: Input I_TERMINATE 
received in state S_RECOVERY from do_recover
Jun 14 20:42:28 h18 pacemaker-controld[7026]:  warning: Sending IPC to lrmd 
disabled until pending reply received
Jun 14 20:42:28 h18 pacemaker-controld[7026]:  error: Couldn't perform 
lrmd_rsc_cancel operation (timeout=0): -114: Connection timed out (110)
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  warning: Sending IPC to lrmd 
disabled until pending reply received
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  error: Couldn't perform 
lrmd_rsc_cancel operation (timeout=0): -114: Connection timed out (110)
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  notice: Stopped 2 recurring 
operations at shutdown (0 remaining)
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  error: 3 resources were active 
at shutdown
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  notice: Disconnected from the 
executor
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  notice: Disconnected from 
Corosync
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  notice: Disconnected from the 
CIB manager
Jun 14 20:42:33 h18 pacemaker-controld[7026]:  error: Could not recover from 
internal error
Jun 14 20:42:33 h18 pacemakerd[7003]:  error: pacemaker-controld[7026] exited 
with status 1 (Error occurred)
Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping pacemaker-schedulerd
Jun 14 20:42:33 h18 pacemaker-schedulerd[7024]:  notice: Caught 'Terminated' 
signal
Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping pacemaker-attrd
Jun 14 20:42:33 h18 pacemaker-attrd[7022]:  notice: Caught 'Terminated' signal
Jun 14 20:42:33 h18 pacemakerd[7003]:  notice: Stopping pacemaker-execd
Jun 14 20:42:34 h18 sbd[6856]:  warning: inquisitor_child: pcmk health check: 
UNHEALTHY
Jun 14 20:42:34 h18 sbd[6856]:  warning: inquisitor_child: Servant pcmk is 
outdated (age: 41877)
(SBD Fencing)

Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: Why not retry a monitor (pacemaker‑execd) that got a segmentation fault?

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 14.06.2022 um 15:49 in
Nachricht
:
> On Tue, 2022‑06‑14 at 14:36 +0200, Ulrich Windl wrote:
>> Hi!
>> 
>> I had a case where a VirtualDomain monitor operation ended in a core
>> dump (actually it was pacemaker‑execd, but it counted as "monitor"
>> operation), and the cluster decided to restart the VM. Wouldn't it be
>> worth to retry the monitor operation first?
> 
> It counts like any other monitor failure
> 
>> Chances are that a re‑tried monitor operation returns a better status
>> than segmentation fault.
>> Or dies the logic just ignore processes dying on signals?
>> 
>> 20201202.ba59be712‑150300.4.21.1.x86_64 (SLES15 SP3)
>> 
>> Jun 14 14:09:16 h19 systemd‑coredump[28788]: Process 28786
>> (pacemaker‑execd) of user 0 dumped core.
>> Jun 14 14:09:16 h19 pacemaker‑execd[7440]:  warning:
>> prm_xen_v04_monitor_60[28786] terminated with signal:
>> Segmentation fault
> 
> This means that the child process forked to execute the resource agent
> segfaulted, which is odd.

Yes it's odd, but isn't the cluster just to protect us from odd situations?
;-)

> 
> Is the agent a compiled program? If not, it's possible the tiny amount
> of pacemaker code that executes the agent is what segfaulted. Do you
> have the actual core, and can you do a backtrace?

Believe me, it's just "odd":
  Stack trace of thread
28786:
  #0  0x7f85589e0bcf
__libc_fork (/lib64/libc-2.31.so + 0xe1bcf)
  #1  0x7f855949b85d n/a
(/usr/lib64/libcrmservice.so.28.2.2 + 0x785d)
  #2  0x7f855949a5e3 n/a
(/usr/lib64/libcrmservice.so.28.2.2 + 0x65e3)
  #3  0x7f8558d470ed n/a
(/usr/lib64/libglib-2.0.so.0.6200.6 + 0x530ed)
  #4  0x7f8558d46624
g_main_context_dispatch (/usr/lib64/libglib-2.0.so.0.6200.6 + 0x52624)
  #5  0x7f8558d469c0 n/a
(/usr/lib64/libglib-2.0.so.0.6200.6 + 0x529c0)
  #6  0x7f8558d46c82
g_main_loop_run (/usr/lib64/libglib-2.0.so.0.6200.6 + 0x52c82)
  #7  0x558c0765930b n/a
(/usr/lib/pacemaker/pacemaker-execd + 0x330b)
  #8  0x7f85589342bd
__libc_start_main (/lib64/libc-2.31.so + 0x352bd)
  #9  0x558c076593da n/a
(/usr/lib/pacemaker/pacemaker-execd + 0x33da)

Rumors say it's Dell's dcdbas module combined with Xen and an AMD CPU plus
some software bugs ;-)

Regards,
Ulrich

> 
>> Jun 14 14:09:16 h19 pacemaker‑controld[7443]:  error: Result of
>> monitor operation for prm_xen_v04 on h19: Error
>> Jun 14 14:09:16 h19 pacemaker‑controld[7443]:  notice: Transition 9
>> action 107 (prm_xen_v04_monitor_60 on h19): expected 'ok' but got
>> 'error'
>> ...
>> Jun 14 14:09:16 h19 pacemaker‑schedulerd[7442]:  notice:  *
>> Recoverprm_xen_v04  ( h19 )
>> 
>> Regards,
>> ulrich
>> 
>> 
>> 
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
> ‑‑ 
> Ken Gaillot 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Why not retry a monitor (pacemaker-execd) that got a segmentation fault?

[Ulrich Windl]

Hi!

I had a case where a VirtualDomain monitor operation ended in a core dump 
(actually it was pacemaker-execd, but it counted as "monitor" operation), and 
the cluster decided to restart the VM. Wouldn't it be worth to retry the 
monitor operation first?
Chances are that a re-tried monitor operation returns a better status than 
segmentation fault.
Or dies the logic just ignore processes dying on signals?

20201202.ba59be712-150300.4.21.1.x86_64 (SLES15 SP3)

Jun 14 14:09:16 h19 systemd-coredump[28788]: Process 28786 (pacemaker-execd) of 
user 0 dumped core.
Jun 14 14:09:16 h19 pacemaker-execd[7440]:  warning: 
prm_xen_v04_monitor_60[28786] terminated with signal: Segmentation fault
Jun 14 14:09:16 h19 pacemaker-controld[7443]:  error: Result of monitor 
operation for prm_xen_v04 on h19: Error
Jun 14 14:09:16 h19 pacemaker-controld[7443]:  notice: Transition 9 action 107 
(prm_xen_v04_monitor_60 on h19): expected 'ok' but got 'error'
...
Jun 14 14:09:16 h19 pacemaker-schedulerd[7442]:  notice:  * Recover
prm_xen_v04  ( h19 )

Regards,
ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] crm status shows CURRENT DC as None

[Ulrich Windl via Users]

>>> Priyanka Balotra  schrieb am 14.06.2022 um 
>>> 07:40
in Nachricht
:
> Hi Folks,
> 
> crm status shows CURRENT DC as None. Please check and let us know why the
> current DC is not pointing to any of the nodes.
> 

Maybe present the complete output of "crm_mon -1Arfj". For example where are 
your nodes in the output?

> 
> 
> *CRM Status:*
> 
> Cluster Summary:
> 
>   * Stack: corosync
> 
> *  * Current DC: NONE*
> 
>   * Last updated: Tue Jun  7 06:14:59 2022
> 
>   * Last change:  Tue Jun  7 05:29:40 2022 by root via cibadmin on FILE-2
> 
>   * 2 nodes configured
> 
>   * 9 resource instances configured
> 
> 
>- How the current DC will be set to any node once we see as *None*?
>- Is there any impact on cluster functionality?
> 
> Thanks
> Priyanka




___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] fencing configuration

[Ulrich Windl]

>>> Zoran Bošnjak  schrieb am 07.06.2022 um 10:26 in
Nachricht <1951254459.265.1654590407828.javamail.zim...@via.si>:
> Hi, I need some help with correct fencing configuration in 5‑node cluster.
> 
> The speciffic issue is that there are 3 rooms, where in addition to node 
> failure scenario, each room can fail too (for example in case of room power

> failure or room network failure).
> 
> room0: [ node0 ]
> roomA: [ node1, node2 ]
> roomB: [ node3, node4 ]

First, it's good that even after a complete room failed, you will still have a
quorum.

> 
> ‑ ipmi board is present on each node
> ‑ watchdog timer is available
> ‑ shared storage is not available

The last one sounds adventuous to me, but I'll read on...

> 
> Please advice, what would be a proper fencing configuration in this case.

sbd using shared storage ;-)

> 
> The intention is to configure ipmi fencing (using "fence_idrac" agent) plus

> watchdog timer as a fallback. In other words, I would like to tell the 
> pacemaker: "If fencing is required, try to fence via ipmi. In case of ipmi 
> fence failure, after some timeout assume watchdog has rebooted the node, so

> it is safe to proceed, as if the (self)fencing had succeeded)."

An interesting question would be how to reach any node in a room if that room
failed.
A perfect solution would be to have a shared storage in every room and
configure 3-way sbd disks.
In addition you could use three-way mirroring of your data, just to be
paranoid ;-)

> 
> From the documentation is not clear to me whether this would be:
> a) multiple fencing where ipmi would be first level and sbd would be a 
> second level fencing (where sbd always succeeds)
> b) or this is considered a single level fencing with a timeout
> 
> I have tried to followed option b) and create stonith resource for each node

> and setup the stonith‑watchdog‑timeout, like this:
> 
> ‑‑‑
> # for each node... [0..4]
> export name=...
> export ip=...
> export password=...
> sudo pcs stonith create "fence_ipmi_$name" fence_idrac \
> lanplus=1 ip="$ip" \
> username="admin"  password="$password" \
> pcmk_host_list="$name" op monitor interval=10m timeout=10s
> 
> sudo pcs property set stonith‑watchdog‑timeout=20
> 
> # start dummy resource
> sudo pcs resource create dummy ocf:heartbeat:Dummy op monitor interval=30s
> ‑‑‑
> 
> I am not sure if additional location constraints have to be specified for 
> stonith resources. For example: I have noticed that pacemaker will start a 
> stonith resource on the same node as the fencing target. Is this OK? 
> 
> Should there be any location constraints regarding fencing and rooms?
> 
> 'sbd' is running, properties are as follows:
> 
> ‑‑‑
> $ sudo pcs property show
> Cluster Properties:
>  cluster‑infrastructure: corosync
>  cluster‑name: debian
>  dc‑version: 2.0.3‑4b1f869f0f
>  have‑watchdog: true
>  last‑lrm‑refresh: 1654583431
>  stonith‑enabled: true
>  stonith‑watchdog‑timeout: 20
> ‑‑‑
> 
> Ipmi fencing (when the ipmi connection is alive) works correctly for each 
> node. The watchdog timer also seems to be working correctly. The problem is

> that dummy resource is not restarted as expected.

My favourite here is "crm_mon -1Arfj" ;-)

> 
> In the test scenario, the dummy resource is currently running on node1. I 
> have simulated node failure by unplugging the ipmi AND host network 
> interfaces from node1. The result was that node1 gets rebooted (by
watchdog), 
> but the rest of the pacemaker cluster was unable to fence node1 (this is 
> expected, since node1's ipmi is not accessible). The problem is that dummy 
> resource remains stopped and node1 unclean. I was expecting that 

"unclean" means fencing is either in progress, or did not succeed (like when
you have no fencing at all).

> stonith‑watchdog‑timeout kicks in, so that dummy resource gets restarted on

> some other node which has quorum. 

So that actually does the fencing. Logs could be interesting to read, too.

> 
> Obviously there is something wrong with my configuration, since this seems 
> to be a reasonably simple scenario for the pacemaker. Appreciate your help.

See above.

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: normal reboot with active sbd does not work

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 03.06.2022 um 17:04 in
Nachricht <99f7746a-c962-33bb-6737-f88ba0128...@gmail.com>:
> On 03.06.2022 16:51, Zoran Bošnjak wrote:
>> Thanks for all your answers. Sorry, my mistake. The ipmi_watchdog is indeed

> OK. I was first experimenting with "softdog", which is blacklisted. So the 
> reasonable question is how to properly start "softdog" on ubuntu.
>> 
> 
> blacklist prevents autoloading of modules by alias during hardware
> detection. Neither softdog or ipmi_watchdog have any alias so they
> cannot be autoloaded and blacklist is irrelevant here.
> 
>> The reason to unload watchdog module (ipmi or softdog) is that there seems

> to be a difference between normal reboot and watchdog reboot.
>> In case of ipmi watchdog timer reboot:
>> - the system hangs at the end of reboot cycle for some time
>> - restart seems to be harder (like power off/on cycle), BIOS runs more 
> diagnostics at startup

maybe kdump is enabled in that case?

>> - it turns on HW diagnostic indication on the server front panel (dell 
> server) which stays on forever
>> - it logs the event to IDRAC, which is unnecessary, because it was not a 
> hardware event, but just a normal reboot

If the hardware watchdog times out and fires, it is consoidered to be an
exceptional event that will be logged and reported.

>> 
>> In case of "sudo reboot" command, I would like to skip this... so the idea

> is to fully stop the watchdog just before reboot. I am not sure how to do 
> this properly.
>> 
>> The "softdog" is better in this respect. It does not trigger nothing from 
> the list above, but I still get the message during reboot
>> [ ... ] watchdog: watchdog0: watchdog did not stop!
>> ... with some small timeout.
>> 
> 
> The first obvious question - is there only one watchdog? Some watchdog
> drivers *are* autoloaded.
> 
> Is there only one user of watchdog? systemd may use it too as example.

Don't mix timers with a watchdog: It makes little sense to habe multipe
watchdogs enabled IMHO.

> 
>> So after some additional testing, the situation is the following:
>> 
>> - without any watchdog and without sbd package, the server reboots
normally
>> - with "softdog" module loaded, I only get "watchdog did not stop message"

> at reboot
>> - with "softdog" loaded, but unloaded with "ExecStop=...rmmod", reboot is 
> normal again
>> - same as above, but with "sbd" package loaded, I am getting "watchdog did

> not stop message" again
>> - switching from "softdog" to "ipmi_watchdog" gets me to the original list

> of problems
>> 
>> It looks like the "sbd" is preventing the watchdog to close, so that 
> watchdog triggers always, even in the case of normal reboot. What am I 
> missing here?

The watchdog may have a "no way out" parameter that prevents disabling it
after enabled once.

> 
> While the only way I can reproduce it on my QEMU VM is "reboot -f"
> (without stopping all services), there is certainly a race condition in
> sbd.service.
> 
> ExecStop=@bindir@/kill -TERM $MAINPID
> 
> 
> systemd will continue as soon as "kill" completes without waiting for
> sbd to actually stop. It means systemd may complete shutdown sequence
> before sbd had chance to react on signal and then simply kill it. Which
> leaves watchdog armed.
> 
> For test purpose try to use script that loops until sbd is actually
> stopped for ExecStop.
> 
> Note that systemd strongly recommends to use synchronous command for
> ExecStop (we may argue that this should be handled by service manager
> itself, but well ...).
> 
>> 
>> Zoran
>> 
>> - Original Message -
>> From: "Andrei Borzenkov" 
>> To: "users" 
>> Sent: Friday, June 3, 2022 11:24:03 AM
>> Subject: Re: [ClusterLabs] normal reboot with active sbd does not work
>> 
>> On 03.06.2022 11:18, Zoran Bošnjak wrote:
>>> Hi all,
>>> I would appreciate an advice about sbd fencing (without shared storage).
>>>
>>> I am using ubuntu 20.04., with default packages from the repository 
> (pacemaker, corosync, fence-agents, ipmitool, pcs...).
>>>
>>> HW watchdog is present on servers. The first problem was to load/unload
the 
> watchdog module. For some reason the module is blacklisted on ubuntu,
>> 
>> What makes you think so?
>> 
>> bor@bor-Latitude-E5450:~$ lsb_release  -d
>> 
>> Description: Ubuntu 20.04.4 LTS
>> 
>> bor@bor-Latitude-E5450:~$ modprobe -c | grep ipmi_watchdog
>> 
>> bor@bor-Latitude-E5450:~$
>> 
>> 
>> 
>> 
>> 
>>> so I've created a service for this purpose.
>>>
>> 
>> man modules-load.d
>> 
>> 
>>> --- file: /etc/systemd/system/watchdog.service
>>> [Unit]
>>> Description=Load watchdog timer module
>>> After=syslog.target
>>>
>> 
>> Without any explicit dependencies stop will be attempted as soon as
>> possible.
>> 
>>> [Service]
>>> Type=oneshot
>>> RemainAfterExit=yes
>>> ExecStart=/sbin/modprobe ipmi_watchdog
>>> ExecStop=/sbin/rmmod ipmi_watchdog
>>>
>> 
>> Why on earth do you need to unload kernel driver when system reboots?
>> 
>>> [Install]
>>> 

[ClusterLabs] Antw: Re: Antw: [EXT] normal reboot with active sbd does not work

[Ulrich Windl]

>>> Zoran Bošnjak  schrieb am 03.06.2022 um 15:16 in
Nachricht <332746042.172.1654262179353.javamail.zim...@via.si>:
> Yes, it's dell power edge. Would you know how to disable front panel 
> indication in case of watchdog reset?

I had a Dell support call: Resetting the iDRAC would reset the alert, but it
would also cause interrupted IPMI communication.
ipmitool -I open sel clear
may work, too, but it will clear the event log.

Regards,
Ulrich



> 
> "echo V >/dev/watchdog" makes no difference.
> 
> - Original Message -
> From: "Ulrich Windl" 
> To: "users" 
> Sent: Friday, June 3, 2022 11:00:18 AM
> Subject: [ClusterLabs] Antw: [EXT] normal reboot with active sbd does not 
> work
> 
>>>> Zoran Bošnjak  schrieb am 03.06.2022 um 10:18 in
> Nachricht <2046503996.272.1654244336372.javamail.zim...@via.si>:
>> Hi all,
>> I would appreciate an advice about sbd fencing (without shared storage).
> 
> Not an answer, but curiosity:
> As sbd needs very little space (like just 1MB), did anybody ever try to use

> a
> small computer like a raspberry pi to privide shared storage for SBD via 
> iSCSI
> for example?
> The disk could be a partition of the flash card (it's written quite
rarely).
> 
> ...
>> After some long timeout, it looks like the watchdog timer expires and
server
> 
>> boots, but the failure indication remains on the front panel of the
server.
> 
> 
> Dell PowerEdge? ;-)
> 
> In SLES I have these (among others) settings:
> SBD_WATCHDOG_DEV=/dev/watchdog
> SBD_WATCHDOG_TIMEOUT=30
> SBD_TIMEOUT_ACTION=flush,reboot
> 
> I did:
> h16:~ # echo iTCO_wdt > /etc/modules-load.d/watchdog.conf
> h16:~ # systemctl restart systemd-modules-load
> h16:~ # lsmod | egrep "(wd|dog)"
> iTCO_wdt   16384  0
> iTCO_vendor_support16384  1 iTCO_wdt
> 
> Later I changed it to:
> h16:~ # echo ipmi_watchdog > /etc/modules-load.d/watchdog.conf
> h16:~ # systemctl restart systemd-modules-load
> 
> After reboot there was a conflict:
> Dec 04 12:07:22 h16 kernel: watchdog: wdat_wdt: cannot register miscdev on
> minor=130 (err=-16).
> Dec 04 12:07:22 h16 kernel: watchdog: wdat_wdt: a legacy watchdog module is
> probably present.
> h16:~ # lsmod | grep wd
> wdat_wdt   20480  0
> h16:~ # modprobe -r wdat_wdt
> h16:~ # modprobe ipmi_watchdog
> h16:~ # lsmod | grep wat
> ipmi_watchdog  32768  1
> ipmi_msghandler   114688  4
ipmi_devintf,ipmi_si,ipmi_watchdog,ipmi_ssif
> 
> h16:/etc/modprobe.d # cat 99-local.conf
> #
> # please add local extensions to this file
> #
> h16:/etc/modprobe.d # echo 'blacklist wdat_wdt' >> 99-local.conf
> 
> Maybe also check whether „echo V >/dev/watchdog“ will stop the watchdig
> properly. SUSE (and upstream meanwhile Iguess) had to fix it.
> 
> Hope this helps a bit.
> 
> Regards,
> Ulrich
> 
>> If I uninstall the 'sbd' package, the "sudo reboot" works normally again.
>> 
>> My question is: How do I configure the system, to have the 'sbd' function 
>> present, but still be able to reboot the system normally.
>> 
>> regards,
>> Zoran
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] Re: normal reboot with active sbd does not work

[Ulrich Windl]

>>> Klaus Wenninger  schrieb am 03.06.2022 um 11:03 in
Nachricht
:
> On Fri, Jun 3, 2022 at 10:19 AM Zoran Bošnjak  wrote:
...
> still opened by sbd. In general I don't see why the watchdog-module should
> be unloaded upon shutdown. So as a first try you just might remove that 

Spcifically if the actual watchdog is a hardware timer that isn't stopped when
the module is unloaded.

> part.
> 
> Klaus
> 
>>
>> regards,
>> Zoran
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
> 
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] normal reboot with active sbd does not work

[Ulrich Windl]

>>> Zoran Bošnjak  schrieb am 03.06.2022 um 10:18 in
Nachricht <2046503996.272.1654244336372.javamail.zim...@via.si>:
> Hi all,
> I would appreciate an advice about sbd fencing (without shared storage).

Not an answer, but curiosity:
As sbd needs very little space (like just 1MB), did anybody ever try to use a
small computer like a raspberry pi to privide shared storage for SBD via iSCSI
for example?
The disk could be a partition of the flash card (it's written quite rarely).

...
> After some long timeout, it looks like the watchdog timer expires and server

> boots, but the failure indication remains on the front panel of the server.


Dell PowerEdge? ;-)

In SLES I have these (among others) settings:
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=30
SBD_TIMEOUT_ACTION=flush,reboot

I did:
h16:~ # echo iTCO_wdt > /etc/modules-load.d/watchdog.conf
h16:~ # systemctl restart systemd-modules-load
h16:~ # lsmod | egrep "(wd|dog)"
iTCO_wdt   16384  0
iTCO_vendor_support16384  1 iTCO_wdt

Later I changed it to:
h16:~ # echo ipmi_watchdog > /etc/modules-load.d/watchdog.conf
h16:~ # systemctl restart systemd-modules-load

After reboot there was a conflict:
Dec 04 12:07:22 h16 kernel: watchdog: wdat_wdt: cannot register miscdev on
minor=130 (err=-16).
Dec 04 12:07:22 h16 kernel: watchdog: wdat_wdt: a legacy watchdog module is
probably present.
h16:~ # lsmod | grep wd
wdat_wdt   20480  0
h16:~ # modprobe -r wdat_wdt
h16:~ # modprobe ipmi_watchdog
h16:~ # lsmod | grep wat
ipmi_watchdog  32768  1
ipmi_msghandler   114688  4 ipmi_devintf,ipmi_si,ipmi_watchdog,ipmi_ssif

h16:/etc/modprobe.d # cat 99-local.conf
#
# please add local extensions to this file
#
h16:/etc/modprobe.d # echo 'blacklist wdat_wdt' >> 99-local.conf

Maybe also check whether „echo V >/dev/watchdog“ will stop the watchdig
properly. SUSE (and upstream meanwhile Iguess) had to fix it.

Hope this helps a bit.

Regards,
Ulrich

> If I uninstall the 'sbd' package, the "sudo reboot" works normally again.
> 
> My question is: How do I configure the system, to have the 'sbd' function 
> present, but still be able to reboot the system normally.
> 
> regards,
> Zoran
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Q: "Multiple attributes match name=target-role"

[Ulrich Windl]

Hi!

We had some issue with a resource in SLES15 SP3, so I cleaned the error:
h16:~ # crm_resource -C -r prm_xen_v07 -N h18 -n start
Cleaned up prm_xen_v07 on h18
Multiple attributes match name=target-role
  Value: Started(id=prm_xen_v07-meta_attributes-target-role)
  Value: Started(id=prm_xen_v07-meta_attributes-0-target-role)
  Value: Started(id=prm_xen_v07-meta_attributes-1-target-role)

There have always been multiple target roles in the past, but I never saw this 
complaint, making me woner whether a recent update introduced that feature.

crmsh-4.3.1+20220505.cf4ab649-150200.5.80.1.noarch
pacemaker-2.0.5+20201202.ba59be712-150300.4.21.1.x86_64

primitive prm_xen_v07 VirtualDomain \
params config="/etc/xen/vm/v07.xml" hypervisor="xen:///system" 
remoteuri="xen+tls://%n.uni-regensburg.de" autoset_utilization_cpu=false 
autoset_utilization_hv_memory=false \
op start timeout=120 interval=0 \
op stop timeout=240 interval=0 \
op monitor interval=600 timeout=90 \
op migrate_to timeout=120 interval=0 \
op migrate_from timeout=120 interval=0 \
utilization utl_cpu=20 utl_ram=2048 \
meta priority=8900 allow-migrate=true target-role=Started \
meta 1: resource-stickiness=5 target-role=Started \
meta 2: rule 0: date spec hours=7-19 weekdays=1-5 
resource-stickiness=1000 target-role=Started

AFAIR a crm resource start/stop set the target-role in every rule in the past.

Finally: The resource is running fine on h16, but pacemaker keeps complaining 
it isn't started.
Jun 02 08:03:21 h16 pacemaker-schedulerd[7494]:  warning: Unexpected result 
(error: Failed to start virtual domain v07.) was recorded for start of 
prm_xen_v07 on h19 at Jun  1 09:19:46 2022
Jun 02 08:03:21 h16 pacemaker-schedulerd[7494]:  warning: Forcing prm_xen_v07 
away from h19 after 100 failures (max=100)

Also it does not try to stop it there. Could it be that this is another effect 
of the RAM corruption introduced with SLES15 SP3? h16 is DC...

Regards,
Ulrich



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] What's the number in "Servant pcmk is outdated (age: 682915)"

[Ulrich Windl]

Hi!

I'm wondering what the number in parentheses is for these messages:
sbd[6809]:  warning: inquisitor_child: pcmk health check: UNHEALTHY
sbd[6809]:  warning: inquisitor_child: Servant pcmk is outdated (age: 682915)

Regards,
Ulrich


___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: [EXT] No node name in corosync‑cmapctl output

[Ulrich Windl]

>>> Andreas Hasenack  schrieb am 31.05.2022 um 15:16 in
Nachricht
:
> Hi,
> 
> corosync 3.1.6
> pacemaker 2.1.2
> crmsh 4.3.1
> 
> TL;DR
> I only seem to get a "name" attribute in the "corosync‑cmapctl | grep
> nodelist" output if I set an explicit name in corosync.conf's
> nodelist. If I rely on the default of "name will be uname ‑n if it's
> not set", I get nothing.
> 
> 
> I formed a test cluster of 3 nodes, and I'm not setting the name
> attribute in the nodelist, so that it defaults to `uname ‑n`:
> nodelist {
> node {
>  nodeid: 1
> ring0_addr: k1
> }
> node {
> nodeid: 2
> ring0_addr: k2
> }
> node {
> nodeid: 3
> ring0_addr: k3
> }
> }
> 
> The addresses "k1", "k2" and "k3" are fully resolvable (I know IPs are
> better, but for this quick test it was simpler to use the hostnames).
> 
> crm status is happy:
> root@k1:~# crm status
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: k3 (version 2.1.2‑ada5c3b36e2) ‑ partition with quorum
>   * Last updated: Tue May 31 12:53:02 2022
>   * Last change:  Tue May 31 12:51:55 2022 by hacluster via crmd on k3
>   * 3 nodes configured
>   * 0 resource instances configured
> 
> Node List:
>   * Online: [ k1 k2 k3 ]
> 
> Full List of Resources:
>   * No resources
> 
> 
> But there is no node name in the corosync‑cmapctl output:
> 
> root@k1:~# corosync‑cmapctl |grep nodelist
> nodelist.local_node_pos (u32) = 0
> nodelist.node.0.nodeid (u32) = 1
> nodelist.node.0.ring0_addr (str) = k1
> nodelist.node.1.nodeid (u32) = 2
> nodelist.node.1.ring0_addr (str) = k2
> nodelist.node.2.nodeid (u32) = 3
> nodelist.node.2.ring0_addr (str) = k3
> 
> I was expecting to have entries like "nodelist.node.0.name = k1" in
> that output. Apparently I only get that if I explicitly set a node
> name in nodelist.
> 
> For example, if I set the name of nodeid 1 to "explicit1":
> node {
> name: explicit1
> nodeid: 1
> ring0_addr: k1
> }

We use a much d9ifferent version, but we use an address for the adress ;-):

...
ring0_addr: 172.20.16.16
ring1_addr: 10.2.2.16
...

The man page here also says:
   ringX_addr
  This specifies IP address of one of the nodes. X is ring
number.

Regards,
Ulrich

> 
> Then I get the name attribute for that nodeid only:
> # corosync‑cmapctl |grep nodelist
> nodelist.local_node_pos (u32) = 0
> nodelist.node.0.name (str) = explicit1
> nodelist.node.0.nodeid (u32) = 1
> nodelist.node.0.ring0_addr (str) = k1
> nodelist.node.1.nodeid (u32) = 2
> nodelist.node.1.ring0_addr (str) = k2
> nodelist.node.2.nodeid (u32) = 3
> nodelist.node.2.ring0_addr (str) = k3
> 
> Why not also use "uname ‑n" when "name" is not explicitly set in the
> corosync nodelist config?
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] More pacemaker oddities while stopping DC

[Ulrich Windl]

Hi!

We are still suffering from kernel RAM corruption on the Xen hypervisor when a 
VM or the hypervisor is doing I/O (three months since the bug report at SUSE, 
but no fix or workaround meaning the whole Xen cluster project was canceled 
after 20 years, but that's a different topic). All VMs will be migrated to 
VMware, dumping the whole SLES15 Xen cluster very soon.

My script that detected RAM corruption tried to shutdown pacemaker, hoping for 
the best (i.e. VMs to be live-migrated away). However there are very strange 
decisions made (pacemaker-2.0.5+20201202.ba59be712-150300.4.21.1.x86_64):

May 24 17:05:07 h16 VirtualDomain(prm_xen_test-jeos7)[24460]: INFO: test-jeos7: 
live migration to h19 succeeded.
May 24 17:05:07 h16 VirtualDomain(prm_xen_test-jeos9)[24463]: INFO: test-jeos9: 
live migration to h19 succeeded.
May 24 17:05:07 h16 pacemaker-execd[7504]:  notice: prm_xen_test-jeos7 
migrate_to (call 321, PID 24281) exited with status 0 (execution time 5500ms, 
queue time 0ms)
May 24 17:05:07 h16 pacemaker-controld[7509]:  notice: Result of migrate_to 
operation for prm_xen_test-jeos7 on h16: ok
May 24 17:05:07 h16 pacemaker-execd[7504]:  notice: prm_xen_test-jeos9 
migrate_to (call 323, PID 24283) exited with status 0 (execution time 5514ms, 
queue time 0ms)
May 24 17:05:07 h16 pacemaker-controld[7509]:  notice: Result of migrate_to 
operation for prm_xen_test-jeos9 on h16: ok

Would you agree that the migration was successful? I'd say YES!

However this is what happened:

May 24 17:05:19 h16 pacemaker-controld[7509]:  notice: Transition 2460 
(Complete=16, Pending=0, Fired=0, Skipped=7, Incomplete=57, 
Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Stopped
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Unexpected result 
(error) was recorded for stop of prm_ping_gw1:1 on h16 at May 24 17:05:02 2022
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Unexpected result 
(error) was recorded for stop of prm_ping_gw1:1 on h16 at May 24 17:05:02 2022
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Cluster node h16 will 
be fenced: prm_ping_gw1:1 failed there
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Unexpected result 
(error) was recorded for stop of prm_iotw-md10:1 on h16 at May 24 17:05:02 2022
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Unexpected result 
(error) was recorded for stop of prm_iotw-md10:1 on h16 at May 24 17:05:02 2022
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_ping_gw1 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_ping_gw1 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_ping_gw1 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_iotw-md10 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_iotw-md10 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Forcing cln_iotw-md10 
away from h16 after 100 failures (max=100)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  notice: Resource 
prm_xen_test-jeos7 can no longer migrate from h16 to h19 (will stop on both 
nodes)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  notice: Resource 
prm_xen_test-jeos9 can no longer migrate from h16 to h19 (will stop on both 
nodes)
May 24 17:05:19 h16 pacemaker-schedulerd[7508]:  warning: Scheduling Node h16 
for STONITH

So the DC considers the migration to have failed, even though it was reported 
as success!
(The ping had dumped core due to RAM corruption before)

May 24 17:03:12 h16 kernel: ping[23973]: segfault at 213e6 ip 000213e6 
sp 7ffc249fab78 error 14 in bash[5655262bc000+f1000]

So it stopped the VMs that were migrated successfully before:
May 24 17:05:19 h16 pacemaker-controld[7509]:  notice: Initiating stop 
operation prm_xen_test-jeos7_stop_0 on h19
May 24 17:05:19 h16 pacemaker-controld[7509]:  notice: Initiating stop 
operation prm_xen_test-jeos9_stop_0 on h19
May 24 17:05:19 h16 pacemaker-controld[7509]:  notice: Requesting fencing 
(reboot) of node h16

Those test VMs were not important, but the important part was that due to the 
failure to stop the ping resource, it did not even try to migrate the other VMs 
(non-test) away, so those were hard-fenced.

For completeness I should add that the RAM corruption also affected pacemaker 
itself:

May 24 17:05:02 h16 kernel: traps: pacemaker-execd[24272] general protection 
fault ip:7fc572327bcf sp:7ffca7cd22d0 error:0 in 
libc-2.31.so[7fc572246000+1e6000]
May 24 17:05:02 h16 kernel: pacemaker-execd[24277]: segfault at 0 ip 
 sp 7ffca7cd22f0 error 14 in 
pacemaker-execd[56347df4e000+b000]
May 24 17:05:02 h16 kernel: Code: Bad RIP value.

That affected the stop of some 

[ClusterLabs] Antw: [EXT] What/how to clean up when bootstrapping new cluster (or: I have a phantom node)

[Ulrich Windl]

>>> Andreas Hasenack  schrieb am 24.05.2022 um 22:05 in
Nachricht
:
> Hi,
> 
> I'm trying to find out the correct steps to start a corosync/pacemaker
> cluster right after installing its packages in Debian or Ubuntu.
> 
> I'm not using crmsh or pcs on purpose, I really wanted to get this
> basic initial step working without those.
> 
> Right after install, the default config has this nodelist:
> nodelist {
> # Change/uncomment/add node sections to match cluster configuration
> 
> node {
> # Hostname of the node
> name: node1
> # Cluster membership node identifier
> nodeid: 1
> # Address of first link
> ring0_addr: 127.0.0.1
> # When knet transport is used it's possible to define
> up to 8 links
> #ring1_addr: 192.168.1.1
> }
> # ...
> }
> 
> 
> (full default pristine config: https://pastebin.ubuntu.com/p/htBkCvBWqr/)
> 
> This results in a crm_mon output of:
> 
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: node1 (version 2.0.3‑4b1f869f0f) ‑ partition with quorum
>   * Last updated: Tue May 24 19:57:05 2022
>   * Last change:  Tue May 24 19:56:59 2022 by hacluster via crmd on node1
>   * 1 node configured
>   * 0 resource instances configured
> 
> Node List:
>   * Online: [ node1 ]
> 
> Active Resources:
>   * No active resources
> 
> I also tried with corosync 3.1.6 and pacemaker 2.1.2, btw.
> 
> I then proceed to making changes to corosync.conf. I give it a real
> hostname, ring IP and node id:
> nodelist {
> # Change/uncomment/add node sections to match cluster configuration
> 
> node {
> # Hostname of the node
> name: f4
> # Cluster membership node identifier
> nodeid: 104
> # Address of first link
> ring0_addr: 10.226.63.102
> # When knet transport is used it's possible to define
> up to 8 links
> #ring1_addr: 192.168.1.1
> }
> # ...
> }
> 
> 
> Restart the services:
> systemctl restart pacemaker corosync
> 
> But now I have this phantom "node1" in the cluster, and the cluster
> thinks it has two nodes:

I guess when starting pacemaker it creates a CIB (containing the nodes).
Changing corosync.conf does not affect the CIB, so you'll have to clean it up
(as Ken noted).
The point is: Why did you start with some fake node?

Regards,
Ulrich

> 
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: f4 (version 2.0.3‑4b1f869f0f) ‑ partition with quorum
>   * Last updated: Tue May 24 19:59:56 2022
>   * Last change:  Tue May 24 19:59:22 2022 by hacluster via crmd on f4
>   * 2 nodes configured
>   * 0 resource instances configured
> 
> Node List:
>   * Node node1: UNCLEAN (offline)
>   * Online: [ f4 ]
> 
> Active Resources:
>   * No active resources
> 
> 
> What is the cleanup step (or steps) that I'm missing? Or are there so
> many details that it's best to leave this to pcs/crmsh?
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 



___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/

[ClusterLabs] Antw: Re: Antw: [EXT] Re: Cluster unable to find back together

[Ulrich Windl]

>>> Klaus Wenninger  schrieb am 23.05.2022 um 19:43 in
Nachricht
:
> On Fri, May 20, 2022 at 7:43 AM Ulrich Windl
>  wrote:
>>
>> >>> Jan Friesse  schrieb am 19.05.2022 um 14:55 in
>> Nachricht
>> <1abb8468-6619-329f-cb01-3f51112db...@redhat.com>:
>> > Hi,
>> >
>> > On 19/05/2022 10:16, Leditzky, Fabian via Users wrote:
>> >> Hello
>> >>
>> >> We have been dealing with our pacemaker/corosync clusters becoming
>> unstable.
>> >> The OS is Debian 10 and we use Debian packages for pacemaker and
corosync,
>> >> version 3.0.1‑5+deb10u1 and 3.0.1‑2+deb10u1 respectively.
>> >
>> > Seems like pcmk version is not so important for behavior you've
>> > described. Corosync 3.0.1 is super old, are you able to reproduce the
>>
>> I'm running corosync-2.4.5-12.7.1.x86_64 (SLES15 SP3) here ;-)
>>
>> Are you mixing "super old" with "super buggy"?
> 
> Actually 3.0.1 is older than 2.4.5 and on top 2.4.5 is the head of a mature
> branch while 3.0.1 is the beginning of a new branch that brought
> substantial changes.

Klaus,

thanks for explaining. Wasn't aware of that.

Regards,
Ulrich


> 
> Klaus
>>
>> Regards,
>> Ulrich
>>
>> > behavior with 3.1.6? What is the version of knet? There were quite a few
>> > fixes so last one (1.23) is really recommended.
>> >
>> > You can try to compile yourself, or use proxmox repo
>> > (http://download.proxmox.com/debian/pve/) which contains newer version
>> > of packages.
>> >
>> >> We use knet over UDP transport.
>> >>
>> >> We run multiple 2‑node and 4‑8 node clusters, primarily managing VIP
>> > resources.
>> >> The issue we experience presents itself as a spontaneous disagreement
of
>> >> the status of cluster members. In two node clusters, each node
>> spontaneously
>> >> sees the other node as offline, despite network connectivity being OK.
>> >> In larger clusters, the status can be inconsistent across the nodes.
>> >> E.g.: node1 sees 2,4 as offline, node 2 sees 1,4 as offline while node
3
>> and
>> > 4 see every node as online.
>> >
>> > This really shouldn't happen.
>> >
>> >> The cluster becomes generally unresponsive to resource actions in this
>> > state.
>> >
>> > Expected
>> >
>> >> Thus far we have been unable to restore cluster health without
restarting
>> > corosync.
>> >>
>> >> We are running packet captures 24/7 on the clusters and have custom
>> tooling
>> >> to detect lost UDP packets on knet ports. So far we could not see
>> > significant
>> >> packet loss trigger an event, at most we have seen a single UDP packet
>> > dropped
>> >> some seconds before the cluster fails.
>> >>
>> >> However, even if the root cause is indeed a flaky network, we do not
>> > understand
>> >> why the cluster cannot recover on its own in any way. The issues
definitely
>>
>> > persist
>> >> beyond the presence of any intermittent network problem.
>> >
>> > Try newer version. If problem persist, it's good idea to monitor if
>> > packets are really passed thru. Corosync always (at least) creates
>> > single node membership.
>> >
>> > Regards,
>> >Honza
>> >
>> >>
>> >> We were able to artificially break clusters by inducing packet loss with
an
>>
>> > iptables rule.
>> >> Dropping packets on a single node of an 8‑node cluster can cause
>> malfunctions
>> > on
>> >> multiple other cluster nodes. The expected behavior would be detecting
that
>>
>> > the
>> >> artificially broken node failed but keeping the rest of the cluster
>> stable.
>> >> We were able to reproduce this also on Debian 11 with more recent
>> > corosync/pacemaker
>> >> versions.
>> >>
>> >> Our configuration basic, we do not significantly deviate from the
>> defaults.
>> >>
>> >> We will be very grateful for any insights into this problem.
>> >>
>> >> Thanks,
>> >> Fabian
>> >>
>> >> // corosync.conf
>> >> totem {
>> >>  version: 2
>> >>  cluster_name: cluster01
>> >>  crypto_cipher: aes256
>> >>  crypto_hash: sha512
>> &g