Re: [ClusterLabs] 'pcs stonith update' takes, then reverts
After a LOT of hassle, I finally got it updated, but OMG it was painful. I degraded the cluster (unsure if needed), set maintenance mode, deleted the stonith levels, deleted the stonith devices, recreated them with the updated values, recreated the stonith levels, and finally disabled maintenance mode. It should not have been this hard, right? Why is heck would it be that pacemaker kept "rolling back" to old configs? I'd delete the stonith levels, delete one PDU stonith resource, delete a second, and suddenly all the levels were back and the resources I had just removed came back with the old configs. Certainly I was doing something wrong, but what? digimer On 2021-07-23 8:04 p.m., Digimer wrote: > Update; > > Appears I can't even delete the damn things. They re-appeared after > doing a [pcs stonith remove '!. > > Wow. > > digimer > > On 2021-07-23 7:56 p.m., Digimer wrote: >> Hi all, >> >> Got a really odd one here... >> >> I had a cluster in the lab where it was built and tested. Then we >> deployed it, and I've been trying to update the stonith config. It >> _seems_ to take at first, then it reverts back to the old config. This >> is super annoying, obviously. :) >> >> So I can confirm that the APC PDUs work; >> >> >> [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.3 -n 3 -o status >> Status: ON >> [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.4 -n 3 -o status >> Status: ON >> >> >> Here's the config as-shipped; >> >> >> # pcs stonith config apc_snmp_node1_an-pdu03 >> Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) >> Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 >> pcmk_off_action=reboot port=5 >> Operations: monitor interval=60 >> (apc_snmp_node1_an-pdu03-monitor-interval-60) >> Target: an-a02n01 >> Level 1 - ipmilan_node1 >> Level 2 - apc_snmp_node1_an-pdu03,apc_snmp_node1_an-pdu04 >> Level 3 - delay_node1 >> Target: an-a02n02 >> Level 1 - ipmilan_node2 >> Level 2 - apc_snmp_node2_an-pdu03,apc_snmp_node2_an-pdu04 >> Level 3 - delay_node2 >> >> >> So in this example, I am trying to update 'apc_snmp_node1_an-pdu03' to >> change the IP from 10.201.2.1 -> 10.201.2.3 and to change the port from >> port 5 -> 3. >> >> >> # pcs stonith update apc_snmp_node1_an-pdu03 ip=10.201.2.3 >> pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 >> # pcs stonith config apc_snmp_node1_an-pdu03 >> Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) >> Attributes: ip=10.201.2.3 pcmk_host_list=an-a02n01 >> pcmk_off_action=reboot port=3 >> >> >> As you can see, initially it appears to work. However, after a minute, >> the config reverts; >> >> >> # pcs stonith config apc_snmp_node1_an-pdu03 >> Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) >> Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 >> pcmk_off_action=reboot port=5 >> >> >> This happens to all four stonith devices (two per node, two nodes). I've >> tried doing a 'pcs stonith disable ' for all four devices, and did a >> 'pcs stonith cleanup' to make sure errors were cleared before updating, >> still no luck. >> >> Any idea what I'm doing wrong? >> >> The logs from the node I run the update on, followed by the logs on the >> peer: >> >> digimer >> >> >> Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: >> Updating all attributes after cib_refresh_notify event >> Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: >> notice: State transition S_IDLE -> S_POLICY_ENGINE >> Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: >> notice: State transition S_ELECTION -> S_INTEGRATION >> Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: >> Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because >> resource parameters have changed >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> warning: Unexpected result (error) was recorded for start of >> apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> warning: Unexpected result (error) was recorded for start of >> apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> warning: Unexpected result (error) was recorded for start of >> apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> warning: Unexpected result (error) was recorded for start of >> apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 >> Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: >> notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because >> resource parameters have
Re: [ClusterLabs] 'pcs stonith update' takes, then reverts
Update; Appears I can't even delete the damn things. They re-appeared after doing a [pcs stonith remove '!. Wow. digimer On 2021-07-23 7:56 p.m., Digimer wrote: > Hi all, > > Got a really odd one here... > > I had a cluster in the lab where it was built and tested. Then we > deployed it, and I've been trying to update the stonith config. It > _seems_ to take at first, then it reverts back to the old config. This > is super annoying, obviously. :) > > So I can confirm that the APC PDUs work; > > > [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.3 -n 3 -o status > Status: ON > [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.4 -n 3 -o status > Status: ON > > > Here's the config as-shipped; > > > # pcs stonith config apc_snmp_node1_an-pdu03 > Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) > Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 > pcmk_off_action=reboot port=5 > Operations: monitor interval=60 > (apc_snmp_node1_an-pdu03-monitor-interval-60) > Target: an-a02n01 > Level 1 - ipmilan_node1 > Level 2 - apc_snmp_node1_an-pdu03,apc_snmp_node1_an-pdu04 > Level 3 - delay_node1 > Target: an-a02n02 > Level 1 - ipmilan_node2 > Level 2 - apc_snmp_node2_an-pdu03,apc_snmp_node2_an-pdu04 > Level 3 - delay_node2 > > > So in this example, I am trying to update 'apc_snmp_node1_an-pdu03' to > change the IP from 10.201.2.1 -> 10.201.2.3 and to change the port from > port 5 -> 3. > > > # pcs stonith update apc_snmp_node1_an-pdu03 ip=10.201.2.3 > pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 > # pcs stonith config apc_snmp_node1_an-pdu03 > Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) > Attributes: ip=10.201.2.3 pcmk_host_list=an-a02n01 > pcmk_off_action=reboot port=3 > > > As you can see, initially it appears to work. However, after a minute, > the config reverts; > > > # pcs stonith config apc_snmp_node1_an-pdu03 > Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) > Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 > pcmk_off_action=reboot port=5 > > > This happens to all four stonith devices (two per node, two nodes). I've > tried doing a 'pcs stonith disable ' for all four devices, and did a > 'pcs stonith cleanup' to make sure errors were cleared before updating, > still no luck. > > Any idea what I'm doing wrong? > > The logs from the node I run the update on, followed by the logs on the > peer: > > digimer > > > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: > Updating all attributes after cib_refresh_notify event > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: > notice: State transition S_IDLE -> S_POLICY_ENGINE > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: > notice: State transition S_ELECTION -> S_INTEGRATION > Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: > Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because > resource parameters have changed > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because > resource parameters have changed > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Unexpected result (error) was recorded for start of > apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 > Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: > warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after > 100 failures (max=100) > Jul 23 16:44:47 an-a02n01.alteeve.com
[ClusterLabs] 'pcs stonith update' takes, then reverts
Hi all, Got a really odd one here... I had a cluster in the lab where it was built and tested. Then we deployed it, and I've been trying to update the stonith config. It _seems_ to take at first, then it reverts back to the old config. This is super annoying, obviously. :) So I can confirm that the APC PDUs work; [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.3 -n 3 -o status Status: ON [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.4 -n 3 -o status Status: ON Here's the config as-shipped; # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=5 Operations: monitor interval=60 (apc_snmp_node1_an-pdu03-monitor-interval-60) Target: an-a02n01 Level 1 - ipmilan_node1 Level 2 - apc_snmp_node1_an-pdu03,apc_snmp_node1_an-pdu04 Level 3 - delay_node1 Target: an-a02n02 Level 1 - ipmilan_node2 Level 2 - apc_snmp_node2_an-pdu03,apc_snmp_node2_an-pdu04 Level 3 - delay_node2 So in this example, I am trying to update 'apc_snmp_node1_an-pdu03' to change the IP from 10.201.2.1 -> 10.201.2.3 and to change the port from port 5 -> 3. # pcs stonith update apc_snmp_node1_an-pdu03 ip=10.201.2.3 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.3 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 As you can see, initially it appears to work. However, after a minute, the config reverts; # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=5 This happens to all four stonith devices (two per node, two nodes). I've tried doing a 'pcs stonith disable ' for all four devices, and did a 'pcs stonith cleanup' to make sure errors were cleared before updating, still no luck. Any idea what I'm doing wrong? The logs from the node I run the update on, followed by the logs on the peer: digimer Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Updating all attributes after cib_refresh_notify event Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_ELECTION -> S_INTEGRATION Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 100 failures (max=100) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 100 failures (max=100) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 100 failures (max=100) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing
Re: [ClusterLabs] How to create the stonith resource in virtualbox
Hi fellow,I saw your post on ClusterLabs and I have a question for you to solve my problem. Pcs Stonith is not started on the node and always Stopped. I did check the corosync.log and it said error is No route to host. Although I have been checking my configuration of IP, firewall, hosts file, I cannot start it. Could you help me out of this problem friend?Please reply me soon.Best regards,Anh Sent from Mail for Windows 10 ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] unexpected fenced node and promotion of the new master PAF ‑ postgres
On Fri, 23 Jul 2021 12:52:00 +0200 damiano giuliani wrote: > the time query isnt the problem, is known that took its time. the network > is 10gbs bonding, quite impossible to sature with queries :=). Everything is possible, it's just harder :) [...] > checking again the logs what for me is not clear its the cause of the loss > of quorum and then fence the node. As said before, according to logs from other nodes, ltaoperdbs02 did not answers to the TOTEM protocol anymore, so it left the communication group. But worse, it did it without saying goodbye properly: > [TOTEM ] Failed to receive the leave message. failed: 1 >From this exact time, the node is then considered "uncleaned", aka its state "unknown". To solve this trouble, the cluster needs to fence it to set a predictable state: OFF. So, the reaction to the trouble is sane. Now, from the starting point of this conversation, the question is what happened? Logs on other nodes will probably not help, as they just witnessed a node disappearing without any explanation. Logs from ltaoperdbs02 might help, but the corosync log you sent stop at 00:38:44, almost 2 minutes before the fencing as reported from other nodes: > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: warning: pe_fence_node: >Cluster node ltaoperdbs02 will be fenced: peer is no longer part of > So the cluster works flawessy as expected: as soon ltaoperdbs02 become > "unreachable", it formed a new quorum, fenced the lost node and promoted > the new master. exact. > What i cant findout is WHY its happened. > there are no useful information into the system logs neither into the > Idrac motherboard logs. Because I suppose some log where not synced to disks when the server has been fenced. Either the server clocks were not synched (I doubt), or you really lost almost 2 minutes of logs. > There is a way to improve or configure a log system for fenced / failed > node? Yes: 1.setup rsyslog to export logs on some dedicated logging servers. Such servers should receive and save logs from your clusters and other hardwares (network?) and keep them safe. You will not loose messages anymore. 2. Gather a lot of system metrics and keep them safe (eg. export them using pcp, collectd, etc). Metrics and visualization are important to cross-compare with logs and pinpoint something behaving outside of the usual scope. Looking at your log, I still find your query time are suspicious. I'm not convinced they are the root cause, they might be just a bad symptom/signal of something going wrong there. Having a one-row INSERT taking 649.754ms is suspicious. Maybe it's just a locking problem, maybe there's some CPU-bound postgis things involved, maybe with some GIN or GiST indexes, but it's still suspicious considering the server is over-sized in performance as you stated... And maybe the network or SAN had a hick-up and corosync has been too sensible to it. Check the retransmit and timeout parameters? Regards, ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Antw: Re: Antw: [EXT] unexpected fenced node and promotion of the new master PAF ‑ postgres
>>> damiano giuliani schrieb am 23.07.2021 um >>> 12:52 in Nachricht : > hi guys thanks for supporting. > the time query isnt the problem, is known that took its time. the network > is 10gbs bonding, quite impossible to sature with queries :=). > the servers are totally overkilled, at database full working loads 20% of > the resources have been used. > checking again the logs what for me is not clear its the cause of the loss > of quorum and then fence the node. > there are no informations into the logs (even into Idrac/ motherboard event > logs). > > the only clear logs are : > 228684] ltaoperdbs03 corosyncnotice [TOTEM ] A processor failed, forming > new configuration. Hi! I wonder: Would the corosync Blackbox (COROSYNC-BLACKBOX(8)) help? As an alternative you could capture TOTEM packages in some rotating files, trying to find out what was going on. As it seems now, the issue is that a remote node cannot "be seen". Regards, Ulrich > [228684] ltaoperdbs03 corosyncnotice [TOTEM ] A new membership ( > 172.18.2.12:227) was formed. Members left: 1 > [228684] ltaoperdbs03 corosyncnotice [TOTEM ] Failed to receive the leave > message. failed: 1 > [228684] ltaoperdbs03 corosyncwarning [CPG ] downlist left_list: 1 > received > [228684] ltaoperdbs03 corosyncwarning [CPG ] downlist left_list: 1 > received > Jul 13 00:40:37 [228695] ltaoperdbs03cib: info: > pcmk_cpg_membership:Group cib event 3: ltaoperdbs02 (node 1 pid > 6136) left via cluster exit > Jul 13 00:40:37 [228695] ltaoperdbs03cib: info: > crm_update_peer_proc: pcmk_cpg_membership: Node ltaoperdbs02[1] - > corosync-cpg is now offline > Jul 13 00:40:37 [228700] ltaoperdbs03 crmd: info: > pcmk_cpg_membership:Group crmd event 3: ltaoperdbs02 (node 1 pid > 6141) left via cluster exit > Jul 13 00:40:37 [228695] ltaoperdbs03cib: notice: > crm_update_peer_state_iter: Node ltaoperdbs02 state is now lost | nodeid=1 > previous=member source=crm_update_peer_proc > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: warning: pe_fence_node: > Cluster node ltaoperdbs02 will be fenced: peer is no longer part of > the cluster > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: warning: > determine_online_status:Node ltaoperdbs02 is unclean > > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: notice: LogNodeActions: > * Fence (reboot) ltaoperdbs02 'peer is no longer part of the cluster' > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: notice: LogAction: * > Promotepgsqld:0 ( Slave -> Master ltaoperdbs03 ) > Jul 13 00:40:37 [228699] ltaoperdbs03pengine: info: LogActions: > Leave pgsqld:1(Slave ltaoperdbs04) > > > So the cluster works flawessy as expected: as soon ltaoperdbs02 become > "unreachable", it formed a new quorum, fenced the lost node and promoted > the new master. > > What i cant findout is WHY its happened. > there are no useful information into the system logs neither into the > Idrac motherboard logs. > > There is a way to improve or configure a log system for fenced / failed > node? > > Thanks > > Damiano > > Il giorno gio 22 lug 2021 alle ore 15:06 Jehan-Guillaume de Rorthais < > j...@dalibo.com> ha scritto: > >> Hi, >> >> On Wed, 14 Jul 2021 07:58:14 +0200 >> "Ulrich Windl" wrote: >> [...] >> > Could it be that a command saturated the network? >> > Jul 13 00:39:28 ltaoperdbs02 postgres[172262]: [20-1] 2021-07-13 >> 00:39:28.936 >> > UTC [172262] LOG: duration: 660.329 ms execute : SELECT >> > xmf.file_id, f.size, fp.full_path FROM ism_x_medium_file xmf JOIN#011 >> > ism_files f ON f.id_file = xmf.file_id JOIN#011 ism_files_path fp ON >> > f.id_file = fp.file_id JOIN ism_online o ON o.file_id = xmf.file_id >> WHERE >> > xmf.medium_id = 363 AND xmf.x_media_file_status_id = 1 AND >> > o.online_status_id = 3GROUP BY xmf.file_id, f.size, fp.full_path >> LIMIT >> > 7265 ; >> >> I doubt such a query could saturate the network. The query time itself >> isn't >> proportional to the result set size. >> >> Moreover, there's only three fields per row and according to their name, I >> doubt the row size is really big. >> >> Plus, imagine the result set is that big, chances are that the frontend >> will >> not be able to cope with it as fast as the network, unless the frontend is >> doing >> nothing really fancy with the dataset. So the frontend itself might >> saturate >> before the network, giving some break to the later. >> >> However, if this query time is unusual, that might illustrate some >> pressure on >> the server by some other mean (CPU ? MEM ? IO ?). Detailed metrics would >> help. >> >> Regards, >> ___ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ >> ___ Manage your subscription:
Re: [ClusterLabs] Antw: [EXT] unexpected fenced node and promotion of the new master PAF ‑ postgres
hi guys thanks for supporting. the time query isnt the problem, is known that took its time. the network is 10gbs bonding, quite impossible to sature with queries :=). the servers are totally overkilled, at database full working loads 20% of the resources have been used. checking again the logs what for me is not clear its the cause of the loss of quorum and then fence the node. there are no informations into the logs (even into Idrac/ motherboard event logs). the only clear logs are : 228684] ltaoperdbs03 corosyncnotice [TOTEM ] A processor failed, forming new configuration. [228684] ltaoperdbs03 corosyncnotice [TOTEM ] A new membership ( 172.18.2.12:227) was formed. Members left: 1 [228684] ltaoperdbs03 corosyncnotice [TOTEM ] Failed to receive the leave message. failed: 1 [228684] ltaoperdbs03 corosyncwarning [CPG ] downlist left_list: 1 received [228684] ltaoperdbs03 corosyncwarning [CPG ] downlist left_list: 1 received Jul 13 00:40:37 [228695] ltaoperdbs03cib: info: pcmk_cpg_membership:Group cib event 3: ltaoperdbs02 (node 1 pid 6136) left via cluster exit Jul 13 00:40:37 [228695] ltaoperdbs03cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node ltaoperdbs02[1] - corosync-cpg is now offline Jul 13 00:40:37 [228700] ltaoperdbs03 crmd: info: pcmk_cpg_membership:Group crmd event 3: ltaoperdbs02 (node 1 pid 6141) left via cluster exit Jul 13 00:40:37 [228695] ltaoperdbs03cib: notice: crm_update_peer_state_iter: Node ltaoperdbs02 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc Jul 13 00:40:37 [228699] ltaoperdbs03pengine: warning: pe_fence_node: Cluster node ltaoperdbs02 will be fenced: peer is no longer part of the cluster Jul 13 00:40:37 [228699] ltaoperdbs03pengine: warning: determine_online_status:Node ltaoperdbs02 is unclean Jul 13 00:40:37 [228699] ltaoperdbs03pengine: notice: LogNodeActions: * Fence (reboot) ltaoperdbs02 'peer is no longer part of the cluster' Jul 13 00:40:37 [228699] ltaoperdbs03pengine: notice: LogAction: * Promotepgsqld:0 ( Slave -> Master ltaoperdbs03 ) Jul 13 00:40:37 [228699] ltaoperdbs03pengine: info: LogActions: Leave pgsqld:1(Slave ltaoperdbs04) So the cluster works flawessy as expected: as soon ltaoperdbs02 become "unreachable", it formed a new quorum, fenced the lost node and promoted the new master. What i cant findout is WHY its happened. there are no useful information into the system logs neither into the Idrac motherboard logs. There is a way to improve or configure a log system for fenced / failed node? Thanks Damiano Il giorno gio 22 lug 2021 alle ore 15:06 Jehan-Guillaume de Rorthais < j...@dalibo.com> ha scritto: > Hi, > > On Wed, 14 Jul 2021 07:58:14 +0200 > "Ulrich Windl" wrote: > [...] > > Could it be that a command saturated the network? > > Jul 13 00:39:28 ltaoperdbs02 postgres[172262]: [20-1] 2021-07-13 > 00:39:28.936 > > UTC [172262] LOG: duration: 660.329 ms execute : SELECT > > xmf.file_id, f.size, fp.full_path FROM ism_x_medium_file xmf JOIN#011 > > ism_files f ON f.id_file = xmf.file_id JOIN#011 ism_files_path fp ON > > f.id_file = fp.file_id JOIN ism_online o ON o.file_id = xmf.file_id > WHERE > > xmf.medium_id = 363 AND xmf.x_media_file_status_id = 1 AND > > o.online_status_id = 3GROUP BY xmf.file_id, f.size, fp.full_path > LIMIT > > 7265 ; > > I doubt such a query could saturate the network. The query time itself > isn't > proportional to the result set size. > > Moreover, there's only three fields per row and according to their name, I > doubt the row size is really big. > > Plus, imagine the result set is that big, chances are that the frontend > will > not be able to cope with it as fast as the network, unless the frontend is > doing > nothing really fancy with the dataset. So the frontend itself might > saturate > before the network, giving some break to the later. > > However, if this query time is unusual, that might illustrate some > pressure on > the server by some other mean (CPU ? MEM ? IO ?). Detailed metrics would > help. > > Regards, > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: [EXT] Re: Two node cluster without fencing and no split brain?
On Fri, Jul 23, 2021 at 8:55 AM Ulrich Windl < ulrich.wi...@rz.uni-regensburg.de> wrote: > >>> "john tillman" schrieb am 22.07.2021 um 16:48 in > Nachricht > <1175ffcec0033015e13d11d7821d5acb.squir...@mail.panix.com>: > > There was a lot of discussion on this topic which might have overshadowed > > this question so I will ask it again in case someone missed it. > > > > It comes from a post (see below) that we were pointed to here by Andrei: > > > > Is there something like the described "ping tiebreaker" in the current > > world of pacemaker/corosync? > > Maybe explain how it should work: > If the two nodes cannot rech each other, but each can reach the ping node, > which node has the quorum then? > Guess both - which is what is played down as 'disadvantage' in the description below ;-) > > > > > Best Regards, > > ‑John > > > >> Interesting read. Thank you for providing it! > >> > >> In this follow up post > >> > > > https://techthoughts.typepad.com/managing_computers/2007/10/more > ‑about‑quor.htm > > > l > >> the author mentions the following: > >> > >> Ping tiebreaker > >> > >> Some HA systems provide a ping tiebreaker. To make this work, you > pick a > >> address outside the cluster to ping, and any partition that can ping > that > >> address has quorum. The obvious advantage is that it's very simple to > set > >> up ‑ doesn't require any additional servers or shared disk. The > >> disadvantage (and it's a big one) is that it's very possible for > multiple > >> partitions to think they have quorum. In the case of split‑site > (disaster > >> recovery) type clusters, it's going to happen fairly often. If you can > >> use this method for a single site in conjunction with fencing, then it > >> will likely work out quite well. It's a lot better than no tiebreaker, > or > >> one that always says "you have quorum". Having said that, it's > >> significantly inferior to any of the other methods. > >> > >> The quote "It's a lot better than no tiebreaker..." is what I am looking > >> for. Is there something like a "ping tiebreaker" in the current world > of > >> pacemaker/corosync? > >> > >> Thanks to all those who have already commented on my question. I > >> appreciate the input/education. > >> > >> Best Regards, > >> ‑John > >> > >> > >> > >>> On Wed, Jul 21, 2021 at 3:55 PM Ulrich Windl > >>> wrote: > > Hi! > > Maybe someone feels motivated to write some article comparing the > concepts > * split brain > * quorum > * fencing > > >>> > >>> Yet another one? Using your own reply "search is free". > >>> > >>> > > > https://techthoughts.typepad.com/managing_computers/2007/10/split > ‑brain‑quo.htm > > > l > >>> > There are eight possible states that I tried to illustrate on the > attached sketch (S="Split Brain", "Q=Quorum, F=Fencing). > > ;‑) > > Regards, > Ulrich > > > >>> Andrei Borzenkov 21.07.2021, 07:52 >>> > > On 21.07.2021 07:28, Strahil Nikolov via Users wrote: > > Hi, > > consider using a 3rd system as a Q disk. > > What was not clear in "Quorum is a different concept and doesn't > remove > the need for fencing"? > > > Also, you can use iscsi from that node as a SBD device, so you will > have proper fencing .If you don't have a hardware watchdog device, you > can use softdog kernel module for that. > > Best Regards,Strahil Nikolov > > > > > > On Wed, Jul 21, 2021 at 1:45, Digimer wrote: On > 2021‑07‑20 6:04 p.m., john tillman wrote: > >> Greetings, > >> > >> Is it possible to configure a two node cluster (pacemaker 2.0) > without > >> fencing and avoid split brain? > > > > No. > > > >> I was hoping there was a way to use a 3rd node's ip address, like > from a > >> network switch, as a tie breaker to provide quorum. A simple > successful > >> ping would do it. > > > > Quorum is a different concept and doesn't remove the need for > fencing. > > > >> I realize that this 'ping' approach is not the bullet proof > solution > that > >> fencing would provide. However, it may be an improvement over two > nodes > >> alone. > > > > It would be, at best, a false sense of security. > > > >> Is there a configuration like that already? Any other ideas? > >> > >> Pointers to useful documents/discussions on avoiding split brain > with > two > >> node clusters would be welcome. > > > > https://www.alteeve.com/w/The_2‑Node_Myth > > > > (note: currently throwing a cert error related to the let's encrypt > > issue, should be cleared up soon). > > > > > > ___ > > Manage your subscription: > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > ClusterLabs home:
[ClusterLabs] Antw: Re: [EXT] Re: Two node cluster without fencing and no split brain?
>>> "john tillman" schrieb am 22.07.2021 um 16:48 in Nachricht <1175ffcec0033015e13d11d7821d5acb.squir...@mail.panix.com>: > There was a lot of discussion on this topic which might have overshadowed > this question so I will ask it again in case someone missed it. > > It comes from a post (see below) that we were pointed to here by Andrei: > > Is there something like the described "ping tiebreaker" in the current > world of pacemaker/corosync? Maybe explain how it should work: If the two nodes cannot rech each other, but each can reach the ping node, which node has the quorum then? > > Best Regards, > ‑John > >> Interesting read. Thank you for providing it! >> >> In this follow up post >> > https://techthoughts.typepad.com/managing_computers/2007/10/more‑about‑quor.htm > l >> the author mentions the following: >> >> Ping tiebreaker >> >> Some HA systems provide a ping tiebreaker. To make this work, you pick a >> address outside the cluster to ping, and any partition that can ping that >> address has quorum. The obvious advantage is that it's very simple to set >> up ‑ doesn't require any additional servers or shared disk. The >> disadvantage (and it's a big one) is that it's very possible for multiple >> partitions to think they have quorum. In the case of split‑site (disaster >> recovery) type clusters, it's going to happen fairly often. If you can >> use this method for a single site in conjunction with fencing, then it >> will likely work out quite well. It's a lot better than no tiebreaker, or >> one that always says "you have quorum". Having said that, it's >> significantly inferior to any of the other methods. >> >> The quote "It's a lot better than no tiebreaker..." is what I am looking >> for. Is there something like a "ping tiebreaker" in the current world of >> pacemaker/corosync? >> >> Thanks to all those who have already commented on my question. I >> appreciate the input/education. >> >> Best Regards, >> ‑John >> >> >> >>> On Wed, Jul 21, 2021 at 3:55 PM Ulrich Windl >>> wrote: Hi! Maybe someone feels motivated to write some article comparing the concepts * split brain * quorum * fencing >>> >>> Yet another one? Using your own reply "search is free". >>> >>> > https://techthoughts.typepad.com/managing_computers/2007/10/split‑brain‑quo.htm > l >>> There are eight possible states that I tried to illustrate on the attached sketch (S="Split Brain", "Q=Quorum, F=Fencing). ;‑) Regards, Ulrich >>> Andrei Borzenkov 21.07.2021, 07:52 >>> On 21.07.2021 07:28, Strahil Nikolov via Users wrote: > Hi, > consider using a 3rd system as a Q disk. What was not clear in "Quorum is a different concept and doesn't remove the need for fencing"? > Also, you can use iscsi from that node as a SBD device, so you will have proper fencing .If you don't have a hardware watchdog device, you can use softdog kernel module for that. > Best Regards,Strahil Nikolov > > > On Wed, Jul 21, 2021 at 1:45, Digimer wrote: On 2021‑07‑20 6:04 p.m., john tillman wrote: >> Greetings, >> >> Is it possible to configure a two node cluster (pacemaker 2.0) without >> fencing and avoid split brain? > > No. > >> I was hoping there was a way to use a 3rd node's ip address, like from a >> network switch, as a tie breaker to provide quorum. A simple successful >> ping would do it. > > Quorum is a different concept and doesn't remove the need for fencing. > >> I realize that this 'ping' approach is not the bullet proof solution that >> fencing would provide. However, it may be an improvement over two nodes >> alone. > > It would be, at best, a false sense of security. > >> Is there a configuration like that already? Any other ideas? >> >> Pointers to useful documents/discussions on avoiding split brain with two >> node clusters would be welcome. > > https://www.alteeve.com/w/The_2‑Node_Myth > > (note: currently throwing a cert error related to the let's encrypt > issue, should be cleared up soon). > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ >>>
