Re: [Linux-HA] Quick 'death match cycle' question.
03.09.2013 07:04, Digimer wrote: ... To solve problem 1, you can set a delay against one of the nodes. Say you set the fence primitive for node 01 to have 'delay=15'. When node 1 goes to fence node 2, it starts immediately. When node 2 starts to fence node 1, it sees the 15 second delay and pauses. Node 1 will power off node 2 long before node 2 finishes the pause. You can further help this problem by disabling acpid on the nodes. Without it, the power-off signal from the BMC will be nearly instant, shortening up the window where both nodes can initiate a fence. Does anybody know for sure how and *why* does it work? I mean why disabling userspace ACPI event reader (which reads just what kernel sends after hardware events) affects how hardware behaves? To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. Enhancement to this would be enabling corosync/pacemaker back during the clean shutdown and disabling it after boot. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Quick 'death match cycle' question.
On 03/09/13 14:14, Vladislav Bogdanov wrote: 03.09.2013 07:04, Digimer wrote: ... To solve problem 1, you can set a delay against one of the nodes. Say you set the fence primitive for node 01 to have 'delay=15'. When node 1 goes to fence node 2, it starts immediately. When node 2 starts to fence node 1, it sees the 15 second delay and pauses. Node 1 will power off node 2 long before node 2 finishes the pause. You can further help this problem by disabling acpid on the nodes. Without it, the power-off signal from the BMC will be nearly instant, shortening up the window where both nodes can initiate a fence. Does anybody know for sure how and *why* does it work? I mean why disabling userspace ACPI event reader (which reads just what kernel sends after hardware events) affects how hardware behaves? Disabling acpid causes, in my experience, the node to instantly power down when it receives a power-button event. How/why this happens is probably buried in the kernel source and/or ACPI definitions. To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. Enhancement to this would be enabling corosync/pacemaker back during the clean shutdown and disabling it after boot. That would be a good idea, actually. I like that. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Quick 'death match cycle' question.
On 2013-09-03T21:14:02, Vladislav Bogdanov bub...@hoster-ok.com wrote: To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. Enhancement to this would be enabling corosync/pacemaker back during the clean shutdown and disabling it after boot. There's something in sbd which does this. See https://github.com/l-mb/sbd/blob/master/man/sbd.8.pod and the -S option. I'm contemplating how do to this in a generic fashion. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Quick 'death match cycle' question.
03.09.2013 21:45, Digimer wrote: On 03/09/13 14:14, Vladislav Bogdanov wrote: 03.09.2013 07:04, Digimer wrote: ... To solve problem 1, you can set a delay against one of the nodes. Say you set the fence primitive for node 01 to have 'delay=15'. When node 1 goes to fence node 2, it starts immediately. When node 2 starts to fence node 1, it sees the 15 second delay and pauses. Node 1 will power off node 2 long before node 2 finishes the pause. You can further help this problem by disabling acpid on the nodes. Without it, the power-off signal from the BMC will be nearly instant, shortening up the window where both nodes can initiate a fence. Does anybody know for sure how and *why* does it work? I mean why disabling userspace ACPI event reader (which reads just what kernel sends after hardware events) affects how hardware behaves? Disabling acpid causes, in my experience, the node to instantly power down when it receives a power-button event. How/why this happens is probably buried in the kernel source and/or ACPI definitions. This assumes some kind of back-events, which are not the part of ACPI iirc. And kernel just translates forward ACPI events (bits in hw port???) to userspace. Interesting enough, how do they do it... To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. Enhancement to this would be enabling corosync/pacemaker back during the clean shutdown and disabling it after boot. That would be a good idea, actually. I like that. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Quick 'death match cycle' question.
03.09.2013 21:36, Lars Marowsky-Bree wrote: On 2013-09-03T21:14:02, Vladislav Bogdanov bub...@hoster-ok.com wrote: To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. Enhancement to this would be enabling corosync/pacemaker back during the clean shutdown and disabling it after boot. There's something in sbd which does this. See https://github.com/l-mb/sbd/blob/master/man/sbd.8.pod and the -S option. Yes, but I thought it is a no-go with just drbd replicated disks (usual case for 2-node clusters). I'm contemplating how do to this in a generic fashion. It is quite straight-forward with SysVinit and its emulation with upstart, but could be tricky with native upstart and systemd. Need to investigate... ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Quick 'death match cycle' question.
I've got a very simple question which I suspect betrays my lack of understanding of something basic. Could someone help me understand? If I have a two-node Pacemaker cluster - say, a really simple cluster of two nodes, A B, with a solitary network connection between them - then I have to set no-quorum-policy to 'ignore'. If the network connection is broken then both A B will attempt to STONITH each other. Is there anything that would stop an endless cycle of each killing the other if the actions of the STONITH agents are set to reboot? I.e.: - A B race to STONITH each other - A kills B - A assumes resources - B reboots - B can't see A - B kills A - B assumes resources - A reboots - A can't see B - A kills B - A assumes resources ... etc. It's to stop this sort of cycle that I've set my STONITH actions to 'off' rather than 'reboot'. But I was reading the 'Fencing topology' document that Digimer referenced and I was reminded in my perusal that many people/clusters use a 'reboot' action. For a simple quorum-less cluster of two nodes how do those clusters avoid a never-ending cycle of each node killing the other, if neither node can 'see' the other via corosync? It's a very basic question; I think I'm forgetting something obvious. Thanks for any help! ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Quick 'death match cycle' question.
On 02/09/13 23:23, Alex Sudakar wrote: I've got a very simple question which I suspect betrays my lack of understanding of something basic. Could someone help me understand? If I have a two-node Pacemaker cluster - say, a really simple cluster of two nodes, A B, with a solitary network connection between them - then I have to set no-quorum-policy to 'ignore'. If the network connection is broken then both A B will attempt to STONITH each other. Is there anything that would stop an endless cycle of each killing the other if the actions of the STONITH agents are set to reboot? I.e.: - A B race to STONITH each other - A kills B - A assumes resources - B reboots - B can't see A - B kills A - B assumes resources - A reboots - A can't see B - A kills B - A assumes resources ... etc. It's to stop this sort of cycle that I've set my STONITH actions to 'off' rather than 'reboot'. But I was reading the 'Fencing topology' document that Digimer referenced and I was reminded in my perusal that many people/clusters use a 'reboot' action. For a simple quorum-less cluster of two nodes how do those clusters avoid a never-ending cycle of each node killing the other, if neither node can 'see' the other via corosync? It's a very basic question; I think I'm forgetting something obvious. Thanks for any help! There are two parts to this problem; 1. Both call a fence, both might win (dual fence) because both can get their fence call started before they die. This is because the fence devices, the IPMI BMCs, are independent. 2. A fence loop is where a fenced node boots, fences the other node. The other node boots, fences the first. Wash, rinse, repeat. To solve problem 1, you can set a delay against one of the nodes. Say you set the fence primitive for node 01 to have 'delay=15'. When node 1 goes to fence node 2, it starts immediately. When node 2 starts to fence node 1, it sees the 15 second delay and pauses. Node 1 will power off node 2 long before node 2 finishes the pause. You can further help this problem by disabling acpid on the nodes. Without it, the power-off signal from the BMC will be nearly instant, shortening up the window where both nodes can initiate a fence. To solve problem 2, simply disable corosync/pacemaker from starting on boot. This way, the fenced node will be (hopefully) back up and running, so you can ssh into it and look at what happened. It won't try to rejoin the cluster though, so no risk of a fence loop. digimer -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
