12.04.2019 15:30, Олег Самойлов пишет: > >> 11 апр. 2019 г., в 20:00, Klaus Wenninger <[email protected]> >> написал(а): >> >> On 4/11/19 5:27 PM, Олег Самойлов wrote: >>> Hi all. I am developing HA PostgreSQL cluster for 2 or 3 >>> datacenters. In case of DataCenter failure (blackout) the fencing >>> will not work and will prevent to switching to working DC. So I >>> disable the fencing. The cluster working is based on a quorum and >>> I added a quorum device on a third DC in case of 2 DC. But I need >>> somehow solve >> Why would you disable fencing? SBD with watchdog-fencing (no >> shared disk) is made for exactly that use-case but you need fencing >> to be enabled and stonith-watchdog-timeout to be set to roughly 2x >> the watchdog-timeout. > > Interesting. There are a lot in documentation about using the sbd > with 1,2,3 block devices, but about using without block devices is > nothing, except a sentence that this is possible. :) >
Yes, stonith-watchdog-timeout does not really ring the bell and does not make it obvious it is related to SBD in any way. The way it seems to works is If stonithd^Wpaxcemaker-fenced receives request to kill node and no suitable stonith device for this node was found and SBD is active and stonith-watchdog-timeout is non-zero, fenced will a) forward request to victim and b) wait for specified timeout and expect node to self fence. If victim is alive it will initiate reboot either via local SBD or via SysRq if it could not contact SBD. If victim is not reachable, it is expected that SBD will commit suicide. You will see in logs something like Apr 14 09:13:45 ha1 pacemaker-fenced [1808] (call_remote_stonith) notice: Waiting 10s for ha2 to self-fence (reboot) for pacemaker-controld.1812.5a81fe48 ((nil)) Unfortunately 1. It is by far not obvious. You only see something during actual stonith attempt. You see Apr 14 09:13:45 ha1 pacemaker-schedulerd[1811] (unpack_config) notice: Watchdog will be used via SBD if fencing is required sprinkled over log file, but it is misleading - it will *NOT* use SBD fencing unless stonith-watchdog-timeout is actually set to non-zero. And you see the following log entry exactly once during normal startup: Apr 14 09:10:15 ha1 pacemaker-controld [1812] (check_sbd_timeout) debug: Using calculated value 10000 for stonith-watchdog-timeout (-1) Apr 14 09:10:15 ha1 pacemaker-controld [1812] (check_sbd_timeout) info: Watchdog configured with stonith-watchdog-timeout -1 and SBD timeout 5000ms And it may be too far in the past and already rotated away. 2. No high-level command showing current pacemaker run-time state shows you whether this mechanism is active. OK, if you know what you are looking for you may use direct CIB query to check values of have-watchdog and stonith-watchdog-timeout. But where pray is have-watchdog documented? Apparently intention was to create special internal stonith device with name "watchdog" which would be visible in at least stonith_admin -L output (not sure whether crm_mon would show it). But as implemented currently, fenced would (attempt to) create this device once very early during initial startup *if* stonith-watchdog-timeout is not zero - but it queries for the value of stonith-watchdog-timeout(or at least receives reply) far later, which means this special device is never created. Which leads to funny duplication of code which contains identical handling of both no device and special "watchdog" device cases. If anything, this is confusing to anyone looking at the code. Apr 14 09:09:54 ha1 pacemaker-fenced [1808] (main) info: Starting pacemaker-fenced mainloop ^^^^^^^^^^^^^^^ this line is already past attempt to create special watchdog device Apr 14 09:10:15 ha1 pacemaker-controld [1812] (check_sbd_timeout) info: Watchdog configured with stonith-watchdog-timeout -1 and SBD timeout 5000ms And of course stonith-watchdog-timeout can be changed to 0 at run-time so this device would now have to be removed even if it had been created successfully in the first place. So it really need to be created in CIB update handler. Which leaves window between changing this value and creation of special device so probably duplication of code is inevitable. If anyone thinks it is a bug, I will open bug report. 3. If node could not be contacted for whatever reasons and has not received self-fence request it will still be assumed to be fenced and pacemaker will start relocating resources. I do not know whether pacemaker cross-checks with actual node state (i.e. node is expected to be lost at the point watchdog timeout is expired). _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
