Re: [ClusterLabs] Corosync 2.4.4 is available at corosync.org!
Jan Friesse writes: > Ferenc Wágner napsal(a): > >> I wonder if c139255 (totemsrp: Implement sanity checks of received >> msgs) has direct security relevance as well. > > Not entirely direct, but quite similar. > >> Should I include that too in the Debian security update? Debian >> stable has 2.4.2, so I'm cherry picking into that version. > > Yes, please include all > fc1d5418533c1faf21616b282c2559bed7d361c4..b25b029fe186bacf089ab8136da58390945eb35c Hi Honza, I'm confused, the commit I mentioned above is not in the range you provided. Besides, I can only include targeted security fixes for exploitable vulnerabilities in a stable security update. A pre- authentication buffer overflow (CVE-2018-1084) most certainly qualifies, while the msgio cleanup does not. Missing checks for messages being sent (08cb237) are hard to judge for me... wouldn't expoiting this require root privileges to start with? Also, how much of these issues can be mitigated by enabling encryption or strict firewalling? Basically, I'll need more ammo to push all these changes through the Security Team. (I'll package 2.4.4 for testing/unstable and eventually provide a stable backport of it, but that goes through different channels.) -- Thanks, Feri ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
Hello all, One of my resources is a pppd process, which is started with the heartbeat/anything RA. That RA just spawn the pppd process with the correct parameters and return OCF_SUCCESS if the process started. The problem is that the service provided by pppd is only available after some time (a few seconds to 30s), ie. when it have successfully negotiated a connection. At this time, the interface it creates is UP. The issue here is that other resources that depend on this connection are started by Pacemaker just after it starts pppd, thus before the interface is UP. This creates various problems. I figured that fixing this would require to add a monitor call inside the start operation, and wait for a successful monitor before returning OCF_SUCCESS, within the start timeout. Is it a correct approach? Are there some other standard way to fix this, like a "wait for condition" Resource Agent? Using Pacemaker 1.1.16 on Debian stretch. -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
On 13/04/18 11:07 +0200, Nicolas Huillard wrote: Hello all, One of my resources is a pppd process, which is started with the heartbeat/anything RA. That RA just spawn the pppd process with the correct parameters and return OCF_SUCCESS if the process started. The problem is that the service provided by pppd is only available after some time (a few seconds to 30s), ie. when it have successfully negotiated a connection. At this time, the interface it creates is UP. The issue here is that other resources that depend on this connection are started by Pacemaker just after it starts pppd, thus before the interface is UP. This creates various problems. I figured that fixing this would require to add a monitor call inside the start operation, and wait for a successful monitor before returning OCF_SUCCESS, within the start timeout. Is it a correct approach? Are there some other standard way to fix this, like a "wait for condition" Resource Agent? You could try using the monitor_hook parameter to check the status, or use the Delay agent between the anything resource and the other resources. Using Pacemaker 1.1.16 on Debian stretch. -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] No slave is promoted to be master
OK, I know what happen. It seems like your standbies were not replicating when the master "crashed", you can find tons of messages like this in the log files: WARNING: No secondary connected to the master WARNING: "db2" is not connected to the primary WARNING: "db3" is not connected to the primary When a standby is not replicating, the master set negative master score to them to forbid the promotion on them, as they are probably lagging for some undefined time. The following command shows the scores just before the simulated master crash: $ crm_simulate -x pe-input-2039.bz2 -s|grep -E 'date|promotion' Using the original execution date of: 2018-04-11 16:23:07Z pgsqld:0 promotion score on db1: 1001 pgsqld:1 promotion score on db2: -1000 pgsqld:2 promotion score on db3: -1000 "1001" score design the master. Streaming standbies always have a positive master score between 1000 and 1000-N*10 where N is the number of connected standbies. On Fri, 13 Apr 2018 01:37:54 + 范国腾 wrote: > The log is in the attachment. > > We make a bug in the PG code in master node to make it not be restarted any > more in order to test the following scenario: One slave could be promoted > when the master crashed, > > -邮件原件- > 发件人: Jehan-Guillaume de Rorthais [mailto:j...@dalibo.com] > 发送时间: 2018年4月12日 17:39 > 收件人: 范国腾 > 抄送: Cluster Labs - All topics related to open-source clustering welcomed > 主题: Re: [ClusterLabs] No slave is promoted to be > master > > Hi, > On Thu, 12 Apr 2018 08:31:39 + > 范国腾 wrote: > > > Thank you very much for help check this issue. The information is in > > the attachment. > > > > I have restarted the cluster after I send my first email. Not sure if > > it affects the checking of "the result of "crm_simulate -sL" > > It does... > > Could you please provide files > from /var/lib/pacemaker/pengine/pe-input-2039.bz2 to pe-input-2065.bz2 ? > > [...] > > Then the master is restarted and it could not start(that is ok and we > > know the reason)。 > > Why couldn't it start ? -- Jehan-Guillaume de Rorthais Dalibo ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
Le vendredi 13 avril 2018 à 11:15 +0200, Oyvind Albrigtsen a écrit : > On 13/04/18 11:07 +0200, Nicolas Huillard wrote: > > One of my resources is a pppd process, which is started with the > > heartbeat/anything RA. That RA just spawn the pppd process with the > > correct parameters and return OCF_SUCCESS if the process started. > > The problem is that the service provided by pppd is only available > > after some time (a few seconds to 30s), ie. when it have > > successfully > > negotiated a connection. At this time, the interface it creates is > > UP. > > > > The issue here is that other resources that depend on this > > connection > > are started by Pacemaker just after it starts pppd, thus before the > > interface is UP. This creates various problems. > > > > I figured that fixing this would require to add a monitor call > > inside > > the start operation, and wait for a successful monitor before > > returning > > OCF_SUCCESS, within the start timeout. > > > > Is it a correct approach? > > Are there some other standard way to fix this, like a "wait for > > condition" Resource Agent? > > You could try using the monitor_hook parameter to check the status, The issue here is the monitor will at first return a "fail", which is considered fatal by Pacemaker unless property start-failure-is-fatal is set to false, which may come with side-effects. That's what I do now with a ping RA inserted before the service which may fail if the interface is not UP. It works, but triggers some "fail" events which are not really "fails" but "not started yet". > or > use the Delay agent between the anything resource and the other > resources. I'll try this. Hoping a sensible delay can be derived from the logs. Thanks, -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
On 13/04/18 11:53 +0200, Nicolas Huillard wrote: Le vendredi 13 avril 2018 à 11:15 +0200, Oyvind Albrigtsen a écrit : On 13/04/18 11:07 +0200, Nicolas Huillard wrote: > One of my resources is a pppd process, which is started with the > heartbeat/anything RA. That RA just spawn the pppd process with the > correct parameters and return OCF_SUCCESS if the process started. > The problem is that the service provided by pppd is only available > after some time (a few seconds to 30s), ie. when it have > successfully > negotiated a connection. At this time, the interface it creates is > UP. > > The issue here is that other resources that depend on this > connection > are started by Pacemaker just after it starts pppd, thus before the > interface is UP. This creates various problems. > > I figured that fixing this would require to add a monitor call > inside > the start operation, and wait for a successful monitor before > returning > OCF_SUCCESS, within the start timeout. > > Is it a correct approach? > Are there some other standard way to fix this, like a "wait for > condition" Resource Agent? You could try using the monitor_hook parameter to check the status, The issue here is the monitor will at first return a "fail", which is considered fatal by Pacemaker unless property start-failure-is-fatal is set to false, which may come with side-effects. That's what I do now with a ping RA inserted before the service which may fail if the interface is not UP. It works, but triggers some "fail" events which are not really "fails" but "not started yet". You might try setting it to e.g. "sleep 30; " and see if that works. or use the Delay agent between the anything resource and the other resources. I'll try this. Hoping a sensible delay can be derived from the logs. Thanks, -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Corosync 2.4.4 is available at corosync.org!
Ferenc Wágner napsal(a): Jan Friesse writes: Ferenc Wágner napsal(a): I wonder if c139255 (totemsrp: Implement sanity checks of received msgs) has direct security relevance as well. Not entirely direct, but quite similar. Should I include that too in the Debian security update? Debian stable has 2.4.2, so I'm cherry picking into that version. Yes, please include all fc1d5418533c1faf21616b282c2559bed7d361c4..b25b029fe186bacf089ab8136da58390945eb35c Hi Honza, Ferenc, I'm confused, the commit I mentioned above is not in the range you provided. Besides, I can only include targeted security fixes for Actually it is. c139255 = master/camelback branch, 50e17ffc736f0052e921c861b6953ba8938e4103 = needle branch. exploitable vulnerabilities in a stable security update. A pre- authentication buffer overflow (CVE-2018-1084) most certainly qualifies, while the msgio cleanup does not. Missing checks for messages being Patch "msgio: Fix reading of msg longer than i32" is not only cleanup. It also fixes real problem when message length > 2^31 . sent (08cb237) are hard to judge for me... wouldn't expoiting this require root privileges to start with? Also, how much of these issues None of these require root privileges can be mitigated by enabling encryption or strict firewalling? All (including the CVE one) can be mitigated by strict firewall. The CVE one and the msgio cannot be mitigated by encryption, other issues can be. Basically, I'll need more ammo to push all these changes through the Security Team. We can probably do CVE for others. Honza (I'll package 2.4.4 for testing/unstable and eventually provide a stable backport of it, but that goes through different channels.) ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
Le vendredi 13 avril 2018 à 11:59 +0200, Oyvind Albrigtsen a écrit : > On 13/04/18 11:53 +0200, Nicolas Huillard wrote: > > Le vendredi 13 avril 2018 à 11:15 +0200, Oyvind Albrigtsen a > > écrit : > > The issue here is the monitor will at first return a "fail", which > > is considered fatal by Pacemaker unless property start-failure-is- > > fatal is set to false, which may come with side-effects. > > That's what I do now with a ping RA inserted before the service > > which may fail if the interface is not UP. It works, but triggers > > some "fail" events which are not really "fails" but "not started > > yet". > > You might try setting it to e.g. "sleep 30; > " and see if that works. I'm using resource-agent package 4.0.0, and just noticed that what I was thinking about was implemented more recently in : https://github.com/ClusterLabs/resource-agents/commit/ee099d62c23e0afd0 442a4febde80412b8ac22f1#diff-07b3e128cbd8576888076cc71c00233b I'll use that one, thanks ! -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [solved] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
Le vendredi 13 avril 2018 à 11:15 +0200, Oyvind Albrigtsen a écrit : > On 13/04/18 11:07 +0200, Nicolas Huillard wrote: > > I figured that fixing this would require to add a monitor call > > inside the start operation, and wait for a successful monitor > > before returning OCF_SUCCESS, within the start timeout. > > > > Is it a correct approach? > > Are there some other standard way to fix this, like a "wait for > > condition" Resource Agent? > > You could try using the monitor_hook parameter to check the status, What I came up with is: monitor_hook="(i=50; while [ $i -gt 0 ]; do ip link show ppp0 && exit 0; sleep 5; i=$((i+5)); done; exit 1)" The "active" part of it being "ip link show ppp0". The rest is a loop to wait the least possible (at 5s interval). This works with resource-agents 4.1.1, not 4.0.0 that I initially used. -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [solved] heartbeat/anything Resource Agent : "wait for proper service before ending the start operation"
On 13/04/18 14:11 +0200, Nicolas Huillard wrote: Le vendredi 13 avril 2018 à 11:15 +0200, Oyvind Albrigtsen a écrit : On 13/04/18 11:07 +0200, Nicolas Huillard wrote: > I figured that fixing this would require to add a monitor call > inside the start operation, and wait for a successful monitor > before returning OCF_SUCCESS, within the start timeout. > > Is it a correct approach? > Are there some other standard way to fix this, like a "wait for > condition" Resource Agent? You could try using the monitor_hook parameter to check the status, What I came up with is: monitor_hook="(i=50; while [ $i -gt 0 ]; do ip link show ppp0 && exit 0; sleep 5; i=$((i+5)); done; exit 1)" You could simplify it by using for: for x in $(seq 1 10); do ip link show ppp0 && exit; sleep 5; done; exit 1 The "active" part of it being "ip link show ppp0". The rest is a loop to wait the least possible (at 5s interval). This works with resource-agents 4.1.1, not 4.0.0 that I initially used. -- Nicolas Huillard ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] HALVM monitor action fail on slave node. Possible bug?
Hello, I'm trying to configure a simple 2 node cluster with drbd and HALVM (ocf:heartbeat:LVM) but I have a problem that I'm not able to solve, to I decided to write this long post. I need to really understand what I'm doing and where I'm doing wrong. More precisely, I'm configuring a pacemaker cluster with 2 nodes and only one drbd resource. Here all operations: - System configuration hostnamectl set-hostname pcmk[12] yum update -y yum install vim wget git -y vim /etc/sysconfig/selinux -> permissive mode systemctl disable firewalld reboot - Network configuration [pcmk1] nmcli connection modify corosync ipv4.method manual ipv4.addresses 192.168.198.201/24 ipv6.method ignore connection.autoconnect yes nmcli connection modify replication ipv4.method manual ipv4.addresses 192.168.199.201/24 ipv6.method ignore connection.autoconnect yes [pcmk2] nmcli connection modify corosync ipv4.method manual ipv4.addresses 192.168.198.202/24 ipv6.method ignore connection.autoconnect yes nmcli connection modify replication ipv4.method manual ipv4.addresses 192.168.199.202/24 ipv6.method ignore connection.autoconnect yes ssh-keyget -t rsa ssh-copy-id root@pcmk[12] scp /etc/hosts root@pcmk2:/etc/hosts - Drbd Repo configuration and drbd installation rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm yum update -y yum install drbd84-utils kmod-drbd84 -y - Drbd Configuration: Creating a new partition on top of /dev/vdb -> /dev/vdb1 of type "Linux" (83) [/etc/drbd.d/global_common.conf] usage-count no; [/etc/drbd.d/myres.res] resource myres { on pcmk1 { device /dev/drbd0; disk /dev/vdb1; address 192.168.199.201:7789; meta-disk internal; } on pcmk2 { device /dev/drbd0; disk /dev/vdb1; address 192.168.199.202:7789; meta-disk internal; } } scp /etc/drbd.d/myres.res root@pcmk2:/etc/drbd.d/myres.res systemctl start drbd <-- only for test. The service is disabled at boot! drbdadm create-md myres drbdadm up myres drbdadm primary --force myres - LVM Configuration [root@pcmk1 ~]# lsblk NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:01 1024M 0 rom vda 252:00 20G 0 disk ├─vda1 252:101G 0 part /boot └─vda2 252:20 19G 0 part ├─cl-root 253:00 17G 0 lvm / └─cl-swap 253:102G 0 lvm [SWAP] vdb 252:16 08G 0 disk └─vdb1 252:17 08G 0 part <--- /dev/vdb1 is the partition I'd like to use as backing device for drbd └─drbd0 147:008G 0 disk [/etc/lvm/lvm.conf] write_cache_state = 0 use_lvmetad = 0 filter = [ "a|drbd.*|", "a|vda.*|", "r|.*|" ] Disabling lvmetad service systemctl disable lvm2-lvmetad.service systemctl disable lvm2-lvmetad.socket reboot - Creating volume group and logical volume systemctl start drbd (both nodes) drbdadm primary myres pvcreate /dev/drbd0 vgcreate havolumegroup /dev/drbd0 lvcreate -n c-vol1 -L1G havolumegroup [root@pcmk1 ~]# lvs LV VGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root cl-wi-ao <17.00g swap cl-wi-ao 2.00g c-vol1 havolumegroup -wi-a- 1.00g - Cluster Configuration yum install pcs fence-agents-all -y systemctl enable pcsd systemctl start pcsd echo redhat | passwd --stdin hacluster pcs cluster auth pcmk1 pcmk2 pcs cluster setup --name ha_cluster pcmk1 pcmk2 pcs cluster start --all pcs cluster enable --all pcs property set stonith-enabled=false<--- Just for test!!! pcs property set no-quorum-policy=ignore - Drbd resource configuration pcs cluster cib drbd_cfg pcs -f drbd_cfg resource create DrbdRes ocf:linbit:drbd drbd_resource=myres op monitor interval=60s pcs -f drbd_cfg resource master DrbdResClone DrbdRes master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true [root@pcmk1 ~]# pcs -f drbd_cfg resource show Master/Slave Set: DrbdResClone [DrbdRes] Stopped: [ pcmk1 pcmk2 ] [root@pcmk1 ~]# Testing the failover with a forced shutoff of pcmk1. When pcmk1 returns up, drbd is slave but logical volume is not active on pcmk2. So I need HALVM [root@pcmk2 ~]# lvs LV VGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root cl-wi-ao <17.00g swap cl-wi-ao 2.00g c-vol1 havolumegroup -wi--- 1.00g [root@pcmk2 ~]# - Lvm resource and constraints pcs cluster cib lvm_cfg pcs -f lvm_cfg resource create HALVM ocf:heartbea
Re: [ClusterLabs] HALVM monitor action fail on slave node. Possible bug?
the first thing that you need to configure is the stonith, because you have this constraint "constraint order promote DrbdResClone then start HALVM" To recover and promote drbd to master when you crash a node, configurare the drbd fencing handler. pacemaker execute monitor in both nodes, so this is normal, to test why monitor fail, use ocf-tester 2018-04-13 15:29 GMT+02:00 Marco Marino : > Hello, I'm trying to configure a simple 2 node cluster with drbd and HALVM > (ocf:heartbeat:LVM) but I have a problem that I'm not able to solve, to I > decided to write this long post. I need to really understand what I'm doing > and where I'm doing wrong. > More precisely, I'm configuring a pacemaker cluster with 2 nodes and only > one drbd resource. Here all operations: > > - System configuration > hostnamectl set-hostname pcmk[12] > yum update -y > yum install vim wget git -y > vim /etc/sysconfig/selinux -> permissive mode > systemctl disable firewalld > reboot > > - Network configuration > [pcmk1] > nmcli connection modify corosync ipv4.method manual ipv4.addresses > 192.168.198.201/24 ipv6.method ignore connection.autoconnect yes > nmcli connection modify replication ipv4.method manual ipv4.addresses > 192.168.199.201/24 ipv6.method ignore connection.autoconnect yes > [pcmk2] > nmcli connection modify corosync ipv4.method manual ipv4.addresses > 192.168.198.202/24 ipv6.method ignore connection.autoconnect yes > nmcli connection modify replication ipv4.method manual ipv4.addresses > 192.168.199.202/24 ipv6.method ignore connection.autoconnect yes > > ssh-keyget -t rsa > ssh-copy-id root@pcmk[12] > scp /etc/hosts root@pcmk2:/etc/hosts > > - Drbd Repo configuration and drbd installation > rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org > rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo. > noarch.rpm > yum update -y > yum install drbd84-utils kmod-drbd84 -y > > - Drbd Configuration: > Creating a new partition on top of /dev/vdb -> /dev/vdb1 of type > "Linux" (83) > [/etc/drbd.d/global_common.conf] > usage-count no; > [/etc/drbd.d/myres.res] > resource myres { > on pcmk1 { > device /dev/drbd0; > disk /dev/vdb1; > address 192.168.199.201:7789; > meta-disk internal; > } > on pcmk2 { > device /dev/drbd0; > disk /dev/vdb1; > address 192.168.199.202:7789; > meta-disk internal; > } > } > > scp /etc/drbd.d/myres.res root@pcmk2:/etc/drbd.d/myres.res > systemctl start drbd <-- only for test. The service is disabled at > boot! > drbdadm create-md myres > drbdadm up myres > drbdadm primary --force myres > > - LVM Configuration > [root@pcmk1 ~]# lsblk > NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sr0 11:01 1024M 0 rom > vda 252:00 20G 0 disk > ├─vda1 252:101G 0 part /boot > └─vda2 252:20 19G 0 part > ├─cl-root 253:00 17G 0 lvm / > └─cl-swap 253:102G 0 lvm [SWAP] > vdb 252:16 08G 0 disk > └─vdb1 252:17 08G 0 part <--- /dev/vdb1 is the partition > I'd like to use as backing device for drbd > └─drbd0 147:008G 0 disk > > [/etc/lvm/lvm.conf] > write_cache_state = 0 > use_lvmetad = 0 > filter = [ "a|drbd.*|", "a|vda.*|", "r|.*|" ] > > Disabling lvmetad service > systemctl disable lvm2-lvmetad.service > systemctl disable lvm2-lvmetad.socket > reboot > > - Creating volume group and logical volume > systemctl start drbd (both nodes) > drbdadm primary myres > pvcreate /dev/drbd0 > vgcreate havolumegroup /dev/drbd0 > lvcreate -n c-vol1 -L1G havolumegroup > [root@pcmk1 ~]# lvs > LV VGAttr LSize Pool Origin Data% Meta% > Move Log Cpy%Sync Convert > root cl-wi-ao <17.00g > > swap cl-wi-ao 2.00g > > c-vol1 havolumegroup -wi-a- 1.00g > > > - Cluster Configuration > yum install pcs fence-agents-all -y > systemctl enable pcsd > systemctl start pcsd > echo redhat | passwd --stdin hacluster > pcs cluster auth pcmk1 pcmk2 > pcs cluster setup --name ha_cluster pcmk1 pcmk2 > pcs cluster start --all > pcs cluster enable --all > pcs property set stonith-enabled=false<--- Just for test!!! > pcs property set no-quorum-policy=ignore > > - Drbd resource configuration > pcs cluster cib drbd_cfg > pcs -f drbd_cfg resource create DrbdRes ocf:linbit:drbd > drbd_resource=myres op monitor interval=60s > pcs -f drbd_cfg resource master DrbdResClone DrbdRes master-max=1 > master-node-max=1 clone-max=2 clone-node-max=1 notify=true > [root@pcmk1 ~]# pcs -f drbd_cfg resource show > Ma
Re: [ClusterLabs] Failing operations immediately when node is known to be down
On Tue, 2018-04-10 at 12:56 -0500, Ryan Thomas wrote: > I’m trying to implement a HA solution which recovers very quickly > when a node fails. It my configuration, when I reboot a node, I see > in the logs that pacemaker realizes the node is down, and decides to > move all resources to the surviving node. To do this, it initiates a > ‘stop’ operation on each of the resources to perform the move. The > ‘stop’ fails as expected after 20s (the default action timeout). > However, in this case, with the node known to be down, I’d like to > avoid this 20 second delay. The node is known to be down, so any > operations sent to the node will fail. It would be nice if > operations sent to a down node would immediately fail, thus reducing > the time it takes the resource to be started on the surviving node. > I do not want to reduce the timeout for the operation, because the > timeout is sensible for when a resource moves due to a non-node- > failure. Is there a way to accomplish this? > > Thanks for your help. How are you rebooting -- cleanly (normal shutdown) or simulating a failure (e.g. power button)? In a normal shutdown, pacemaker will move all resources off the node before it shuts down. These operations shouldn't fail, because the node isn't down yet. When a node fails, corosync should detect this and notify pacemaker. Pacemaker will not try to execute any operations on a failed node. Instead, it will fence it. What log messages do you see from corosync and pacemaker indicating that the node is down? Do you have fencing configured and tested? -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] General Capabilities Question
Hi, I had a general question about Pacemaker to see if it would work for a somewhat unique situation. I have a cluster of 10 active machines + 2 standby that each have 3 interfaces (2 control, 1 management). I want each of the control interfaces to use virtual IPs, such that if any of those 10 nodes fail, one of the two standby nodes can assume the virtual IPs of the failed node. For the virtual IPs, I wanted the failover to happen entirely in userspace and essentially just receive a notification that a node failed, and allow my application on the standby node to do all of the virtual IP reconfiguration. By userspace I mean that these interfaces are typically not configured by Linux, so the failure of those interfaces and the configuration would be done by a user script. The reason I'm looking at pacemaker is it appears to be robust on node failure detections, and the STONITH is something I'd like to do. Is this something pacemaker could be configured to do, and if so, which part of the documentation should I focus on? ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] HAProxy resource agent
Hello, I'm planning to install an active\active HAProxy cluster on CentOS 7 I didn't found that there is any RA for HAproxy. I found some on the net but I'm not sure if I need it. For example: https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy I can always use the systemd service RA What is your recommendation? Thanks, Tomer. ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] HAProxy resource agent
On 13/04/2018 17:56, Tomer Azran wrote: > Hello, > I'm planning to install an active\active HAProxy cluster on CentOS 7 > I didn't found that there is any RA for HAproxy. > I found some on the net but I'm not sure if I need it. For example: > https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy > I can always use the systemd service RA > What is your recommendation? Systemd is fine for a service like haproxy, go for it! -- RaSca Mia Mamma Usa Linux: Niente è impossibile da capire, se lo spieghi bene! ra...@miamammausalinux.org http://www.miamammausalinux.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Pacemaker resources are not scheduled
My cluster version: Corosync 2.4.0 Pacemaker 1.1.16 There are many resource anomalies. Some resources are only monitored and not recovered. Some resources are not monitored or recovered. Only one resource of vnm is scheduled normally, but this resource cannot be started because other resources in the cluster are abnormal. Just like a deadlock. I have been plagued by this problem for a long time. I just want a stable and highly available resource with infinite recovery for everyone. Is my resource configure correct? $ cat /etc/corosync/corosync.conf mpatibility: whitetank quorum { provider: corosync_votequorum two_node: 0 } nodelist { node { ring0_addr: 122.0.1.8 name: 122.0.1.8 nodeid: 01008 } node { ring0_addr: 122.0.1.9 name: 122.0.1.9 nodeid: 01009 } node { ring0_addr: 122.0.1.10 name: 122.0.1.10 nodeid: 01010 } } totem { version: 2 token: 3000 token_retransmits_before_loss_const: 10 join:60 consensus: 3600 vsftype: none max_messages:20 clear_node_high_bit: yes rrp_mode:none secauth: off threads: 2 transport: udpu interface { ringnumber: 0 bindnetaddr: 122.0.1.8 mcastport: 5405 } } logging { fileline:off to_stderr: no to_logfile: yes logfile: /root/info/logs/pacemaker_cluster/corosync.log to_syslog: yes syslog_facility: daemon syslog_priority: info debug: off function_name: on timestamp: on logger_subsys { subsys: AMF debug: off tags: enter|leave|trace1|trace2|trace3|trace4|trace6 } } amf { mode: disabled } aisexec { user: root group: root } $ crm configure show node 1008: 122.0.1.8 node 1009: 122.0.1.9 node 1010: 122.0.1.10 primitive apigateway apigateway \ op monitor interval=20s timeout=220 \ op stop interval=0 timeout=120s on-fail=restart \ op start interval=0 timeout=120s on-fail=restart \ meta failure-timeout=60s target-role=Started primitive apigateway_vip IPaddr2 \ params ip=122.0.1.203 cidr_netmask=24 \ op start interval=0 timeout=20 \ op stop interval=0 timeout=20 \ op monitor timeout=20s interval=10s depth=0 \ meta migration-threshold=3 failure-timeout=60s primitive inetmanager inetmanager \ op monitor interval=10s timeout=160 \ op stop interval=0 timeout=60s on-fail=restart \ op start interval=0 timeout=60s on-fail=restart \ meta migration-threshold=2 failure-timeout=60s resource-stickiness=100 primitive inetmanager_vip IPaddr2 \ params ip=122.0.1.201 cidr_netmask=24 \ op start interval=0 timeout=20 \ op stop interval=0 timeout=20 \ op monitor timeout=20s interval=10s depth=0 \ meta migration-threshold=3 failure-timeout=60s primitive logserver logserver \ op monitor interval=20s timeout=220 \ op stop interval=0 timeout=120s on-fail=restart \ op start interval=0 timeout=120s on-fail=restart \ meta failure-timeout=60s target-role=Started primitive mariadb_vip IPaddr2 \ params ip=122.0.1.204 cidr_netmask=24 \ op start interval=0 timeout=20 \ op stop interval=0 timeout=20 \ op monitor timeout=20s interval=10s depth=0 \ meta migration-threshold=3 failure-timeout=60s primitive mysql mysql \ op monitor interval=20s timeout=220 \ op stop interval=0 timeout=120s on-fail=restart \ op start interval=0 timeout=120s on-fail=restart \ meta failure-timeout=60s target-role=Started primitive p_rdbserver hardb_docker \ op start timeout=3600 interval=0 \ op stop timeout=1260 interval=0 \ op promote timeout=3600 interval=0 \ op monitor role=Master interval=1 timeout=30 \ op monitor interval=15 timeout=7200 \ meta migration-threshold=3 failure-timeout=3600s primitive p_rdbvip IPaddr2 \ params ip=100.0.1.203 cidr_netmask=24 \ op start interval=0 timeout=20 \ op stop interval=0 timeout=20 \ op monitor timeout=20s interval=10s depth=0 \ meta migration-threshold=3 failure-timeout=60s resource-stickiness=100 primitive rabbitmq rabbitmq \ op monitor interval=20s timeout=220 \ op stop interval=0 timeout=120s on-fail=restart \ op start interval=0 timeout=120s on-fail=restart \ meta failure-timeout=60s target-role=Started primitive rabbitmq_vip IPaddr2 \ params ip=122.0.1.200 \ op start interval=0 timeout=20 \ op stop interval=0 timeout=20 \ op monitor timeout=20s interval=10s depth=0 \ meta migration-t
Re: [ClusterLabs] General Capabilities Question
On Thu, 2018-04-12 at 21:09 -0700, Cliff Burdick wrote: > Hi, I had a general question about Pacemaker to see if it would work > for a somewhat unique situation. I have a cluster of 10 active > machines + 2 standby that each have 3 interfaces (2 control, 1 > management). I want each of the control interfaces to use virtual > IPs, such that if any of those 10 nodes fail, one of the two standby > nodes can assume the virtual IPs of the failed node. For the virtual > IPs, I wanted the failover to happen entirely in userspace and > essentially just receive a notification that a node failed, and allow > my application on the standby node to do all of the virtual IP > reconfiguration. By userspace I mean that these interfaces are > typically not configured by Linux, so the failure of those interfaces > and the configuration would be done by a user script. > > The reason I'm looking at pacemaker is it appears to be robust on > node failure detections, and the STONITH is something I'd like to do. > Is this something pacemaker could be configured to do, and if so, > which part of the documentation should I focus on? Yes, that's what Pacemaker is designed for. I recommend starting with the "Clusters from Scratch" document to get familiar with the basics. Then you can look at "Pacemaker Explained" to get detailed info about particular configuration options (especially constraints). I'm not sure what your interface/VIP stuff does, but you probably want to write your own OCF resource agent (see IPaddr2 as an example) to manage the IP, and let Pacemaker call it as needed. -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Booth fail-over conditions
Hey all, new user to pacemaker/booth and I'm fumbling my way through my first proof of concept. I have a 2 site configuration setup with local pacemaker clusters at each site (running rabbitmq) and a booth arbitrator. I've successfully validated the base failover when the "granted" site has failed. My question is if there are any other ways to configure failover, i.e. using resource health checks or the like? Thanks! ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org