Re: [Linux-HA] Antw: Question around resources constraints (pacemaker on RHE7.1)
> On 2 Sep 2015, at 9:42 pm, MOULLE, ALAIN <alain.mou...@atos.net> wrote: > > Hi Andrew > > in fact, my real problem was simplier like this one described as a simplified > example of my configuration : > (I'm working with pacemaker from RHEL7.1 GA (1.1.12-22) ) > > Let's say for the example that we have in a two-nodes configuration these 5 > resources : > 1 resource VGA (ocf:heartbeat:LVM) > 1 resource fs-img-for-vm1 (ocf:heartbeat:Filesystem) > 1 resource vm1 (ocf:heartbeat:VirtualDomain) > 1 resource fs-img-for-vm2 (ocf:heartbeat:Filesystem) > 1 resource vm2 (ocf:heartbeat:VirtualDomain) > knowing that both fs-img are on LVs in VGA > > So I have to set these constraints : > prefered location VGA on node1 > order fs-img-vm1 after VGA > order fs-img-vm2 after VGA > order vm1 after fs-img-vm1 > order vm2 after fs-img-vm2 > colocation fs-img-vm1 with VGA > colocation fs-img-vm2 with VGA > colocation vm1 with fs-img-vm1 > colocation vm2 with fs-img-vm2 > > but I unfornutately, I want that the vm2 (and fs-img-vm2) never starts on > node2, said in another way : I want that if VGA is on node1 all resources > fs-img-vm1, fs-img-vm2, vm1 and vm2 must be started on node1, > but if VGA for whatever reason has been migrated on node2, I want that only > resources VGA, fs-img-vm1 and vm1 are started on node2, both others > fs-img-vm2 and vm2 remainig "Stopped" > > So I add the constraint location fs-img-vm2 avoids node2 (hence +INF) > > But testing this configuration, none of the 5 resources can never start on > vm2. You mean “node2” here right? > > In my understanding, due to my choice of setting colocation that way : fs > with VGA , and vm with fs, I was thinking that pacemaker, when migrating on > node2, could start VGA, fs-img-vm1 and vm1 on node2 and leave fs-img-vm2 and > vm2 "Stopped". > > Am I wrong ? > And is there a way to get this behavior ? Should be, attach your cib? > > Thanks a lot > Alain > > > De : linux-ha-boun...@lists.linux-ha.org > [linux-ha-boun...@lists.linux-ha.org] de la part de Andrew Beekhof > [and...@beekhof.net] > Envoyé : vendredi 28 août 2015 04:31 > À : Please subscribe to us...@clusterlabs.org instead > Objet : Re: [Linux-HA] Antw: Question around resources constraints > (pacemaker on RHE7.1) > >> On 25 Aug 2015, at 6:18 pm, Ulrich Windl <ulrich.wi...@rz.uni-regensburg.de> >> wrote: >> >>>>> "MOULLE, ALAIN" <alain.mou...@atos.net> schrieb am 21.08.2015 um 15:27 in >> Nachricht >> <df84cff8a85ab546b2d53fff12267727022...@frauvj99ex5msx.ww931.my-it-solutions.net >> >> : >>> Hi >>> >>> I can't find a way to configure constraints in pacemaker so that with these >> >>> resources: >>> >>> Res1 >>> Res2 >>> Res3 >>> Res4 >>> Res5 >>> >>> with current colocation constraints : >>> Res2 with Res1 >>> Res3 with Res2 >> >> I think pacemaker still cannot do transitive location constraints; > > ? > >> can you >> try >> R2 with R1 >> R3 with R1 >> (and related) instead? > > No no no. > This has the opposite effect, a failure of R3 could easily result in /none/ > of the resources moving. > > Better to just change: > >>> Res4 with Res1 > > to: > Res4 with Res3 > > Newer versions might do better with the existing config too. > If not, attach a crm_report :) > >>> Res5 with Res4 >>> >>> and current order symmetrical constraints : >>> Res2 after Res1 >>> Res3 after Res2 >>> >>> Res4 after Res1 >>> Res5 after Res4 >>> >>> and migration-threshold=1 on all resources . >>> >>> What I want it that if I have a failure for example on Res3 is that all the >> >>> 5 Ressources are migrated. >>> >>> Is there a solution ? >>> >>> For example , with an HA LVM configuration and VM resources : >>> >>> Res1=VGA >>> Res2=FS-img-VM1 (where are the VM1 image and .xml) (FS-img-VM1 is on VGA >> LV) >>> Res3=VM1 >>> Res4=FS-img-VM2 (where are the VM2 image and .xml) (FS-img-VM1 is on another >> VGA >>> LV) >>> Res5=VM2 >>> >>> So current constraints are so that VGA is activated before the FS-img is >>> mounted and before the VM is started. >>> >>> But I want that if the VM1 fails, it can migrate on another node (toge
Re: [Linux-HA] Antw: Question around resources constraints (pacemaker on RHE7.1)
On 25 Aug 2015, at 6:18 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: MOULLE, ALAIN alain.mou...@atos.net schrieb am 21.08.2015 um 15:27 in Nachricht df84cff8a85ab546b2d53fff12267727022...@frauvj99ex5msx.ww931.my-it-solutions.net : Hi I can't find a way to configure constraints in pacemaker so that with these resources: Res1 Res2 Res3 Res4 Res5 with current colocation constraints : Res2 with Res1 Res3 with Res2 I think pacemaker still cannot do transitive location constraints; ? can you try R2 with R1 R3 with R1 (and related) instead? No no no. This has the opposite effect, a failure of R3 could easily result in /none/ of the resources moving. Better to just change: Res4 with Res1 to: Res4 with Res3 Newer versions might do better with the existing config too. If not, attach a crm_report :) Res5 with Res4 and current order symmetrical constraints : Res2 after Res1 Res3 after Res2 Res4 after Res1 Res5 after Res4 and migration-threshold=1 on all resources . What I want it that if I have a failure for example on Res3 is that all the 5 Ressources are migrated. Is there a solution ? For example , with an HA LVM configuration and VM resources : Res1=VGA Res2=FS-img-VM1 (where are the VM1 image and .xml) (FS-img-VM1 is on VGA LV) Res3=VM1 Res4=FS-img-VM2 (where are the VM2 image and .xml) (FS-img-VM1 is on another VGA LV) Res5=VM2 So current constraints are so that VGA is activated before the FS-img is mounted and before the VM is started. But I want that if the VM1 fails, it can migrate on another node (together with all other 4 resources) but with only the constraints above, the VGA stalls the migration of the VM ... Is there any solution by constraints configuration ? Note : we can't use group resources for now, because VMs are remote-nodes. Thanks Alain Moullé ___ Linux-HA mailing list is closing down. Please subscribe to us...@clusterlabs.org instead. http://clusterlabs.org/mailman/listinfo/users ___ Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha ___ Linux-HA mailing list is closing down. Please subscribe to us...@clusterlabs.org instead. http://clusterlabs.org/mailman/listinfo/users ___ Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha ___ Linux-HA mailing list is closing down. Please subscribe to us...@clusterlabs.org instead. http://clusterlabs.org/mailman/listinfo/users ___ Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha
Re: [Linux-HA] Ping nodes and cluster rules
On 17 Apr 2015, at 10:58 pm, Adam Błaszczykowski adam.blaszczykow...@gmail.com wrote: Hello, I am using two nodes in cluster with corosync 2.3.4 and pacemaker 1.1.12 and each node has access to 2 ping nodes. I would like to know if it is possible to set following cluster rules: Rule 1 - do nothing with resources if 1 of 2 ping nodes are unreachable for one cluster node Rule 2 - move resources if 2 of 2 ping nodes are unreachable for one cluster node Rule 3 - do nothing with resources if all ping nodes are unreachable for both cluster nodes With the current implementation, you can’t satisfy rules 2 and 3 OR rules 1 and 3. NOTE: In my current configuration resources are exported when I lose access to all ping nodes on both cluster nodes. I would like to change this behavior as described in Rule 3 Thank you in advance ! Best Regards Adam Blaszczykowski ___ Linux-HA mailing list is closing down. Please subscribe to us...@clusterlabs.org instead. http://clusterlabs.org/mailman/listinfo/users ___ Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha ___ Linux-HA mailing list is closing down. Please subscribe to us...@clusterlabs.org instead. http://clusterlabs.org/mailman/listinfo/users ___ Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha
Re: [Linux-HA] Q: Resource migration (Xen live migration)
On 13 Feb 2015, at 8:38 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hello! I have some questions on pacemakers's resource migration. We have a Xen host that has some problems (still to be investigated) that causes some VM disk not be be ready for use. When tyring to migrate a VM frem the bad host to a good host through pacemaker, migration seemed to hang. At some state the source VM was no longer present on the bad host (Unable to find domain 'v09'), but pacemaker still tried a migration: crmd[6779]: notice: te_rsc_command: Initiating action 100: migrate_from prm_xen_v09_migrate_from_0 on h05 Only after the timeout CRM realized that there is a problem: crmd[6779]: warning: status_from_rc: Action 100 (prm_xen_v09_migrate_from_0) on h05 failed (target: 0 vs. rc: 1): Error After that CRM still stried a stop on the source host (h10) (and on the destination host): crmd[6779]: notice: te_rsc_command: Initiating action 98: stop prm_xen_v09_stop_0 on h10 crmd[6779]: notice: te_rsc_command: Initiating action 26: stop prm_xen_v09_stop_0 on h05 Q1: Is this the way it should work? Mostly, but the agent should have detected the condition earlier and returned an error (instead of timing out). Before that we had the same situation (thae bad host had been set to standby) when someone tired of waiting so long destroyed the affected Xen VMS on the source host while the cluster was migrating. Eventually the VMs came up (restarted instead of being live migrated) on the good hosts. Then we shutdown OpenAIS on the bad host, installed updates and rebooted the bad host (during reboot OpenAIS was started (still standby)). To my surprise pacemaker thought the VMS were still running on the bad host and initiated a migration. That would be coming from the resource agent. As there were no source VMs on the bad host, but alle the affected VMs were running on some good host, CRM stutdown the VMs on the good hostss, just to restart them. Q2: Ist this expected behavior? I can hardly believe! Nope, fix the agent :) Software is SLES11 SP3 with pacemaker-1.1.11-0.7.53 (and related) on all hosts. Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Announcing the Heartbeat 3.0.6 Release
On 11 Feb 2015, at 8:24 am, Lars Ellenberg lars.ellenb...@linbit.com wrote: TL;DR: If you intend to set up a new High Availability cluster using the Pacemaker cluster manager, you typically should not care for Heartbeat, but use recent releases (2.3.x) of Corosync. If you don't care for Heartbeat, don't read further. Unless you are beekhof... there's a question below ;-) After 3½ years since the last officially tagged release of Heartbeat, I have seen the need to do a new maintenance release. The Heartbeat 3.0.6 release tag: 3d59540cf28d and the change set it points to: cceeb47a7d8f The main reason for this was that pacemaker more recent than somewhere between 1.1.6 and 1.1.7 would no longer work properly on the Heartbeat cluster stack. Because some of the daemons have moved from glue to pacemaker proper, and changed their paths. This has been fixed in Heartbeat. And because during that time, stonith-ng was refactored, and would still reliably fence, but not understand its own confirmation message, so it was effectively broken. This I fixed in pacemaker. If you chose to run new Pacemaker with the Heartbeat communication stack, it should be at least 1.1.12 with a few patches, see my December 2014 commits at the top of https://github.com/lge/pacemaker/commits/linbit-cluster-stack-pcmk-1.1.12 I'm not sure if they got into pacemaker upstream yet. beekhof? Do I need to rebase? Or did I miss you merging these? Merged now :-) We're about to start the 1.1.13 release cycle, so it wont be far away --- If you have those patches, consider setting this new ha.cf configuration parameter: # If pacemaker crmd spawns the pengine itself, # it sometimes forgets to kill the pengine on shutdown, # which later may confuse the system after cluster restart. # Tell the system that Heartbeat is supposed to # control the pengine directly. crmd_spawns_pengine off Here is the shortened Heartbeat changelog, the longer version is available in mercurial: http://hg.linux-ha.org/heartbeat-STABLE_3_0/shortlog - fix emergency shutdown due to broken update_ackseq - fix node dead detection problems - fix converging of membership (ccm) - fix init script startup glitch (caused by changes in glue/resource-agents) - heartbeat.service file for systemd platforms - new ucast6 UDP IPv6 communication plugin - package ha_api.py in standard package - update some man pages, specifically the example ha.cf - also report ccm membership status for cl_status hbstatus -v - updated some log messages, or their log levels - reduce max_delay in broadcast client_status query to one second - apply various (mostly cosmetic) patches from Debian - drop HBcompress compression plugins: they are part of cluster glue - drop openais HBcomm plugin - better support for current pacemaker versions - try to not miss a SIGTERM (fix problem with very fast respawn/stop cycle) - dopd: ignore dead ping nodes - cl_status improvements - api internals: reduce IPC round-trips to get at status information - uid=root is sufficient to use heartbeat api (gid=haclient remains sufficient) - fix /dev/null as log- or debugfile setting - move daemon binaries into libexecdir - document movement of compression plugins into cluster-glue - fix usage of SO_REUSEPORT in ucast sockets - fix compile issues with recent gcc and -Werror Note that a number of the mentioned fixes have been created two years ago already, and may have been released in packages for a long time, where vendors have chosen to package them. As to future plans for Heartbeat: Heartbeat is still useful for non-pacemaker, haresources-mode clusters. We (Linbit) will maintain Heartbeat for the foreseeable future. That should not be too much of a burden, as it is stable, and due to long years of field exposure, all bugs are known ;-) The most notable shortcoming when using Heartbeat with Pacemaker clusters would be the limited message size. There are currently no plans to remove that limitation. With its wide choice of communications paths, even exotic communication plugins, and the ability to run arbitrarily many paths, some deployments may even favor it over Corosync still. But typically, for new deployments involving Pacemaker, in most cases you should chose Corosync 2.3.x as your membership and communication layer. For existing deployments using Heartbeat, upgrading to this Heartbeat version is strongly recommended. Thanks, Lars Ellenberg ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org
Re: [Linux-ha-dev] [Linux-HA] Announcing the Heartbeat 3.0.6 Release
On 11 Feb 2015, at 8:24 am, Lars Ellenberg lars.ellenb...@linbit.com wrote: TL;DR: If you intend to set up a new High Availability cluster using the Pacemaker cluster manager, you typically should not care for Heartbeat, but use recent releases (2.3.x) of Corosync. If you don't care for Heartbeat, don't read further. Unless you are beekhof... there's a question below ;-) After 3½ years since the last officially tagged release of Heartbeat, I have seen the need to do a new maintenance release. The Heartbeat 3.0.6 release tag: 3d59540cf28d and the change set it points to: cceeb47a7d8f The main reason for this was that pacemaker more recent than somewhere between 1.1.6 and 1.1.7 would no longer work properly on the Heartbeat cluster stack. Because some of the daemons have moved from glue to pacemaker proper, and changed their paths. This has been fixed in Heartbeat. And because during that time, stonith-ng was refactored, and would still reliably fence, but not understand its own confirmation message, so it was effectively broken. This I fixed in pacemaker. If you chose to run new Pacemaker with the Heartbeat communication stack, it should be at least 1.1.12 with a few patches, see my December 2014 commits at the top of https://github.com/lge/pacemaker/commits/linbit-cluster-stack-pcmk-1.1.12 I'm not sure if they got into pacemaker upstream yet. beekhof? Do I need to rebase? Or did I miss you merging these? Merged now :-) We're about to start the 1.1.13 release cycle, so it wont be far away --- If you have those patches, consider setting this new ha.cf configuration parameter: # If pacemaker crmd spawns the pengine itself, # it sometimes forgets to kill the pengine on shutdown, # which later may confuse the system after cluster restart. # Tell the system that Heartbeat is supposed to # control the pengine directly. crmd_spawns_pengine off Here is the shortened Heartbeat changelog, the longer version is available in mercurial: http://hg.linux-ha.org/heartbeat-STABLE_3_0/shortlog - fix emergency shutdown due to broken update_ackseq - fix node dead detection problems - fix converging of membership (ccm) - fix init script startup glitch (caused by changes in glue/resource-agents) - heartbeat.service file for systemd platforms - new ucast6 UDP IPv6 communication plugin - package ha_api.py in standard package - update some man pages, specifically the example ha.cf - also report ccm membership status for cl_status hbstatus -v - updated some log messages, or their log levels - reduce max_delay in broadcast client_status query to one second - apply various (mostly cosmetic) patches from Debian - drop HBcompress compression plugins: they are part of cluster glue - drop openais HBcomm plugin - better support for current pacemaker versions - try to not miss a SIGTERM (fix problem with very fast respawn/stop cycle) - dopd: ignore dead ping nodes - cl_status improvements - api internals: reduce IPC round-trips to get at status information - uid=root is sufficient to use heartbeat api (gid=haclient remains sufficient) - fix /dev/null as log- or debugfile setting - move daemon binaries into libexecdir - document movement of compression plugins into cluster-glue - fix usage of SO_REUSEPORT in ucast sockets - fix compile issues with recent gcc and -Werror Note that a number of the mentioned fixes have been created two years ago already, and may have been released in packages for a long time, where vendors have chosen to package them. As to future plans for Heartbeat: Heartbeat is still useful for non-pacemaker, haresources-mode clusters. We (Linbit) will maintain Heartbeat for the foreseeable future. That should not be too much of a burden, as it is stable, and due to long years of field exposure, all bugs are known ;-) The most notable shortcoming when using Heartbeat with Pacemaker clusters would be the limited message size. There are currently no plans to remove that limitation. With its wide choice of communications paths, even exotic communication plugins, and the ability to run arbitrarily many paths, some deployments may even favor it over Corosync still. But typically, for new deployments involving Pacemaker, in most cases you should chose Corosync 2.3.x as your membership and communication layer. For existing deployments using Heartbeat, upgrading to this Heartbeat version is strongly recommended. Thanks, Lars Ellenberg ___ Linux-HA mailing list linux...@lists.linux-ha.org
Re: [Linux-ha-dev] quorum status
Check corosync.conf, I'm guessing pcs enabled two node mode. On 29 Jan 2015, at 5:21 pm, Yan, Xiaoping (NSN - CN/Hangzhou) xiaoping@nsn.com wrote: Hi, I used this command: pcs cluster stop rhel1 I think both is shutdown. Br, Rip -Original Message- From: ext Andrew Beekhof [mailto:and...@beekhof.net] Sent: Thursday, January 29, 2015 1:51 PM To: Yan, Xiaoping (NSN - CN/Hangzhou) Cc: Linux-HA-Dev@lists.linux-ha.org Subject: Re: quorum status Did you shut down pacemaker or corosync or both? On 29 Jan 2015, at 4:18 pm, Yan, Xiaoping (NSN - CN/Hangzhou) xiaoping@nsn.com wrote: Hi, Any suggestion please? Br, Rip _ From: Yan, Xiaoping (NSN - CN/Hangzhou) Sent: Wednesday, January 28, 2015 4:04 PM To: Linux-HA-Dev@lists.linux-ha.org Subject: quorum status Hi experts: I’m making 2 node (rhel1 and rhel2) Linux cluster following Pacemaker-1.1-Clusters_from_Scratch-en-US After I shutdown one of the node (pcs cluster stop rhel1) , the other partition still have quorum. While according to document, chapter 5.3.1 ,it should be partition without quorum. (A cluster is said to have quorum when total_nodes2*active_nodes) What could be the problem? Thank you. [root@rhel2 ~]# pcs status Cluster name: mycluster Last updated: Wed Jan 28 10:49:46 2015 Last change: Wed Jan 28 10:07:10 2015 via cibadmin on rhel1 Stack: corosync Current DC: rhel2 (2) - partition with quorum Version: 1.1.10-29.el7-368c726 2 Nodes configured 1 Resources configured Online: [ rhel2 ] OFFLINE: [ rhel1 ] Full list of resources: ClusterIP (ocf::heartbeat:IPaddr2): Started rhel2 PCSD Status: rhel1: Unable to authenticate rhel2: Unable to authenticate Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled [root@rhel2 ~]# pcs status corosync Membership information -- Nodeid Votes Name 2 1 rhel2 (local) Br, Rip ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] quorum status
Did you shut down pacemaker or corosync or both? On 29 Jan 2015, at 4:18 pm, Yan, Xiaoping (NSN - CN/Hangzhou) xiaoping@nsn.com wrote: Hi, Any suggestion please? Br, Rip _ From: Yan, Xiaoping (NSN - CN/Hangzhou) Sent: Wednesday, January 28, 2015 4:04 PM To: Linux-HA-Dev@lists.linux-ha.org Subject: quorum status Hi experts: I’m making 2 node (rhel1 and rhel2) Linux cluster following Pacemaker-1.1-Clusters_from_Scratch-en-US After I shutdown one of the node (pcs cluster stop rhel1) , the other partition still have quorum. While according to document, chapter 5.3.1 ,it should be partition without quorum. (A cluster is said to have quorum when total_nodes2*active_nodes) What could be the problem? Thank you. [root@rhel2 ~]# pcs status Cluster name: mycluster Last updated: Wed Jan 28 10:49:46 2015 Last change: Wed Jan 28 10:07:10 2015 via cibadmin on rhel1 Stack: corosync Current DC: rhel2 (2) - partition with quorum Version: 1.1.10-29.el7-368c726 2 Nodes configured 1 Resources configured Online: [ rhel2 ] OFFLINE: [ rhel1 ] Full list of resources: ClusterIP (ocf::heartbeat:IPaddr2): Started rhel2 PCSD Status: rhel1: Unable to authenticate rhel2: Unable to authenticate Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled [root@rhel2 ~]# pcs status corosync Membership information -- Nodeid Votes Name 2 1 rhel2 (local) Br, Rip ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-HA] SLES11 SP3: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321
On 21 Jan 2015, at 3:38 am, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! When a SLES11SP3 node joined a 3-node cluster after reboot (and preceeding update), a node with up-to-date software showed these messages (I feel these should not appear): Jan 20 17:12:38 h10 corosync[13220]: [MAIN ] Completed service synchronization, ready to provide service. Jan 20 17:12:38 h10 cib[13257]: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321 Jan 20 17:12:38 h10 cib[13257]: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321 Jan 20 17:12:38 h10 cib[13257]: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321 We fixed this upstream a little while back. I think the fix even came from suse. ### So why may not the same nodes have the same nodeid? Jan 20 17:12:38 h10 attrd[13260]: warning: crm_dump_peer_hash: crm_find_peer: Node 84939948/h05 = 0x61ae90 - b6cabbb3-8332-4903-85be-0c06272755ac Jan 20 17:12:38 h10 attrd[13260]: warning: crm_dump_peer_hash: crm_find_peer: Node 17831084/h01 = 0x61e300 - 11693f38-8125-45f2-b397-86136d5894a4 Jan 20 17:12:38 h10 attrd[13260]: warning: crm_dump_peer_hash: crm_find_peer: Node 739512330/h10 = 0x614400 - 302e33d8-7cee-4f3b-97da-b38f0d51b0f6 ### above are the three nodes of the cluster Jan 20 17:12:38 h10 attrd[13260]: crit: crm_find_peer: Node 739512321 and 17831084 share the same name 'h01' ### Now there are different nodeids it seems... Jan 20 17:12:38 h10 attrd[13260]: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321 Jan 20 17:12:38 h10 cib[13257]: warning: crm_find_peer: Node 'h01' and 'h01' share the same cluster nodeid: 739512321 ### The same again... (pacemaker-1.1.11-0.7.53, corosync-1.4.7-0.19.6) As a result the node h01 is offline now. Before updating the software the node was member of the cluster. On node h01 I see messages like these: cib[7439]: notice: get_node_name: Could not obtain a node name for classic openais (with plugin) nodeid 84939948 cib[7439]: notice: crm_update_peer_state: plugin_handle_membership: Node (null)[84939948] - state is now member (was (null)) cib[7439]: notice: get_node_name: Could not obtain a node name for classic openais (with plugin) nodeid 739512330 cib[7439]: notice: crm_update_peer_state: plugin_handle_membership: Node (null)[739512330] - state is now member (was (null)) crmd[7444]: warning: crmd_cs_dispatch: Receiving messages from a node we think is dead: rksaph05[84939948] crmd[7444]: notice: get_node_name: Could not obtain a node name for classic openais (with plugin) nodeid 739512330 corosync[7402]: [MAIN ] Completed service synchronization, ready to provide service. crmd[7444]: notice: do_state_transition: State transition S_PENDING - S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ] An attempt to restart openais did hang with this messages: attrd[7442]: notice: attrd_perform_update: Sent update 7: shutdown=1421771193 corosync[7402]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=7444, seq=6) to terminate... [message repeats] So I killed crmd (pid 7444)m and openais shut down. Unfortunately the problem still persists... Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Support for DRDB
On 18 Jan 2015, at 3:45 am, Lars Marowsky-Bree l...@suse.com wrote: On 2015-01-16T16:25:15, EXTERNAL Konold Martin (erfrakon, RtP2/TEF72) external.martin.kon...@de.bosch.com wrote: I am glad to hear that SLE HA has no plans to drop support for DRBD. Unfortunately I currently cannot disclose who is spreading this false information. Too bad. Do let them know they're quite wrong though ;-) And ask them to stop repeating it. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Support for DRDB
On 17 Jan 2015, at 4:19 am, Digimer li...@alteeve.ca wrote: On 16/01/15 10:43 AM, Dmitri Maziuk wrote: On 1/16/2015 8:39 AM, Lars Marowsky-Bree wrote: On 2015-01-16T11:56:04, EXTERNAL Konold Martin (erfrakon, RtP2/TEF72) external.martin.kon...@de.bosch.com wrote: I have been told that support for DRBD is supposed to be phased out from both SLES and RHEL in the near future. This is massively incorrect for SLE HA. (drbd is part of the HA add-on, not SLES.) We have absolutely no such plans, and will continue to support drbd as part of our offerings. Where did you hear that? Don't know about RHEL but on Centos you get DRBD from ELRepo, not even EPEL, so I expect it has not been in the RHEL for quite some time. Unless RedHat's offering it as a separate add-on to their paying customers (but the last rumour I heard was they wouldn't do it 'cause they didn't want to share the loot with linbit). Dima When RHEL 6 was released, Red Hat wanted to reduce their support overhead a lot. So many things that used to be supported were dropped. To my knowledge, DRBD was never actually supported. Which would mean Red Hat's co-operation with Linbit has only ever increased over time - something I can't imagine changing. DRBD, unlike most other dropped programs, is still supported, just not by RH directly. They worked out an agreement with LINBIT to allow officially supported RHEL systems to be fully supported when they ran DRBD. The agreement was that RHEL users who needed DRBD support specifically would be routed to LINBIT for that support. So though it's changed, it is still supported. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Monitor a Pacemaker Cluster with ocf:pacemaker:ClusterMon and/or external-agent
On 26 Nov 2014, at 4:15 pm, Ranjan Gajare ranjangajar...@gmail.com wrote: I have CentOs 9.3 9.3? that would be impressive can you tell me the version of pacemaker you have installed? i suspect you're lacking this patch: https://github.com/beekhof/pacemaker/commit/3df6aff installed and using two VMs. That is I have two nodes HA configuration. I configured one node as master and another as standby. I have created three resources like vip-master, vip-rep and pgsql and again creating resource as in following command. I want to configure HA failover notification. When failover happen I want notification about fail node and new master. Once I execute following command pcs resource create test ClusterMon ocf:pacemaker:ClusterMon params user=root update=30 extra_options=-E /usr/local/bin/test.sh op monitor on-fail=restart interval=10 clone test-clone ClusterMon meta target-role=Started Then I did # cibadmin --query tmp.xml then check with tmp.xml. It has entries for whatever resources created for HA along with test resource created in above command. Now I am using test.sh as external agent as echo $CRM_notify_rc echo $CRM_notify_task echo $CRM_notify_recipient echo $CRM_notify_node echo $CRM_notify_rsc echo $CRM_notify_desc echo $CRM_notify_status echo $CRM_notify_target_rc After that I shut down cluster and start it again but no such notification occur. I think test.sh is not get trigger. As per my knowledge ocf:pacemaker:ClusterMon resource runs crm_mon at background. Then why environment variables from test.sh not able to read cluster status from crm_mon output. I have also set permission to all for test.sh Thank You, Ranjan -- View this message in context: http://linux-ha.996297.n3.nabble.com/Monitor-a-Pacemaker-Cluster-with-ocf-pacemaker-ClusterMon-and-or-external-agent-tp15951p15973.html Sent from the Linux-HA mailing list archive at Nabble.com. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: Avoid resource restart after configuration change
On 3 Dec 2014, at 6:55 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 03.12.2014 um 06:45 in Nachricht 6da1e43b-b83a-4441-9fb0-88bfe409d...@beekhof.net: On 1 Dec 2014, at 7:46 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I'd like to change a resource's configuration, but don't want a restart of the resource. That is, I want the configuration to be effective the next time the resource is started. I know in general it doesn't make sense, but for cLVM it does: Per default debug logging is on and it floods the syslog with junk. If I want to turn off debug logging, I'll have to pass an extra parameter to clvmd, or I could do clvmd -d0 -C to fix the problem until next resource start. As cLVMd is clones to all nodes in the cluster, a configuration change would effectively halt the whole cluster (all users of cLVM). I want to avoid that. As you can see, I can change the running resources, and I can configure the resources to be fine next time they restart. So all I need is way to prevent a restart of clvmd when changing its resource configuration. You could make it in a scratch copy of the cib, note the new digests in the status section and include those in your update. Could you provide some more details, please? cibadmin -Q old.xml cibadmin -Q new.xml CIB_file=new.xml cibadmin -M --xml-text 'whatever you want to change/' crm_simulate -SX new.xml compare old.xml to new.xml looking for new values for op restart-digest=.../ and op reload-digest=.../ now change the real cluster and include changes to restart-digest and reload-digest Alternatively, use maintenance-mode and crm_resource -C I don't understand: in maintenance mode I cannot start/stop anything, right? You don't need to. Just make the change, then use crm_resource -C to erase the status section, the probes will run with the new set of parameters. At that point, it should be safe to turn maintenance mode off again. So the changes to the CIB will just be delayed until management mode is off again, right? What is the role of crm_resource -C? Does cleanup make CRM forget that it has to restart resources when the management mode is turned off again? Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Q: Avoid resource restart after configuration change
On 1 Dec 2014, at 7:46 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I'd like to change a resource's configuration, but don't want a restart of the resource. That is, I want the configuration to be effective the next time the resource is started. I know in general it doesn't make sense, but for cLVM it does: Per default debug logging is on and it floods the syslog with junk. If I want to turn off debug logging, I'll have to pass an extra parameter to clvmd, or I could do clvmd -d0 -C to fix the problem until next resource start. As cLVMd is clones to all nodes in the cluster, a configuration change would effectively halt the whole cluster (all users of cLVM). I want to avoid that. As you can see, I can change the running resources, and I can configure the resources to be fine next time they restart. So all I need is way to prevent a restart of clvmd when changing its resource configuration. You could make it in a scratch copy of the cib, note the new digests in the status section and include those in your update. Alternatively, use maintenance-mode and crm_resource -C ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Cluster-devel] [ha-wg] [RFC] Organizing HA Summit 2015
On 25 Nov 2014, at 8:54 pm, Lars Marowsky-Bree l...@suse.com wrote: On 2014-11-24T16:16:05, Fabio M. Di Nitto fdini...@redhat.com wrote: Yeah, well, devconf.cz is not such an interesting event for those who do not wear the fedora ;-) That would be the perfect opportunity for you to convert users to Suse ;) I´d prefer, at least for this round, to keep dates/location and explore the option to allow people to join remotely. Afterall there are tons of tools between google hangouts and others that would allow that. That is, in my experience, the absolute worst. It creates second class participants and is a PITA for everyone. I agree, it is still a way for people to join in tho. I personally disagree. In my experience, one either does a face-to-face meeting, or a virtual one that puts everyone on the same footing. Mixing both works really badly unless the team already knows each other. I know that an in-person meeting is useful, but we have a large team in Beijing, the US, Tasmania (OK, one crazy guy), various countries in Europe etc. Yes same here. No difference.. we have one crazy guy in Australia.. Yeah, but you're already bringing him for your personal conference. That's a bit different. ;-) OK, let's switch tracks a bit. What *topics* do we actually have? Can we fill two days? Where would we want to collect them? Personally I'm interested in talking about scaling - with pacemaker-remoted and/or a new messaging/membership layer. Other design-y topics: - SBD - degraded mode - improved notifications - containerisation of services (cgroups, docker, virt) - resource-agents (upstream releases, handling of pull requests, testing) User-facing topics could include recent features (ie. pacemaker-remoted, crm_resource --restart) and common deployment scenarios (eg. NFS) that people get wrong. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [ha-wg-technical] [Cluster-devel] [ha-wg] [RFC] Organizing HA Summit 2015
On 26 Nov 2014, at 10:06 am, Digimer li...@alteeve.ca wrote: On 25/11/14 04:31 PM, Andrew Beekhof wrote: Yeah, but you're already bringing him for your personal conference. That's a bit different. ;-) OK, let's switch tracks a bit. What *topics* do we actually have? Can we fill two days? Where would we want to collect them? Personally I'm interested in talking about scaling - with pacemaker-remoted and/or a new messaging/membership layer. Other design-y topics: - SBD - degraded mode - improved notifications This my be something my company can bring to the table. We just hired a dev whose principle goal is to develop and alert system for HA. We're modelling it heavily on the fence/resource agent model with a scan core and scan agents. It's sort of like existing tools, but designed specifically for HA clusters and heavily focused on not interfering with the host more than at all necessary. By Feb., it should be mostly done. We're doing this for our own needs, but it might be a framework worth talking about, if nothing else to see if others consider it a fit. Of course, it will be entirely open source. *If* there is interest, I could put together a(n informal) talk on it with a demo. Definitely interesting - containerisation of services (cgroups, docker, virt) - resource-agents (upstream releases, handling of pull requests, testing) User-facing topics could include recent features (ie. pacemaker-remoted, crm_resource --restart) and common deployment scenarios (eg. NFS) that people get wrong. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [ha-wg-technical] [ha-wg] [RFC] Organizing HA Summit 2015
On 26 Nov 2014, at 4:51 pm, Fabio M. Di Nitto fabbi...@fabbione.net wrote: On 11/25/2014 10:54 AM, Lars Marowsky-Bree wrote: On 2014-11-24T16:16:05, Fabio M. Di Nitto fdini...@redhat.com wrote: Yeah, well, devconf.cz is not such an interesting event for those who do not wear the fedora ;-) That would be the perfect opportunity for you to convert users to Suse ;) I´d prefer, at least for this round, to keep dates/location and explore the option to allow people to join remotely. Afterall there are tons of tools between google hangouts and others that would allow that. That is, in my experience, the absolute worst. It creates second class participants and is a PITA for everyone. I agree, it is still a way for people to join in tho. I personally disagree. In my experience, one either does a face-to-face meeting, or a virtual one that puts everyone on the same footing. Mixing both works really badly unless the team already knows each other. I know that an in-person meeting is useful, but we have a large team in Beijing, the US, Tasmania (OK, one crazy guy), various countries in Europe etc. Yes same here. No difference.. we have one crazy guy in Australia.. Yeah, but you're already bringing him for your personal conference. That's a bit different. ;-) OK, let's switch tracks a bit. What *topics* do we actually have? Can we fill two days? Where would we want to collect them? I´d say either a google doc or any random etherpad/wiki instance will do just fine. -ENOGOOGLE As for the topics: - corosync qdevice and plugins (network, disk, integration with sdb?, others?) - corosync RRP / libknet integration/replacement - fence autodetection/autoconfiguration For the user facing topics (that is if there are enough participants and I only got 1 user confirmation so far): - demos, cluster 101, tutorials - get feedback - get feedback - get more feedback Fabio ___ ha-wg-technical mailing list ha-wg-techni...@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/ha-wg-technical ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] time_longclock illumos
On 17 Nov 2014, at 5:17 am, Randy S sim@live.nl wrote: Hi all, new user here. We have been testing an older version of the heartbeat / pacemaker combination compiled for illumos (an opensolaris follow-up). Versions: Heartbeat-3-0-STABLE-3.0.5 Pacemaker-1-0-Pacemaker-1.0.11 The 1.0.x series is now long departed. Best to look into a recent 1.1.x build It all works ok while testing (several months now) but I have noticed that every so often (and sometimes quite frequently) I see the following console message appear: crmd: [ID 996084 daemon.crit] [12637]: CRIT: time_longclock: old value was 298671305, new value is 298671304, diff is 1, callcount 141814 Now from what I have been able to find about this, is that this type of occurence should have been fixed in heartbeat post 2.1.4 versions. At that time this occurence could make a cluster start behaving irratically. We have two test implementions of a cluster, 1 in vmware and 1 on standard hardware. All just for testing. We have made sure that timesync is done via ntp with the internet. The hardware implementation doesn't show this message as many times as the vmware implementation, but still it appears (sometimes about three times per 24 hours). We haven't had any strange behaviour yet in the cluster, but my questions about this are as follows: should we worry about this 'time_longclock' crit error eventhough it should have been fixed in version post HA 3? Is there something (simple) that can be done to prevent this type of error, or should we expect normal cluster behaviour since ntp is used. The above message should make clear that I'm not a programmer, nor am I a heartbeat specialist ;-) Regards, S. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] New user can't get cman to recognize other systems
On 22 Oct 2014, at 7:36 am, jayknowsu...@gmail.com wrote: Yep, my network engineer and I found that the multicast packets were being blocked by the underlying hypervisor for the VM systems. Yeah, that'll happen :-( I believe its fixed in newer kernels, but for a while there multicast would appear to work and then stop for no good reason. Putting the device into promiscuous mode seemed to help IIRC. This is the bug I knew it as: https://bugzilla.redhat.com/show_bug.cgi?id=1090670 At first we thought it was just iptables on the servers, but i was certain I had actually turned that off. The issue has been bumped up to the operations team for a fixing this, but since I've gotten it to work with unicast, there's no pressure Sent from my iPad On Oct 21, 2014, at 3:15 PM, Digimer li...@alteeve.ca wrote: Glad you sorted it out! So then, it was almost certainly a multicast issue. I would still strongly recommend trying to source and fix the problem, and reverting to mcast if you can. More efficient. :) digimer On 21/10/14 02:59 PM, John Scalia wrote: Ok, got it working after a little more effort, and the cluster is now properly reporting. On Tue, Oct 21, 2014 at 1:34 PM, John Scalia jayknowsu...@gmail.com wrote: So, I set transport=udpi' in the cluster.conf file, and it now looks like this: cluster config_version=11 name=pgdb_cluster transport=udpu fence_daemon/ clusternodes clusternode name=csgha1 nodeid=1 fence method name=pcmk-redirect device name=pcmk port=csgha1/ /method /fence /clusternode clusternode name=csgha2 nodeid=2 fence method name=pcmk-redirect device name=pcmk port=csgha2/ /method /fence /clusternode clusternode name=csgha3 nodeid=3 fence method name=pcmk-redirect device name=pcmk port=csgha3/ /method /fence /clusternode /clusternodes cman/ fencedevices fencedevice agent=fence_pcmk name=pcmk/ /fencedevices rm failoverdomains/ resources/ /rm /cluster But, after restarting the cluster I don't see any difference. Did I do something wrong? -- Jay On Tue, Oct 21, 2014 at 12:25 PM, Digimer li...@alteeve.ca wrote: No, you don't need to specify anything in cluster.conf for unicast to work. Corosync will divine the IPs by resolving the node names to IPs. If you set multicast and don't want to use the auto-selected mcast IP, then you can specify the mcast IP group to use via multicast... /. digimer On 21/10/14 12:22 PM, John Scalia wrote: OK, looking at the cman man page on this system, I see the line saying the corosync.conf file is not used. So, I'm guessing I need to set a unicast address somewhere in the cluster.conf file, but the man page only mentions the multicast addr=.../ parameter. What can I use to set this to a unicast address for ports 5404 and 5405? I'm assuming I can't just put a unicast address for the multicast parameter, and the man page for cluster.conf wasn't much help either. We're still working on having the security team permit these 3 systems to use multicast. On 10/21/2014 11:51 AM, Digimer wrote: Keep us posted. :) On 21/10/14 08:40 AM, John Scalia wrote: I've been check hostname resolution this morning, and all the systems are listed in each /etc/hosts file (No DNS in this environment.) and ping works on every system both to itself and all the other systems. At least it's working on the 10.10.1.0/24 network. I ran tcpdump trying to see what traffic is on port 5405 on each system, and I'm only seeing outbound on each, even though netstat shows each is listening on the multicast address. My suspicion is that the router is eating the multicast broadcasts, so I may try the unicast address instead, but I'm waiting on one of our network engineers to see if my suspicion is correct about the router. He volunteered to help late yesterday. On 10/20/2014 4:34 PM, Digimer wrote: It looks sane on the surface. The 'gethostip' tool comes from the 'syslinux' package, and it's really handy! The '-d' says to give the IP in dotted-decimanl notation only. What I was trying to see was whether the 'uname -n' resolved to the IP on the same network card as the other nodes. This is how corosync decides which interface to send cluster traffic onto. I suspect you might have a general network issue, possibly related to multicast. (Some switches and some hypervisor virtual networks don't play nice with corosync). Have you tried unicast? If not, try setting the cman ../ element to have the cman transport=udpu ... / attribute. Do note that unicast isn't as efficient as multicast, so thought it might work, I'd personally treat it as a debug tool to isolate the source of the problem. cheers digimer PS - Can you share your pacemaker configuration? On 20/10/14 03:40 PM, John Scalia wrote: Sure,
Re: [Linux-HA] New user can't get cman to recognize other systems
On 22 Oct 2014, at 9:16 am, Digimer li...@alteeve.ca wrote: Blocked for me, too. Possible to clone - client data? Needless paranoia more likely. This is the original fedora bug (nothing marked private): https://bugzilla.redhat.com/show_bug.cgi?id=880035 and the kbase: https://access.redhat.com/solutions/784373 On 21/10/14 06:14 PM, jayknowsu...@gmail.com wrote: Sure! But i can't seem to get Redhat to let me see the bug, even though I have an account. Sent from my iPad On Oct 21, 2014, at 5:51 PM, Andrew Beekhof and...@beekhof.net wrote: On 22 Oct 2014, at 7:36 am, jayknowsu...@gmail.com wrote: Yep, my network engineer and I found that the multicast packets were being blocked by the underlying hypervisor for the VM systems. Yeah, that'll happen :-( I believe its fixed in newer kernels, but for a while there multicast would appear to work and then stop for no good reason. Putting the device into promiscuous mode seemed to help IIRC. This is the bug I knew it as: https://bugzilla.redhat.com/show_bug.cgi?id=1090670 At first we thought it was just iptables on the servers, but i was certain I had actually turned that off. The issue has been bumped up to the operations team for a fixing this, but since I've gotten it to work with unicast, there's no pressure Sent from my iPad On Oct 21, 2014, at 3:15 PM, Digimer li...@alteeve.ca wrote: Glad you sorted it out! So then, it was almost certainly a multicast issue. I would still strongly recommend trying to source and fix the problem, and reverting to mcast if you can. More efficient. :) digimer On 21/10/14 02:59 PM, John Scalia wrote: Ok, got it working after a little more effort, and the cluster is now properly reporting. On Tue, Oct 21, 2014 at 1:34 PM, John Scalia jayknowsu...@gmail.com wrote: So, I set transport=udpi' in the cluster.conf file, and it now looks like this: cluster config_version=11 name=pgdb_cluster transport=udpu fence_daemon/ clusternodes clusternode name=csgha1 nodeid=1 fence method name=pcmk-redirect device name=pcmk port=csgha1/ /method /fence /clusternode clusternode name=csgha2 nodeid=2 fence method name=pcmk-redirect device name=pcmk port=csgha2/ /method /fence /clusternode clusternode name=csgha3 nodeid=3 fence method name=pcmk-redirect device name=pcmk port=csgha3/ /method /fence /clusternode /clusternodes cman/ fencedevices fencedevice agent=fence_pcmk name=pcmk/ /fencedevices rm failoverdomains/ resources/ /rm /cluster But, after restarting the cluster I don't see any difference. Did I do something wrong? -- Jay On Tue, Oct 21, 2014 at 12:25 PM, Digimer li...@alteeve.ca wrote: No, you don't need to specify anything in cluster.conf for unicast to work. Corosync will divine the IPs by resolving the node names to IPs. If you set multicast and don't want to use the auto-selected mcast IP, then you can specify the mcast IP group to use via multicast... /. digimer On 21/10/14 12:22 PM, John Scalia wrote: OK, looking at the cman man page on this system, I see the line saying the corosync.conf file is not used. So, I'm guessing I need to set a unicast address somewhere in the cluster.conf file, but the man page only mentions the multicast addr=.../ parameter. What can I use to set this to a unicast address for ports 5404 and 5405? I'm assuming I can't just put a unicast address for the multicast parameter, and the man page for cluster.conf wasn't much help either. We're still working on having the security team permit these 3 systems to use multicast. On 10/21/2014 11:51 AM, Digimer wrote: Keep us posted. :) On 21/10/14 08:40 AM, John Scalia wrote: I've been check hostname resolution this morning, and all the systems are listed in each /etc/hosts file (No DNS in this environment.) and ping works on every system both to itself and all the other systems. At least it's working on the 10.10.1.0/24 network. I ran tcpdump trying to see what traffic is on port 5405 on each system, and I'm only seeing outbound on each, even though netstat shows each is listening on the multicast address. My suspicion is that the router is eating the multicast broadcasts, so I may try the unicast address instead, but I'm waiting on one of our network engineers to see if my suspicion is correct about the router. He volunteered to help late yesterday. On 10/20/2014 4:34 PM, Digimer wrote: It looks sane on the surface. The 'gethostip' tool comes from the 'syslinux' package, and it's really handy! The '-d' says to give the IP in dotted-decimanl notation only. What I was trying to see was whether the 'uname -n' resolved to the IP on the same network card as the other nodes. This is how corosync decides which interface to send cluster traffic onto. I
Re: [Linux-HA] Configuring corosync on a CentOS 6.5
On 22 Oct 2014, at 2:15 am, John Scalia jayknowsu...@gmail.com wrote: Hi all, again, My network engineer and I have found that the VM's hypervisor was set up to block multicast broadcasts by our security team. Blocked or lost? These links might be worth a look: https://access.redhat.com/solutions/784373 https://bugzilla.redhat.com/show_bug.cgi?id=880035 We're not really certain why or if we can change that for at least my 3 systems. He's speaking with them now. Anyway, as you don't have to configure corosync on CentOS or Redhat, and there isn't even an /etc/corosync/corosync.conf on these systems, what problems could I cause by creating a config file and would the system actually use it on a restart? I want to try setting the multicast address to a unicast one, at least for testing. This whole setup seems a little odd since CentOS uses CMAN and pacemaker, but corosync is getting started and I see all the systems listening on port 5404 and 5405 similar to as follows: udp00 10.10.1.129:54040.0.0.0:* udp00 10.10.1.129:54050.0.0.0:* udp00 239.192.143.91:5405 0.0.0.0* So, if CentOS uses CMAN and pacemaker, why is corosync still in the mix? CMAN == corosync + some magic that gets loaded as a plugin -- Jay ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] New user can't get cman to recognize other systems
On 21 Oct 2014, at 7:17 am, John Scalia jayknowsu...@gmail.com wrote: Thanks, but on centOS are you saying to use pcs cluster start rather than using service cman start and service pacemaker start? I was just going by the tutorial, which doesn't mention this. 'service pacemaker start' and 'pcs cluster start' are pretty much equivalent. both will start cman if its not running already On 10/20/2014 3:44 PM, Maciej Rostański wrote: Hello, In my experience such problems were the effect of my mistakes, such as not having all hosts in /etc/hosts file. Check this, please, I know it sounds simple. Also, commands: pcs cluster setup --name clustername node1 node2 node3 pcs cluster enable pcs cluster start are much more pleasant to run than ccs method you use, and they work on Centos6.5 Regards, Maciej 2014-10-20 20:50 GMT+02:00 John Scalia jayknowsu...@gmail.com: Hi all, I'm trying to build my first ever HA cluster and I'm using 3 VMs running CentOS 6.5. I followed the instructions to the letter at: http://clusterlabs.org/quickstart-redhat.html and everything appears to start normally, but if I run cman_tool nodes -a, I only see: Node StsInc Joined Name 1 M 64 2014-10--20 14:00:00 csgha1 Addresses: 10.10.1.128 2 X 0 csgha2 3 X 0 csgha3 In the other systems, the output is the same except for which system is shown as joined. Each shows just itself as belonging to the cluster. Also, pcs status reflects similarly with non-self systems showing offline. I've checked netstat -an and see each machine listening on ports 5405 and 5405. And the logs are rather involved, but I'm not seeing errors in it. Any ideas for where to look for what's causing them to not communicate? -- Jay ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Two node Pacemaker with one Corosync only quorum node
On 3 Sep 2014, at 9:29 am, Brian Campbell brian.campb...@editshare.com wrote: I'm wondering if there are any problems that would occur if you ran a cluster with only two nodes running Pacemaker, but add a third Corosync only node to provide quorum. I tried this setup, and it appears to work fine after some brief testing; I configured Corosync and votequorum appropriately on all three nodes, but only ever started Pacemaker on two of them. After enabling no-quorum-policy=stop, if I disconnected one of the nodes it would stop itself and the other would take over like I expect, rather than both nodes trying to promote themselves as occurs when there are only two nodes and no-quorum-policy=ignore (for the purposes of debugging and development, I don't have stonith enabled in order to make it easier to monitor what's going on at each node, without my connection dropping due to rebooting the machine). I'm now wondering if there will be any problems I haven't anticipated with this setup, or anything I should look out for. Its supposed to work and I've not heard anyone complain lately that it doesn't (possibly because no-one tried recently). If you encounter any problems, definitely let us know. Of course, other options would involve having the third node simply running Pacemaker but permanently in standby, or making it an asymmetric cluster and only allowing any resources to run on the first two nodes. But I'm curious if it's possible to go the simplest possible route and just have corosync running on a third quorum node; or possibly even more. Our setup has a couple of master nodes with large amounts of RAM so that all of the metadata can fit into RAM, and then a number of cheap storage nodes to store the actual bulk data. Because we have the cheap storage nodes, we have a number of machines we can run as quorum-only nodes, but don't want to ever accidentally select them as a master or slave node. Thanks, Brian (as an aside, I'm wondering if this is the right list for this question, or if I should be asking it on the Pacemaker list instead; I haven't quite figured out where questions about interactions between Pacemaker and Corosync should be asked) ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: dampening explained?
On 10 Sep 2014, at 6:43 pm, Lars Ellenberg lars.ellenb...@linbit.com wrote: On Wed, Sep 10, 2014 at 03:00:15PM +1000, Andrew Beekhof wrote: What are the precise semantics of dampening (attrd_updater -d)? Basic idea of attr_updater -d delay is in fact wait for the dust to settle, especially for attribute based constraints, as used with the ping RA. Current behaviour (afai understand it): If a new update comes in while a previous update is still pending, the timer of the new update is *ignored*, the new value is queued instead, and the original timer is restarted. I think thats what I wrote but longer :) Absolutely :) (UNLESS the new value is the same as the currently pending value. Otherwise values would never reach the cib if the update interval is shorter than the dampening interval.) Value is put into the cib only if the original timer actually expires, or attrd flushes out all current values for some other reason. - Problem: for flapping or otherwise continually changing values, new values won't make it into the cib for a long time. Do we have many attributes like that? I've seen it once for ping attributes with a dieing switch. I guess that qualifies as Not around here.. That would surely converge on 0 though right? Workaround: use a dampening interval shorter than the update interval. Problem with that workaround: you may still hit the same undesired situations you could reach with immediately updating the values, you did not wait for the dust to settle. Proposal: add a update deadline along with the dampening, which would normally be sufficiently larger, and count from the original update (this timer would not be restarted). Optionally feed all updates into an aggregate function. Default aggregate function may be latest value. Once the update deadline is reached, (submitted values still changing; flapping? ...) write out the current aggregate value anyways. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: dampening explained?
On 9 Sep 2014, at 4:11 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 09.09.2014 um 00:25 in Nachricht c19365b9-b626-471e-a92a-001950d01...@beekhof.net: On 8 Sep 2014, at 5:19 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I remember having asked this before, but I'l still missing a good explanation: What are the precise semantics of dampening (attrd_updater -d)? The manual page just says: -d, --delay=value The time to wait (dampening) in seconds further changes occur Who is waiting? attrd before it updates the cib So the RA sees an extra delay of -d seconds? No. attrd does the waiting, not attrd_updater What changes? other processes or nodes calling attrd_updater So it means that if you call attr_updater -d X and then attr_updater -d Y the second update will be delayed until at least X seconds since the first update passed? No. It means wait X seconds for another update (locally or on another node) before writing the value to the CIB. If another update does arrive, reset the clock and wait another X seconds. Repeat. Usually only the -d from the first call to attrd_updater has any affect. Subsequent calls re-use the -d value from the first. If a third attr_updater -d Z is called, will the system ensure that at least X + Y seconds passed since the first update before appying the third update? No. If the delay is not over, does the next update wait (as I guessed), or is the next update simply ignored? Please explain! As you see, there are too many questions that are still unanswered. (pacemaker-1.1.10-0.15.25 of SLES11 SP3) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: dampening explained?
On 9 Sep 2014, at 11:12 pm, Lars Ellenberg lars.ellenb...@linbit.com wrote: On Tue, Sep 09, 2014 at 05:10:33PM +1000, Andrew Beekhof wrote: On 9 Sep 2014, at 4:11 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 09.09.2014 um 00:25 in Nachricht c19365b9-b626-471e-a92a-001950d01...@beekhof.net: On 8 Sep 2014, at 5:19 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I remember having asked this before, but I'l still missing a good explanation: What are the precise semantics of dampening (attrd_updater -d)? The manual page just says: -d, --delay=value The time to wait (dampening) in seconds further changes occur Who is waiting? attrd before it updates the cib So the RA sees an extra delay of -d seconds? No. attrd does the waiting, not attrd_updater What changes? other processes or nodes calling attrd_updater So it means that if you call attr_updater -d X and then attr_updater -d Y the second update will be delayed until at least X seconds since the first update passed? No. It means wait X seconds for another update (locally or on another node) before writing the value to the CIB. If another update does arrive, reset the clock and wait another X seconds. Repeat. Usually only the -d from the first call to attrd_updater has any affect. Subsequent calls re-use the -d value from the first. If a third attr_updater -d Z is called, will the system ensure that at least X + Y seconds passed since the first update before appying the third update? No. Some of this has been discussed before: http://www.gossamer-threads.com/lists/linuxha/dev/75740?do=post_view_threaded#75740 Basic idea of attr_updater -d delay is in fact wait for the dust to settle, especially for attribute based constraints, as used with the ping RA. Current behaviour (afai understand it): If a new update comes in while a previous update is still pending, the timer of the new update is *ignored*, the new value is queued instead, and the original timer is restarted. I think thats what I wrote but longer :) (UNLESS the new value is the same as the currently pending value. Otherwise values would never reach the cib if the update interval is shorter than the dampening interval.) Value is put into the cib only if the original timer actually expires, or attrd flushes out all current values for some other reason. - Problem: for flapping or otherwise continually changing values, new values won't make it into the cib for a long time. Do we have many attributes like that? Workaround: use a dampening interval shorter than the update interval. Problem with that workaround: you may still hit the same undesired situations you could reach with immediately updating the values, you did not wait for the dust to settle. Proposal: add a update deadline along with the dampening, which would normally be sufficiently larger, and count from the original update (this timer would not be restarted). Optionally feed all updates into an aggregate function. Default aggregate function may be latest value. Once the update deadline is reached, (submitted values still changing; flapping? ...) write out the current aggregate value anyways. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Q: dampening explained?
On 8 Sep 2014, at 5:19 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I remember having asked this before, but I'l still missing a good explanation: What are the precise semantics of dampening (attrd_updater -d)? The manual page just says: -d, --delay=value The time to wait (dampening) in seconds further changes occur Who is waiting? attrd before it updates the cib What changes? other processes or nodes calling attrd_updater Please explain! (pacemaker-1.1.10-0.15.25 of SLES11 SP3) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] unable to recover from split-brain in a two-node cluster
On 25 Jun 2014, at 12:03 am, Lars Ellenberg lars.ellenb...@linbit.com wrote: On Tue, Jun 24, 2014 at 12:23:30PM +1000, Andrew Beekhof wrote: On 24 Jun 2014, at 1:52 am, f...@vmware.com wrote: Hi, I understand that initially the split-brain is caused by heartbeat messaging layer and there is nothing much can be done when packets are dropped. However, the problem is sometimes when the load is gone (or when iptables allows all traffic in my test setup), it doesn't recover. In the second case I provided, the heartbeat on both nodes did find each other and both were active, but pacemaker in both nodes still thinks peer is offline. I don't know if this is heartbeat's problem or Pacemaker's problem though. Do you see any messages from 'crmd' saying the node left/returned? If you only see the node going away, then its almost certainly a heartbeat problem. You may have better luck with a corosync based cluster, or even a newer version of pacemaker (or both! the 1.0.x codebase is quite old at this point). I was never all that happy with heartbeat's membership code, it was a near-abandoned mystery box even at the point I started Pacemaker 10 years ago. Corosync membership had its problems in the beginning, but personally I take comfort in the fact that its actively being worked on. Opinions differ, but IMHO it surpassed heartbeat for reliability 3-4 years ago. Possibly. But especially with nodes unexpectedly returning after having been declared dead, I've still seen more problems with corosync than with heartbeat, even within the last few years. Unfortunately a fair share of those have also been pacemaker bugs :( Yan is working on another one related to slow fencing devices. Anyways: Andrew is right, you should use (recent!) corosync and recent pacemaker. And working node level fencing aka stonith. That said, you said earlier you are using heartbeat 3.0.5, and that heartbeat successfully re-established membership. So you can confirm ccm_testclient on both nodes reports the expected and same membership? Is that 3.0.5 release tag, or a more recent hg checkout? You need heartbeat up to at least this commit: http://hg.linux-ha.org/heartbeat-STABLE_3_0/rev/fd1b907a0de6 (I meant to add a 3.0.6 release tag since at least I pushed that commit, but because of packaging inconsistencies I want to fix, and other commitments, I deferred that much too long). -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] unable to recover from split-brain in a two-node cluster
On 24 Jun 2014, at 1:52 am, f...@vmware.com wrote: Hi, I understand that initially the split-brain is caused by heartbeat messaging layer and there is nothing much can be done when packets are dropped. However, the problem is sometimes when the load is gone (or when iptables allows all traffic in my test setup), it doesn't recover. In the second case I provided, the heartbeat on both nodes did find each other and both were active, but pacemaker in both nodes still thinks peer is offline. I don't know if this is heartbeat's problem or Pacemaker's problem though. Do you see any messages from 'crmd' saying the node left/returned? If you only see the node going away, then its almost certainly a heartbeat problem. You may have better luck with a corosync based cluster, or even a newer version of pacemaker (or both! the 1.0.x codebase is quite old at this point). I was never all that happy with heartbeat's membership code, it was a near-abandoned mystery box even at the point I started Pacemaker 10 years ago. Corosync membership had its problems in the beginning, but personally I take comfort in the fact that its actively being worked on. Opinions differ, but IMHO it surpassed heartbeat for reliability 3-4 years ago. Thanks, -Kaiwei - Original Message - From: Andrew Beekhof and...@beekhof.net To: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, June 22, 2014 3:45:00 PM Subject: Re: [Linux-HA] unable to recover from split-brain in a two-node cluster On 21 Jun 2014, at 5:18 am, f...@vmware.com wrote: Hi, New to this list and hope I can get some help here. I'm using pacemaker 1.0.10 and heartbeat 3.0.5 for a two-node cluster. I'm having split-brain problem when heartbeat messages sometimes get dropped when system is under high load. However the problem is it never recover back when system load became low. I created a test setup to test this by setting dead time to 6 seconds, and continuously dropping one-way heartbeat packets (udp dst port 694) for 5~8 seconds and resume the traffic for 1~2 seconds using iptables. After the system got into split-brain state, I stop the test and allow all heartbeat traffic to go through. Sometimes the system recovered by sometimes it didn't. There are various symptoms when the system didn't recovered from split-brain: 1. In one instance, cl_status listnodes becomes empty. The syslog keeps showing 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.warning] [2853]: WARN: Message hist queue is filling up (436 messages in queue) 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: hist-ackseq =12111 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: hist-lowseq =12111, hist-hiseq=12547 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: expecting from node-1 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: it's ackseq=12111 2. In another instance, cl_status nodestatus node shows both nodes are active, but crm_mon -1 shows that each of the two nodes thinks itself is the DC, and peer node is offline. Pengine process is running on one node only. The node not running pengine (but still thinks itself is DC) has log shows crmd terminated pengine because it detected peer is active. After that, the peer status keeps flapping between dead and active, but pengine has never being started again. The last log shows the peer is active (after I stopped the test and allow all traffic). However crm_mon -1 shows itself is the DC and peer is offline as: [root@node-1 ~]# crm_mon -1 Last updated: Fri Jun 20 19:12:23 2014 Stack: Heartbeat Current DC: node-1 (bf053fc5-5afd-b483-2ad2-3c9fc354f7fa) - partition with quorum Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, unknown expected votes 1 Resources configured. Online: [ node-1 ] OFFLINE: [ node-0 ] cluster (heartbeat:ha): Started node-1 Any help, like pointer to the source code where the problem might be, or any existing bug filed for this (I did some search but didn't find matched symptoms) is appreciated. This is happening at the heartbeat level. Not much pacemaker can do I'm afraid. Perhaps look to see if heartbeat is real time scheduled, if not that may explain why its being staved of CPU and can't get its messages out. Thanks, -Kaiwei ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org https://urldefense.proofpoint.com/v1/url?u=http://lists.linux-ha.org/mailman/listinfo/linux-hak=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=9IPI1z37RqWr21klX9jnPw%3D%3D%0Am=wkHclXpXSRj4jN%2FF6yECfGjmY217uMGOFlJmf%2B1H4FA%3D%0As=1816bc839d2eb1e28a3d00afaecf7d0ad1eb371fc314b0acf875b0c3e6c9add8 See also: https://urldefense.proofpoint.com/v1/url?u=http://linux-ha.org
Re: [Linux-HA] unable to recover from split-brain in a two-node cluster
On 21 Jun 2014, at 5:18 am, f...@vmware.com wrote: Hi, New to this list and hope I can get some help here. I'm using pacemaker 1.0.10 and heartbeat 3.0.5 for a two-node cluster. I'm having split-brain problem when heartbeat messages sometimes get dropped when system is under high load. However the problem is it never recover back when system load became low. I created a test setup to test this by setting dead time to 6 seconds, and continuously dropping one-way heartbeat packets (udp dst port 694) for 5~8 seconds and resume the traffic for 1~2 seconds using iptables. After the system got into split-brain state, I stop the test and allow all heartbeat traffic to go through. Sometimes the system recovered by sometimes it didn't. There are various symptoms when the system didn't recovered from split-brain: 1. In one instance, cl_status listnodes becomes empty. The syslog keeps showing 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.warning] [2853]: WARN: Message hist queue is filling up (436 messages in queue) 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: hist-ackseq =12111 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: hist-lowseq =12111, hist-hiseq=12547 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: expecting from node-1 2014-06-19T18:59:57+00:00 node-0 heartbeat: [daemon.debug] [2853]: debug: it's ackseq=12111 2. In another instance, cl_status nodestatus node shows both nodes are active, but crm_mon -1 shows that each of the two nodes thinks itself is the DC, and peer node is offline. Pengine process is running on one node only. The node not running pengine (but still thinks itself is DC) has log shows crmd terminated pengine because it detected peer is active. After that, the peer status keeps flapping between dead and active, but pengine has never being started again. The last log shows the peer is active (after I stopped the test and allow all traffic). However crm_mon -1 shows itself is the DC and peer is offline as: [root@node-1 ~]# crm_mon -1 Last updated: Fri Jun 20 19:12:23 2014 Stack: Heartbeat Current DC: node-1 (bf053fc5-5afd-b483-2ad2-3c9fc354f7fa) - partition with quorum Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, unknown expected votes 1 Resources configured. Online: [ node-1 ] OFFLINE: [ node-0 ] cluster (heartbeat:ha): Started node-1 Any help, like pointer to the source code where the problem might be, or any existing bug filed for this (I did some search but didn't find matched symptoms) is appreciated. This is happening at the heartbeat level. Not much pacemaker can do I'm afraid. Perhaps look to see if heartbeat is real time scheduled, if not that may explain why its being staved of CPU and can't get its messages out. Thanks, -Kaiwei ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] How to restart cluster ?
On 9 Jun 2014, at 7:58 pm, jarek ja...@poczta.srv.pl wrote: Hello! Thank you for the answer, but this answer didn't solve my problem. I have simple two-node cluster with virtual ip address and Postgres with streaming replication, created with this tutorial: http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster I have two problems to solve: 1. I need some script, which will restart cluster on user demand. This script should stop postgres resource on both nodes and next restart them in that way, that postgres will be work without any additional operations (like removing lock files, cleaning resources etc). This sounds like a secondary problem. Perhaps back up and explain why you need this? Also, the RA should be cleaning up lock files when it is asked to stop. 2. I have a virtual model of this cluster working under VMWare. VMWare is restarted from time to time, and I have no control when master or slave will be restarted. I would like to create script, which will be called from runlevel 6 and will safely stop postgres resource. I tried to do it with: crm configure property stop-all-resources=true but after reboot I had to remove PGSQL.lock manually, and also master node has been changed. Do you have any idea how to do it ? Taktoshi MATSUO wrote: Do you use pgsql RA with Master/Slave setting ? I recommend you to stop slave node's pacemaker at first because pgsql RA removes PGSQL.lock automatically if the node is master and there is no slaves. Stop procedure 1. stop slave node - suppose nodeB 2. stop master node (PGSQL.lock file is removed) - suppose nodeA Start procedure 3. start the nodeA because it has the newest data. 4. start the nodeB If PGSQL.lock exists, the data may be inconsistent. See http://www.slideshare.net/takmatsuo/2012929-pg-study-16012253 (P36, P37) best regards Jarek ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] getting proper sources
On 1 Jun 2014, at 2:15 am, Dmitri Maziuk dmaz...@bmrb.wisc.edu wrote:
On 5/30/2014 6:20 PM, Andrew Beekhof wrote:
Is there a reason you keep spouting nonsense?
Yes: I have a memory and it remembers. For example, this:
http://www.gossamer-threads.com/lists/linuxha/users/81573?do=post_view_threaded#81573
I don't remember that being an isolated incident either.
There is a very large distinction between:
- you don't pay for ${X} so I won't help diagnose your issue and get it fixed
upstream, and
- you don't pay for ${X} so you don't get to make demands about what goes into
packages for ${X}.
That you can't or won't see the difference says more about you than Lars.
[snip]
Heartbeat has been deprecated for some time already,
Nevertheless, when someone asks you how to get to the library, everyone's
using kindle these days is rarely the best answer.
More like our directions weren't convenient so instead you came to our house
and got abusive because we didn't have Harry Potter in Latin.
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] getting proper sources
On 31 May 2014, at 12:56 am, Dmitri Maziuk dmaz...@bmrb.wisc.edu wrote: On 5/29/2014 4:20 PM, Digimer wrote: On 29/05/14 01:43 PM, Dimitri Maziuk wrote: Support for free is 50% chance Lars will ask you if you're a paying Suse customer. Jay was asking about RHEL, but even then, I've seen Lars offer lots of help in #linux-ha without asking that. Yes he asked about centos and yes, I said 50% chance, not 100%. 50% is still grossly unfair. Not once have I seen Lars (or anyone here) refuse to help someone based on their being or not being a paying customer. Is there a reason you keep spouting nonsense? ... the level of support is: Digimer will tell you upgrade to pacemaker. Heartbeat has been deprecated for some time already, Nevertheless, when someone asks you how to get to the library, everyone's using kindle these days is rarely the best answer. Dima ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] getting proper sources
On 30 May 2014, at 3:01 am, Jay G. Scott g...@arlut.utexas.edu wrote: On Wed, May 28, 2014 at 07:42:58PM -0400, Digimer wrote: On 28/05/14 05:05 PM, Jay G. Scott wrote: Greetings, I'm a noob. If this isn't the right place to ask this, let me know. I took general configuration questions to include compiling. OS = RHEL6 (I hope) x86_64 I downloaded: Reusable-Cluster-Components-glue--glue-1.0.9.tar.bz2 Heartbeat-3-0-7e3a82377fa8.tar.bz2 ClusterLabs-resource-agents-v3.9.2-0-ge261943.tar.gz Errr I just now noticed this. Is there a Pacemaker source tarball somewhere? 'Cause I guess I don't have it. I wrote a book on using autotools and making RPMs. Looks like the existing packages could use some work. Heartbeat can't find the libltdl.tar file to install a local version, but I have a feeling the packages I have installed already should make that a moot point. Back to my problem: I compiled and installed Reusable-Cluster-Components-glue--glue-1.0.9.tar.bz2 Then I pressed on to Heartbeat. According to what I read online that's the proper order. 1. I have a libltdl.so.7 library but the configure script claims it does not have lt_ldopen(). I decided to bluster my way past that, in the hopes that the routine wouldn't really be needed. 2. But now it can't find ltdl.h, and I can't readily find a way around that. Ie, I can't find a RH package that has that header file. libtool and libtool-ltdl are both installed. [root@w1dns ~]# rpm -q -a | grep libtool libtool-2.2.6-15.5.el6.x86_64 libtool-ltdl-2.2.6-15.5.el6.x86_64 Are there pre-built versions of these things somewhere? I'm not anxious to re-invent the wheel. Are there more parts to this than I have? j. I think you might want to be sure you're using the right stack. Trick is, there are two answers for this. Under RHEL 6, the fully supported stack is corosync + cman + rgmanager. This has been Red Hat's stack since it got into the HA game, and will be supported until 2020 when RHEL 6 support ends. The other answer is corosync + pacemaker (+cman). This support is nearly complete, but not totally complete. It got out of Tech Preview at the end of the 6.4 stream and is effectively fully supported on 6.5 an onward. Further, this will be the only supported stack on RHEL 7 and forward. In either case, heartbeat is long deprecated (though still supported by Linbit, whom Red Hat has a support deal with). It has not been actively supported in years and there is no plan to restart development in the future. So please, do not use it. If you're curious about the reasons for all this, I have an incomplete-but-still-hopefully-helpful history of HA coming together here: https://alteeve.ca/w/History_of_HA_Clustering i made a note to go check this. So to best help you, can I ask what your near and long-term goals are? joke's on me: yeah, you can ask. and as of about 15 minutes ago they changed. near term -- now, nothing. longer term -- looks like i'd need to move to Centos or one of the other cost-free OSes. sigh. so, thanks -- sincerely -- for the RH answer. what's the answer for ... Centos, I guess...? And it does embarrass me to have to ask that. For paid fix this right now! support, then SLES and RHEL (6.5+) are good options (both pay people to work on pacemaker). There is also the option of talking to a cross-distro vendor like Linbit. CentOS is a good free option as you get something enterprise ready and can piggy back off the fixes that go into RHEL. The downside is that your problems need to be likely to affect paying customers before they can be prioritised and there can be a decent delay between a fix being available and shipped. In some ways though, you can be better off running your favourite distro plus the latest from upstream. Thats what the developers are using every day and all fixes are available immediately. The project's regression tests get performed on every commit and are expansive enough that pretty much anything that makes it into the public source tree is quite usable. The source trees for most things are at: http://github.com/ClusterLabs j. Once you've identified which stack is best for your use-case, we can better help you identify compilation issues (or if you need to be compiling anything at all). Cheers -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems -- Jay Scott 512-835-3553g...@arlut.utexas.edu Head of Sun Support, Sr. System Administrator Applied Research Labs, Computer Science Div. S224 University of
Re: [Linux-HA] getting proper sources
On 30 May 2014, at 7:20 am, Digimer li...@alteeve.ca wrote: On 29/05/14 01:43 PM, Dimitri Maziuk wrote: On 05/29/2014 12:01 PM, Jay G. Scott wrote: what's the answer for ... Centos, I guess...? And it does embarrass me to have to ask that. Pacemaker/corosync -- 2+-node clusters, active-active clusters, active development. Support for free is 50% chance Lars will ask you if you're a paying Suse customer. Jay was asking about RHEL, but even then, I've seen Lars offer lots of help in #linux-ha without asking that. The HA community helped me learn everything I know about HA (the quality of that education I will leave to others to determine) and not once was I asked by any RH or SUSE employee if I was a paying customer. The only context in which I've asked this question, or seen other do so, is to know how to get a fix to them efficiently. Customers of enterprise distros sometimes need to go via specific processes in order to fast track the creation of packages that include the fix from upstream. It's honestly been one of the best communities I've seen in the OSS world. Heartbeat 'R1' (i.e. as long as you don't use 'crm' mode) -- simple, stupid, has been rock solid (and, consequently, untouched) for years. 2-node active/passive clusters only, DIY external resource monitoring (mon), the level of support is: Digimer will tell you upgrade to pacemaker. Or cman+rgmanager, either one. The argument for pacemaker is, as I said in the first reply, that it is the only stack with long-term plans. Heartbeat has been deprecated for some time already, and RH stopped development of cman+rgmanager a year or two ago with the release of 3.2, though it will be actively supported until at least 2020. Cheers -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] A question of pgsql resource
On 19 May 2014, at 8:12 pm, Naoya Anzai anzai-na...@mxu.nes.nec.co.jp wrote:
Hi Matsuo-san
Thank you for your response.
But I would like you to keep current stopping process
because I think it's safer to use STONITH.
Could you implement it adding new parameter if you implement?
Sorry,
I don't fully understand all of pacemaker yet.
In pacemaker,
Is there a function that a node executes STONITH when
a resource agent of the other node timed out ?
yes. Specify on-fail=fence for that operation
or
Is there a function that a node executes harakiri when
own resource agent timed out ?
I'm thinking that the resource agent must implement that functions,
because pacemaker does not provide them.
Is this wrong ??
BTW, is it true that the cause of time-out is not while but pg_ctl(-m i)?
If pg_ctl (-m i), you need to use time-out parameter or you can use
exec_with_timeout().
In My test pattern,
I issued SIGSTOP signals to all of MASTER-PostgreSQL processes.
# killall -SIGSTOP postgres
In fact,
it looks like a resouce agent timed out in pgsql_real_monitor() after pg_ctl(
-m i) has timed out.
--- real suspend point of the pgsql resource
output=`su $OCF_RESKEY_pgdba -c cd $OCF_RESKEY_pgdata; \
$OCF_RESKEY_psql $psql_options -U $OCF_RESKEY_pgdba \
-Atc \${CHECK_MS_SQL}\`
---
It's a psql.
Regards,
Naoya
---
Naoya Anzai
Engineering Department
NEC Solution Inovetors, Ltd.
E-Mail: anzai-na...@mxu.nes.nec.co.jp
---
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] OSX and HA
See if http://clusterlabs.org/wiki/SourceInstall#Darwin.2FMacOS_X helps. On 9 May 2014, at 5:38 am, Andrew Marks andrew.mar...@icloud.com wrote: I am looking for information on compiling this HA to be used with OSX Server (10.8/10.9) I am looking to do a simple active/passive cluster for IP failover. I was hoping for find some OSX Specific instructions. Thanks you for any information. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] How I can create unordered group of resources
On 5 May 2014, at 10:06 pm, Fabian Herschel fabian.hersc...@arcor.de wrote: On 05/05/2014 02:36 AM, Andrew Beekhof wrote: On 4 May 2014, at 4:22 pm, Fabian Herschel fabian.hersc...@arcor.de wrote: I would create the group with the meta attributr for unordered resources. Meta odered=false N. Use a colocation set. Could you explain your No? Whats wrong in using the unordered feature? Why was this meta attribute added to groups, if we shouldn't use it? Its an abomination that I should never have implemented but now cannot remove. Von Samsung-Tablet gesendet Ursprüngliche Nachricht Von: Vladimir Romanov vroma...@gmail.com Datum:03.05.2014 10:29 (GMT+01:00) An: linux-ha@lists.linux-ha.org Betreff: [Linux-HA] How I can create unordered group of resources Hello! I try create Master/Slave cluster using Pacemaker on Centos 6.5 (CRM+PCS). I create master/slave statefull resource. My setup also have many other resources (IPs, Routes, LSB...). I one of resources is failed on first mode I want to move all resources to another node. Now I use group to create this setup. But when I kill -9 some process all processes listen below also restarted. That is best practice for this task? -- Vladimir Romanov ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] How I can create unordered group of resources
On 4 May 2014, at 4:22 pm, Fabian Herschel fabian.hersc...@arcor.de wrote: I would create the group with the meta attributr for unordered resources. Meta odered=false N. Use a colocation set. Von Samsung-Tablet gesendet Ursprüngliche Nachricht Von: Vladimir Romanov vroma...@gmail.com Datum:03.05.2014 10:29 (GMT+01:00) An: linux-ha@lists.linux-ha.org Betreff: [Linux-HA] How I can create unordered group of resources Hello! I try create Master/Slave cluster using Pacemaker on Centos 6.5 (CRM+PCS). I create master/slave statefull resource. My setup also have many other resources (IPs, Routes, LSB...). I one of resources is failed on first mode I want to move all resources to another node. Now I use group to create this setup. But when I kill -9 some process all processes listen below also restarted. That is best practice for this task? -- Vladimir Romanov ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Resource blocked
On 23 Apr 2014, at 4:15 am, Tom Parker tpar...@cbnco.com wrote: Good morning I am trying to restart resources on one of my clusters and I am getting the message pengine[13397]: notice: LogActions: Start domtcot1-qa(qaxen1 - blocked) How can I find out why this resource is blocked. Depending on your version 'crm_simulate -SL -VV' might provide more information. (add more -V options for more detail) signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Info messages in syslog: Are they normal?
Your mail client has butchered your message beyond recognition. Pacemaker did send some INFO logs to syslog in the past, so yes it is normal. On 5 Apr 2014, at 1:55 pm, Anna Hegedus akh...@hotmail.com wrote: Hi Everyone, I have to apologize in advance. I have used other solutions before in regards to monitoring servers and having redundancy, but this is my first experience with heartbeat. I've inherited a set of servers from someone that is using drbd to provide a redundant installation of mysql and another set of disks for lucene. After installing some monitoring software I started receiving these messages in emails: Apr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation lucene-disk:0_last_failure_0 found resource lucene-disk:0 active on staircase.bup.prod.localApr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation db-master-mysql_last_failure_0 found resource db-master-mysql active on drawers.bup.prod.localApr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation lucene-server_last_failure_0 found resource lucene-server active on drawers.bup.prod.localApr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation lucene-disk:1_last_failure_0 found resource lucene-disk:1 active in master mode on drawers.bup.prod.localApr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation lucene-fs_last_failure_0 found resource lucene-fs active on drawers.bup.prod.localApr 4 22:14:15 staircase pengine: [31090]: notice: unpack_rsc_op: Operation lucene-ip_last_failure_0 found resource lucene-ip active on drawers.bup.p rod.localApr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave db-master-ip#011(Started drawers.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave db-master-mysql#011(Started drawers.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave db-slave-ip#011(Started staircase.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave db-slave-mysql#011(Started staircase.bup.prod.local))Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave lucene-disk:0#011(Slave staircase.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave lucene-disk:1#011(Master drawers.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave lucene-fs#011(Started drawers.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: LogActions: Leave lucene-ip#011(Started drawers.bup.prod.local)Apr 4 22:14:15 staircas e pengine: [31090]: notice: LogActions: Leave lucene-server#011(Started drawers.bup.prod.local)Apr 4 22:14:15 staircase pengine: [31090]: notice: process_pe_message: Transition 17386: PEngine Input stored in: /var/lib/pengine/pe-input-478.bz2Apr 4 22:14:15 staircase crmd: [31085]: info: do_state_transition: State transition S_POLICY_ENGINE - S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]Apr 4 22:14:15 staircase crmd: [31085]: info: unpack_graph: Unpacked transition 17386: 0 actions in 0 synapsesApr 4 22:14:15 staircase crmd: [31085]: info: do_te_invoke: Processing graph 17386 (ref=pe_calc-dc-1396664055-18368) derived from /var/lib/pengine/pe-input-478.bz2Apr 4 22:14:15 staircase crmd: [31085]: info: run_graph: Apr 4 22:14:15 staircase crmd: [31085]: notice: run_graph: Transition 17386 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-478.bz2): CompleteApr 4 22:14:15 staircase crmd: [31085]: info: te_graph_trigger: Transition 17386 is now completeApr 4 22:14:15 staircase crmd: [31085]: info: notify_crmd: Transition 17386 status: done - nullApr 4 22:14:15 staircase crmd: [31085]: info: do_state_transition: State transition S_TRANSITION_ENGINE - S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]Apr 4 22:14:15 staircase crmd: [31085]: info: do_state_transition: Starting PEngine Recheck TimerApr 4 22:14:22 staircase lrmd: [31082]: info: rsc:lucene-disk:0 monitor[286] (pid 12190)Apr 4 22:14:22 staircase lrmd: [31082]: info: operation monitor[286] on lucene-disk:0 for client 31085: pid 12190 exited with return code 0 Everything seems fine, and the services are now working after I found some issues with the daemon on one box, fixed it, then cleaned and reprobed with the crmadmin command. I am not sure if anything is actually wrong, or if these are harmless warnings. Is it anything I should worry about? If so, I can provide my config files. I just wasn't sure if this is some sort of harmless thing or if I should genuinely be worried. Thanks for your help!Kathy ___
Re: [Linux-HA] How to tell pacemaker to process a new event during a long-running resource operation
On 17 Mar 2014, at 6:49 pm, Lars Marowsky-Bree l...@suse.com wrote: On 2014-03-14T15:50:18, David Vossel dvos...@redhat.com wrote: in-flight operations always have to complete before we can process a new transition. The only way we can transition earlier is by killing the in-flight process, which results in failure recovery and possibly fencing depending on what operation it is. There's really nothing that can be done to speed this up except work on lowering the startup time of that resource. We keep getting similar requests though - some services take a long time, and during that interval, the cluster is essentially stuck. As the density of resources in the cluster increases and the number of nodes goes up, this becomes more of an issue. It *would* be possible with changes to the TE/PE - assume in-flight operations will complete as planned, so that any further changes to in-flight resources would be ordered after them, the ability to accept actions completing from previous transitions -, but it's also non-trivial. Thats probably the most lucid thinking anyone has done on the subject... you should put that into a bugzilla so we don't collectively forget. I'd agree that this is probably going to increase in importance over time, so we'll likely have to address it sooner or later (just not right now). Pacemaker 2.0 material? ;-) Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple colocation with same resource group
On 25 Feb 2014, at 1:29 am, Tony Stocker tony.stoc...@nasa.gov wrote: On Mon, 24 Feb 2014, Andrew Beekhof wrote: On 22 Feb 2014, at 2:16 am, Greg Woods wo...@ucar.edu wrote: On Fri, 2014-02-21 at 12:37 +, Tony Stocker wrote: colocation inf_ftpd inf: infra_group ftpd or do I need to use an 'order' statement instead, i.e.: order ftp_infra mandatory: infra_group:start ftpd I'm far from a leading expert on this, but in my experience, colocation and order are completely separate concepts. If you want both, you have to state both. So I would say you need both colocation and order statements to get what you want. Exactly So are is **this** what my configuration should like given that information?: primitive ip ocf:heartbeat:IPaddr2 params ip=1.2.3.4 primitive job ocf:pps:jobfile params role=test job=first primitive pwd ocf:pps:pwdfile params role=test primitive ftpd ocf:pps:proftpd primitive httpd ocf:heartbeat:apache primitive smtpd ocf:heartbeat:postfix primitive bes ocf:pps:besServer group infra_group ip job pwd colocation inf_ftpd inf: ftpd infra_group colocation inf_http inf: httpd infra_group colocation inf_mail inf: smtpd infra_group colocation inf_odap inf: bes infra_group order ftpd_infra mandatory: infra_group:start ftpd order http_infra mandatory: infra_group:start httpd order smtp_infra mandatory: infra_group:start smtpd order odap_infra mandatory: infra_group:start bes Is my syntax above correct for the situation where I need all elements of 'infra_group' started first, and then the various other primitives started? In other words am I correctly stating my colocation requirements? From what I recall of crmsh, it is correct Or do I need to reverse the order, like so?: colocation inf_ftpd inf: infra_group ftpd Will the order statements ensure that the infra_group is completed startup before starting ftpd? yes In other words, since part of the infra_group is to set the password file, and since the ftpd daemon depends on the existence of UID's in said password file, the ftpd primitive is not going to start until the infra_group has fully completed startup, correct? Am I allowed to separately list the pairs of colocation and order statements as I've done above? yes Or will that cause issues? no By separately stating the pairs, as opposed to creating a single line, have I avoided creating ad hoc resource sets that I didn't explicitly define? its shouldn't matter either way. Thanks, Tony -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Multiple colocation with same resource group
On 22 Feb 2014, at 2:16 am, Greg Woods wo...@ucar.edu wrote: On Fri, 2014-02-21 at 12:37 +, Tony Stocker wrote: colocation inf_ftpd inf: infra_group ftpd or do I need to use an 'order' statement instead, i.e.: order ftp_infra mandatory: infra_group:start ftpd I'm far from a leading expert on this, but in my experience, colocation and order are completely separate concepts. If you want both, you have to state both. So I would say you need both colocation and order statements to get what you want. Exactly I have similar scenarios, where virtual machines depend on the underlying DRBD device, so I colocate the DRBD master-slave resource and the VM resource, then have an order statement to say the DRBD resource must be started before the VM. --Greg ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] pgsql resource agent in status Stopped after crm resource cleanup
On 21 Feb 2014, at 10:55 pm, Lukas Grossar lukas.gros...@adfinis-sygroup.ch wrote: Hi I'm currently building a 2 node DRBD backed PostgreSQL on Debian Wheezy and I'm testing how Pacemaker reacts to specific failure scenarios. One thing I did test that currently drives me crazy is when I manually stop PostgreSQL trough pg_ctl or just kill the master process to simulate a crash the pgsql resource agent correctly detects the error and restarts PostgreSQL. The problem is have arises when I later call 'crm resource cleanup pgsql' to delete the failcount and the failed tasks the pgsql resources shows up as Stopped, but in reality it is still running fine. I'm having the same problem when I delete the failcount separately and then do the cleanup. The problem seems to be that psql_monitor runs into a timeout: Feb 21 12:47:59 vm-db-01 crmd: [6494]: WARN: cib_action_update: rsc_op 44: pgsql_monitor_3 on vm-db-01 timed out After the timeout pgsql is being restarted, and the interesting thing is that I can delete the failed action from the timeout without a problem. Does anyone have an idea what the problem could be in this case? Not without more logs. You'd probably want to turn on 'set -x' in the resource agent to see why it can't complete. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: crm configure edit/show regex
On 20 Feb 2014, at 1:42 am, Lars Marowsky-Bree l...@suse.com wrote: On 2014-02-19T10:31:45, Andrew Beekhof and...@beekhof.net wrote: Unifying this might be difficult, as far as I know pcs doesn't have an interactive mode or anything similar to the configure interface of crmsh.. It does have bash completion for the command line. FWIW, so does the crm shell ;-) Wasn't trying to imply it didn't. Only that pcs wasn't completely devoid of such functionality. (It works just like in the interactive mode too, probably the fully internal mode is a bit faster, since it avoids bash stuff.) Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: crm configure edit/show regex
On 19 Feb 2014, at 12:13 am, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Kristoffer Grönlundkgronl...@suse.com schrieb am 18.02.2014 um 14:07 in Nachricht 20140218140726.1b2dfd0f@ultralix: On Tue, 18 Feb 2014 13:48:21 +0100 Lars Marowsky-Bree l...@suse.com wrote: On 2014-02-18T09:55:44, Kristoffer Grönlund kgronl...@suse.com wrote: Sounds like a good idea to me, too. I'll see what I can do :) At least a simple approximation of glob-syntax should be doable. Constraints can already refer to regex now too. And we're talking about being able to match on tags (or even attributes) on start/stop etc. It'd make sense to then merge all this consistently, right? (Perhaps even via a crm_resource -l option, so that pcs/crmsh use the same matching code as the PE and as each other, if that makes any sense.) I also realised that there is another complication, which is that unless in interactive mode, the bash shell will consume the regex syntax, and it would need to be escaped. So that would make the feature even more confusing and doesn't seem entirely clean - although this is a limitation of shell syntax. In theory the shell could use the completion information to expand 'crm configure show prim*' into all the completions for prim*. I don't know whether this is a real restriction for people who are used to use the command line: Even when using grep, sed or awk, some REs have to be escaped, and the shell has no idea how to complete them. Unifying this might be difficult, as far as I know pcs doesn't have an interactive mode or anything similar to the configure interface of crmsh.. It does have bash completion for the command line. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] openstack fencing agent
On 19 Feb 2014, at 9:15 am, JR botem...@gmail.com wrote: Greetings, Someone on the linux-ha irc channel suggested that perhaps this agent might be of use to others using openstack. Note: it's based on fence_virsh and was written in about 20 mins; it seems to work for me but YMMV. Question... this is for openstack instances acting as cluster nodes? If so, how did you get them to use predictable IPs that corosync could bind to? Also, how did you provide authentication for the nova client? I created https://github.com/beekhof/fence_openstack but gave up on it because corosync wasn't able to function. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Why does o2cb RA remove module ocfs2?
On 17 Feb 2014, at 6:39 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 17.02.2014 um 02:33 in Nachricht 7619a7e9-f006-4098-90f9-5c5b8bc84...@beekhof.net: On 11 Feb 2014, at 10:38 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: [...] I did a quick check: It seems only ocf:ocfs2:o2cb does such (IMHO) nonsense like removing a module on stop (I can guess it's a leftover from o2cb module hacking when the developer was too lazy to remove the module by hand when wanting to try a newer version): seems pretty reasonable to me. stop == remove all trace of the active resource. [...] But why doesn't the LVM RA try to remove the lvm module, and why doesn't the NFS RA try to remove the nfs module, etc. then? No idea, I didn't write those either. Perhaps they should signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Why does o2cb RA remove module ocfs2?
On 11 Feb 2014, at 10:38 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de
wrote:
Lars Marowsky-Bree l...@suse.com schrieb am 05.02.2014 um 15:11 in
Nachricht
20140205141140.gu13...@suse.de:
On 2014-02-05T15:06:47, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de
wrote:
I guess the kernel update is more common than the just the ocfs2-kmp
update
Well, some customers do apply updates in the recommended way, and thus
don't encounter this ;-) In any case, since at this time the cluster
services are already stopped, at least the service impact is minimal.
This would avoid this error too. Or keeping multiple kernel versions in
parallel (which also helps if a kernel update no longer boots for some
reason). Removing the running kernel package is usually not a great
idea; I prefer to remove them after having successfully rebooted only,
because you *never* know if you may have to reload a module.
There's another way: (Like HP-UX learned to do it): Defer changes to the
running kernel until shutdown/reboot.
True. Hence: activate multi-versions for the kernel in
/etc/zypp/zypp.conf and only remove the old kernel after the reboot. I
do that manually, but I do think we even have a script for that
somewhere. I honestly don't remember where though; I like to keep
several kernels around for testing anyway.
I think this is the default going forward, but as always: zypper gained
this ability during the SLE 11 cycle, and we couldn't just change
existing behaviour in a simple update, it has to be manually
activated.
I did a quick check: It seems only ocf:ocfs2:o2cb does sucj (IMHO) nonsense
like removing a module on stop (I can guess it's a leftover from o2cb module
hacking when the developer was too lazy to remove the module by hand when
wanting to try a newer version):
seems pretty reasonable to me.
stop == remove all trace of the active resource.
--
# egrep 'modprobe|rmmod' /usr/lib/ocf/resource.d/*/*
/usr/lib/ocf/resource.d/heartbeat/drbd: do_cmd modprobe -s drbd `$DRBDADM
sh-mod-parms` || {
/usr/lib/ocf/resource.d/heartbeat/iface-vlan: error=$(modprobe
8021q 21)
/usr/lib/ocf/resource.d/linbit/drbd:do_cmd modprobe -s drbd
`$DRBDADM sh-mod-parms` || {
/usr/lib/ocf/resource.d/ocfs2/o2cb:modprobe -rs $FSNAME
/usr/lib/ocf/resource.d/ocfs2/o2cb:modprobe -rs $MODNAME
/usr/lib/ocf/resource.d/ocfs2/o2cb: modprobe -s ocfs2_stackglue
/usr/lib/ocf/resource.d/ocfs2/o2cb:modprobe -s ocfs2_stack_user
/usr/lib/ocf/resource.d/ocfs2/o2cb: modprobe -s ocfs2
/usr/lib/ocf/resource.d/pacemaker/controld: modprobe configfs
/usr/lib/ocf/resource.d/pacemaker/controld: modprobe dlm
--
Regards,
Ulrich
Regards,
Lars
--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer,
HRB 21284 (AG Nürnberg)
Experience is the name everyone gives to their mistakes. -- Oscar Wilde
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: crmd (?) becomes unresponsive
On 22 Jan 2014, at 10:58 pm, Thomas Schulte tho...@cupracer.de wrote: Hi Lars, I thought that was what I just did? This is likely a pacemaker problem, and more pacemaker experts are subscribed to the clusterlabs list than here. I think we're mostly all on both. Also, the right bugzilla for the attachments/logs. Regards, Lars thank you very much for your suggestion, I'm fine with that. I created a bug report @ClusterLabs for that and attached my files there. http://bugs.clusterlabs.org/show_bug.cgi?id=5192 Thanks. I'm still in catchup mode at the moment, but I will get to this and other such requests asap. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] SLE11 SP3: attrd[13911]: error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0)
Looks like corosync is dying underneath pacemaker. On 15 Jan 2014, at 6:49 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I feel the current clusterstack for SLES11 SP3 has several problems. I'm fighting for a day to get my test cluster up again after having installed the latest updates. I still cannot find out what's going on, but I suspect there are too many bugs in the software (again). For example I just saw these messages: Jan 15 08:39:50 o4 attrd[13911]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 attrd[13911]: crit: attrd_cs_destroy: Lost connection to Corosync service! Jan 15 08:39:50 o4 cib[13908]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 cib[13908]:error: cib_cs_destroy: Corosync connection lost! Exiting. Jan 15 08:39:50 o4 attrd[13911]: notice: main: Exiting... Jan 15 08:39:50 o4 attrd[13911]: notice: main: Disconnecting client 0x611eb0, pid=13913... Jan 15 08:39:50 o4 stonith-ng[13909]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 stonith-ng[13909]:error: stonith_peer_cs_destroy: Corosync connection terminated Jan 15 08:39:50 o4 attrd[13911]:error: attrd_cib_connection_destroy: Connection to the CIB terminated... Jan 15 08:39:50 o4 crmd[13913]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 crmd[13913]:error: crmd_cs_destroy: connection terminated I'm afraid nobody except the developer can make any sense from these error messages, except that there is an error. Seriously? Lost connection to Corosync service! and Corosync connection lost! Exiting. are too cryptic for you? Here's the list of software installed recently: drbd-pacemaker-8.4.4-0.20.2 Mon Jan 13 12:58:49 2014 drbd-8.4.4-0.20.2 Mon Jan 13 12:58:49 2014 sleha-bootstrap-0.3-0.26.1Mon Jan 13 12:58:48 2014 pacemaker-mgmt-2.1.2-0.11.4 Mon Jan 13 12:58:48 2014 crmsh-1.2.6-0.25.4Mon Jan 13 12:58:48 2014 pacemaker-1.1.10-0.9.28 Mon Jan 13 12:58:47 2014 pacemaker-mgmt-client-2.1.2-0.11.4Mon Jan 13 12:58:43 2014 cluster-glue-1.0.11-0.19.4Mon Jan 13 12:58:43 2014 sbd-1.2.1-0.7.22 Mon Jan 13 12:58:42 2014 libpacemaker3-1.1.10-0.9.28 Mon Jan 13 12:58:41 2014 resource-agents-3.9.5-0.32.22 Mon Jan 13 12:58:38 2014 openais-1.1.4-5.17.5 Mon Jan 13 11:12:19 2014 libglue2-1.0.11-0.19.4Mon Jan 13 11:12:18 2014 drbd-xen-8.4.4-0.20.2 Mon Jan 13 11:12:18 2014 drbd-udev-8.4.4-0.20.2Mon Jan 13 11:12:18 2014 drbd-heartbeat-8.4.4-0.20.2 Mon Jan 13 11:12:18 2014 drbd-bash-completion-8.4.4-0.20.2 Mon Jan 13 11:12:18 2014 libqb0-0.16.0-0.7.4 Mon Jan 13 11:12:17 2014 libopenais3-1.1.4-5.17.5 Mon Jan 13 11:12:17 2014 drbd-utils-8.4.4-0.20.2 Mon Jan 13 11:12:17 2014 Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: SLE11 SP3: attrd[13911]: error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0)
On 15 Jan 2014, at 11:13 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 15.01.2014 um 10:27 in Nachricht 59b0ba57-84bd-4ed9-be06-22c41bc21...@beekhof.net: Looks like corosync is dying underneath pacemaker. On 15 Jan 2014, at 6:49 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I feel the current clusterstack for SLES11 SP3 has several problems. I'm fighting for a day to get my test cluster up again after having installed the latest updates. I still cannot find out what's going on, but I suspect there are too many bugs in the software (again). For example I just saw these messages: Jan 15 08:39:50 o4 attrd[13911]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 attrd[13911]: crit: attrd_cs_destroy: Lost connection to Corosync service! Jan 15 08:39:50 o4 cib[13908]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 cib[13908]:error: cib_cs_destroy: Corosync connection lost! Exiting. Jan 15 08:39:50 o4 attrd[13911]: notice: main: Exiting... Jan 15 08:39:50 o4 attrd[13911]: notice: main: Disconnecting client 0x611eb0, pid=13913... Jan 15 08:39:50 o4 stonith-ng[13909]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 stonith-ng[13909]:error: stonith_peer_cs_destroy: Corosync connection terminated Jan 15 08:39:50 o4 attrd[13911]:error: attrd_cib_connection_destroy: Connection to the CIB terminated... Jan 15 08:39:50 o4 crmd[13913]:error: plugin_dispatch: Receiving message body failed: (2) Library error: Success (0) Jan 15 08:39:50 o4 crmd[13913]:error: crmd_cs_destroy: connection terminated I'm afraid nobody except the developer can make any sense from these error messages, except that there is an error. Seriously? Lost connection to Corosync service! and Corosync connection lost! Exiting. are too cryptic for you? I'm talking about ...error: Success (0): What type of error is that? (2) Library error is from corosync, Success (0) is the value of errno. Yes the errno part is confusing, but its also valuable information :( [...] Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Better way to change master in 3 node pgsql cluster
On 13 Jan 2014, at 8:32 pm, Andrey Rogovsky a.rogov...@gmail.com wrote: Hi I have 3 node postgresql cluster. It work well. But I have some trobule with change master. For now, if I need change master, I must: 1) Stop PGSQL on each node and cluster service 2) Start Setup new manual PGSQL replication 3) Change attributes on each node for point to new master 4) Stop PGSQL on each node 5) Celanup resource and start cluster service It take a lot of time. Is it exist better way to change master? Newer versions support: crm_resource --resource msPostgresql --ban --master --host a.geocluster.e-autopay.com This is my cluster service status: Node Attributes: * Node a.geocluster.e-autopay.com: + master-pgsql:0 : 1000 + pgsql-data-status : LATEST + pgsql-master-baseline : 2F90 + pgsql-status : PRI * Node c.geocluster.e-autopay.com: + master-pgsql:0 : 1000 + pgsql-data-status : SYNC + pgsql-status : STOP * Node b.geocluster.e-autopay.com: + master-pgsql:0 : 1000 + pgsql-data-status : SYNC + pgsql-status : STOP I was use http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster for my 3 nodes cluster without hard stik. Now I got strange situation all nodes stay slave: Last updated: Sat Dec 7 04:33:47 2013 Last change: Sat Dec 7 12:56:23 2013 via crmd on a Stack: openais Current DC: c - partition with quorum Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff 5 Nodes configured, 3 expected votes 4 Resources configured. Online: [ a c b ] Master/Slave Set: msPostgresql [pgsql] Slaves: [ a c b ] My config is: node a \ attributes pgsql-data-status=DISCONNECT node b \ attributes pgsql-data-status=DISCONNECT node c \ attributes pgsql-data-status=DISCONNECT primitive pgsql ocf:heartbeat:pgsql \ params pgctl=/usr/lib/postgresql/9.3/bin/pg_ctl psql=/usr/bin/psql pgdata=/var/lib/postgresql/9.3/main start_opt=-p 5432 rep_mode=sync node_list=a b c restore_command=cp /var/lib/postgresql/9.3/pg_archive/%f %p master_ip=192.168.10.200 restart_on_promote=true config=/etc/postgresql/9.3/main/postgresql.conf \ op start interval=0s timeout=60s on-fail=restart \ op monitor interval=4s timeout=60s on-fail=restart \ op monitor interval=3s role=Master timeout=60s on-fail=restart \ op promote interval=0s timeout=60s on-fail=restart \ op demote interval=0s timeout=60s on-fail=stop \ op stop interval=0s timeout=60s on-fail=block \ op notify interval=0s timeout=60s primitive pgsql-master-ip ocf:heartbeat:IPaddr2 \ params ip=192.168.10.200 nic=peervpn0 \ op start interval=0s timeout=60s on-fail=restart \ op monitor interval=10s timeout=60s on-fail=restart \ op stop interval=0s timeout=60s on-fail=block group master pgsql-master-ip ms msPostgresql pgsql \ meta master-max=1 master-node-max=1 clone-max=3 clone-node-max=1 notify=true colocation set_ip inf: master msPostgresql:Master order ip_down 0: msPostgresql:demote master:stop symmetrical=false order ip_up 0: msPostgresql:promote master:start symmetrical=false property $id=cib-bootstrap-options \ dc-version=1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff \ cluster-infrastructure=openais \ expected-quorum-votes=3 \ no-quorum-policy=ignore \ stonith-enabled=false \ crmd-transition-delay=0 \ last-lrm-refresh=1386404222 rsc_defaults $id=rsc-options \ resource-stickiness=100 \ migration-threshold=1 ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: does heartbeat 3.0.4 use IP aliases under CentOS 6.5?
On 7 Jan 2014, at 6:27 pm, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Dimitri Maziuk dmaz...@bmrb.wisc.edu schrieb am 04.01.2014 um 18:39 in Nachricht 52c8473b.8000...@bmrb.wisc.edu: On 1/4/2014 10:39 AM, Lars Marowsky-Bree wrote: On 2014-01-03T20:56:42, Digimer li...@alteeve.ca wrote: causing a lot of reinvention of the wheel. In the last 5~6 years, both teams have been working hard to unify under one common open-source HA stack. Pacemaker + corosync v2+ is the result of all that hard work. :) Yes. We know finally have one stack everywhere. Yay! Yah, ein stack uber alles! You'd note that even the cpu market has settled on two kinds of music, not one... It's getting off-topic, but very recently I discovered that some demo software I wanted to try required SSE3E while my CPU only provides SSE3... I mean: Which average user knows what command set extensions his CPU actually supports? Maybe the war Intel vs. AMD starts a new round also ;-) It wouldn't be fair if we had all the fun signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] FW cluster fails at 4am
On 28 Dec 2013, at 3:34 pm, Tracy Reed tr...@ultraviolet.org wrote: Hello all, First, thanks in advance for any help anyone may provide. I've been battling this problem off and on for months and it is driving me mad: Once every week or two my cluster fails. For reasons unknown it seems to initiate a failover and then the shorewall service (lsb) does not get started (or is stopped). The majority of the time it happens just after 4am. Although it has happened at other times, although much less frequently. Tonight I am going to have to be up at 4am to poke around on the cluster and observe what is happening, if anything. One theory is some sort of resource starvation such as CPU but I've stress tested it and run a backup and a big file copy through the firewall at the same time and never get more than 1 core of cpu (almost all due to the backup) out of 4 utilized and nothing interesting happening to pacemaker/resources. My setup is a bit complicated in that I have 63 IPaddr2 resources plus the shorewall resource. Plus order and colocation rules to make sure it all sticks together the IPs come up before shorewall. I am running the latest RHEL/CentOS RPMs in CentOS 6.5: [root@new-fw1 shorewall]# rpm -qa |grep -i corosync corosync-1.4.1-17.el6.x86_64 corosynclib-1.4.1-17.el6.x86_64 [root@new-fw1 shorewall]# rpm -qa |grep -i pacemaker pacemaker-1.1.10-14.el6.x86_64 pacemaker-libs-1.1.10-14.el6.x86_64 pacemaker-cluster-libs-1.1.10-14.el6.x86_64 pacemaker-cli-1.1.10-14.el6.x86_64 I am a little concerned about how pacemaker manages the shorewall resource. It usually fails to bring up shorewall after a failover event. Shorewall could fail to start if the IP addresses shorewall is expecting to be on the interfaces are not there yet. But I have dependencies to prevent this from ever happening such as: order shorewall-after-dmz-gw inf: dmz-gw shorewall I also wonder if the shorewall init script is properly LSB compatible. It wasn't out of the box and I had to make a minor change. But now it does seem to be LSB compatible: [root@new-fw2 ~]# /etc/init.d/shorewall status ; echo result: $? Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:14 PST 2013 Shorewall is running State:Started (Fri Dec 27 04:11:14 PST 2013) from /etc/shorewall/ result: 0 [root@new-fw2 ~]# /etc/init.d/shorewall stop ; echo result: $? Shutting down shorewall: [ OK ] result: 0 [root@new-fw2 ~]# /etc/init.d/shorewall status ; echo result: $? Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:48 PST 2013 Shorewall is stopped State:Stopped (Fri Dec 27 16:57:47 PST 2013) result: 3 [root@new-fw2 ~]# /etc/init.d/shorewall start ; echo result: $? Starting shorewall:Shorewall is already running [ OK ] result: 0 [root@new-fw2 ~]# /etc/init.d/shorewall status ; echo result: $? Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:58:04 PST 2013 Shorewall is running State:Started (Fri Dec 27 16:57:53 PST 2013) from /etc/shorewall/ result: 0 So it shouldn't be an LSB issue at this point... I have a very hard time making heads or tails of the /var/log/cluster/corosync.log log files. For example, I just had this appear in the log files: Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave spider2-eth0-40 (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave spider2-eth0-41 (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave corpsites (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave dbrw(Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave mjhdev (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave datapass1-ssl-eth0-2(Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave datapass2-ssl-eth0-2(Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave datapass2-ssl-eth0-1(Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave datapass2-ssl-eth0 (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave rrdev2 (Started new-fw2.mydomain.com) Dec 27 19:56:31 [1551] new-fw1.mydomain.compengine: info: LogActions: Leave webmail-resumepromotion (Started
Re: [Linux-HA] FW cluster fails at 4am
On 7 Jan 2014, at 10:52 am, Tracy Reed tr...@ultraviolet.org wrote: On Sat, Dec 28, 2013 at 12:42:28AM PST, Jefferson Ogata spake thusly: Is it possible that it's a coincidence of log rotation after patching? In certain circumstances i've had library replacement or subsequent prelink activity on libraries lead to a crash of some services during log rotation. This hasn't happened to me with pacemaker/cman/corosync, but it might conceivably explain why it only happens to you once in a while. I just caught the cluster in the middle of crashing again and noticed it had a system load of 9. Although it isn't clear why. See my other reply: Consider though, what effect 63 IPaddr monitor operations running at the same time might have on your system. A backup was running but after the cluster failed over the backup continues and the load went to very nearly zero. So it doesn't seem like the backup was causing the issue. But the system was noticeably performance impacted. I've never noticed this situation before. One thing I really need to learn more about is how the cluster knows when something has failed and it needs to fail over. We ask the resources by calling their script with $action=monitor For node-level failures, corosync tells us. I first setup a linux-ha firewall cluster back around 2001 and we used simple heartbeat and some scripts to pass around IP addresses and start/stop the firewall. It would ping its upstream gateway and communicate with its partner via a serial cable. If the active node couldn't ping its upstream it killed the local heartbeat and the partner took over. If the active node wasn't sending heartbeats the passive node took over. Once working it stayed working and was much much simpler than the current arrangement. I have no idea how the current system actually communicates or what the criteria for failover really are. What are the chances that the box gets overloaded and drops a packet and the partner takes over? What if I had an IP conflict with another box on the network and one of my VIP IP addresses didn't behave as expected? What would any of these look like in the logs? One of my biggest difficulties in diagnosing this is that the logs are huge and noisy. It is hard to tell what is normal, what is an error, and what is the actual test that failed which caused the failover. You might take a look at the pacct data in /var/account/ for the time of the crash; it should indicate exit status for the dying process as well as what other processes were started around the same time. Process accounting wasn't running but /var/log/audit/audit.log is which has the same info. What dying process are we talking about here? I haven't been able to identify any processes which died. I think there was an assumption that your resources were long running daemons. Yes, you're supposed to switch to cman. Not sure if it's related to your problem, tho. I suspect the cman issue is unrelated Quite likely. so I'm not going to mess with it until I get the current issue figure out. I've had two more crashes since I started this thread: One around 3am and one just this afternoon around 1pm. A backup was running but after the cluster failed over the backup kept running and the load returned to normal (practically zero). -- Tracy Reed ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Attribures was not changed via crm_attribute
On 27 Dec 2013, at 3:24 am, Andrey Rogovsky a.rogov...@gmail.com wrote: This command help me. I return pg cluster in normal state. Thanks a lot. Now I was reboot a node for test and got this state: Resource Group: master pgsql-master-ip (ocf::heartbeat:IPaddr2): Started b.mydomain.com Master/Slave Set: msPostgresql [pgsql] Masters: [ b.mydomain.com ] Slaves: [ c.mydomain.com ] Stopped: [ pgsql:2 ] apache-master-ip (ocf::heartbeat:IPaddr2): Started c.mydomain.com apache (ocf::heartbeat:apache): Started c.mydomain.com Node Attributes: * Node a.mydomain.com: + master-pgsql:2 : -INFINITY + pgsql-data-status : DISCONNECT + pgsql-status : STOP * Node c.mydomain.com: + master-pgsql:0 : -INFINITY + master-pgsql:2 : -INFINITY + pgsql-data-status : SYNC + pgsql-status : HS:async * Node b.mydomain.com: + master-pgsql:0 : 1000 + master-pgsql:1 : 1000 + pgsql-data-status : SYNC + pgsql-master-baseline : 3200 + pgsql-status : PRI I was maunal sync pgsql from a node to b node and remove /var/lib/pgsql/tmp/PGSQL.lock /var/lib/postgresql/9.3/main/recovery.conf on a node. Now, how to better way ask cluster about migrate master pgsql on a node? crm_resource --ban --resource msPostgresql --host c.mydomain.com 2013/12/25 Jeff Frost j...@pgexperts.com So, does this not work: crm_attribute -l reboot -N c.mydomain.com --name master-pgsql:2 --delete ? On Dec 25, 2013, at 8:27 AM, Andrey Rogovsky a.rogov...@gmail.com wrote: I run similar commands. But I have unwanted attributes and want to delete they. 2013/12/25 Jeff Frost j...@pgexperts.com I usually do something like this on the master I want to bring up: #!/bin/bash rm -f /var/lib/pgsql/9.2/data/recovery.conf rm -f ~postgres/tmp/PGSQL.lock crm_attribute -l reboot -N $(uname -n) -n pgsql-data-status -v LATEST crm_attribute -l reboot -N $(uname -n) -n master-pgsql -v 1000 crm resource cleanup msPostgresql crm resource cleanup master-group On Dec 24, 2013, at 11:45 PM, Andrey Rogovsky a.rogov...@gmail.com wrote: Thanks a lot! It really help me! Now I have this status: Node Attributes: * Node a.mydomain.com: + master-pgsql:0 : 1000 + master-pgsql:1 : -INFINITY + pgsql-data-status : LATEST + pgsql-master-baseline : 2F90 + pgsql-status : PRI * Node c.mydomain.com: + master-pgsql:0 : 1000 + master-pgsql:2 : -INFINITY + pgsql-data-status : SYNC + pgsql-status : STOP * Node b.mydomain.com: + master-pgsql:0 : 1000 + master-pgsql:1 : -INFINITY + pgsql-data-status : SYNC + pgsql-status : STOP I can begin start pgsql replication, but don't like this: + master-pgsql:1 : -INFINITY + master-pgsql:2 : -INFINITY I want removie this attributes. I try run: root@b:~# crm_attribute -N c.mydomain.com --name master-pgsql:2 --delete And got answer: Error performing operation: cib object missing May be I miss something... How I can delete unwanted attribuies? 2013/12/24 Jeff Frost j...@pgexperts.com On Dec 23, 2013, at 9:47 PM, Andrey Rogovsky a.rogov...@gmail.com wrote: crm_attribute -l forever -N a.mydomain.com -n pgsql-data-status -v LATEST crm_attribute -l forever -N b.mydomain.com -n pgsql-data-status -v SYNC crm_attribute -l forever -N c.mydomain.com -n pgsql-data-status -v SYNC Try: crm_attribute -l reboot instead of -l forever --- Jeff Frost j...@pgexperts.com CTO, PostgreSQL Experts, Inc. Phone: 1-888-PG-EXPRT x506 FAX: 415-762-5122 http://www.pgexperts.com/ ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems --- Jeff Frost j...@pgexperts.com CTO, PostgreSQL Experts, Inc. Phone: 1-888-PG-EXPRT x506 FAX: 415-762-5122 http://www.pgexperts.com/ ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha
Re: [Linux-HA] Problem not in our membership
On 6 Dec 2013, at 6:57 pm, Moullé Alain alain.mou...@bull.net wrote: Hi, I've found a thread talking about this problem on 1.1.7, but at the end , is the patch : https://github.com/ClusterLabs/pacemaker/commit/03f6105592281901cc10550b8ad19af4beb5f72f sufficient and correct to solve the problem ? yes Thanks Alain Le 03/12/2013 10:15, Moullé Alain a écrit : Hi, with : pacemaker-1.1.7-6 corosync-1.4.1-15 On crm migrate , I'm randomly facing this problem : ... node1 daemon warning cib warning: cib_peer_callback: Discarding cib_apply_diff message (342) from node2: not in our membership whereas the node2 is healthy and always member of the cluster. Is-it a known problem ? Is there already a patch ? Thanks Alain ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] FYI: resource-agents-3.9.2-40.el6.x86_64 kills heartbeat-3.0.4
On Wed, Nov 27, 2013, at 06:15 PM, Jefferson Ogata wrote: On 2013-11-28 01:55, Andrew Beekhof wrote: On 28 Nov 2013, at 11:29 am, Jefferson Ogata linux...@antibozo.net wrote: On 2013-11-28 00:12, Dimitri Maziuk wrote: Just so you know: RedHat's (centos, actually) latest build of resource-agents sets $HA_BIN to /usr/libexec/heartbeat. The daemon in heartbeat-3.0.4 RPM is /usr/lib64/heartbeat/heartbeat so $HA_BIN/heartbeat binary does not exist. (And please hold the upgrade to pacemaker comments: I'm hoping if I wait just a little bit longer I can upgrade to ceph and openstack -- or retire, whichever comes first ;) Hey, upgrading to pacemaker wouldn't necessarily help. Red Hat broke that last month by dropping most of the resource agents they'd initially shipped. (Don't you love Technology Previews?) Thats the whole point behind the tech preview label... it means the software is not yet in a form that Red Hat will support and is subject to changes _exactly_ like the one made to resource-agents. Um, yes, i know. That's why i mentioned it. Ok, sorry, I wasn't sure. It's nicer, however, when Red Hat takes a conservative position with the Tech Preview. They could have shipped a minimal set of resource agents in the first place, 3 years ago we didn't know if pacemaker would _ever_ be supported in RHEL-6, so stripping out agents wasn't on our radar. I'm sure the only reason it and the rest of pacemaker shipped at all was to humor the guy they'd just hired. It was only at the point that supporting pacemaker in 6.5 became likely that someone took a look at the full list and had a heart-attack. so people would have a better idea what they had to provide on their own end, instead of pulling the rug out with nary a mention of what they were doing. Yes, that was not good. One of the challenges I find at Red Hat is the gaps between when a decision is made, when we're allowed to talk about it and when customers find out about it. As a developermore its the things we spent significant time on that first come to mind when writing release notes, not the 3s it took to remove some files from the spec file - even though the latter is going to have a bigger affect :-( We can only say that lessons have been learned and that we will do better if there is a similar situation next time. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] FYI: resource-agents-3.9.2-40.el6.x86_64 kills heartbeat-3.0.4
On Thu, Nov 28, 2013, at 10:26 AM, Dimitri Maziuk wrote: On 2013-11-27 20:15, Jefferson Ogata wrote: It's nicer, however, when Red Hat takes a conservative position with the Tech Preview. They could have shipped a minimal set of resource agents in the first place, so people would have a better idea what they had to provide on their own end, instead of pulling the rug out with nary a mention of what they were doing. The other issue is epel packaging: unsupported obsolete heartbeat-3.0.4 should arguably not depend on bleeding edge tech preview packages in the first place. If anything, it should conflict with pacemaker co. Technically pacemaker can still be built to support heartbeat, so making it a conflict is not strictly correct. The complications came from the combination of some of EPELs policies, build ordering/dependancies and heartbeat being overly monolithic to begin with. However, if epel maintainer has no time to make 30-second edit to /etc/init.d script I don't expect him to repackage the whole thing right in my lifetime. Dima ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] fence_apc always fails after some time and resources remains stopped
On Thu, Nov 28, 2013, at 12:11 AM, RaSca wrote: Il giorno Ven 22 Nov 2013 10:26:08 CET, RaSca ha scritto: [...] After this resources remains in stopped state. Why this happens? Am I in this case: https://github.com/ClusterLabs/pacemaker/pull/334 ? What kind of workaround can I use? Thanks a lot, as usual. I don't know if my problem is the one described in the errata above, It shouldn't be, that problem was specific to those using non-RH agents. but what I found for resolving my problem was to use fence_apc_snmp instead of fence_apc. Since it is using snmp it is faster and it never timeout. All the other questions are still open, so if you want to give a suggestion I am open for the discussion. -- RaSca Mia Mamma Usa Linux: Niente è impossibile da capire, se lo spieghi bene! ra...@miamammausalinux.org http://www.miamammausalinux.org ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] FYI: resource-agents-3.9.2-40.el6.x86_64 kills heartbeat-3.0.4
On 28 Nov 2013, at 11:29 am, Jefferson Ogata linux...@antibozo.net wrote: On 2013-11-28 00:12, Dimitri Maziuk wrote: Just so you know: RedHat's (centos, actually) latest build of resource-agents sets $HA_BIN to /usr/libexec/heartbeat. The daemon in heartbeat-3.0.4 RPM is /usr/lib64/heartbeat/heartbeat so $HA_BIN/heartbeat binary does not exist. (And please hold the upgrade to pacemaker comments: I'm hoping if I wait just a little bit longer I can upgrade to ceph and openstack -- or retire, whichever comes first ;) Hey, upgrading to pacemaker wouldn't necessarily help. Red Hat broke that last month by dropping most of the resource agents they'd initially shipped. (Don't you love Technology Previews?) Thats the whole point behind the tech preview label... it means the software is not yet in a form that Red Hat will support and is subject to changes _exactly_ like the one made to resource-agents. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Node affinity across reboots
On 15 Nov 2013, at 4:37 am, Michael Jones michael.jo...@quantum.com wrote: Hello, I'm attempting to use pacemaker/corosync (1.1.10/2.31) in a two node active/passive embedded application where all resources should run only on one node. I'm looking for a configuration option that will remember the last active node when the appliance is rebooted. I'm using postgresql replication and would like the appliance to always boot to the most recently active node! Do you mean that after the cluster was completely stopped and starts on both nodes, you want the application to remember which node it was on prior to the cluster stopping? That's correct. Sorry, I should have clarified, the two nodes that I'm talking about are in the same appliance connected by a backplane. It seems like I need to set some attribute on a node that can be used in a rule, maybe? We don't natively support such a feature, but you could have your resource agent create/update a location constraint to push it to one node or the other. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Node affinity across reboots
On 14 Nov 2013, at 9:25 am, Michael Jones michael.jo...@quantum.com wrote: Hello, I'm attempting to use pacemaker/corosync (1.1.10/2.31) in a two node active/passive embedded application where all resources should run only on one node. I'm looking for a configuration option that will remember the last active node when the appliance is rebooted. I'm using postgresql replication and would like the appliance to always boot to the most recently active node! Do you mean that after the cluster was completely stopped and starts on both nodes, you want the application to remember which node it was on prior to the cluster stopping? signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] iSCSI corruption during interconnect failure with pacemaker+tgt+drbd+protocol C
On 13 Nov 2013, at 2:10 pm, Jefferson Ogata linux...@antibozo.net wrote: Here's a problem i don't understand, and i'd like a solution to if possible, or at least i'd like to understand why it's a problem, because i'm clearly not getting something. I have an iSCSI target cluster using CentOS 6.4 with stock pacemaker/CMAN/corosync and tgt, and DRBD 8.4 which i've built from source. Both DRBD and cluster comms use a dedicated crossover link. The target storage is battery-backed RAID. DRBD resources all use protocol C. stonith is configured and working. tgtd write cache is disabled using mode_page in additional_params. This is correctly reported using sdparm --get WCE on initiators. Here's the question: if i am writing from an iSCSI initiator, and i take down the crossover link between the nodes of my cluster, i end up with corrupt data on the target disk. I know this isn't the formal way to test pacemaker failover. Everything's fine if i fence a node or do a manual migration or shutdown. But i don't understand why taking the crossover down results in corrupted write operations. In greater detail, assuming the initiator sends a write request for some block, here's the normal sequence as i understand it: - tgtd receives it and queues it straight for the device backing the LUN (write cache is disabled). - drbd receives it, commits it to disk, sends it to the other node, and waits for an acknowledgement (protocol C). - the remote node receives it, commits it to disk, and sends an acknowledgement. - the initial node receives the drbd acknowledgement, and acknowledges the write to tgtd. - tgtd acknowledges the write to the initiator. Now, suppose an initiator is writing when i take the crossover link down, and pacemaker reacts to the loss in comms by fencing the node with the currently active target. It then brings up the target on the surviving, formerly inactive, node. This results in a drbd split brain, since some writes have been queued on the fenced node but never made it to the surviving node, and must be retransmitted by the initiator; once the surviving node becomes active it starts committing these writes to its copy of the mirror. I'm fine with a split brain; i can resolve it by discarding outstanding data on the fenced node. But in practice, the actual written data is lost, and i don't understand why. AFAICS, none of the outstanding writes should have been acknowledged by tgtd on the fenced node, so when the surviving node becomes active, the initiator should simply re-send all of them. But this isn't what happens; instead most of the outstanding writes are lost. No i/o error is reported on the initiator; stuff just vanishes. I'm writing directly to a block device for these tests, so the lost data isn't the result of filesystem corruption; it simply never gets written to the target disk on the survivor. What am i missing? iSCSI, drbd, etc are not really my area of expertise, but it may be worth taking the cluster out of the loop and manually performing the equivalent actions. If the underlying drbd and iSCSI setups have a problem, then the cluster isn't going to do much about it. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: How many primitives, groups can I have
On 12 Nov 2013, at 12:01 am, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I guess there is no direct limit for the number of primitives, but performance may depend on that number: Maybe O(1), mybe (O(n), hopefully not (On^2) or worse. Immediately I suspect the communication protocol (TOTEM) may run into problems if a lot of data is exchanged, probably even after tuning some parameters. Unfortunately the CIB and to some extent the PE are more of a bottleneck. There is some experimental code in git that tries to automagically throttle the flood of actions to keep the CIB under control though. I know, it's not the answer you wanted to read, but... Regards, Ulrich Michael Brookhuis mimabr...@googlemail.com schrieb am 11.11.2013 um 13:57 in Nachricht 5280d42d.8030...@googlemail.com: Hi, Is there a limit in the number of proimitives, etc you can have? What maximum number is recommended based on best-practices? Are 1500 to many? Thanks Mima ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Two fencing devices, long timeout - why?
I can't really comment until I know which version of pacemaker this is... On 1 Nov 2013, at 2:57 am, Jakob Curdes j...@info-systems.de wrote: Hi , I have a cman-based cluster that uses pcmk-fencing. we have configured an ipmilan fencing device and an apc fencing device with stonith. I set a fencing order like this: fencing-topology \ fencing-level devices=ipmi_gw2 id=fencing-gw2-2 index=2 target=gw2/ \ fencing-level devices=apc_power_gw2 id=fencing-gw2-1 index=1 target=gw2/ \ /fencing-topology This all works as intended, i.e. the apc is used as first device and shuts down the server. But when I try to simulate a failure of the APC device by setting an alternate IP (that is not reachable), fencing takes a very long time before it succeeds. stonith-timeout for cluster is 20 secs. So I would expect that after 20 secs it moves on to the second device. However what I see is this: Oct 31 16:09:59 corosync [TOTEM ] A processor failed, forming new configuration. # machine powered off manually Oct 31 16:10:00 [5875] gw1 stonith-ng: info: stonith_action_create: Initiating action list for agent fence_apc (target=(null)) Oct 31 16:10:00 [5875] gw1 stonith-ng: info: internal_stonith_action_execute: Attempt 2 to execute fence_apc (list). remaining timeout is 120 Oct 31 16:10:01 [5875] gw1 stonith-ng: info: update_remaining_timeout: Attempted to execute agent fence_apc (list) the maximum number of times (2) allowed (...) Oct 31 16:10:06 [5879] gw1 crmd: notice: te_fence_node: Executing reboot fencing operation (151) on gw2 (timeout=2) Oct 31 16:10:06 [5875] gw1 stonith-ng: notice: handle_request: Client crmd.5879.4fe77863 wants to fence (reboot) 'gw2' with device '(any)' Oct 31 16:10:06 [5875] gw1 stonith-ng: notice: merge_duplicates: Merging stonith action reboot for node gw2 originating from client crmd.5879.aa3600b2 with identical request from stonith_admin.27011@gw1.ad19c5b3 (24s) Oct 31 16:10:06 [5875] gw1 stonith-ng: info: initiate_remote_stonith_op: Initiating remote operation reboot for gw2: aa3600b2-7b6d-4243-b525-de0a0a7399a8 (duplicate) Oct 31 16:10:06 [5875] gw1 stonith-ng: info: stonith_command: Processed st_fence from crmd.5879: Operation now in progress (-115) *Oct 31 16:11:30 [5879] gw1 crmd:error: stonith_async_timeout_handler:Async call 4 timed out after 84000ms* Oct 31 16:11:30 [5879] gw1 crmd: notice: tengine_stonith_callback: Stonith operation 4/151:40:0:304a2845-8177-4018-9fb6-7b94d0d1288a: Timer expired (-62) Oct 31 16:11:30 [5879] gw1 crmd: notice: tengine_stonith_callback: Stonith operation 4 for gw2 failed (Timer expired): aborting transition. Oct 31 16:11:30 [5879] gw1 crmd: info: abort_transition_graph: tengine_stonith_callback:447 - Triggered transition abort (complete=0) : Stonith failed Oct 31 16:11:30 [5879] gw1 crmd: notice: te_fence_node: Executing reboot fencing operation (151) on gw2 (timeout=2) Oct 31 16:11:30 [5875] gw1 stonith-ng: notice: handle_request: Client crmd.5879.4fe77863 wants to fence (reboot) 'gw2' with device '(any)' Oct 31 16:11:30 [5875] gw1 stonith-ng: notice: merge_duplicates: Merging stonith action reboot for node gw2 originating from client crmd.5879.429b7850 with identical request from stonith_admin.27011@gw1.ad19c5b3 (24s) Oct 31 16:11:30 [5875] gw1 stonith-ng: info: initiate_remote_stonith_op: Initiating remote operation reboot for gw2: 429b7850-9d23-49f0-abe4-1f18eb8d122a (duplicate) Oct 31 16:11:30 [5875] gw1 stonith-ng: info: stonith_command: Processed st_fence from crmd.5879: Operation now in progress (-115) Oct 31 16:12:24 [5875] gw1 stonith-ng: info: call_remote_stonith: Total remote op timeout set to 240 for fencing of node gw2 for stonith_admin.27011.ad19c5b3 *Oct 31 16:12:24 [5875] gw1 stonith-ng: info: call_remote_stonith: Requesting that gw1 perform op reboot gw2 with ipmi_gw2 for stonith_admin.27011 (144s)* Oct 31 16:12:24 [5875] gw1 stonith-ng: info: stonith_command: Processed st_fence from gw1: Operation now in progress (-115) Oct 31 16:12:24 [5875] gw1 stonith-ng: info: stonith_action_create: Initiating action reboot for agent fence_ipmilan (target=gw2) *Oct 31 16:12:26 [5875] gw1 stonith-ng: notice: log_operation: Operation 'reboot' [27473] (call 0 from stonith_admin.27011) for host 'gw2' with device 'ipmi_gw2' returned: 0 (OK)* So it does not take 25 seconds to reboot gw2 withthe fallback stonith device but more than two minutes, although we even see the 20 seconds timeout values. What am I doing wrong? Here is the device config : primitive apc_power_gw2 stonith:fence_apc \ params ipaddr=192.168.33.64 pcmk_host_list=gw2 pcmk_host_check=static-list pcmk_host_argument=none
Re: [Linux-HA] RH and gfs2 and Pacemaker
On 16/10/2013, at 4:50 PM, Moullé Alain alain.mou...@bull.net wrote: Hi Andrew, thanks. and when switching to last option 4: For RHEL7+, option 4: corosync + cpg + quorumd + mcp what will be the status around the two binaries to use ? gfs_controld goes away but dlm_controld remains Thanks Alain Le 15/10/2013 22:40, Andrew Beekhof a écrit : On 16/10/2013, at 1:35 AM, Moullé Alain alain.mou...@bull.net wrote: OK , I was following this documentation : http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/ch08s02.html the key pieces are the dlm_controld and gfs_controld helpers -- no .pcmk suffix Part of the reason for supporting cman is to avoid needing special versions of those daemons. is it obsolete ? or is there a new documentation more recent elsewhere ? Thanks again Alain Le 15/10/2013 16:34, Lars Marowsky-Bree a écrit : On 2013-10-15T16:25:37, Moullé Alain alain.mou...@bull.net wrote: Hi Lars, thanks a lot for information. I 'll try, but the documentation asks for gfs2-cluster rpm installation, and for now I don't find this rpm on RHEL6.4, and don't know if it is always required ... but it is not in your side ;-) gfs2-cluster? I thought all that was needed was gfs2-utils. Without the gfs2-controld anymore, I don't think you need that package. Regards, Lars ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] RH and gfs2 and Pacemaker
On 16/10/2013, at 1:35 AM, Moullé Alain alain.mou...@bull.net wrote: OK , I was following this documentation : http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/ch08s02.html the key pieces are the dlm_controld and gfs_controld helpers -- no .pcmk suffix Part of the reason for supporting cman is to avoid needing special versions of those daemons. is it obsolete ? or is there a new documentation more recent elsewhere ? Thanks again Alain Le 15/10/2013 16:34, Lars Marowsky-Bree a écrit : On 2013-10-15T16:25:37, Moullé Alain alain.mou...@bull.net wrote: Hi Lars, thanks a lot for information. I 'll try, but the documentation asks for gfs2-cluster rpm installation, and for now I don't find this rpm on RHEL6.4, and don't know if it is always required ... but it is not in your side ;-) gfs2-cluster? I thought all that was needed was gfs2-utils. Without the gfs2-controld anymore, I don't think you need that package. Regards, Lars ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Max number of resources under Pacemaker ?
On 30/09/2013, at 10:49 PM, Moullé Alain alain.mou...@bull.net wrote: Hi, sorry for the delay on this thread, I was unavailable a few weeks, but just FYI, I wanted to share some results I got a few weeks ago: I've tried some tests on a configuration and start/stop of 500 Dummy resources, and I got these time values : 1/ configuration with successive crm commands crm configure primitive ... :**it takes about 1H so it is not usable 2/ with a unique crm command crm configure File with all dummy primitives in File : it takes 7s / that's OK 3/ add just one location constraint for each dummy primitive with crm configure File with all constraints in File : it takes 27s / strange but acceptable 4/ Start of the 500 primitives with successive crm commands crm resource start ... :it takes 7mn28 / seems not acceptable moreover for dummy resources ... 5/ Start of the 500 primitives with parallel (background) crm commands crm resource start ... : not possible, lots of commands exit in errors and anyway it takes also long time 6/ Start of the 500 primitives in parallel by setting all target-roles to Started in Pacemaker: = with crm configure edit : s/Stopped/Started on the 500 primitives Result : around6 mn for all primitives to be started. Seems not acceptable moreover for dummy resources , and it will take let's say about 3 mn for a failover if primitives are well located half on one node and half on the other. These results are with dummy resources, and we can imagine that with real resources it will take much longer, not speaking about the periodic monitoring of 500 primitives ... So, based on these results, I think that the limit in number of resources is far below 500 resources ... Its not so much the pure number of resources, but how many are doing things at the same time. As Dejan mentioned, there is batch-limit but finding some way to auto-tune this is on the short-term agenda. But I wanted to give these results just to keep going on this subject and perhaps get some ideas ... Thanks Alain Le 05/09/2013 10:58, Lars Marowsky-Bree a écrit : On 2013-09-04T08:26:14, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: In my experience network traffic grows somewhat linear with the size of the CIB. At some point you probably have to change communication parameters to keep the cluster in a happy comminication state. Yes, I wish corosync would auto-tune to a higher degree. Apparently though, that's a slightly harder problem. We welcome any feedback on required tunables. Those that we ship on SLE HA worked for us (and even for rather largeish configurations), but they may not be appropriate everywhere. Despite of the cluster internals, there may be problems if a node goes online and hundreds of resources are started in parallel, specifically if those resources weren't designed for it. I suspect IP addresses, MD-RAIDs, LVM stuff, drbd, filesystems, exportfs, etc. No, most of these resource scripts *are* supposed to be concurrency-safe. If you find something that breaks, please share the feedback. It's true that the way how concurrent load limitation is implemented in Pacemaker/LRM isn't perfect yet. batch-limit is rather coarse. The per-node LRM child limit is probably the best bet right now. But it doesn't differentiate between starting many light-weight resources in parallel (such as IPaddr) versus heavy-weights (VMs with Oracle databases). (migration-threshold goes in the same direction.) Historical context matters. Pacemaker comes from the HA world; we still believe 3-7 node clusters are the largest anyone ought to reasonably build, considering the failure/admin/security domain issues with single point of failures and the increasing likelihood of double failures etc. But there's several trends - Even those 3-7 nodes become increasingly powerful multi-core kick-ass boxes. 7 nodes might well host hundreds of resources nowadays (say, above 70 VMs with all their supporting resources). People build much larger clusters because there's no good way to divide and conquer yet - e.g., if you build several 3 or 5 node clusters, there's no support for managing those clusters-of-clusters. And people use Pacemaker for HPC style deployments (e.g., private clouds with tons of VMs) - because while our HPC support is suboptimal, it is better than the HA support in most of the Cloud offerings. As a note: Just recently we had a failure in MD-RAID activation with no real reason to be found in syslog, and the cluster got quite confused. (I had reported this to my favourite supporter (SR 10851868591), but haven't heard anything since then...) I'll try to dig that out of the support system and give it a look. Regards, Lars ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org
Re: [Linux-HA] problems eliminating the use of multicast (fwd)
On 19/09/2013, at 8:52 PM, Jakob Curdes j...@info-systems.de wrote: Am 19.09.2013 11:49, schrieb David Lang: On Thu, 19 Sep 2013, Jakob Curdes wrote: That's the direction we started, but apparently the centos pacemaker/corosync packages don't look at the corosync.conf file, they expect to extract everything out of cluster.conf. Ok but you have noted that the transport parameter in our config is a parameter of cman, not of totem? We also have a totem definition but this only sets some timeouts. Also it seems that at least some parameters in corosync.conf are read, but I cannot recall which ones. No. When cman is in use, corosync.conf is not touched. cman make be setting some defaults to the same values though JC ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10
On 06/09/2013, at 5:51 PM, Lars Ellenberg lars.ellenb...@linbit.com wrote: On Tue, Aug 27, 2013 at 06:51:45AM +0200, Andreas Mock wrote: Hi Andrew, as this is a real showstopper at the moment I invested some other hours to be sure (as far as possible) not having made an error. Some additions: 1) I mirrored the whole mini drbd config to another pacemaker cluster. Same result: pacemaker 1.1.8 works, pacemaker 1.1.10 not 2) When I remove the target role Stopped from the drbd ms resource and insert the config snippet related to the drbd device via crm -f file to a lean running pacemaker config (pacemaker cluster options, stonith resources), it seems to work. That means one of the nodes gets promoted. Then after stopping 'crm resource stop ms_drbd_xxx' and starting again I see the same promotion error as described. The drbd resource agent is using /usr/sbin/crm_master. Is there a possibility that feedback given through this client tool is changing the timing behaviour of pacemaker? Or the way transitions are scheduled? Any idea that may be related to a change in pacemaker? I think that recent pacemaker allows for start and promote in the same transition. At least in the one case I saw logs of, this wasn't the case. The PE computed: Current cluster status: Online: [ db05 db06 ] r_stonith-db05 (stonith:fence_imm):Started db06 r_stonith-db06 (stonith:fence_imm):Started db05 Master/Slave Set: ms_drbd_fodb [r_drbd_fodb] Slaves: [ db05 db06 ] Master/Slave Set: ms_drbd_fodblog [r_drbd_fodblog] Slaves: [ db05 db06 ] Transition Summary: * Promote r_drbd_fodb:0 (Slave - Master db05) * Promote r_drbd_fodblog:0 (Slave - Master db05) and it was the promotion of r_drbd_fodb:0 that failed. Or at least has promote follow start much faster than before. To be able to really tell why DRBD has problems to promote, I'd need the drbd (kernel!) logs, the agent logs are not good enough. Maybe you hit some interesting race between DRBD establishing the connection, and pacemaker trying to promote one of the nodes. Correlating DRBD kernel logs with pacemaker logs should tell. Do you have DRBD resource level fencing enabled? I guess right now you can reproduce it easily by: crm resource stop ms_drbd crm resource start ms_drbd I suspect you would not be able to reproduce by: crm resource stop ms_drbd crm resource demote ms_drbd (will only make drbd Secondary stuff) ... meanwhile, DRBD will establish the connection ... crm resource promote ms_drbd (will then promote one node) Hth, Lars -Ursprüngliche Nachricht- Von: Andrew Beekhof [mailto:and...@beekhof.net] Gesendet: Dienstag, 27. August 2013 05:02 An: General Linux-HA mailing list Cc: pacema...@oss.clusterlabs.org Betreff: Re: [Pacemaker] [Linux-HA] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10 On 27/08/2013, at 3:31 AM, Andreas Mock andreas.m...@web.de wrote: Hi all, while the linbit drbd resource agent seems to work perfectly on pacemaker 1.1.8 (standard software repository) we have problems with the last release 1.1.10 and also with the newest head 1.1.11.xxx. As using drbd is not so uncommon I really hope to find interested people helping me out. I can provide as much debug information as you want. Environment: RHEL 6.4 clone (Scientific Linux 6.4) cman based cluster. DRBD 8.4.3 compiled from sources. 64bit - A drbd resource configured following the linbit documentation. - Manual start and stop (up/down) and setting primary of drbd resource working smoothly. - 2 nodes dis03-test/dis04-test - Following simple config on pacemaker 1.1.8 configure property no-quorum-policy=stop property stonith-enabled=true rsc_defaults resource-stickiness=2 primitive r_stonith-dis03-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis03-test pcmk_host_list=dis03-test primitive r_stonith-dis04-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis04-test pcmk_host_list=dis04-test location r_stonith-dis03_hates_dis03 r_stonith-dis03-test \ rule $id=r_stonith-dis03_hates_dis03-test_rule -inf: #uname eq dis03-test location r_stonith-dis04_hates_dis04 r_stonith-dis04-test \ rule $id=r_stonith-dis04_hates_dis04-test_rule -inf: #uname eq dis04-test primitive r_drbd_postfix ocf:linbit:drbd \ params drbd_resource=postfix drbdconf=/usr/local/etc/drbd.conf \ op monitor interval=15s timeout=60s role=Master \ op monitor interval=45s timeout=60s
Re: [Linux-HA] Max number of resources under Pacemaker ?
On 04/09/2013, at 4:09 PM, Vladislav Bogdanov bub...@hoster-ok.com wrote: 04.09.2013 07:16, Andrew Beekhof wrote: On 03/09/2013, at 9:20 PM, Moullé Alain alain.mou...@bull.net wrote: Hello, A simple question : is there a maximum number of resources (let's say simple primitives) that Pacemaker can support at first at configuration of ressources via crm, and of course after configuration when Pacemaker has to monitor all the primitives ? Simple answer: it depends (more precisely, could we envisage around 500 or 600 primitives, or is it completely mad ? ;-) ) (I know it is dependant on node power, CPU, mem, etc., but I'm speaking here only of eventual Pacemaker limitations) There is no inherent limit, the policy engine can cope with many thousands. The CIB is less able to cope - for which batch-limit is useful (to throttle the number of operation updates being thrown at the CIB which limits its CPU usage). The other limit is local and cluster messaging sizes - once the compressed cib gets too big for either or both transports you can no longer even run 'cibadmin -Q' For IPC, the limit is tuneable via the environment. For corosync, its high (1Mb) but (I think) only tuneable at compile time. Are there any possibilities/plans to implement partial messages? This is part of the reason we send only the changes that result from an update. Message fragmenting would have to happen at the corosync/libqb level. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Max number of resources under Pacemaker ?
On 04/09/2013, at 4:26 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! In my experience network traffic grows somewhat linear with the size of the CIB. At some point you probably have to change communication parameters to keep the cluster in a happy comminication state. Yes. Tuning corosync.conf for larger clusters is still crucial. Despite of the cluster internals, there may be problems if a node goes online and hundreds of resources are started in parallel, specifically if those resources weren't designed for it. I suspect IP addresses, MD-RAIDs, LVM stuff, drbd, filesystems, exportfs, etc. Thats what batch-limit is for, it slows down the flood (same 'Mb' of traffic but greater 's'). On the downside however, it slows down the flood causing recovery to take longer. As a note: Just recently we had a failure in MD-RAID activation with no real reason to be found in syslog, and the cluster got quite confused. (I had reported this to my favourite supporter (SR 10851868591), but haven't heard anything since then...) Regards, Ulrich Andrew Beekhof and...@beekhof.net schrieb am 04.09.2013 um 06:16 in Nachricht 3703-9464-458e-9024-8119c8066...@beekhof.net: On 03/09/2013, at 9:20 PM, Moullé Alain alain.mou...@bull.net wrote: Hello, A simple question : is there a maximum number of resources (let's say simple primitives) that Pacemaker can support at first at configuration of ressources via crm, and of course after configuration when Pacemaker has to monitor all the primitives ? Simple answer: it depends (more precisely, could we envisage around 500 or 600 primitives, or is it completely mad ? ;-) ) (I know it is dependant on node power, CPU, mem, etc., but I'm speaking here only of eventual Pacemaker limitations) There is no inherent limit, the policy engine can cope with many thousands. The CIB is less able to cope - for which batch-limit is useful (to throttle the number of operation updates being thrown at the CIB which limits its CPU usage). The other limit is local and cluster messaging sizes - once the compressed cib gets too big for either or both transports you can no longer even run 'cibadmin -Q' For IPC, the limit is tuneable via the environment. For corosync, its high (1Mb) but (I think) only tuneable at compile time. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Max number of resources under Pacemaker ?
On 04/09/2013, at 5:50 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 04.09.2013 um 08:27 in Nachricht ffc283dc-1f66-4b41-a919-1d66e1bf8...@beekhof.net: On 04/09/2013, at 4:09 PM, Vladislav Bogdanov bub...@hoster-ok.com wrote: [...] Are there any possibilities/plans to implement partial messages? This is part of the reason we send only the changes that result from an update. I haven't checked recently, but when a lot of communication happened, applying the diffs frequently failed , and a full copy was transferred. I guess it's due to the fact that the protocol is not a two-way (request response) diff protocol, but just a one-way diff broadcast protocol. So if a node didn't have the previous version it cannot process the latest diff and needs a full copy instead. In this case it asks for and is sent a full copy to get it back on track. Frequently failing diffs is symptomatic of a bug[1], not a design flaw. [1] Eg. the one fixed by + Andrew Beekhof (11 months ago) 19589ee: High: Core: Prevent ordering changes when applying xml diffs I don't know how well this scales for big clusters with massive updates. BTW: What are the reasons why a DC changes node while current DC-node is fine? Communication difficulties and cib replace operations (its a low cost way to regenerate the status section). Occasionally this also happens, causing some extra communication... Message fragmenting would have to happen at the corosync/libqb level. Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Max number of resources under Pacemaker ?
On 03/09/2013, at 9:20 PM, Moullé Alain alain.mou...@bull.net wrote: Hello, A simple question : is there a maximum number of resources (let's say simple primitives) that Pacemaker can support at first at configuration of ressources via crm, and of course after configuration when Pacemaker has to monitor all the primitives ? Simple answer: it depends (more precisely, could we envisage around 500 or 600 primitives, or is it completely mad ? ;-) ) (I know it is dependant on node power, CPU, mem, etc., but I'm speaking here only of eventual Pacemaker limitations) There is no inherent limit, the policy engine can cope with many thousands. The CIB is less able to cope - for which batch-limit is useful (to throttle the number of operation updates being thrown at the CIB which limits its CPU usage). The other limit is local and cluster messaging sizes - once the compressed cib gets too big for either or both transports you can no longer even run 'cibadmin -Q' For IPC, the limit is tuneable via the environment. For corosync, its high (1Mb) but (I think) only tuneable at compile time. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Quick 'death match cycle' question.
On 03/09/2013, at 4:32 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! I don't have a real answer for this, but I can report other bad experience with 2-node cluster like yours: If the DC is fenced, the other node tries to become DC, but if the other node (who still thinks he's DC) reboots just before the other node has completed his ego trip, both nodes cannot agree on who's becoming DC. I'll have to reboot (or shut down OpenAIS) on one of these nodes. Seen in SLES11 SP2 (latest updates). I hope you've reported that to suse. Definitely should not be happening. An idea for you problem: If the cluster would count reboots within a timeframe (e.g. as node attribute), the fencong operation could change from reboot to poweroff. I don't know how to do it, though. Regards, Ulrich Alex Sudakar alex.suda...@gmail.com schrieb am 03.09.2013 um 05:23 in Nachricht calq2s-hxkq5ghv9bs1snnojk4gtnl1su-nzujpdxwosv2ap...@mail.gmail.com: I've got a very simple question which I suspect betrays my lack of understanding of something basic. Could someone help me understand? If I have a two-node Pacemaker cluster - say, a really simple cluster of two nodes, A B, with a solitary network connection between them - then I have to set no-quorum-policy to 'ignore'. If the network connection is broken then both A B will attempt to STONITH each other. Is there anything that would stop an endless cycle of each killing the other if the actions of the STONITH agents are set to reboot? I.e.: - A B race to STONITH each other - A kills B - A assumes resources - B reboots - B can't see A - B kills A - B assumes resources - A reboots - A can't see B - A kills B - A assumes resources ... etc. It's to stop this sort of cycle that I've set my STONITH actions to 'off' rather than 'reboot'. But I was reading the 'Fencing topology' document that Digimer referenced and I was reminded in my perusal that many people/clusters use a 'reboot' action. For a simple quorum-less cluster of two nodes how do those clusters avoid a never-ending cycle of each node killing the other, if neither node can 'see' the other via corosync? It's a very basic question; I think I'm forgetting something obvious. Thanks for any help! ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Quick 'death match cycle' question.
On 04/09/2013, at 3:02 AM, Lars Marowsky-Bree l...@suse.com wrote: On 2013-09-03T10:25:58, Digimer li...@alteeve.ca wrote: I've run only 2-node clusters and I've not seen this problem. That said, I've long-ago moved off of openais in favour of corosync. Given that membership is handled there, I would look at openais as the source of your trouble. This is, sorry, entirely wrong. openais does not handle membership. This sounds more like a DC election race window or some such in pacemaker. At worst that should only cause another election though. If do_election_count_vote() is reporting different winners to different people however... that would be very bad. If Ulrich files a bug report with proper logs, I'm sure we can resolve this (perhaps with an update to SP3 and a more recent pacemaker release ;-). Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Quick 'death match cycle' question.
On 04/09/2013, at 3:10 AM, Digimer li...@alteeve.ca wrote: On 03/09/13 13:08, Lars Marowsky-Bree wrote: On 2013-09-03T13:04:52, Digimer li...@alteeve.ca wrote: My mistake then. I had assumed that corosync was just a stripped down openais, so I figured openais provided the same functions. My personal experience with openais is limited to my early days of learning HA clustering on EL5. Yes and no. SLE HA 11 ships openais as a corosync add-on, because it still uses the AIS CKPT service for OCFS2 (which can't be changed w/o breaking wire-compatibility). But it's already corosync underneath. It still calls the init script openais for compatibility reasons (so that existing scripts that call that don't fail). So that can be confusing, because one still starts/stops openais ... In openSUSE Factory, we're this close to basing the stack on latest upstream of everything and cleanly so, I hope ;-) Regards, Lars Aaaah, so in this context, openais is referring to the plugin, not the stand-alone that I was thinking of. Makes much more sense now. Also it is a pacemaker plugin that is doing membership calculations. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] error: te_connect_stonith: Sign-in failed: triggered a retry
On 30/08/2013, at 1:37 PM, Tom Parker tpar...@cbnco.com wrote: This is happening when I am using the really large CIB Its really hard to imagine one causing the other. Next time, can you set PCMK_blackbox=yes in your environment and retest? The log file will indicate a file with more information. http://blog.clusterlabs.org/blog/2013/pacemaker-logging/ has details on what to look for and what to do with the result. and no. There doesn't seem to be anything else. 3 of my 6 nodes were showing this error. Now that I have deleted and recreated my CIB this log message seems to have gone away. On 08/29/2013 10:16 PM, Andrew Beekhof wrote: On 30/08/2013, at 5:51 AM, Tom Parker tpar...@cbnco.com wrote: Hello Since my upgrade last night I am also seeing this message in the logs on my servers. error: te_connect_stonith: Sign-in failed: triggered a retry Old mailing lists seem to imply that this is an issue with heartbeat which I don't think I am running. My software stack is this at the moment: cluster-glue-1.0.11-0.15.28 libcorosync4-1.4.5-0.18.15 corosync-1.4.5-0.18.15 pacemaker-mgmt-2.1.2-0.7.40 pacemaker-mgmt-client-2.1.2-0.7.40 pacemaker-1.1.9-0.19.102 Does anyone know where this may be coming from? were there any other errors? ie. did stonithd crash? Thanks Tom Parker. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] error: filter_colocation_constraint: both allocated but to different nodes
On 02/09/2013, at 11:24 PM, Thomas Schulte tho...@cupracer.de wrote: Hi Andrew, did you already find the time to have a look at the pe-input-59.bz2 file? I did. Its seems the latest version doesn't produce any errors. Might be a good opportunity to upgrade to 1.1.10 Thank you. Regards, Thomas Am 2013-08-28 06:25, schrieb Thomas Schulte: Hi Andrew, thank you! The latest file is attached (pe-input-59.bz2). Regards, Thomas Am 28.08.2013 um 01:40 schrieb Andrew Beekhof and...@beekhof.net: On 27/08/2013, at 8:54 PM, Thomas Schulte tho...@cupracer.de wrote: Hi list, I'm experiencing a strange problem an I can't figure out what's wrong. I'm running openSUSE 12.3 on a 2-node-cluster with pacemaker-1.1.9-55.2.x86_64. Every 15 minutes the following messages are logged. If stonith is enabled, my hosts get fenced afterwards. The messages appeared on both nodes until I temporarily activated stonith. After some fencing, now only the second node s00202 keeps logging these (both nodes online again): Aug 27 10:21:13 s00202 crmd[3390]: notice: do_state_transition: State transition S_IDLE - S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Aug 27 10:21:13 s00202 pengine[3389]: notice: unpack_config: On loss of CCM Quorum: Ignore Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_vmail and ms_drbd_vmail are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_mysql and ms_drbd_mysql are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_redis and ms_drbd_redis are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_bind and ms_drbd_bind are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_squid and ms_drbd_squid are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]: last message repeated 3 times Aug 27 10:21:13 s00202 crmd[3390]: notice: do_te_invoke: Processing graph 61 (ref=pe_calc-dc-1377591673-1345) derived from /var/lib/pacemaker/pengine/pe-input-53.bz2 Aug 27 10:21:13 s00202 crmd[3390]: notice: run_graph: Transition 61 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-53.bz2): Complete Aug 27 10:21:13 s00202 crmd[3390]: notice: do_state_transition: State transition S_TRANSITION_ENGINE - S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Aug 27 10:21:13 s00202 pengine[3389]: notice: process_pe_message: Calculated Transition 61: /var/lib/pacemaker/pengine/pe-input-53.bz2 Could you attach this file please? I'll be able to see if the current version behaves any better. I believe that the errors about filter_colocation_contraint all have the same cause, so I'm going to reduce the problem to the following: Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_vmail and ms_drbd_vmail are both allocated but to different nodes: s00202 vs. n/a This is my configuration (relevant extract): # crm configure show node s00201 node s00202 primitive pri_drbd_vmail ocf:linbit:drbd \ operations $id=pri_drbd_vmail-operations \ op monitor interval=20 role=Slave timeout=20 \ op monitor interval=10 role=Master timeout=20 \ params drbd_resource=vmail primitive pri_fs_vmail ocf:heartbeat:Filesystem \ params device=/dev/drbd5 fstype=ext4 directory=/var/vmail \ op monitor interval=30 primitive pri_ip_vmail ocf:heartbeat:IPaddr2 \ operations $id=pri_ip_vmail-operations \ op monitor interval=10s timeout=20s \ params ip=10.0.1.105 nic=br0 primitive pri_svc_postfix ocf:heartbeat:postfix \ operations $id=pri_svc_postfix-operations \ op monitor interval=60s timeout=20s \ params config_dir=/etc/postfix_vmail group grp_vmail pri_fs_vmail pri_ip_vmail pri_svc_postfix \ meta target-role=Started ms ms_drbd_vmail pri_drbd_vmail \ meta notify=true target-role=Started master-max=1 is-managed=true colocation col_grp_vmail_ON_drbd_vmail inf: grp_vmail:Started ms_drbd_vmail:Master order ord_ms_drbd_vmail_BEFORE_grp_vmail inf: ms_drbd_vmail:promote grp_vmail:start property $id
Re: [Linux-HA] Antw: A couple of questions regarding STONITH fencing ...
On 29/08/2013, at 4:18 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! After some short thinking I find that using ssh as STONITH is probably the wrong thing to do, because it can never STONITH if the target is down already. Correct (Is it documented that a STONITH operation should work independent from the target being up or down?) Stonith must be reliable. Anything that doesn't work when the target is down is not reliable. Maybe some shared storage and a mechanism like sbd is the way to go. With everything VMs, shared storage shouldn't be a problem. So if your VMs fail because the host failed, the cluster will see the loss of comunication, will try to fence the affected VMs (to be sure they are down), and then start to rerun failed resources. Right? Assuming the VMs are cluster nodes. Yes. In which case you could use fence_xvm or whatever the suse equivalent is. Regards, Ulrich Alex Sudakar alex.suda...@gmail.com schrieb am 29.08.2013 um 04:11 in Nachricht calq2s-f-m_b92quh0kbb4ozufstfk2gavyq19+xvsb7of8k...@mail.gmail.com: Hi. I'm running a simple two-node cluster with Red Hat Enterprise Linux 6, corosync 1.4.1, pacemaker 1.1.7 and crm. Each node (A B) are KVM VMs in separate physical machines/hypervisors. I have two separate networks between the machines; network X is a bonded pair, network Y a direct link between the two hypervisors. I've set up corosync to use X for ring #0 and Y for ring #1. The clustered application is a tomcat web application running on top of a DRBD block device. I've configured DRBD to use X for its communication link (DRBD apparently can't handle more than one network/link). I've set up the DRBD 'fence-peer' and 'after-resync-target' hooks to call the scripts to manage location constraints when network X is down. I also appreciate the documentation out there that teaches one all of this. :-) Nodes A B each have two STONITH resources set up for the other. Node A, for example, has resources 'fence_B_X' and 'fence_B_Y' which are instances of the stonith:fence_virsh resource agent configured to use either network X or network Y to ssh to to B's physical host and use 'virsh' to power off B. Everything's currently working as expected, but I have a couple of questions about how STONITH works and how I might be able to tweak it. 1. How can I add STONITH for the physical machines into the mix? In one of my tests I powered off B's hypervisor while B was running the clustered application. A tried to stonith B but both of its fence resources failed, since neither could ssh to that hypervisor to run virsh - stonith-ng[1785]: notice: can_fence_host_with_device: Disabling port list queries for fence_B_X (0/-1): Unable to connect/login to fencing device stonith-ng[1785]: info: can_fence_host_with_device: fence_B_X can not fence B (aka. 'B'): dynamic-list That's as expected, of course. As was the cluster's 'freeze' on starting the application on A; I've been told on this list that pacemaker won't move resources if/while its STONITH operations have failed. How could I add, say, a 'fence_ilo' stonith resource agent to the mix? The idea being that, if a node can't successfully use fence_virsh to power down the other VM, it can then escalate to trying to power off that node's entire hypervisor? Q 1.1. Could I set up a stonith fence_ilo resource with suitable pcmk* parameters to 'map' the cluster node names A B to resolve to the hypervisors' names that the fence_ilo resource would understand and use to take down the hypervisor? Q 1.2. How can I prioritize fence resources? I haven't come across that sort of thing; please forgive me if its a basic configuration facility (of crm) that I've forgotten (it's not the same as priority weights in allocating a resource to a node, but rather the order in which each stonith agent is invoked in a stonith situation on that node). Can I set up the cluster so fence_A_X and fence_A_Y will be tried first before the cluster will attempt 'fence_A_hypervisor'? Q2. How can I set up STONITH to proceed even if STONITH wasn't achieved? I understand the golden rule of a pacemaker cluster is not to proceed - to 'freeze' - if STONITH fails. And why that rule is in place, to avoid the dangers of split-brain. Really. :-) But in this case - the improbable event that both networks are down - my preference is for each node (if up) to independently run the application. I.e. for each to assume 'primary' role in its side of the DRBD resource and run the tomcat application. I'm okay with that, I want that, I'll accept the burden of reconciling antagonistic data updates after the crisis is over. Really. :-) Is there an accepted/standard way to tell a STONITH agent - or fence_virsh - to return 'success' even if the STONITH operation failed? Or a configuration item that instructs Pacemaker to proceed
Re: [Linux-HA] Antw: A couple of questions regarding STONITH fencing ...
On 29/08/2013, at 5:28 PM, Alex Sudakar alex.suda...@gmail.com wrote: On Thu, Aug 29, 2013 at 4:18 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: After some short thinking I find that using ssh as STONITH is probably the wrong thing to do, because it can never STONITH if the target is down already. Maybe some shared storage and a mechanism like sbd is the way to go. With everything VMs, shared storage shouldn't be a problem. Sadly I'm stuck with what I've got, which is just network connectivity to the hypervisors - which is all fence_virsh needs - and connectivity to the ILO 'lights out' controllers for the hypervisors (which fence_ilo can use. I should have mentioned that the physical machines are HP machines with ILO controllers.) It seems reasonable to me that one could want a cluster to use an ordered set of STONITH agents if a cluster node goes down; in my case: + try and use fence_virsh to shut down the VM; otherwise + try and use fence_ilo to shut down the entire hypervisor Can pacemaker do this? Have an ordered/prioritized list of STONITH resources? I realise you got the answer in your other thread, but for those following this one: yes - http://clusterlabs.org/wiki/Fencing_topology So if your VMs fail because the host failed, the cluster will see the loss of comunication, will try to fence the affected VMs (to be sure they are down), and then start to rerun failed resources. Right? Yes. At the moment this will all happen only if the fencing succeeds. As per my Q2, I'd like this to happen even if the chain of STONITH resources _weren't_ successful in getting through. I'm content to accept a split-brain situation in preference to the application being unavailable entirely. Thanks! ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] error: te_connect_stonith: Sign-in failed: triggered a retry
On 30/08/2013, at 5:51 AM, Tom Parker tpar...@cbnco.com wrote: Hello Since my upgrade last night I am also seeing this message in the logs on my servers. error: te_connect_stonith: Sign-in failed: triggered a retry Old mailing lists seem to imply that this is an issue with heartbeat which I don't think I am running. My software stack is this at the moment: cluster-glue-1.0.11-0.15.28 libcorosync4-1.4.5-0.18.15 corosync-1.4.5-0.18.15 pacemaker-mgmt-2.1.2-0.7.40 pacemaker-mgmt-client-2.1.2-0.7.40 pacemaker-1.1.9-0.19.102 Does anyone know where this may be coming from? were there any other errors? ie. did stonithd crash? Thanks Tom Parker. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Pacemaker 1.19 cannot manage more than 127 resources
On 30/08/2013, at 5:49 AM, Tom Parker tpar...@cbnco.com wrote: Hello. Las night I updated my SLES 11 servers to HAE-SP3 which contains the following versions of software: cluster-glue-1.0.11-0.15.28 libcorosync4-1.4.5-0.18.15 corosync-1.4.5-0.18.15 pacemaker-mgmt-2.1.2-0.7.40 pacemaker-mgmt-client-2.1.2-0.7.40 pacemaker-1.1.9-0.19.102 With the previous versions of openais/corosync I could run over 200 resources with no problems and with very little lag with the management commands (crm_mon, crm configure, etc) Today I am unable to configure more than 127 resources. When I commit my 128th resource all the crm commands start to fail (crm_mon just hangs) or timeout (ERROR: running cibadmin -Ql: Call cib_query failed (-62): Timer expired) I have attached my original crm config with 201 primitives to this e-mail. If anyone has any ideas as to what may have changed between pacemaker versions that would cause this please let me know. If I can't get this solved this week I will have to downgrade to SP2 again. Thanks for any information. I suspect you've hit an IPC buffer limit. Depending on exactly what went into the SUSE builds, you should have the following environment variables (documentation from /etc/syconfig/pacemaker on RHEL) to play with: # Force use of a particular class of IPC connection # PCMK_ipc_type=shared-mem|socket|posix|sysv # Specify an IPC buffer size in bytes # Useful when connecting to really big clusters that exceed the default 20k buffer # PCMK_ipc_buffer=20480 signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Pacemaker 1.19 cannot manage more than 127 resources
On 30/08/2013, at 1:42 PM, Tom Parker tpar...@cbnco.com wrote: My pacemaker config contains the following settings: LRMD_MAX_CHILDREN=8 export PCMK_ipc_buffer=3172882 perhaps go higher This is what I had today to get to 127 Resources defined. I am not sure what I should choose for the PCMK_ipc_type. Do you have any suggestions for large clusters? shm is the new upstream default, but it may not have propagated to suse yet. Thanks Tom On 08/29/2013 11:19 PM, Andrew Beekhof wrote: On 30/08/2013, at 5:49 AM, Tom Parker tpar...@cbnco.com wrote: Hello. Las night I updated my SLES 11 servers to HAE-SP3 which contains the following versions of software: cluster-glue-1.0.11-0.15.28 libcorosync4-1.4.5-0.18.15 corosync-1.4.5-0.18.15 pacemaker-mgmt-2.1.2-0.7.40 pacemaker-mgmt-client-2.1.2-0.7.40 pacemaker-1.1.9-0.19.102 With the previous versions of openais/corosync I could run over 200 resources with no problems and with very little lag with the management commands (crm_mon, crm configure, etc) Today I am unable to configure more than 127 resources. When I commit my 128th resource all the crm commands start to fail (crm_mon just hangs) or timeout (ERROR: running cibadmin -Ql: Call cib_query failed (-62): Timer expired) I have attached my original crm config with 201 primitives to this e-mail. If anyone has any ideas as to what may have changed between pacemaker versions that would cause this please let me know. If I can't get this solved this week I will have to downgrade to SP2 again. Thanks for any information. I suspect you've hit an IPC buffer limit. Depending on exactly what went into the SUSE builds, you should have the following environment variables (documentation from /etc/syconfig/pacemaker on RHEL) to play with: # Force use of a particular class of IPC connection # PCMK_ipc_type=shared-mem|socket|posix|sysv # Specify an IPC buffer size in bytes # Useful when connecting to really big clusters that exceed the default 20k buffer # PCMK_ipc_buffer=20480 ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10
On 27/08/2013, at 2:51 PM, Andreas Mock andreas.m...@web.de wrote: Hi Andrew, as this is a real showstopper at the moment I invested some other hours to be sure (as far as possible) not having made an error. Some additions: 1) I mirrored the whole mini drbd config to another pacemaker cluster. Same result: pacemaker 1.1.8 works, pacemaker 1.1.10 not The version of drbd is the same too? 2) When I remove the target role Stopped from the drbd ms resource and insert the config snippet related to the drbd device via crm -f file to a lean running pacemaker config (pacemaker cluster options, stonith resources), it seems to work. That means one of the nodes gets promoted. Then after stopping 'crm resource stop ms_drbd_xxx' and starting again I see the same promotion error as described. The drbd resource agent is using /usr/sbin/crm_master. Is there a possibility that feedback given through this client tool is changing the timing behaviour of pacemaker? Or the way transitions are scheduled? Any idea that may be related to a change in pacemaker? # git diff --stat Pacemaker-1.1.8..Pacemaker-1.1.10 | tail -n 1 1610 files changed, 109697 insertions(+), 62940 deletions(-) Needle, meet haystack. Particularly since I have no idea what that drbd error means. If you want me to have a look, you'll need to create a crm_report archive of works and not works. Logs aren't enough. Best regards Andreas Mock -Ursprüngliche Nachricht- Von: Andrew Beekhof [mailto:and...@beekhof.net] Gesendet: Dienstag, 27. August 2013 05:02 An: General Linux-HA mailing list Cc: pacema...@oss.clusterlabs.org Betreff: Re: [Pacemaker] [Linux-HA] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10 On 27/08/2013, at 3:31 AM, Andreas Mock andreas.m...@web.de wrote: Hi all, while the linbit drbd resource agent seems to work perfectly on pacemaker 1.1.8 (standard software repository) we have problems with the last release 1.1.10 and also with the newest head 1.1.11.xxx. As using drbd is not so uncommon I really hope to find interested people helping me out. I can provide as much debug information as you want. Environment: RHEL 6.4 clone (Scientific Linux 6.4) cman based cluster. DRBD 8.4.3 compiled from sources. 64bit - A drbd resource configured following the linbit documentation. - Manual start and stop (up/down) and setting primary of drbd resource working smoothly. - 2 nodes dis03-test/dis04-test - Following simple config on pacemaker 1.1.8 configure property no-quorum-policy=stop property stonith-enabled=true rsc_defaults resource-stickiness=2 primitive r_stonith-dis03-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis03-test pcmk_host_list=dis03-test primitive r_stonith-dis04-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis04-test pcmk_host_list=dis04-test location r_stonith-dis03_hates_dis03 r_stonith-dis03-test \ rule $id=r_stonith-dis03_hates_dis03-test_rule -inf: #uname eq dis03-test location r_stonith-dis04_hates_dis04 r_stonith-dis04-test \ rule $id=r_stonith-dis04_hates_dis04-test_rule -inf: #uname eq dis04-test primitive r_drbd_postfix ocf:linbit:drbd \ params drbd_resource=postfix drbdconf=/usr/local/etc/drbd.conf \ op monitor interval=15s timeout=60s role=Master \ op monitor interval=45s timeout=60s role=Slave \ op start timeout=240 \ op stop timeout=240 \ meta target-role=Stopped migration-threshold=2 ms ms_drbd_postfix r_drbd_postfix \ meta master-max=1 master-node-max=1 \ clone-max=2 clone-node-max=1 \ notify=true \ meta target-role=Stopped commit - Pacemaker is started from scratch - Config above is applied by crm -f file where file has the above config snippet. - After that crm_mon shows the following status --8- Last updated: Mon Aug 26 18:42:47 2013 Last change: Mon Aug 26 18:42:42 2013 via cibadmin on dis03-test Stack: cman Current DC: dis03-test - partition with quorum Version: 1.1.10-1.el6-9abe687 2 Nodes configured 4 Resources configured Online: [ dis03-test dis04-test ] Full list of resources: r_stonith-dis03-test (stonith:fence_mock): Started dis04-test r_stonith-dis04-test (stonith:fence_mock): Started dis03-test Master/Slave Set: ms_drbd_postfix [r_drbd_postfix] Stopped: [ dis03-test dis04-test ] Migration summary: * Node dis04-test: * Node dis03-test: --8
Re: [Linux-HA] error: filter_colocation_constraint: both allocated but to different nodes
On 27/08/2013, at 8:54 PM, Thomas Schulte tho...@cupracer.de wrote: Hi list, I'm experiencing a strange problem an I can't figure out what's wrong. I'm running openSUSE 12.3 on a 2-node-cluster with pacemaker-1.1.9-55.2.x86_64. Every 15 minutes the following messages are logged. If stonith is enabled, my hosts get fenced afterwards. The messages appeared on both nodes until I temporarily activated stonith. After some fencing, now only the second node s00202 keeps logging these (both nodes online again): Aug 27 10:21:13 s00202 crmd[3390]: notice: do_state_transition: State transition S_IDLE - S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Aug 27 10:21:13 s00202 pengine[3389]: notice: unpack_config: On loss of CCM Quorum: Ignore Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_vmail and ms_drbd_vmail are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_mysql and ms_drbd_mysql are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_redis and ms_drbd_redis are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_bind and ms_drbd_bind are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_squid and ms_drbd_squid are both allocated but to different nodes: s00202 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_www and ms_drbd_www are both allocated but to different nodes: s00201 vs. n/a Aug 27 10:21:13 s00202 pengine[3389]: last message repeated 3 times Aug 27 10:21:13 s00202 crmd[3390]: notice: do_te_invoke: Processing graph 61 (ref=pe_calc-dc-1377591673-1345) derived from /var/lib/pacemaker/pengine/pe-input-53.bz2 Aug 27 10:21:13 s00202 crmd[3390]: notice: run_graph: Transition 61 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-53.bz2): Complete Aug 27 10:21:13 s00202 crmd[3390]: notice: do_state_transition: State transition S_TRANSITION_ENGINE - S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Aug 27 10:21:13 s00202 pengine[3389]: notice: process_pe_message: Calculated Transition 61: /var/lib/pacemaker/pengine/pe-input-53.bz2 Could you attach this file please? I'll be able to see if the current version behaves any better. I believe that the errors about filter_colocation_contraint all have the same cause, so I'm going to reduce the problem to the following: Aug 27 10:21:13 s00202 pengine[3389]:error: filter_colocation_constraint: pri_fs_vmail and ms_drbd_vmail are both allocated but to different nodes: s00202 vs. n/a This is my configuration (relevant extract): # crm configure show node s00201 node s00202 primitive pri_drbd_vmail ocf:linbit:drbd \ operations $id=pri_drbd_vmail-operations \ op monitor interval=20 role=Slave timeout=20 \ op monitor interval=10 role=Master timeout=20 \ params drbd_resource=vmail primitive pri_fs_vmail ocf:heartbeat:Filesystem \ params device=/dev/drbd5 fstype=ext4 directory=/var/vmail \ op monitor interval=30 primitive pri_ip_vmail ocf:heartbeat:IPaddr2 \ operations $id=pri_ip_vmail-operations \ op monitor interval=10s timeout=20s \ params ip=10.0.1.105 nic=br0 primitive pri_svc_postfix ocf:heartbeat:postfix \ operations $id=pri_svc_postfix-operations \ op monitor interval=60s timeout=20s \ params config_dir=/etc/postfix_vmail group grp_vmail pri_fs_vmail pri_ip_vmail pri_svc_postfix \ meta target-role=Started ms ms_drbd_vmail pri_drbd_vmail \ meta notify=true target-role=Started master-max=1 is-managed=true colocation col_grp_vmail_ON_drbd_vmail inf: grp_vmail:Started ms_drbd_vmail:Master order ord_ms_drbd_vmail_BEFORE_grp_vmail inf: ms_drbd_vmail:promote grp_vmail:start property $id=cib-bootstrap-options \ no-quorum-policy=ignore \ placement-strategy=balanced \ dc-version=1.1.9-2db99f1 \ cluster-infrastructure=classic openais (with plugin) \ expected-quorum-votes=2 \ last-lrm-refresh=1377585341 \ stonith-enabled=false rsc_defaults $id=rsc-options \ resource-stickiness=100 \ migration-threshold=3 op_defaults $id=op-options \ timeout=600 \ record-pending=false # crm_resource -L Master/Slave Set: ms_drbd_vmail [pri_drbd_vmail] Masters: [ s00202 ]
Re: [Linux-HA] cibadmin --query wiping out resource configuration
Seems to be fixed in the latest version: [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ [root@pcmk-1 ~]# cibadmin --query --local --scope=resources --no-children resources/ On 23/08/2013, at 6:30 PM, Ferenc Wagner wf...@niif.hu wrote: Hi, Under Pacemaker 1.1.7: # cibadmin --query --local --scope=resources --no-children resources clone id=storage-clone group id=storage primitive class=ocf id=dlm provider=pacemaker type=controld operations op id=dlm-monitor-120 interval=120 name=monitor/ /operations /primitive [...] /resources # cibadmin --query --local --scope=resources --no-children Call cib_query failed (-22): The object/attribute does not exist null And indeed, the resources element is gone from the CIB, everything is orphaned. The output of crm configure show lacks all resources, but their colocation and order constraints are still there. crm configure edit xml throws an exception after pasting back the resources element: Traceback (most recent call last): File /usr/sbin/crm, line 45, in module main.run() File /usr/lib/python2.7/dist-packages/crm/main.py, line 300, in run if not parse_line(levels,shlex.split(inp)): File /usr/lib/python2.7/dist-packages/crm/main.py, line 148, in parse_line rv = d() # execute the command File /usr/lib/python2.7/dist-packages/crm/main.py, line 147, in lambda d = lambda: cmd[0](*args) File /usr/lib/python2.7/dist-packages/crm/ui.py, line 1434, in edit return set_obj.edit() File /usr/lib/python2.7/dist-packages/crm/cibconfig.py, line 153, in edit return self.edit_save(s) File /usr/lib/python2.7/dist-packages/crm/cibconfig.py, line 138, in edit_save if not self.save(s): File /usr/lib/python2.7/dist-packages/crm/cibconfig.py, line 460, in save doc.unlink() File /usr/lib/python2.7/xml/dom/minidom.py, line 1578, in unlink Node.unlink(self) File /usr/lib/python2.7/xml/dom/minidom.py, line 264, in unlink child.unlink() File /usr/lib/python2.7/xml/dom/minidom.py, line 672, in unlink Node.unlink(self) File /usr/lib/python2.7/xml/dom/minidom.py, line 264, in unlink child.unlink() File /usr/lib/python2.7/xml/dom/minidom.py, line 672, in unlink Node.unlink(self) File /usr/lib/python2.7/xml/dom/minidom.py, line 264, in unlink child.unlink() File /usr/lib/python2.7/xml/dom/minidom.py, line 672, in unlink Node.unlink(self) File /usr/lib/python2.7/xml/dom/minidom.py, line 264, in unlink child.unlink() File /usr/lib/python2.7/xml/dom/minidom.py, line 668, in unlink for attr in self._attrs.values(): AttributeError: 'NoneType' object has no attribute 'values' Manual editing and cibadmin --replace was able to fix the config. Do I misunderstand something again, or is this a bug, maybe a known one? (I know crm was replaced by crmsh, I'm more worried about cibadmin.) -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Storing arbitrary metadata in the CIB
On 23/08/2013, at 5:54 PM, Ferenc Wagner wf...@niif.hu wrote: Andrew Beekhof and...@beekhof.net writes: On 22/08/2013, at 10:08 PM, Ferenc Wagner wf...@niif.hu wrote: Our setup uses some cluster wide pieces of meta information. Think access control lists for resource instances used by some utilities or some common configuration data used by the resource agents. Currently this info is stored in local files on the nodes or replicated in each primitive as parameters. Are you aware that resources can - have multiple sets of parameters, and No, I'm not sure what you're referring to here. http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/#s-reusing-config-elements A typical resource: primitive id=ping-1 class=ocf type=ping provider=pacemaker instance_attributes id=ping-1-params nvpair id=ping-1-debug name=debug value=true/ nvpair id=ping-1-host_list name=host_list value=172.16.1.4/ nvpair id=ping-1-name name=name value=connected/ /instance_attributes operations op id=ping-1-monitor-60s interval=60s name=monitor/ /operations /primitive One with multiple sets of attributes: primitive id=ping-1 class=ocf type=ping provider=pacemaker instance_attributes id=ping-1-params nvpair id=ping-1-debug name=debug value=true/ nvpair id=ping-1-host_list name=host_list value=172.16.1.4/ nvpair id=ping-1-name name=name value=connected/ /instance_attributes instance_attributes id=ping-1-extra-params nvpair id=ping-1-extra name=extra value=something/ /instance_attributes operations op id=ping-1-monitor-60s interval=60s name=monitor/ /operations /primitive One with attributes defined by the previous resource: primitive id=ping-2 class=ocf type=ping provider=pacemaker instance_attributes id=ping-2-params nvpair id=ping-2-host_list name=host_list value=172.16.1.3/ nvpair id=ping-2-name name=name value=connected/ /instance_attributes instance_attributes id-ref=ping-1-extra-params/ operations op id=ping-2-monitor-60s interval=60s name=monitor/ /operations /primitive I guess not rsc_defaults or node specific parameters of clones. Maybe rules? I can't see them immediately useful. Could you please provide a pointer into Pacemaker 1.1 explained? (Btw. 5.7.2.3. Disabling a Monitor Operation misses the re-enable example at the end of the section.) - share sets of parameters Do you mean by using resource templates? Those indeed sound useful, thanks for the pointer! Used templates could store shared resource parameters, and unused templates could store global cluster parameters used by our management utilities. I'll give this a shot. -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] cibadmin --delete disregarding attributes
On 23/08/2013, at 5:32 PM, Ferenc Wagner wf...@niif.hu wrote: Andrew Beekhof and...@beekhof.net writes: On 22/08/2013, at 10:22 PM, Ferenc Wagner wf...@niif.hu wrote: man cibadmin says: the tagname and all attributes must match in order for the element to be deleted, for the element to be deleted --- not the children of the element to be deleted Ah, so these are XML attributes, not instance attributes. Thanks for the clarification! I was confused. # cibadmin --create --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' [...] # cibadmin --delete --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' tagname, class, provider, type and id all match here. so deletion is allowed to proceed Fine, then I can still use the description attribute to restrict deletion, right? yes I have to use a fixed tag name (id) to exclude the possibility of having multiple such resources in the CIB. -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Q: groups of groups
On 23/08/2013, at 5:15 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Andrew Beekhof and...@beekhof.net schrieb am 23.08.2013 um 02:14 in Nachricht 7e68fa3b-6c15-4e39-ba43-f9a76647f...@beekhof.net: On 22/08/2013, at 7:31 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! Suppose you have an application A that needs two filesystems F1 and F2. The filesystems are on separate LVM VGs VG1 and VG2 with LVs L1 and L2, respectively. The RAID R1 and R2 provide the LVM PVs. (Actually we have one group that has 58 primitives in them with both dimensions being wider than in this example) So you can configure group grp_A R1 R2 VG1 VG2 F1 F2 A (assuming the elements are primitives already configured) Now for example if R2 has a problem, the cluster will restart the whole group of resources, even that sequence that is unaffected (R1 VG1 F1). This causes extra operations and time for recovery what you don't like. So don't put them in a group? What you can do now is having parallel execution like this group grp_A (R1 R2) (VG1 VG2) (F1 F2) A You're saying this is currently possible? As it turned out: not yet ;-) It only works for explicit ordering. That makes more sense. If so, crmsh must be re-writing this into something other than a group. This should not be much of a problem; the problem seems to be: How to convert such structures back to the original notation? Quite hard to know what the original form was (Note that this is probably a bad idea as the RAIDs and VGs (and maybe mount also) most likely use a common lock each that forces serialization) For the same failure scenario R2 wouldn't be restarted, so the gain is small. A better approach seems to be group grp_A (R1 VG1 F1) (R2 VG2 F2) A Now for the same failure R1, VG1, and F1 will survive; unfortunately if R1 fails, then everything will be restarted, like in the beginning. So what you really want is group grp_A ((R1 VG1 F1) (R2 VG2 F2)) A Now if R2 fails, then R1, VG1, and F1 will survive, and if R1 fails, then R2, VG2 and F2 will survive Unfortunately the syntax of the last example is not supported. I'm surprised the one before it is even supported. Groups of groups have never been supported. I know, and the PDP-8 had 128kB RAM. But things can change. Not groups of groups. Hell, I'd get rid of groups completely if I could This one isn't either: group grp_1 R1 VG1 F1 group grp_2 R2 VG2 F2 group grp_A (grp_1 grp_2) A So a group of groups would be nice to have. I thought about that long time ago, but only yesterday I learned about the syntax of netgroups which has exactly that: a netgroup can contain another netgroup ;-) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10
On 27/08/2013, at 3:31 AM, Andreas Mock andreas.m...@web.de wrote: Hi all, while the linbit drbd resource agent seems to work perfectly on pacemaker 1.1.8 (standard software repository) we have problems with the last release 1.1.10 and also with the newest head 1.1.11.xxx. As using drbd is not so uncommon I really hope to find interested people helping me out. I can provide as much debug information as you want. Environment: RHEL 6.4 clone (Scientific Linux 6.4) cman based cluster. DRBD 8.4.3 compiled from sources. 64bit - A drbd resource configured following the linbit documentation. - Manual start and stop (up/down) and setting primary of drbd resource working smoothly. - 2 nodes dis03-test/dis04-test - Following simple config on pacemaker 1.1.8 configure property no-quorum-policy=stop property stonith-enabled=true rsc_defaults resource-stickiness=2 primitive r_stonith-dis03-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis03-test pcmk_host_list=dis03-test primitive r_stonith-dis04-test stonith:fence_mock \ meta resource-stickiness=INFINITY target-role=Started \ op monitor interval=180 timeout=300 requires=nothing \ op start interval=0 timeout=300 \ op stop interval=0 timeout=300 \ params vmname=dis04-test pcmk_host_list=dis04-test location r_stonith-dis03_hates_dis03 r_stonith-dis03-test \ rule $id=r_stonith-dis03_hates_dis03-test_rule -inf: #uname eq dis03-test location r_stonith-dis04_hates_dis04 r_stonith-dis04-test \ rule $id=r_stonith-dis04_hates_dis04-test_rule -inf: #uname eq dis04-test primitive r_drbd_postfix ocf:linbit:drbd \ params drbd_resource=postfix drbdconf=/usr/local/etc/drbd.conf \ op monitor interval=15s timeout=60s role=Master \ op monitor interval=45s timeout=60s role=Slave \ op start timeout=240 \ op stop timeout=240 \ meta target-role=Stopped migration-threshold=2 ms ms_drbd_postfix r_drbd_postfix \ meta master-max=1 master-node-max=1 \ clone-max=2 clone-node-max=1 \ notify=true \ meta target-role=Stopped commit - Pacemaker is started from scratch - Config above is applied by crm -f file where file has the above config snippet. - After that crm_mon shows the following status --8- Last updated: Mon Aug 26 18:42:47 2013 Last change: Mon Aug 26 18:42:42 2013 via cibadmin on dis03-test Stack: cman Current DC: dis03-test - partition with quorum Version: 1.1.10-1.el6-9abe687 2 Nodes configured 4 Resources configured Online: [ dis03-test dis04-test ] Full list of resources: r_stonith-dis03-test (stonith:fence_mock): Started dis04-test r_stonith-dis04-test (stonith:fence_mock): Started dis03-test Master/Slave Set: ms_drbd_postfix [r_drbd_postfix] Stopped: [ dis03-test dis04-test ] Migration summary: * Node dis04-test: * Node dis03-test: --8- cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dis03-test, 2013-07-24 17:19:24 on both nodes. The drbd resource was shutdown previously in a clean state, so that any node can be the primary. - Now the weird behaviour when trying to start the drbd with crm resource start ms_drbd_postfix Output of crm_mon -1rf --8- Last updated: Mon Aug 26 18:46:33 2013 Last change: Mon Aug 26 18:46:30 2013 via cibadmin on dis04-test Stack: cman Current DC: dis03-test - partition with quorum Version: 1.1.10-1.el6-9abe687 2 Nodes configured 4 Resources configured Online: [ dis03-test dis04-test ] Full list of resources: r_stonith-dis03-test (stonith:fence_mock): Started dis04-test r_stonith-dis04-test (stonith:fence_mock): Started dis03-test Master/Slave Set: ms_drbd_postfix [r_drbd_postfix] Slaves: [ dis03-test ] Stopped: [ dis04-test ] Migration summary: * Node dis04-test: r_drbd_postfix: migration-threshold=2 fail-count=2 last-failure='Mon Aug 26 18:46:30 2013' * Node dis03-test: Its hard to imagine how pacemaker could cause drbdadm to fail, short of leaving the other side promoted while trying to promote another. Perhaps the drbd folks could comment on what the error means. Failed actions: r_drbd_postfix_promote_0 (node=dis04-test, call=34, rc=1, status=complete, last-rc-change=Mon Aug 26 18:46:29 2013 , queued=1212ms, exec=0ms ): unknown error --8- In the log of the drbd agent I can find the following when the promoting request is handled on dis03-test
Re: [Linux-HA] cman-controlled cluster takes an hour to start !?
Chrissie: Perhaps you have some insight here? On 23/08/2013, at 8:42 PM, Jakob Curdes j...@info-systems.de wrote: Hmmm, the problem turns out to DNS-related. At startup, some of the virtual interfaces are inactive and the DNS servers are unreachable. And CMAN seems to do a lookup for all ip addresses on the machine; I have the names of all cluster members in the hosts file but not all names of all other addresses (i.e. the ones managed by the cluster). Anyway I wonder whiy even with -d64 it doesn't tell me anything about what it is doing. I think the timespan of an hour is just because we have lots of VLAN interfaces the he wants to get a DNS name for Hi, we have a simple 2-node cluster running CMAN and pacemaker under CentOS 6. The problem is that upon startup the machines (even if alone, i.e. second machine is off), will give a cman timeout on startup saying Timed-out waiting for cluster. *If I start the services manually an hour later or so, everything works fine* until I reboot one machine, that machine will typically timeout again on startup. The issue seems not to be related to firewalling (I retried the failed start with open firewalls- no change) nor to multicast communication as the cluster is setup to use unicast via an bonded interface that has no other purposes. For the machine that timeouts, I do not see ANY communication attempts with the other node (running tcpdump on both machines) so I suppose it is waiting for something local to happen. Also, I do not see ANY log entries related to the failed start; the logfiles only get updated once the cluster has started successfully. What might be the cause here? Could that be a fencing problem (we have IPMI fencing configured which works)? Here is the cluster.conf: cluster config_version=13 name=fw-cluster logging logfile=/var/log/cluster/cluster.log logfile_priority=info syslog_priority=crit to_logfile=yes to_syslog=yes/ dlm protocol=sctp/ fence_daemon clean_start=0 post_fail_delay=0 post_join_delay=3 skip_undefined=1/ clusternodes clusternode name=gw1 nodeid=1 fence method name=pcmk-redirect device name=pcmk port=gw1/ /method /fence /clusternode clusternode name=gw2 nodeid=2 fence method name=pcmk-redirect device name=pcmk port=gw2/ /method /fence /clusternode /clusternodes cman expected_votes=1 transport=udpu two_node=1/ fencedevices fencedevice agent=fence_pcmk name=pcmk/ /fencedevices rm failoverdomains/ resources/ /rm /cluster Regards, Jakob Curdes ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Q: groups of groups
On 22/08/2013, at 7:31 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! Suppose you have an application A that needs two filesystems F1 and F2. The filesystems are on separate LVM VGs VG1 and VG2 with LVs L1 and L2, respectively. The RAID R1 and R2 provide the LVM PVs. (Actually we have one group that has 58 primitives in them with both dimensions being wider than in this example) So you can configure group grp_A R1 R2 VG1 VG2 F1 F2 A (assuming the elements are primitives already configured) Now for example if R2 has a problem, the cluster will restart the whole group of resources, even that sequence that is unaffected (R1 VG1 F1). This causes extra operations and time for recovery what you don't like. So don't put them in a group? What you can do now is having parallel execution like this group grp_A (R1 R2) (VG1 VG2) (F1 F2) A You're saying this is currently possible? If so, crmsh must be re-writing this into something other than a group. (Note that this is probably a bad idea as the RAIDs and VGs (and maybe mount also) most likely use a common lock each that forces serialization) For the same failure scenario R2 wouldn't be restarted, so the gain is small. A better approach seems to be group grp_A (R1 VG1 F1) (R2 VG2 F2) A Now for the same failure R1, VG1, and F1 will survive; unfortunately if R1 fails, then everything will be restarted, like in the beginning. So what you really want is group grp_A ((R1 VG1 F1) (R2 VG2 F2)) A Now if R2 fails, then R1, VG1, and F1 will survive, and if R1 fails, then R2, VG2 and F2 will survive Unfortunately the syntax of the last example is not supported. I'm surprised the one before it is even supported. Groups of groups have never been supported. This one isn't either: group grp_1 R1 VG1 F1 group grp_2 R2 VG2 F2 group grp_A (grp_1 grp_2) A So a group of groups would be nice to have. I thought about that long time ago, but only yesterday I learned about the syntax of netgroups which has exactly that: a netgroup can contain another netgroup ;-) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Storing arbitrary metadata in the CIB
On 22/08/2013, at 10:08 PM, Ferenc Wagner wf...@niif.hu wrote: Hi, Our setup uses some cluster wide pieces of meta information. Think access control lists for resource instances used by some utilities or some common configuration data used by the resource agents. Currently this info is stored in local files on the nodes or replicated in each primitive as parameters. Are you aware that resources can - have multiple sets of parameters, and - share sets of parameters The combination might be useful here. I find this suboptimal, as keeping them in sync is a hassle. It is possible to store such stuff in the fake parameter of unmanaged Dummy resources, but that clutters the status output. Can somebody offer some advice in this direction? Or is this idea a pure heresy? -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] cibadmin --delete disregarding attributes
On 22/08/2013, at 10:22 PM, Ferenc Wagner wf...@niif.hu wrote: Hi, man cibadmin says: the tagname and all attributes must match in order for the element to be deleted, for the element to be deleted --- not the children of the element to be deleted but experience says otherwise: the primitive is deleted even if it was created with different attributes than those provided to the --delete call, cf. 'foo' vs 'bar' in the example below. Do I misinterpret the documentation or is this a bug in Pacemaker 1.1.7? # cibadmin --create --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='foo'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes # cibadmin --delete --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' tagname, class, provider, type and id all match here. so deletion is allowed to proceed instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='bar'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] establishing a new resource-agent package provider
On 13/08/2013, at 7:41 PM, Lars Marowsky-Bree l...@suse.com wrote: On 2013-08-07T19:16:24, Lars Ellenberg lars.ellenb...@linbit.com wrote: Hi all, sorry for being a bit late to the game. I was on vacation for 2,5 weeks with no internet-enabled equipment. I can highly recommend the experience ;-) These are the 'core' resource agents that the ocf community supports. Agents outside of the 'core' provider are supported by different projects and subsets of the community (like linbit and the drbd agent). I'm convinced. Lars? I'm sure LMB has an opinion on this as well. Of course. I can see that the reference to heartbeat here is confusing for new people coming to the project. (And I have a subtle feeling that some old-timers want to complete cutting the cord to the heartbeat project everywhere, too.) Personally I don't give a toss beyond not wanting to re-answer why they're called heartbeat agents a hundred times. So if that was intended to be a reference to me, I'd rather speak for myself. That could be solved by using core instead as a provider name. What I worry about is that this will create confusion with the existing (and not-yet-reworked) documentation if the reference to heartbeat suddenly no longer works - even if that happened only on a single distribution. So I'd like to mandate the compatibility code. wfm I'd: - Rename the provider to core - Rework our own documentation and as we find it - Transparently support references to ocf:heartbeat forever: - Re-map s/heartbeat/core/ in the LRM (silently, or it'd get really bloody annoying) Why not just create a symlink? It doesn't even really matter in which direction. Then crmsh/pcs just needs to filter whatever they choose to. No mapping needed. - When a new resource created with that provider, rewrite it to core with an LOG_INFO message given to the user - Hide ocf:heartbeat from the normal list (or show it as depreciated, use core instead); but when someone types ocf:heartTAB, auto-complete to it and of course auto-complete all parameters. - Pacemaker's CIB can do this via an XSLT upgrade rule automatically too, right? But that might mess with existing customer scripts, so maybe we don't want to. We could I guess, but why would we need to? If RHEL dropped the support for the heartbeat name completely, If that happens it would be a discussion for RHEL8 and one that doesn't really benefit anyone. I think this would really suck, because then RHEL would end up with configuration snippets that suddenly no longer worked on RHEL but everywhere else. Like, where have we seen that before? ;-) But besides being bad for RHEL, I think it'd also be bad for the impression of the community as a whole and the friendliness to users. So in short: Rename, but remain backwards-compatible (since the price is low). Was anyone proposing anything different? Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] {SPAM 04.2} Re: Many location on ping resources and best practice for connectivity monitoring
On 09/08/2013, at 6:42 PM, RaSca ra...@miamammausalinux.org wrote: Il giorno Ven 09 Ago 2013 04:42:28 CEST, Andrew Beekhof ha scritto: [...] That sounds like something playing with the virt bridge when the vm starts. Is the host trying to ping through the bridge too? Yes. Is this not correct? Sure, but it does mean that ping would fail if libvirt (or whatever) is messing with the bridge at the time. Many location constraints can reference the attribute created by a single ping resource. Its still not clear to me if you have one ping resource or one ping resource per vm... don't do the second one. I've got many locations based on the same cloned resource (which is named ping). -- RaSca Mia Mamma Usa Linux: Niente è impossibile da capire, se lo spieghi bene! ra...@miamammausalinux.org http://www.miamammausalinux.org ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
