Re: [Pacemaker] SLES 11 SP3 boothd behaviour
Booth version 0.1.0 has no retry method for packets, one single packet loss and the election does not work anymore. Also it has a stupid checking of ballot values against promised values, in which case it communicates but does not do things right. Booth version 0.1.0 is very sensitive about start order and saved ballot values as ticket attributes. The fault might come from your start procedure and the state it remembers from previous runs. And it communicates through two different ways, one is the UDP port you configure in booth.conf used for renew and the other is a fixed TCP port somewhere in 23000 used for catchup. I might give you a hint if you attach the complete log from Aug 25 10:07:10 until Aug 25 10:08:10 for all three sites and attach the snipplet of booth.conf to see the order of lines with site and arbitrator. Anyhow booth 0.1.0 is really unusable for productive environment, it has so many errors in it, you cant rely on it. The booth version from GIT after 4th august 2014 behaves very well even in case of double fault (restart during network down) and builds under SLES and RHEL quite well now. Rainer Gesendet:Dienstag, 26. August 2014 um 14:58 Uhr Von:Sutherland, Rob rsutherl...@broadviewnet.com An:pacemaker@oss.clusterlabs.org pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] SLES 11 SP3 boothd behaviour All nodes in question NTP from the same time source (yes, we have run into synchronicity issues in the past). Interestingly, increasing the lease from 60 seconds to 120 seconds did not affect the behaviour. Rob From: John Lauro [mailto:john.la...@covenanteyes.com] Sent: Monday, August 25, 2014 6:17 PM To: Sutherland, Rob Subject: Re: [Pacemaker] SLES 11 SP3 boothd behaviour You probably already checked this, but just in case... No experience at all with geo-redundancy, but this sounds suspiciously like it could be a time sync problem. Have you tried something like ntpq -np on all 3 nodes and verify the offsets are all low (ie: +/- 10) and times are in sync? (Assuming you are running ntpd, and the process didnt stop.) From: Rob Sutherland rsutherl...@broadviewnet.com To: pacemaker@oss.clusterlabs.org Sent: Monday, August 25, 2014 3:43:34 PM Subject: [Pacemaker] SLES 11 SP3 boothd behaviour Hello all, Were in the process of implementing geo-redundancy on SLES 11 SP3 (version 0.1.0). We are seeing behavior in which site 2 in a geo-cluster decides that the ticket has expired long before actual expiry. Heres an example time-line: 1 - All sites (site 1, site 2 and arbitrator) agree on ticket owner and expiry. i.e. site 2 has the ticket with a 60-second expiry: Aug 25 10:07:10 linux-4i31 booth-arbitrator: [22526]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975690 was executed Aug 25 10:07:10 bb5Btas0 booth-site: [27782]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975690 was executed Aug 25 10:07:10 bb5Atas1 booth-site: [7826]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975690 was executed 2 - After 48 seconds (80% into lease), all three nodes are still in agreement: Site 2: Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: crm_ticket -t geo-ticket -S owner -v 2 was executed Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975738 was executed The arbitrator: Aug 25 10:07:58 linux-4i31 crm_ticket[23836]: notice: crm_log_args: Invoked: crm_ticket -t geo-ticket -S owner -v 2 Aug 25 10:07:58 linux-4i31 booth-arbitrator: [22526]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975738 was executed Site 1: Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: crm_ticket -t geo-ticket -S owner -v 2 was executed Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: crm_ticket -t geo-ticket -S expires -v 1408975738 was executed 3 - Site 2 decides that the ticket has expired (at the expiry time set in step 1) Aug 25 10:08:10 bb5Btas0 booth-site: [27782]: debug: lease expires ... 4 - At 10:08:58, both site 1 and the arbitrator expire the lease and pick a new master. I presume that there was some missed communication between site 2 and the rest of the geo-cluster. There is nothing in the logs to help debug this, though. Any hints on debugging this? BTW: we only ever see this on a site 2 never a site 1. This is consistent across several labs. Is there a bias towards site 1? Thanks in advance, Rob ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
Re: [Pacemaker] Interval-origin in monitor operations does not work
When do we need negative durations ?
Is it because iso8601 must be able to handle negative durations ?
What is the expected reaction on setting a negative interval in the monitor operation ?
Just to know what i can expect from the test.
Rainer
Gesendet:Montag, 05. Mai 2014 um 06:37 Uhr
Von:Andrew Beekhof and...@beekhof.net
An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff:Re: [Pacemaker] Interval-origin in monitor operations does not work
On 2 May 2014, at 4:55 pm, Andrew Beekhof and...@beekhof.net wrote:
On 15 Apr 2014, at 4:12 am, Rainer Brestan rainer.bres...@gmx.net wrote:
Of course, I can.
primitive class=ocf id=resD provider=heartbeat type=Dummy
operations
op id=resD-start-0 interval=0 name=start timeout=20/
op id=resD-stop-0 interval=0 name=stop timeout=20/
op id=resD-monitor-1h interval=1h interval-origin=00:34 name=monitor timeout=60/
/operations
meta_attributes id=resD-meta_attributes
nvpair id=resD-meta_attributes-failure-timeout name=failure-timeout value=15m/
nvpair id=resD-meta_attributes-migration-threshold name=migration-threshold value=3/
/meta_attributes
/primitive
Yes, the origin is in the future, but consider above monitor configuration.
The monitor operation shall run every hour at 34 minutes.
If i would specifiy a full date in the past then pengine has to run a number of while(rc0) loops in unpack_operation.
One year after full date exactly 8760 and this for every call of unpack_operation.
Thats why i specified the first possible run time every day and then they are maximum of 23 while loop runs.
If unpack_operation is called between 00:00 and 00:34 the described situation happens.
Origin is later than now.
Applying this patch will help.
It will, but as I suspected it will also cause:
iso8601 -d 2014-01-01 00:00:30Z -D P-1D -E 2013-12-31 00:00:30Z
to fail with:
Date: 2014-01-01 00:00:30Z
Duration: -01--01 00:00:00Z
Duration ends at: 2014-01-00 00:00:30Z
which isnt right :)
Im working on a true fix now...
These are the resulting patches in https://github.com/beekhof/pacemaker:
+ Andrew Beekhof (14 seconds ago) 44af669: Test: PE: Correctly handle origin offsets in the future (HEAD, master)
+ Andrew Beekhof (27 minutes ago) 3f20485: Fix: PE: Correctly handle origin offsets in the future
+ Andrew Beekhof (4 minutes ago) d39bad6: Test: iso8601: Improved logging of durations
+ Andrew Beekhof (29 minutes ago) afb6c16: Fix: iso8601: Different logic is needed when logging and calculating durations
diff --git a/lib/common/iso8601.c b/lib/common/iso8601.c
index 7dc2495..742de70 100644
--- a/lib/common/iso8601.c
+++ b/lib/common/iso8601.c
@@ -1137,7 +1137,7 @@ crm_time_add_days(crm_time_t * a_time, int extra)
ydays = crm_time_leapyear(a_time-years) ? 366 : 365;
}
- while (a_time-days = 0) {
+ while (a_time-days 0) {
a_time-years--;
a_time-days += crm_time_leapyear(a_time-years) ? 366 : 365;
}
Rainer
Gesendet: Mittwoch, 09. April 2014 um 08:57 Uhr
Von: Andrew Beekhof and...@beekhof.net
An: The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] Interval-origin in monitor operations does not work
On 1 Apr 2014, at 5:10 am, Rainer Brestan rainer.bres...@gmx.net wrote:
Using interval-origin in monitor operation definition does not work any more.
Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.
Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.
The call to crm_time_subtract with
origin=2014-03-31 19:20:00Z
date_set-now=2014-03-31 17:31:04Z
result in
delay=-0001-12-31 01:48:56Z
delay_s=31456136
start_delay=31456136000
which is almost a year in the future.
To be fair, the origin was also in the future.
I dont think that was expected.
Can you supply your cib so I can experiment?
The function crm_time_subtract calculates this by the crm_time_add_* functions.
The buggy statement is in crm_time_add_days.
If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.
But if a_time-days is zero, it must not do anything.
The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.
There are two solutions.
One is the add handling on negative years to crm_time_get_seconds.
The other is to exchange line 1140 in iso8601.c
while (a_time-days = 0) {
by
while (a_time-days 0) {
Second solution is verified to bring the expected result, start-delay of little less than two hours.
Handling of negative years in crm_time_get_seconds might not be a proper solution as the return value of the function is unsigned long long and what to report if the complete calculation gives a negative number of seconds.
Rainer
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http
Re: [Pacemaker] Interval-origin in monitor operations does not work
Of course, I can.
primitive class=ocf id=resD provider=heartbeat type=Dummy
operations
op id=resD-start-0 interval=0 name=start timeout=20/
op id=resD-stop-0 interval=0 name=stop timeout=20/
op id=resD-monitor-1h interval=1h interval-origin=00:34 name=monitor timeout=60/
/operations
meta_attributes id=resD-meta_attributes
nvpair id=resD-meta_attributes-failure-timeout name=failure-timeout value=15m/
nvpair id=resD-meta_attributes-migration-threshold name=migration-threshold value=3/
/meta_attributes
/primitive
Yes, the origin is in the future, but consider above monitor configuration.
The monitor operation shall run every hour at 34 minutes.
If i would specifiy a full date in the past then pengine has to run a number of while(rc0) loops in unpack_operation.
One year after full date exactly 8760 and this for every call of unpack_operation.
Thats why i specified the first possible run time every day and then they are maximum of 23 while loop runs.
If unpack_operation is called between 00:00 and 00:34 the described situation happens.
Origin is later than now.
Applying this patch will help.
diff --git a/lib/common/iso8601.c b/lib/common/iso8601.c
index 7dc2495..742de70 100644
--- a/lib/common/iso8601.c
+++ b/lib/common/iso8601.c
@@ -1137,7 +1137,7 @@ crm_time_add_days(crm_time_t * a_time, int extra)
ydays = crm_time_leapyear(a_time-years) ? 366 : 365;
}
- while (a_time-days = 0) {
+ while (a_time-days 0) {
a_time-years--;
a_time-days += crm_time_leapyear(a_time-years) ? 366 : 365;
}
Rainer
Gesendet:Mittwoch, 09. April 2014 um 08:57 Uhr
Von:Andrew Beekhof and...@beekhof.net
An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff:Re: [Pacemaker] Interval-origin in monitor operations does not work
On 1 Apr 2014, at 5:10 am, Rainer Brestan rainer.bres...@gmx.net wrote:
Using interval-origin in monitor operation definition does not work any more.
Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.
Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.
The call to crm_time_subtract with
origin=2014-03-31 19:20:00Z
date_set-now=2014-03-31 17:31:04Z
result in
delay=-0001-12-31 01:48:56Z
delay_s=31456136
start_delay=31456136000
which is almost a year in the future.
To be fair, the origin was also in the future.
I dont think that was expected.
Can you supply your cib so I can experiment?
The function crm_time_subtract calculates this by the crm_time_add_* functions.
The buggy statement is in crm_time_add_days.
If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.
But if a_time-days is zero, it must not do anything.
The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.
There are two solutions.
One is the add handling on negative years to crm_time_get_seconds.
The other is to exchange line 1140 in iso8601.c
while (a_time-days = 0) {
by
while (a_time-days 0) {
Second solution is verified to bring the expected result, start-delay of little less than two hours.
Handling of negative years in crm_time_get_seconds might not be a proper solution as the return value of the function is unsigned long long and what to report if the complete calculation gives a negative number of seconds.
Rainer
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
[Pacemaker] Interval-origin in monitor operations does not work
Using interval-origin in monitor operation definition does not work any more.
Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.
Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.
The call to crm_time_subtract with
origin=2014-03-31 19:20:00Z
date_set-now=2014-03-31 17:31:04Z
result in
delay=-0001-12-31 01:48:56Z
delay_s=31456136
start_delay=31456136000
which is almost a year in the future.
The function crm_time_subtract calculates this by the crm_time_add_* functions.
The buggy statement is in crm_time_add_days.
If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.
But if a_time-days is zero, it must not do anything.
The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.
There are two solutions.
One is the add handling on negative years to crm_time_get_seconds.
The other is to exchange line 1140 in iso8601.c
while (a_time-days = 0) {
by
while (a_time-days 0) {
Second solution is verified to bring the expected result, start-delay of little less than two hours.
Handling of negative years in crm_time_get_seconds might not be a proper solution as the return value of the function is unsigned long long and what to report if the complete calculation gives a negative number of seconds.
Rainer
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Kamailio managed by Pacemaker
Correct. If we are using syntax which is only valid in bash extensions, we should point to bash and not to sh. Rainer Gesendet:Montag, 03. Februar 2014 um 18:16 Uhr Von:Sherwood McGowan sherwood.mcgo...@gmail.com An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Kamailio managed by Pacemaker I found that on Ubuntu servers, the /bin/sh designation fails, but if you use /bin/bash, it works fine On 1/30/2014 3:25 AM, Rainer Brestan wrote: The resource agent was developed by Stefan Wenk an me. Plan is to include it into GIT Repo resource-agents by pull request after some short testing period outside or own labs. Rainer *Gesendet:* Donnerstag, 30. Januar 2014 um 00:25 Uhr *Von:* Vladimir Broz vladik...@centrum.cz *An:* The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org *Betreff:* Re: [Pacemaker] Kamailio managed by Pacemaker Hi, try kamailio mailing list. The OCF script for kamailio was posted and discussed in this thread: [sr-dev] kamailio OCF resource agent for pacemaker - Jan 7 10:12:58 CET 2014 http://lists.sip-router.org/pipermail/sr-dev/2014-January/022639.html hope that helps, -Vlada On 29.1.2014 23:35, Sherwood McGowan wrote: Greetings all, I am curious to know if any of you have ever set up a cluster in which Pacemaker was managing Kamailio? Id be very interested to discuss it with you. Im currently working on a project that involves 2 nodes, with MySQL databases synchronized via DRBD, and a virtual ip...pretty standard stuff... Now, the problem is, I also need to manage Kamailio as well. Of course, I tried the LSB resource and the upstart resource, and have not gotten anywhere. Id be happy to post configuration details and log snippets if someone would like. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Sherwood McGowan sherwood.mcgo...@gmail.com ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Announce: SNMP agent for pacemaker
I have got the SNMP subagent from pacemaker-mgmt 2.1.2 working with corosync 2.3 and pacemaker 1.1.10. Some modification are implemented because of wrong attach method to CIB and one nasty bug, where hbagent crashes, when it does not find an operation on parsing a change. As for all versions of hbagent with corosync it provides only the resource table of LINUX-HA MIB, but not the node tabel Also i have created a very simple resource agent for hbagent to manage it as cluster resource (monitor method looks for SNMP result, so it can detect hanging hbagent, but still needs improvement). When attaching a cluster IP address to this resource, cluster resources can be monitored through this address as long as resource is running anywhere. The plan is (when i find time to do) to integrate the SNMP table part (which works quite well) of this agent into crm_mon with an option, to let crm_mon (when running as daemon) attaching to SNMP through AgentX. Rainer Gesendet:Mittwoch, 22. Januar 2014 um 11:39 Uhr Von:Michael Schwartzkopff m...@sys4.de An:Andrey Groshev gre...@yandex.ru, pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Announce: SNMP agent for pacemaker Am Mittwoch, 22. Januar 2014, 14:31:57 schrieben Sie: 22.01.2014, 12:43, Michael Schwartzkopff m...@sys4.de: Hi, I am working on a SNMP agent for pacemaker. it is written in perl. At the moment it is in an alpha stadium. On each node local call crm_mon/crm_resource ? Partly yes. But in most cases I read the CIB and parse the config and status part. I included memshare caching to minimize impact. Mit freundlichen Gren, Michael Schwartzkopff -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044 Franziskanerstrae 15, 81669 Mnchen Sitz der Gesellschaft: Mnchen, Amtsgericht Mnchen: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Kamailio managed by Pacemaker
The resource agent was developed by Stefan Wenk an me. Plan is to include it into GIT Repo resource-agents by pull request after some short testing period outside or own labs. Rainer Gesendet:Donnerstag, 30. Januar 2014 um 00:25 Uhr Von:Vladimir Broz vladik...@centrum.cz An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Kamailio managed by Pacemaker Hi, try kamailio mailing list. The OCF script for kamailio was posted and discussed in this thread: [sr-dev] kamailio OCF resource agent for pacemaker - Jan 7 10:12:58 CET 2014 http://lists.sip-router.org/pipermail/sr-dev/2014-January/022639.html hope that helps, -Vlada On 29.1.2014 23:35, Sherwood McGowan wrote: Greetings all, I am curious to know if any of you have ever set up a cluster in which Pacemaker was managing Kamailio? Id be very interested to discuss it with you. Im currently working on a project that involves 2 nodes, with MySQL databases synchronized via DRBD, and a virtual ip...pretty standard stuff... Now, the problem is, I also need to manage Kamailio as well. Of course, I tried the LSB resource and the upstart resource, and have not gotten anywhere. Id be happy to post configuration details and log snippets if someone would like. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments
Please do not merge colocation and order together in a way that only none or both is present.
Example 1: Resource A communicates with resource B over network but A must run before B.
In this case only order is needed without colocation.
Example 2: Resource A and B share a local directory structure.
In this case they must run on the same node with colocation, but order is not important.
chain A B
does neither fit example 1 nor 2.
It is OK to have chain additional, but dont remove order and/or colocation.
Or chain is extended to allow specify only order or only colocation.
Rainer
Gesendet:Freitag, 13. Dezember 2013 um 11:57 Uhr
Von:Lars Marowsky-Bree l...@suse.com
An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff:Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments
On 2013-12-13T11:46:05, Kristoffer Grnlund kgronl...@suse.com wrote:
This worries me as well, however the current syntax for constraints is
confusing and error-prone.
Right. At least the { } would make it clear to users that its now a
resource set and not merely more than 2 in the same sequence.
It would be great to be able to do something
to make this easier to use, but exactly what it would be is hard to
say. Making a change that would silently invert the functionality of
existing configurations is, I agree, not a good idea. However, maybe it
would be acceptable if a version: 2 header is required in the
document to enable the new syntax?
Yeah. Its one of those I wish we had done it differently moments, but
I guess were stuck here.
But another thing we discussed is hiding ids for dependencies by
default, since except for resource objects they really dont matter in
99% of the cases. That would condense the configuration significantly.
Yet another option is to come up with some entirely new construct to
supplement colocation and order which does what I think most people
intuitively expects by default, which is enforces both colocation and
ordering, so that foo depends on bar means foo will start after bar,
and only where bar is running.
chain a b c
(specifying the id or a score would be optional; score would default to
mandatory.)
Im all in favor. ;-) Id love it if this had backend support (so pcs
could use it too), but if we cant get it, we may just merge colocation
and order internally to crmsh.
Regards,
Lars
--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendrffer, HRB 21284 (AG Nrnberg)
Experience is the name everyone gives to their mistakes. -- Oscar Wilde
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
I can also agree patch is working.
To be sure, that it had to do with notify, i have created a clone resource with notify=true and it happened to same way, after notify monitor was not called again.
With patch applied it works also for clone resources.
And from my output of the modified resource agents i can see on the timestamps of applied calls that there is no interruption of monitor operation.
Thu Aug 1 10:34:53 CEST 2013 resABC: operation monitor, type , operation
Thu Aug 1 10:34:59 CEST 2013 resABC: operation notify, type pre, operation start
Thu Aug 1 10:34:59 CEST 2013 resABC: operation notify, type post, operation start
Thu Aug 1 10:35:13 CEST 2013 resABC: operation monitor, type , operation
Monitor interval is set to 20 seconds and it is called at this intervals even if notify is in between.
Some hint for the check of sufficency:
On the original 1.1.10 version (without patch) i have tried some resource configuration change on clone resource with notify=true, which result in a reload call of the resource agent.
After logging reload, monitor starts again on both nodes.
Thu Aug 1 09:28:31 CEST 2013 resX: operation monitor, type , operation
Thu Aug 1 09:28:48 CEST 2013 resX: operation notify, type pre, operation start
Thu Aug 1 09:28:48 CEST 2013 resX: operation notify, type post, operation start
Thu Aug 1 09:38:47 CEST 2013 resX: operation reload, type , operation
Thu Aug 1 09:38:47 CEST 2013 resX: operation monitor, type , operation
Will there be a new tag (like 1.1.10-2) for version 1.1.10 with applied patch ?
Rainer
Gesendet:Donnerstag, 01. August 2013 um 05:56 Uhr
Von:Takatoshi MATSUO matsuo@gmail.com
An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff:Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
Hi Andrew
This patch works fine.
2013/8/1 Andrew Beekhof and...@beekhof.net:
On 01/08/2013, at 10:18 AM, Takatoshi MATSUO matsuo@gmail.com wrote:
Hi Andrew
Im about to collect logs of crm_report,
but Rainer already provides it.
Could you see his reports ?
I had just written:
I can but theyre insufficiently helpful.
when a thought struck me
Can you try the following patch?
It would explain why I couldnt reproduce it locally earlier today.
diff --git a/crmd/lrm.c b/crmd/lrm.c
index d6b0dd0..4bce39a 100644
--- a/crmd/lrm.c
+++ b/crmd/lrm.c
@@ -1744,7 +1744,9 @@ do_lrm_rsc_op(lrm_state_t * lrm_state, lrmd_rsc_info_t * rsc, const char *operat
CRM_CHECK(op != NULL, return);
/* stop any previous monitor operations before changing the resource state */
- if (op-interval == 0 strcmp(operation, CRMD_ACTION_STATUS) != 0) {
+ if (op-interval == 0
+ strcmp(operation, CRMD_ACTION_STATUS) != 0
+ strcmp(operation, CRMD_ACTION_NOTIFY) != 0) {
guint removed = 0;
struct stop_recurring_action_s data;
Thanks,
Takatoshi MATSUO
2013/8/1 Rainer Brestan rainer.bres...@gmx.net:
Base situation for the logs:
Pacemaker stop on int2node1 and int2node2
Master/slave resource msABC already configured.
Included in the crm_report is also per node a file a, this is the one,
which the modified Stateful RA writes to log each action performed.
1.) 19:22:25 start Pacemaker on int2node1
https://www.dropbox.com/s/ftbdl71ol2iyi42/step1.log.tar.bz2
monitor on master is called
2.) 19:32:14 start Pacemaker on int2node2
https://www.dropbox.com/s/s3jnxqvod9mlyz1/step2.log.tar.bz2
monitor on master is not called any more
3.) 19:37:14 stop Pacemaker on int2node2
https://www.dropbox.com/s/w75myab6fxh7mak/step3.log.tar.bz2
monitor on master is still not called any more
4.) 19:42:14 start Pacemaker on in2node2
https://www.dropbox.com/s/p00wl9kx4vwhilh/step4.log.tar.bz2
monitor on master is called normally
Hope this gives a clearer picture which component has forgotten the monitor
action.
Rainer
Gesendet: Mittwoch, 31. Juli 2013 um 14:19 Uhr
Von: Andrew Beekhof and...@beekhof.net
An: The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
On 31/07/2013, at 5:17 PM, Rainer Brestan rainer.bres...@gmx.net wrote:
Modified the RA to log each action call performed and from this log there
is no call of monitor action.
From the logs i do not think it is the policy engine, it might be the LRM
part of crmd (the is the only relevant change be seen after git diff between
1.1.10-rc7 and 1.1.10).
Ok. Can you still send me a crm_report though?
Even if the PE isnt at fault, it shows me what the cib looked like at the
time which can be surprisingly helpful.
And it would have all the logs...
Explanation of the below log:
primitive resABC ocf:heartbeat:Stateful
op start interval=0s timeout=60s on-fail=restart
op monitor interval=30s timeout=60s on-fail=restart
op promote interval=0s timeout=60s on-fail=restart
op demote interval=0 timeout=60s on-fail=restart
op stop interval=0 timeout=60s on-fail=restart
op
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
[16444] notice: notice: run_graph: Transition 86 (Complete=13, Pending=0, Fired=0, Skipped=2, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-125.bz2): Stopped 2013-07-31T08:37:51.705+02:00 int2node1 pengine[16443] notice: notice: unpack_config: On loss of CCM Quorum: Ignore 2013-07-31T08:37:51.705+02:00 int2node1 pengine[16443] notice: notice: stage6: Scheduling Node int2node2 for shutdown 2013-07-31T08:37:51.706+02:00 int2node1 pengine[16443] notice: notice: process_pe_message: Calculated Transition 87: /var/lib/pacemaker/pengine/pe-input-126.bz2 2013-07-31T08:37:51.707+02:00 int2node1 crmd[16444] notice: notice: run_graph: Transition 87 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-126.bz2): Complete 2013-07-31T08:37:51.707+02:00 int2node1 crmd[16444] notice: notice: do_state_transition: State transition S_TRANSITION_ENGINE - S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] 2013-07-31T08:37:51.720+02:00 int2node1 crmd[16444] notice: notice: peer_update_callback: do_shutdown of int2node2 (op 45) is complete Output from RA on int2node1: Wed Jul 31 08:30:52 CEST 2013 resABC: operation notify, type pre, operation start Wed Jul 31 08:30:52 CEST 2013 resABC: operation notify, type post, operation start Wed Jul 31 08:30:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:31:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:31:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:31:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:32:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:32:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:32:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:33:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:33:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:33:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:34:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:34:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:34:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:35:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:35:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:35:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:36:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:36:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:36:53 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:37:13 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:37:33 CEST 2013 resABC: operation monitor, type , operation Wed Jul 31 08:37:51 CEST 2013 resABC: operation notify, type pre, operation stop Wed Jul 31 08:37:51 CEST 2013 resABC: operation notify, type post, operation stop After 08:37:51 no log output from Pacemaker for resABC, nor any output from RA on int2node1. Gesendet:Mittwoch, 31. Juli 2013 um 02:10 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available On 30/07/2013, at 9:13 PM, Rainer Brestan rainer.bres...@gmx.net wrote: I can agree, Master monitor operation is broken in 1.1.10 release. When the slave monitor action is started, the master monitor action is not called any more. Based on? I have created a setup with Stateful resource with two nodes. Then the Pacemaker installation is changed to different versions without changing the configuration part of the CIB. Result: 1.1.10-rc5, 1.1.10-rc6 and 1.1.10-rc7 does not have this error 1.1.10-1 release has the error Installation order (just that anybody know how it was done): 1.1.10-1 - error 1.1.10-rc5 - no error 1.1.10-rc6 - no error 1.1.10-rc7 - no error 1.1.10-1 - error Rainer Gesendet: Freitag, 26. Juli 2013 um 09:32 Uhr Von: Takatoshi MATSUO matsuo@gmail.com An: The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available Hi I used Stateful RA and caught a same issue. 1. before starting slave # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1543.bz2 grep Resource action * Resource action: stateful monitor=2000 on 16-sl6 2. starting slave # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1544.bz2 grep Resource action * Resource action: stateful monitor on 17-sl6 * Resource action: stateful notify on 16-sl6 * Resource action: stateful start on 17-sl6 * Resource action: stateful notify on 16-sl6 * Resource action: stateful notify on 17-sl6 * Resource action: stateful monitor=3000 on 17-sl6 3. after # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1545.bz2
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
Base situation for the logs: Pacemaker stop on int2node1 and int2node2 Master/slave resource msABC already configured. Included in the crm_report is also per node a file a, this is the one, which the modified Stateful RA writes to log each action performed. 1.) 19:22:25 start Pacemaker on int2node1 https://www.dropbox.com/s/ftbdl71ol2iyi42/step1.log.tar.bz2 monitor on master is called 2.) 19:32:14 start Pacemaker on int2node2 https://www.dropbox.com/s/s3jnxqvod9mlyz1/step2.log.tar.bz2 monitor on master is not called any more 3.) 19:37:14 stop Pacemaker on int2node2 https://www.dropbox.com/s/w75myab6fxh7mak/step3.log.tar.bz2 monitor on master is still not called any more 4.) 19:42:14 start Pacemaker on in2node2 https://www.dropbox.com/s/p00wl9kx4vwhilh/step4.log.tar.bz2 monitor on master is called normally Hope this gives a clearer picture which component has forgotten the monitor action. Rainer Gesendet:Mittwoch, 31. Juli 2013 um 14:19 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available On 31/07/2013, at 5:17 PM, Rainer Brestan rainer.bres...@gmx.net wrote: Modified the RA to log each action call performed and from this log there is no call of monitor action. From the logs i do not think it is the policy engine, it might be the LRM part of crmd (the is the only relevant change be seen after git diff between 1.1.10-rc7 and 1.1.10). Ok. Can you still send me a crm_report though? Even if the PE isnt at fault, it shows me what the cib looked like at the time which can be surprisingly helpful. And it would have all the logs... Explanation of the below log: primitive resABC ocf:heartbeat:Stateful op start interval=0s timeout=60s on-fail=restart op monitor interval=30s timeout=60s on-fail=restart op promote interval=0s timeout=60s on-fail=restart op demote interval=0 timeout=60s on-fail=restart op stop interval=0 timeout=60s on-fail=restart op monitor interval=20 role=Master timeout=60 ms msABC resABC meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true crm_mon at begin of log: Last updated: Wed Jul 31 08:30:57 2013 Last change: Tue Jul 30 13:01:36 2013 via crmd on int2node1 Stack: corosync Current DC: int2node1 (1743917066) - partition with quorum Version: 1.1.10-1.el6-368c726 2 Nodes configured 5 Resources configured Online: [ int2node1 int2node2 ] Master/Slave Set: msABC [resABC] Masters: [ int2node1 ] Slaves: [ int2node2 ] crm_mon at end of log: Last updated: Wed Jul 31 08:55:29 2013 Last change: Tue Jul 30 13:01:36 2013 via crmd on int2node1 Stack: corosync Current DC: int2node1 (1743917066) - partition with quorum Version: 1.1.10-1.el6-368c726 2 Nodes configured 5 Resources configured Online: [ int2node1 ] OFFLINE: [ int2node2 ] Master/Slave Set: msABC [resABC] Masters: [ int2node1 ] int2node1 is running, int2node2 is started 2013-07-31T08:30:52.631+02:00 int2node1 pengine[16443] notice: notice: LogActions: Start resABC:1 (int2node2) 2013-07-31T08:30:52.638+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 9: monitor resABC:1_monitor_0 on int2node2 2013-07-31T08:30:52.638+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 54: notify resABC_pre_notify_start_0 on int2node1 (local) 2013-07-31T08:30:52.681+02:00 int2node1 crmd[16444] notice: notice: process_lrm_event: LRM operation resABC_notify_0 (call=64, rc=0, cib-update=0, confirmed=true) ok 2013-07-31T08:30:52.780+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 25: start resABC:1_start_0 on int2node2 2013-07-31T08:30:52.940+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 55: notify resABC_post_notify_start_0 on int2node1 (local) 2013-07-31T08:30:52.943+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 56: notify resABC:1_post_notify_start_0 on int2node2 2013-07-31T08:30:52.982+02:00 int2node1 crmd[16444] notice: notice: process_lrm_event: LRM operation resABC_notify_0 (call=67, rc=0, cib-update=0, confirmed=true) ok 2013-07-31T08:30:52.992+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 24: monitor resABC_monitor_2 on int2node1 (local) 2013-07-31T08:30:52.996+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 26: monitor resABC:1_monitor_3 on int2node2 2013-07-31T08:30:53.035+02:00 int2node1 crmd[16444] notice: notice: process_lrm_event: LRM operation resABC_monitor_2 (call=70, rc=8, cib-update=149, confirmed=false) master At this point int2node2 is stopped. 2013-07-31T08:37:51.457+02:00 int2node1 crmd[16444] notice: notice: do_state_transition: State transition S_IDLE - S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] 2013-07-31T08:37:51.462+02:00 int2node1
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
I can agree, Master monitor operation is broken in 1.1.10 release. When the slave monitor action is started, the master monitor action is not called any more. I have created a setup with Stateful resource with two nodes. Then the Pacemaker installation is changed to different versions without changing the configuration part of the CIB. Result: 1.1.10-rc5, 1.1.10-rc6 and 1.1.10-rc7 does not have this error 1.1.10-1 release has the error Installation order (just that anybody know how it was done): 1.1.10-1 - error 1.1.10-rc5 - no error 1.1.10-rc6 - no error 1.1.10-rc7 - no error 1.1.10-1 - error Rainer Gesendet:Freitag, 26. Juli 2013 um 09:32 Uhr Von:Takatoshi MATSUO matsuo@gmail.com An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available Hi I used Stateful RA and caught a same issue. 1. before starting slave # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1543.bz2 grep Resource action * Resource action: stateful monitor=2000 on 16-sl6 2. starting slave # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1544.bz2 grep Resource action * Resource action: stateful monitor on 17-sl6 * Resource action: stateful notify on 16-sl6 * Resource action: stateful start on 17-sl6 * Resource action: stateful notify on 16-sl6 * Resource action: stateful notify on 17-sl6 * Resource action: stateful monitor=3000 on 17-sl6 3. after # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1545.bz2 grep Resource action * Resource action: stateful monitor=3000 on 17-sl6 Monitor=2000 is deleted. Is this correct ? My setting property no-quorum-policy=ignore stonith-enabled=false rsc_defaults resource-stickiness=INFINITY migration-threshold=1 ms msStateful stateful meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true primitive stateful ocf:heartbeat:Stateful op start timeout=60s interval=0s on-fail=restart op monitor timeout=60s interval=3s on-fail=restart op monitor timeout=60s interval=2s on-fail=restart role=Master op promote timeout=60s interval=0s on-fail=restart op demote timeout=60s interval=0s on-fail=stop op stop timeout=60s interval=0s on-fail=block Regards, Takatoshi MATSUO 2013/7/26 Takatoshi MATSUO matsuo@gmail.com: Hi My report is late for 1.1.10 :( I am using pacemaker 1.1.10-0.1.ab2e209.git. It seems that masters monitor is stopped when slave is started. Does someone encounter same problem ? I attach a log and settings. Thanks, Takatoshi MATSUO 2013/7/26 Digimer li...@alteeve.ca: Congrats!! I know this was a long time in the making. digimer On 25/07/13 20:43, Andrew Beekhof wrote: Announcing the release of Pacemaker 1.1.10 https://github.com/ClusterLabs/pacemaker/releases/Pacemaker-1.1.10 There were three changes of note since rc7: + Bug cl#5161 - crmd: Prevent memory leak in operation cache + cib: Correctly read back archived configurations if the primary is corrupted + cman: Do not pretend we know the state of nodes weve never seen Along with assorted bug fixes, the major topics for this release were: - stonithd fixes - fixing memory leaks, often caused by incorrect use of glib reference counting - supportability improvements (code cleanup and deduplication, standardized error codes) Release candidates for the next Pacemaker release (1.1.11) can be expected some time around Novemeber. A big thankyou to everyone that spent time testing the release candidates and/or contributed patches. However now that Pacemaker is perfect, anyone reporting bugs will be shot :-) To build rpm packages: 1. Clone the current sources: # git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git # cd pacemaker 1. Install dependancies (if you havent already) [Fedora] # sudo yum install -y yum-utils [ALL] # make rpm-dep 1. Build Pacemaker # make release 1. Copy and deploy as needed ## Details - 1.1.10 - final Changesets: 602 Diff: 143 files changed, 8162 insertions(+), 5159 deletions(-) ## Highlights ### Features added since Pacemaker-1.1.9 + Core: Convert all exit codes to positive errno values + crm_error: Add the ability to list and print error symbols + crm_resource: Allow individual resources to be reprobed + crm_resource: Allow options to be set recursively + crm_resource: Implement --ban for moving resources away from nodes and --clear (replaces --unmove) + crm_resource: Support OCF tracing when using --force-(checkstartstop) + PE: Allow active nodes in our current membership to be fenced without quorum + PE: Suppress meaningless IDs when displaying anonymous clone status + Turn off auto-respawning of systemd services when the cluster starts them + Bug cl#5128 - pengine: Support maintenance mode for a single node ### Changes since Pacemaker-1.1.9 + crmd: cib: stonithd: Memory leaks resolved and improved use of glib
Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?
The bug is in the function is_normal_node. This function checks the attribute type for state normal. But this attribute is not used any more. CIB output from Pacemaker 1.1.8 nodes node id=int2node1 uname=int2node1 instance_attributes id=nodes-int2node1 nvpair id=nodes-int2node1-standby name=standby value=off/ /node node id=int2node2 uname=int2node2 instance_attributes id=nodes-int2node2 nvpair id=nodes-int2node2-standby name=standby value=on/ /node /nodes CIB output from Pacemaker 1.1.7 nodes node id=int1node1 type=normal uname=int1node1 /node node id=int1node2 type=normal uname=int1node2 /node /nodes Therefore, function listnodes will not return any node and function standby will use the current node as node and the first argument as lifetime. In case of specified both (node and lifetime) it works because of other else path. Rainer Gesendet:Mittwoch, 15. Mai 2013 um 21:31 Uhr Von:Lars Ellenberg lars.ellenb...@linbit.com An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9? On Wed, May 15, 2013 at 03:34:14PM +0200, Dejan Muhamedagic wrote: On Tue, May 14, 2013 at 10:03:59PM +0200, Lars Ellenberg wrote: On Tue, May 14, 2013 at 09:59:50PM +0200, Lars Ellenberg wrote: On Mon, May 13, 2013 at 01:53:11PM +0200, Michael Schwartzkopff wrote: Hi, crm tells me it is version 1.2.4 pacemaker tell me it is verison 1.1.9 So it should work since incompatibilities are resolved in crm higher that version 1.2.1. Anywas crm tells me nonsense: # crm crm(live)# node crm(live)node# standby node1 ERROR: bad lifetime: node1 Your node is not named node1. check: crm node list Maybe a typo, maybe some case-is-significant nonsense, maybe you just forgot to use the fqdn. maybe the check for is this a known node name is (now) broken? standby with just one argument checks if that argument happens to be a known node name, and assumes that if it is not, it has to be a lifetime, and the current node is used as node name... Maybe we should invert that logic, and instead compare the single argument against allowed lifetime values (reboot, forever), and assume it is supposed to be a node name otherwise? Then the error would become ERROR: unknown node name: node1 Which is probably more useful most of the time. Dejan? Something like this maybe: diff --git a/modules/ui.py.in b/modules/ui.py.in --- a/modules/ui.py.in +++ b/modules/ui.py.in @@ -1185,7 +1185,7 @@ class NodeMgmt(UserInterface): if not args: node = vars.this_node if len(args) == 1: - if not args[0] in listnodes(): + if args[0] in (reboot, forever): Yes, I wanted to look at it again. Another complication is that the lifetime can be just about anything in that date ISO format. That may well be, but right now those would be rejected by crmsh anyways: if lifetime not in (None,reboot,forever): common_err(bad lifetime: %s % lifetime) return False -- : Lars Ellenberg : LINBIT Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?
Seems that it requires now the lifetime crm node standby node1 forever The error message is just nonsense. Rainer Gesendet:Montag, 13. Mai 2013 um 13:53 Uhr Von:Michael Schwartzkopff mi...@clusterbau.com An:pacemaker@oss.clusterlabs.org Betreff:[Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9? Hi, crm tells me it is version 1.2.4 pacemaker tell me it is verison 1.1.9 So it should work since incompatibilities are resolved in crm higher that version 1.2.1. Anywas crm tells me nonsense: # crm crm(live)# node crm(live)node# standby node1 ERROR: bad lifetime: node1 Any ideas? -- Dr. Michael Schwartzkopff Guardinistr. 63 81375 Mnchen Tel: (0163) 172 50 98 ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?
Hi Andrew, yes, this clarifies a lot. Seems that it is really time to throw away the plugin. The CMAN solution wont be able (at least from the documentation) to attach new nodes without reconfiguration and restart CMAN on the existing nodes. The alternative is corosync 2.x. ClusterLabs has a quite long list of corosync versions from branch 2.0, 2.1, 2.2 und 2.3. Beside the current reported issue of version 2.3, which version does ClusterLabs use for its regression test. I found somewhere a note for 2.1.x, is this true ? Rainer Gesendet:Donnerstag, 09. Mai 2013 um 04:31 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] 1.1.8 not compatible with 1.1.7? On 08/05/2013, at 4:53 PM, Andrew Beekhof and...@beekhof.net wrote: On 08/05/2013, at 4:08 PM, Andrew Beekhof and...@beekhof.net wrote: On 03/05/2013, at 8:46 PM, Rainer Brestan rainer.bres...@gmx.net wrote: Now i have all the logs for some combinations. Corosync: 1.4.1-7 for all the tests on all nodes Base is always fresh installation of each node with all packages equal except pacemaker version. int2node1 node id: 1743917066 int2node2 node id: 1777471498 In each ZIP file log from both nodes and the status output of crm_mon and cibadmin -Q is included. 1.) 1.1.8-4 attaches to running 1.1.7-6 cluster https://www.dropbox.com/s/06oyrle4ny47uv9/attach_1.1.8-4_to_1.1.7-6.zip Result: join outstanding 2.) 1.1.9-2 attaches to running 1.1.7-6 cluster https://www.dropbox.com/s/fv5kcm2yb5jz56z/attach_1.1.9-2_to_1.1.7-6.zip Result: join outstanding Neither side is seeing anything from the other, which is very unexpected. I notice youre using the plugin... which acts as a message router. So I suspect something in there has changed (though Im at a loss to say what) and that cman based clusters are unaffected. Confirmed, cman clusters are unaffected. Im yet to work out what changed in the plugin. I worked it out... The Red Hat changelog for 1.1.8-2 originally contained +- Cman is the only supported membership quorum provider, do not ship the corosync plugin When this decision was reversed (when I realised no-one was seeing the ERROR logs indicating it was going away), I neglected to re-instate the following distro specific patch (which avoided conflicts between the ID used by CMAN and Pacemaker): diff --git a/configure.ac b/configure.ac index a3784d5..dafa9e2 100644 --- a/configure.ac +++ b/configure.ac @@ -1133,7 +1133,7 @@ AC_MSG_CHECKING(for native corosync) COROSYNC_LIBS= CS_USES_LIBQB=0 -PCMK_SERVICE_ID=9 +PCMK_SERVICE_ID=10 LCRSODIR=libdir if test SUPPORT_CS = no; then So Pacemaker 6.4 is talking on slot 10, while Pacemaker == 6.4 is using slot 9. This is why the two versions cannot see each other :-( Im very sorry. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] [Patch] pacemaker-mgmt/hbagent avoid coredump with pacemaker=1.1.8/corosync
SNMP agent hbagent from pacemaker-mgmt produces segmentation fault if used with pacemaker=1.1.8 and corosync. The reason is function get_cib_fd in file hbagentv2.c. It tries to get the file descriptor with function pointer inputfd, which is not initialized any more since change of IPC to libqb. The patch uses the already present conditional define HAVE_CRM_IPC_NEW to get the file descriptor from function signon_raw and passes this as return value of get_cib_fd. Rainer hbagentv2.c.patch Description: Binary data ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?
Now i have all the logs for some combinations. Corosync: 1.4.1-7 for all the tests on all nodes Base is always fresh installation of each node with all packages equal except pacemaker version. int2node1 node id: 1743917066 int2node2 node id: 1777471498 In each ZIP file log from both nodes and the status output of crm_mon and cibadmin -Q is included. 1.) 1.1.8-4 attaches to running 1.1.7-6 cluster https://www.dropbox.com/s/06oyrle4ny47uv9/attach_1.1.8-4_to_1.1.7-6.zip Result: join outstanding 2.) 1.1.9-2 attaches to running 1.1.7-6 cluster https://www.dropbox.com/s/fv5kcm2yb5jz56z/attach_1.1.9-2_to_1.1.7-6.zip Result: join outstanding 3.) 1.1.9-2 attaches to running 1.1.8-4 cluster https://www.dropbox.com/s/y9o4yo8g8ahwjga/attach_1.1.9-2_to_1.1.8-4.zip Result: join successful Rainer Gesendet:Freitag, 03. Mai 2013 um 01:30 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] 1.1.8 not compatible with 1.1.7? On 03/05/2013, at 4:46 AM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi Lars, i have tried 1.1.9-2 from download area at clusterlabs for RHEL6 with corosync 1.4.1-17, also running with 1.1.7-6 at the other node. I have to go deeper in details later on (with logs), but the first try was worse than 1.1.8-7. When the node with 1.1.9-2 joins the cluster, it could not even decode the ais_message to get the node name of the node running 1.1.7-6. Logs? It states a new node has joined with the correct node id, but as name it could only decode (null) as node name. Just as first impression. Rainer Gesendet: Dienstag, 30. April 2013 um 17:16 Uhr Von: Lars Marowsky-Bree l...@suse.com An: pacemaker@oss.clusterlabs.org Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7? On 2013-04-24T11:44:57, Rainer Brestan rainer.bres...@gmx.net wrote: Current DC: int2node2 - partition WITHOUT quorum Version: 1.1.8-7.el6-394e906 This may not be the answer you want, since it is fairly unspecific. But I think we noticed something similar when we pulled in 1.1.8, I dont recall the bug number, but I *think* it worked out with a later git version. Can you try a newer build than 1.1.8? Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendrffer, HRB 21284 (AG Nrnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?
Hi Lars, i have tried 1.1.9-2 from download area at clusterlabs for RHEL6 with corosync 1.4.1-17, also running with 1.1.7-6 at the other node. I have to go deeper in details later on (with logs), but the first try was worse than 1.1.8-7. When the node with 1.1.9-2 joins the cluster, it could not even decode the ais_message to get the node name of the node running 1.1.7-6. It states a new node has joined with the correct node id, but as name it could only decode (null) as node name. Just as first impression. Rainer Gesendet:Dienstag, 30. April 2013 um 17:16 Uhr Von:Lars Marowsky-Bree l...@suse.com An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] 1.1.8 not compatible with 1.1.7? On 2013-04-24T11:44:57, Rainer Brestan rainer.bres...@gmx.net wrote: Current DC: int2node2 - partition WITHOUT quorum Version: 1.1.8-7.el6-394e906 This may not be the answer you want, since it is fairly unspecific. But I think we noticed something similar when we pulled in 1.1.8, I dont recall the bug number, but I *think* it worked out with a later git version. Can you try a newer build than 1.1.8? Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendrffer, HRB 21284 (AG Nrnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?
I have tried to make this test, because I had the same problem. Origin: One node cluster, node int2node1 running with IP address 10.16.242.231, quorum ignore, DC int2node1 [root@int2node1 sysconfig]# crm_mon -1 Last updated: Wed Apr 24 09:49:32 2013 Last change: Wed Apr 24 09:44:55 2013 via crm_resource on int2node1 Stack: openais Current DC: int2node1 - partition WITHOUT quorum Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14 1 Nodes configured, 2 expected votes 1 Resources configured. Online: [ int2node1 ] Clone Set: cloneSysInfo [resSysInfo] Started: [ int2node1 ] Next step: Node int2node2 with IP address 10.16.242.233 joins the cluster. Result: [root@int2node1 sysconfig]# crm_mon -1 Last updated: Wed Apr 24 10:14:18 2013 Last change: Wed Apr 24 10:05:20 2013 via crmd on int2node1 Stack: openais Current DC: int2node1 - partition WITHOUT quorum Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14 2 Nodes configured, 2 expected votes 1 Resources configured. Online: [ int2node1 ] OFFLINE: [ int2node2 ] Clone Set: cloneSysInfo [resSysInfo] Started: [ int2node1 ] [root@int2node1 sysconfig]# corosync-objctl grep member runtime.totem.pg.mrp.srp.members.1743917066.ip=r(0) ip(10.16.242.231) runtime.totem.pg.mrp.srp.members.1743917066.join_count=1 runtime.totem.pg.mrp.srp.members.1743917066.status=joined runtime.totem.pg.mrp.srp.members.1777471498.ip=r(0) ip(10.16.242.233) runtime.totem.pg.mrp.srp.members.1777471498.join_count=1 runtime.totem.pg.mrp.srp.members.1777471498.status=joined [root@int2node1 sysconfig]# crm_node -l 1743917066 int2node1 member [root@int2node2 ~]# crm_mon -1 Last updated: Wed Apr 24 11:27:39 2013 Last change: Wed Apr 24 10:07:45 2013 via crm_resource on int2node2 Stack: classic openais (with plugin) Current DC: int2node2 - partition WITHOUT quorum Version: 1.1.8-7.el6-394e906 2 Nodes configured, 2 expected votes 1 Resources configured. Online: [ int2node2 ] OFFLINE: [ int2node1 ] Clone Set: cloneSysInfo [resSysInfo] Started: [ int2node2 ] [root@int2node2 ~]# corosync-objctl grep member runtime.totem.pg.mrp.srp.members.1743917066.ip=r(0) ip(10.16.242.231) runtime.totem.pg.mrp.srp.members.1743917066.join_count=1 runtime.totem.pg.mrp.srp.members.1743917066.status=joined runtime.totem.pg.mrp.srp.members.1777471498.ip=r(0) ip(10.16.242.233) runtime.totem.pg.mrp.srp.members.1777471498.join_count=1 runtime.totem.pg.mrp.srp.members.1777471498.status=joined [root@int2node2 ~]# crm_node -l 1777471498 int2node2 member Pacemaker log of int2node2 with trace setting. https://www.dropbox.com/s/04ciy2g6dfbauxy/pacemaker.log?n=165978094 On int2node1 (1.1.7) the trace setting did not create the pacemaker.log file. Below the excerpt of cib with node information from int2node2. [root@int2node2 ~]# cibadmin -Q cib epoch=17 num_updates=51 admin_epoch=0 validate-with=pacemaker-1.2 crm_feature_set=3.0.7 update-origin=int2node2 update-client=crm_resource cib-last-written=Wed Apr 24 10:07:45 2013 have-quorum=0 dc-uuid=int2node2 configuration crm_config cluster_property_set id=cib-bootstrap-options ... /cluster_property_set /crm_config nodes node id=int2node2 uname=int2node2/ node id=int2node1 uname=int2node1/ /nodes resources ... /resources rsc_defaults ... /rsc_defaults /configuration status node_state id=int2node2 uname=int2node2 in_ccm=true crmd=online crm-debug-origin=do_update_resource join=member expected=member transient_attributes id=int2node2 instance_attributes id=status-int2node2 ... /instance_attributes /transient_attributes lrm id=int2node2 lrm_resources ... /lrm_resources /lrm /node_state node_state id=int2node1 uname=int2node1 in_ccm=true crmd=online join=down crm-debug-origin=do_state_transition/ /status /cib On int2node2 the node state in the cib is different. status node_state id=int2node1 uname=int2node1 ha=active in_ccm=true crmd=online join=member expected=member crm-debug-origin=do_state_transition shutdown=0 transient_attributes id=int2node1 /transient_attributes lrm id=int2node1 lrm_resources ... /lrm_resources /lrm /node_state node_state id=int2node2 uname=int2node2 crmd=online crm-debug-origin=do_state_transition ha=active in_ccm=true join=pending/ /status Rainer Gesendet:Mittwoch, 17. April 2013 um 07:32 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] 1.1.8 not compatible with 1.1.7? On 15/04/2013, at 7:08 PM, Pavlos Parissis pavlos.paris...@gmail.com wrote: Hoi, I upgraded 1st node and here are the logs https://dl.dropboxusercontent.com/u/1773878/pacemaker-issue/node1.debuglog https://dl.dropboxusercontent.com/u/1773878/pacemaker-issue/node2.debuglog Enabling tracing on the mentioned functions didnt give at least to me any more information. 10:22:08 pacemakerd[53588]: notice:
Re: [Pacemaker] attrd waits one second before doing update
OK, and where is the difference between 1.1.8 and 1.1.7. I am currently testing this on a one node cluster, so attrd wait for the message come back from himself. This cant take one second, or is attrd waiting this time anyhow to be sure to get it from all nodes back? Rainer Gesendet:Freitag, 12. April 2013 um 02:03 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] attrd waits one second before doing update On 12/04/2013, at 7:17 AM, Rainer Brestan rainer.bres...@gmx.net wrote: In pacemaker 1.1.7-6 with corosync 1.4.1-7 update of attributes works almost online. Used with SysInfo resource agent and manual commands like attrd_updater -U 4 -n test. In the logfile there is one line attrd[...] notice: attrd_trigger_update: Sending flush up to all hosts for: ... and a few milliseconds later attrd[...] notice: attrd_perform_update: Sent update ... with the same content. After upgrade to version 1.1.8-6 there is always nearly exact one second between trigger and perform. 2013-04-11T22:51:55.389+02:00 int2node2 attrd[28370] notice: notice: attrd_trigger_update: Sending flush op to all hosts for: text (81) 2013-04-11T22:51:56.397+02:00 int2node2 attrd[28370] notice: notice: attrd_perform_update: Sent update 5814: text=81 And what i found out having several updates running, they have a single queue. All attrd_updater processes are waiting for the next to be finished, so there cant be more than one update per second any more. Has this something to do with attrd: Have single-shot clients wait for an ack before disconnecting stated in the Changelog for 1.1.8 ? No, nothing at all. If yes, is it intended to have a single queue ? More like unavoidable, since we need to talk to the other nodes and messages between them are ordered. And is this 1 second fixed ? From where does this 1 second come, i dont think that it takes one second to get the ack. When the timer expires, attrd sends a cluster message to all nodes (including itself) telling them to update the CIB with their current value. The delay comes from waiting for the cluster message we sent to arrive back again before sending our own updates, this helps ensure all the updates arrive in the CIB at almost the same time. This can run into heavy delays (and therefore timeouts) for monitor functions of RA performing attribute updates. Rainer ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] attrd waits one second before doing update
In pacemaker 1.1.7-6 with corosync 1.4.1-7 update of attributes works almost online. Used with SysInfo resource agent and manual commands like attrd_updater -U 4 -n test. In the logfile there is one line attrd[...] notice: attrd_trigger_update: Sending flush up to all hosts for: ... and a few milliseconds later attrd[...] notice: attrd_perform_update: Sent update ... with the same content. After upgrade to version 1.1.8-6 there is always nearly exact one second between trigger and perform. 2013-04-11T22:51:55.389+02:00 int2node2 attrd[28370] notice: notice: attrd_trigger_update: Sending flush op to all hosts for: text (81) 2013-04-11T22:51:56.397+02:00 int2node2 attrd[28370] notice: notice: attrd_perform_update: Sent update 5814: text=81 And what i found out having several updates running, they have a single queue. All attrd_updater processes are waiting for the next to be finished, so there cant be more than one update per second any more. Has this something to do with attrd: Have single-shot clients wait for an ack before disconnecting stated in the Changelog for 1.1.8 ? If yes, is it intended to have a single queue ? And is this 1 second fixed ? From where does this 1 second come, i dont think that it takes one second to get the ack. This can run into heavy delays (and therefore timeouts) for monitor functions of RA performing attribute updates. Rainer ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Master Slave Resource Agent won't promote
Hi Felix, maybe my hint is worthless, but have you implemented the crm_master calls in your RA ? See Stateful RA demo CRM_MASTER calls. Rainer Gesendet:Mittwoch, 10. April 2013 um 09:58 Uhr Von:Felix Zachlod fz.li...@sis-gmbh.info An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:[Pacemaker] Master Slave Resource Agent wont promote Hello! I have another problem with my resource agent which should run in a master slave fashion. I successfully tested the RA with ocf-tester and it completes any promote or demote action: fctarget[14659]: DEBUG: Resource is running fctarget[14659]: DEBUG: Resource is currently running as Slave fctarget[14659]: DEBUG: fctargettest demote : 0 Checking for promote action fctarget[14677]: DEBUG: Resource is running fctarget[14677]: DEBUG: Resource is currently running as Slave fctarget[14677]: DEBUG: Resource is running MASTER fctarget[14677]: DEBUG: Resource promoted fctarget[14677]: DEBUG: fctargettest promote : 0 Testing: demotion of started resource fctarget[14717]: DEBUG: Resource is running MASTER fctarget[14717]: DEBUG: Resource is currently running as Master fctarget[14717]: DEBUG: Resource is running fctarget[14717]: DEBUG: Resource demoted fctarget[14717]: DEBUG: fctargettest demote : 0 Testing: promote fctarget[14746]: DEBUG: Resource is running fctarget[14746]: DEBUG: Resource is currently running as Slave fctarget[14746]: DEBUG: Resource is running MASTER fctarget[14746]: DEBUG: Resource promoted fctarget[14746]: DEBUG: fctargettest promote : 0 Testing: demote fctarget[14790]: DEBUG: Resource is running MASTER fctarget[14790]: DEBUG: Resource is currently running as Master fctarget[14790]: DEBUG: Resource is running fctarget[14790]: DEBUG: Resource demoted fctarget[14790]: DEBUG: fctargettest demote : 0 Testing: demotion of demoted resource fctarget[14819]: DEBUG: Resource is running fctarget[14819]: DEBUG: Resource is currently running as Slave fctarget[14819]: DEBUG: fctargettest demote : 0 [...] fctarget passed all tests I added the resource to the crm and it successfully starts on both nodes. But it does not promote anywhere although I have configured target-role Master The Cluster Manager does not even try to promote as I dont get any log output about something failing. It seems the crm does not like my RA for some reason. I can successfully promote a Stateful Dummy resource or a DRBD device with the CRM. Configuration of resource looks like the following: primitive p_fctarget_7701 ocf:onesty:fctarget params wwnn=77:00:00:00:00:00:01:00 wwpn=77:00:00:00:00:00:01:01 parentwwpns=storage-test-a;21:00:00:e0:8b:86:0e:cf storage-test-b;21:00:00:1b:32:88:66:c5 op stop interval=0 timeout=40s ms ms_fctarget_7701 p_fctarget_7701 meta master-max=1 clone-max=2 clone-node-max=1 target-role=Master My resource agent does support the promote and demote action and advertises them in the meta data: actions action name=start timeout=15 / action name=stop timeout=40 / action name=promote timeout=20 / action name=demote timeout=20 / action name=status timeout=10 interval=10 / action name=monitor timeout=10 interval=10 role=Master/ action name=monitor timeout=10 interval=10 role=Slave/ action name=meta-data timeout=5 / action name=validate-all timeout=10 / /actions case __OCF_ACTION in start) fctarget_start;; stop) fctarget_stop;; promote) fctarget_promote;; demote) fctarget_demote;; monitorstatus) fctarget_monitor;; validate-all) fctarget_validate_all;; *) fctarget_usage exit OCF_ERR_UNIMPLEMENTED ;; This ist he output of crm status: Master/Slave Set: ms_fctarget_7701 [p_fctarget_7701] Slaves: [ storage-test-a storage-test-b ] I tried manually promoting and demoting the resource which simply seems to do NOTHING. Can anyone give me a hint what I am missing? Thank you all in advance ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Clone Resources Individual Configuration per Node
Hi Felix, thats exactly the reason why I took the meta attribute variant. It is currently available neither via crm_resource nor via crm. Maybe a good point to submit an request to Dejan about extension of crmsh. Rainer Gesendet:Dienstag, 09. April 2013 um 15:32 Uhr Von:Felix Zachlod fz.li...@sis-gmbh.info An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] Clone Resources Individual Configuration per Node Hi again, Id be interested to hear your conclusions. I am currently trying your suggestion with using rules. Can anyone tell how to apply such a config via the crm shell? I usually do not edit the xml directly. But I cant find a way to do via the crm shell and did not find any documentation how rules can be applied to attributes via the shell. Thanks in advance ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Clone Resources Individual Configuration per Node
Hi Felix,
basically you have three option to provide information to the resource agent.
- Resource parameters
- Resource meta attributes
- Node attributes
Let me assume some information for an example.
Your nodes are named nodeA and nodeB.
The hardware address for nodeA shall be 0x3 and for nodeB 0x6.
Resource parameter should be defined in the meta information part of the resource agent. As long as you have always a defined number of nodes, this can work as follows.
crm configure ... param node1name=nodeA node1addr=0x3 node2name=nodeB node2addr=0x6
The resource agent can go through all nodeXname parameter and pick the number out with its own hostname and read the address with the same number.
Be aware, that all those parameters have to be defined in the mta-data.
Resource meta attributes are more relaxed, as you can assign every name and every value without defining them in the meta-data.
crm configure ... meta node1name=nodeA node1addr=0x3 node2name=nodeB node2addr=0x6
Read the meta data with crm_resource inside the resource agent.
Node attributes are set and read with crm_attribute and can be persistent.
Set the attribute with
crm_attribute -n hwaddr -N nodeA -l forever -v 0x3
crm_attribute -n hwaddr -N nodeB -l forever -v 0x6
Inside the resource agent read the value with
crm_attribute -n hwaddr -N HOSTNAME -l forever -G -q
and you get the value for the current host.
I had a similar approach to do and used resource meta attributes mainly for two reasons.
- Resource configuration is bound to the resource and does not influence anybody else.
- Using crm configure show also shows the configuration of the resource, using crm_attribute would not.
Rainer
Gesendet:Montag, 08. April 2013 um 09:34 Uhr
Von:Felix Zachlod fz.li...@sis-gmbh.info
An:Pacemaker@oss.clusterlabs.org
Betreff:[Pacemaker] Clone Resources Individual Configuration per Node
Hello List,
I am up to writing a resource agent for Pacemaker which shall be used in a
Master/Slave manner. So the cluster shall always run two instances and one
of them shall be Master. My clone instances need a different configuration
depending on the node that they are running on. I know I could accomplish
this by having some kind of configuration file on each node but I want to
manage as much as possible through the cluster manager. So what I want is a
Master / Slave (or clone) resource running with instance attributes tied to
the cluster node which they are running on.
Let me explain what the purpose is:
The resource is dependent on specific hardware on the cluster node, which
has different hardware addresses on each node. So resource instance A DOES
deliver the same service as resource instance B at least when being in the
master state each but anyways has to be started with a different parameter.
I understood that I can configure a location constraint for a resource so
that it can be run only on one node Lets say my RA expects a param
hwaddr={string}. I can set up a resource for Node a with hwaddr=a and one
for node b with hwaddr=b but how can I tell pacemaker that these two
resources now shall be treated as a Master Slave set? Or am I thinking about
this too complicated?
Thank you for any hint in advance,
Felix
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] issues when installing on pxe booted environment
Puh, i havent thought the discussion became this direction. Corosync is not the only software, which need shared memory, so it is part of the OS startup to provide it, not part of Corosync or Pacemaker. And yes, it is too late to mount it in Pacemaker startup. Even in most embedded Linux installations share memory is available, just with smaller size. So, there is no need to include anything in Corosync or Pacemaker startup directly. My answer about shared memory is valid and neccessary only for anaconda based installations. When you install a RedHat system, the installation process is done by anaconda with a running miniroot (install.img) as ram disk. Within this miniroot shared memory is enabled, but they (RedHat) missed the mount. If somebody wants to correct it, RedHat (more specific the maintainer of install.img) is the correct address for this gap. To have world read access to CRM_CORE_DIR should be enough in case a core pattern set explicit to another directory (not tested yet). When calling resource agents, CRM_CORE_DIR is the current PWD (tested with echo all environment variables to a file inside monitor call). If this directory is not readable during switch user inside resource agent, stupid things could happen. Anyhow, it is not a good method of writing resource agents, switching user without setting the environment of the switched user. So, this is no fault of Pacemaker, more of loose programmed resource agents. When the resource agent or any of its childs produces a core dump, it goes either to the current directory or to the directory specified with kernel core pattern. If the core will go into the current directory and it is written by a switched user, who is not member of pcmk_gid, the core get lost. As i am not sure every resource agent is written with proper switch user environment and not rewriting my core pattern, enable world write on CRM_CCORE_DIR was the easy work around for it. Rainer Gesendet:Freitag, 29. Mrz 2013 um 09:51 Uhr Von:Jacek Konieczny jaj...@jajcus.net An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] issues when installing on pxe booted environment On Fri, 29 Mar 2013 11:37:37 +1100 Andrew Beekhof and...@beekhof.net wrote: On Thu, Mar 28, 2013 at 10:43 PM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi John, to get Corosync/Pacemaker running during anaconda installation, i have created a configuration RPM package which does a few actions before starting Corosync and Pacemaker. An excerpt of the post install of this RPM. # mount /dev/shm if not already existing, otherwise openais cannot work if [ ! -d /dev/shm ]; then mkdir /dev/shm mount /dev/shm fi Perhaps mention this to the corosync guys, it should probably go into their init script. I dont think so. It is just a part of modern Linux system environment. corosync is not supposed to mount the root filesystem or /proc mounting /dev/shm is not its responsibility either. BTW The excerpt above assumes there is a /dev/shm entry in /etc/fstab. Should this be added there by the corosync init script too? Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] PGSQL resource promotion issue
Look at the order. Master/replication VIP ist started after promote and stopped after demote. listen_addresses = * means dont take care about specific interfaces, so it will listen on all interfaces not matter if they are coming up or switching down after postgres start. The master either sets the promotion score to CAN_PROMOTE=100 or CAN_NOT_PROMOTE=-INF. For every slave which is in sync it sets 100 and for each missing sync -INF. When the master is away all slaves fall back to the funtion have_master_right. Now the normal election begins. Every slave writes its own xlog location and the one with the highest value set promotion score to 1000. All slaves which are not in sync dont participate in the master election any more. This is the starting code of function have_master_right. Rainer Gesendet:Freitag, 29. Mrz 2013 um 12:33 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] PGSQL resource promotion issue On Mar 28, 2013, at 8:13 AM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi Steve, i think, you have misunderstood how ip addresses are used with this setup, PGVIP should start after promotion. Take a look at Takatoshis Wiki. https://github.com/t-matsuo/resource-agents/wiki/Resource-Agent-for-PostgreSQL-9.1-streaming-replication I see that he has the master/replication VIPs with a resource order to force promotion before moving the VIPs to the new master. I dont get how the postgres service is going to listen on those interfaces if they have not already migrated to the new master. Even with setting the listen_addresses = * The promotion sequency is very simple. When no master is existing, all slaves write their current replay xlog into the node attribute PGSQL-xlog-loc during monitor call. Does this also hold true if a Master fails? From the looks of it, if there was a Master before the failure that the master score is set from the function that grabs the data_status from the master (STREAMINGSYNC,STREAMINGASYNC,STREAMINGPOTENTIAL, etc ). The reason I ask is if the master fails and the slaves dont then compare their xlog location, there is a potential for data loss if the incorrect slave is promoted. You can see all them with crm_mon -A1f. Each slave gets these attributes from all node configured in parameter node_list (hopefully your node names in Pacemaker are the same as in node_list) and compares them to get the highest. If the highest is this list is the own one, it sets the master-score to 1000, on other nodes to 100. Pacemaker then selects the node with the highest master score and promote this. Rainer Gesendet:Mittwoch, 27. Mrz 2013 um 14:37 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] PGSQL resource promotion issue In talking with andreask from IRC, I miss understood the need to include the op monitor. I figured it was pulled from the resource script by default. I used pcs to add the new attributes and one was then promoted to master pcs resource add_operation PGSQL monitor interval=5s role=Master pcs resource add_operation PGSQL monitor interval=7s v/r STEVE On Mar 27, 2013, at 7:08 AM, Steven Bambling smbambl...@arin.net wrote: Ive built and installed the lastest resource-agents from github on Centos 6 and configured two resources 1 primitive PGVIP: pcs resource create PGVIP ocf:heartbeat:IPaddr2 ip=10.1.22.48 cidr_netmask=25 op monitor interval=1 Before setting up the PGSQL resource I manually configured sync/streaming replication on the three nodes with p1.example.com as the master and verified that replication was working. I think removed the synchronous_standby_name from my postgresql.conf and stop all postgres services on all nodes 1 master/slave PGSQL: -- Ive the resource to use sync replication. Also I am using PGSQL 9.2.3 pcs resource create PGSQL ocf:heartbeat:pgsql params pgctl=/usr/pgsql-9.2/bin/pg_ctl pgdata=/var/lib/pgsql/9.2/data config=/var/lib/pgsql/9.2/data/postgresql.conf stop_escalate=5 rep_mode=sync node_list=p1.example.com p2.example.comp3.example.com restore_command=cp /var/lib/pgsql/9.2/archive/%f %p master_ip=10.1.22.48 repuser=postgres restart_on_promote=true tmpdir=/var/lib/pgsql/9.2/tmpdir xlog_check_count=3 crm_attr_timeout=5 check_wal_receiver=true --master Im able to successfully get all the nodes in the cluster started and the PGVIP resource starts on the 1st node and the PGSQL:[012] resource start on each node in the cluster. The one thing I dont understand is why none of the slaves is taking over the master role. Also how would I go about force promoting one of the slaves into the master role via the PCS command line utility. v/r STEVE ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http
Re: [Pacemaker] issues when installing on pxe booted environment
Hi John, to get Corosync/Pacemaker running during anaconda installation, i have created a configuration RPM package which does a few actions before starting Corosync and Pacemaker. An excerpt of the post install of this RPM. # mount /dev/shm if not already existing, otherwise openais cannot work if [ ! -d /dev/shm ]; then mkdir /dev/shm mount /dev/shm fi # resource agents might run as different user chmod -R go+rwx /var/lib/heartbeat/cores Rainer Gesendet:Donnerstag, 28. Mrz 2013 um 00:46 Uhr Von:Andrew Beekhof and...@beekhof.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] issues when installing on pxe booted environment What about /dev/shm ? Libqb tries to create some shared memory in that location by default. On Thu, Mar 28, 2013 at 8:50 AM, John White jwh...@lbl.gov wrote: Yup: -bash-4.1 cd /var/run/crm/ -bash-4.1 ls lost+found pcmk pengine st_callback st_command -bash-4.1 touch blah -bash-4.1 ls -l total 16 -rw-r--r-- 1 hacluster haclient 0 Mar 27 14:50 blah drwx-- 2 root root 16384 Mar 14 15:00 lost+found srwxrwxrwx 1 root root 0 Mar 22 11:25 pcmk srwxrwxrwx 1 hacluster root 0 Mar 22 11:25 pengine srwxrwxrwx 1 root root 0 Mar 22 11:25 st_callback srwxrwxrwx 1 root root 0 Mar 22 11:25 st_command -bash-4.1 ls -l /var/run/ grep crm drwxr-xr-x 3 hacluster haclient 4096 Mar 27 14:50 crm -bash-4.1 whoami hacluster -bash-4.1 John White HPC Systems Engineer (510) 486-7307 One Cyclotron Rd, MS: 50C-3209C Lawrence Berkeley National Lab Berkeley, CA 94720 On Mar 25, 2013, at 4:21 PM, Andreas Kurz andr...@hastexo.com wrote: On 2013-03-22 19:31, John White wrote: Hello Folks, Were trying to get a corosync/pacemaker instance going on a 4 node cluster that boots via pxe. There have been a number of state/file system issues, but those appear to be *mostly* taken care of thus far. Were running into an issue now where cib just isnt staying up with errors akin to the following (sorry for the lengthy dump, note the attrd and cib connection errors). Any ideas would be greatly appreciated: Mar 22 11:25:18 n0014 cib: [25839]: info: validate_with_relaxng: Creating RNG parser context Mar 22 11:25:18 n0014 attrd: [25841]: info: Invoked: /usr/lib64/heartbeat/attrd Mar 22 11:25:18 n0014 attrd: [25841]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Starting up Mar 22 11:25:18 n0014 attrd: [25841]: info: get_cluster_type: Cluster type is: corosync Mar 22 11:25:18 n0014 attrd: [25841]: notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: init_cpg_connection: Could not connect to the Cluster Process Group API: 2 Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: main: HA Signon failed Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Cluster connection active Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Accepting attribute updates Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: main: Aborting startup Mar 22 11:25:18 n0014 pengine: [25842]: info: Invoked: /usr/lib64/heartbeat/pengine Mar 22 11:25:18 n0014 pengine: [25842]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Mar 22 11:25:18 n0014 pengine: [25842]: debug: main: Checking for old instances of pengine Mar 22 11:25:18 n0014 pengine: [25842]: debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/pengine That /var/run/crm directory is available and owned by hacluster.haclient ... and writable by at least the hacluster user? Regards, Andreas -- Need help with Pacemaker? http://www.hastexo.com/now Mar 22 11:25:18 n0014 pacemakerd: [25834]: ERROR: pcmk_child_exit: Child process attrd exited (pid=25841, rc=100) Mar 22 11:25:18 n0014 pacemakerd: [25834]: notice: pcmk_child_exit: Child process attrd no longer wishes to be respawned Mar 22 11:25:18 n0014 pacemakerd: [25834]: info: update_node_processes: Node n0014.lustre now has process list: 00110312 (was 00111312) Mar 22 11:25:18 n0014 pengine: [25842]: debug: init_client_ipc_comms_nodispatch: Could not init comms on: /var/run/crm/pengine Mar 22 11:25:18 n0014 pengine: [25842]: debug: main: Init server comms Mar 22 11:25:18 n0014 pengine: [25842]: info: main: Starting pengine Mar 22 11:25:18 n0014 stonith-ng: [25838]: debug: init_cpg_connection: Adding fd=4 to mainloop Mar 22 11:25:18 n0014 stonith-ng: [25838]: info: init_ais_connection_once: Connection to corosync: established Mar 22 11:25:18 n0014 stonith-ng: [25838]: debug: crm_new_peer: Creating entry for node n0014.lustre/247988234 Mar 22 11:25:18 n0014 stonith-ng: [25838]: info: crm_new_peer: Node n0014.lustre now has id: 247988234 Mar 22 11:25:18 n0014 stonith-ng: [25838]: info: crm_new_peer: Node 247988234 is now known as n0014.lustre Mar 22
Re: [Pacemaker] PGSQL resource promotion issue
Hi Steve, i think, you have misunderstood how ip addresses are used with this setup, PGVIP should start after promotion. Take a look at Takatoshis Wiki. https://github.com/t-matsuo/resource-agents/wiki/Resource-Agent-for-PostgreSQL-9.1-streaming-replication The promotion sequency is very simple. When no master is existing, all slaves write their current replay xlog into the node attribute PGSQL-xlog-loc during monitor call. You can see all them with crm_mon -A1f. Each slave gets these attributes from all node configured in parameter node_list (hopefully your node names in Pacemaker are the same as in node_list) and compares them to get the highest. If the highest is this list is the own one, it sets the master-score to 1000, on other nodes to 100. Pacemaker then selects the node with the highest master score and promote this. Rainer Gesendet:Mittwoch, 27. Mrz 2013 um 14:37 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] PGSQL resource promotion issue In talking with andreask from IRC, I miss understood the need to include the op monitor. I figured it was pulled from the resource script by default. I used pcs to add the new attributes and one was then promoted to master pcs resource add_operation PGSQL monitor interval=5s role=Master pcs resource add_operation PGSQL monitor interval=7s v/r STEVE On Mar 27, 2013, at 7:08 AM, Steven Bambling smbambl...@arin.net wrote: Ive built and installed the lastest resource-agents from github on Centos 6 and configured two resources 1 primitive PGVIP: pcs resource create PGVIP ocf:heartbeat:IPaddr2 ip=10.1.22.48 cidr_netmask=25 op monitor interval=1 Before setting up the PGSQL resource I manually configured sync/streaming replication on the three nodes with p1.example.com as the master and verified that replication was working. I think removed the synchronous_standby_name from my postgresql.conf and stop all postgres services on all nodes 1 master/slave PGSQL: -- Ive the resource to use sync replication. Also I am using PGSQL 9.2.3 pcs resource create PGSQL ocf:heartbeat:pgsql params pgctl=/usr/pgsql-9.2/bin/pg_ctl pgdata=/var/lib/pgsql/9.2/data config=/var/lib/pgsql/9.2/data/postgresql.conf stop_escalate=5 rep_mode=sync node_list=p1.example.com p2.example.comp3.example.com restore_command=cp /var/lib/pgsql/9.2/archive/%f %p master_ip=10.1.22.48 repuser=postgres restart_on_promote=true tmpdir=/var/lib/pgsql/9.2/tmpdir xlog_check_count=3 crm_attr_timeout=5 check_wal_receiver=true --master Im able to successfully get all the nodes in the cluster started and the PGVIP resource starts on the 1st node and the PGSQL:[012] resource start on each node in the cluster. The one thing I dont understand is why none of the slaves is taking over the master role. Also how would I go about force promoting one of the slaves into the master role via the PCS command line utility. v/r STEVE ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] OCF Resource agent promote question
Hi Steve, when pre promote notify is called, Pacemaker has already selected a node to promote, you cannot stop this sequency any more. As the node with the highest score should have been promoted, this is a code to fail slaves coming up after promote node selection. If you try to force another node to promote, the running master must be first demoted (and then unavailable). But to make this part of the resource agent can be dangerous as you dont know which state a new slave is joining the cluster comes from. If you start all nodes at the same time and not the latest xlog becomes master, then your settings for monitor interval and xlog_check_count is too small. Master promotion score will be set after slave monitor interval * xlog_check_count starting the second of the first slave resource comes up. As it is not predictable any more, which server comes up how fast and when the first slave instance comes up, this time should be not smaller then 30 seconds. Rainer Gesendet:Donnerstag, 28. Mrz 2013 um 12:41 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] OCF Resource agent promote question Im reading the additions that you added to the pgsql resource agent to allow for streaming replication in Postgres 9.1+. Im trying to determine if your resource agent will compensate if node promoted ( new master ) does not have the newest data. From the looks of the pgsql_pre_promote function it seems that it will just fail other replicas (slaves) that have newer data, but will continue with the promotion of the new master even though it does not have the latest data. If this is correct is there a way to force the promotion of the node with the newest data? v/r STEVE On Mar 26, 2013, at 8:19 AM, Steven Bambling smbambl...@arin.net wrote: Excellent thanks so much for the clarification. Ill drop this new RA in and see if I can get things working. STEVE On Mar 26, 2013, at 7:38 AM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi Steve, pgsql RA does the same, it compares the last_xlog_replay_location of all nodes for master promotion. Doing a promote as a restart instead of promote command to conserve timeline id is also on configurable option (restart_on_promote) of the current RA. And the RA is definitely capable of having more than two instances. It goes through the parameter node_list and doing its actions for every member in the node list. Originally it might be planned only to have only one slave, but the current implementation does not have this limitation. It has code for sync replication of more than two nodes, when some of them fall back into async to not promote them. Of course, i will share the extension with the community, when they are ready for use. And the feature of having more than two instances is not removed. I am not running more than two instances on one site, current usage is to have two instances on one site and having two sites and manage master by booth. But it also under discussion to have more than two instances on one site, just to have no availability interruption in case of one server down and the other promote with restart. The implementation is nearly finished, then begins the stress tests of failure scenarios. Rainer Gesendet:Dienstag, 26. Mrz 2013 um 11:55 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] OCF Resource agent promote question On Mar 26, 2013, at 6:32 AM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi Steve, when Pacemaker does promotion, it has already selected a specific node to become master. It is far too late in this state to try to update master scores. But there is another problem with xlog in PostgreSQL. According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced. This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value. This issue was solved in 9.2.3, where wal receiver always counts the end of applied records. We are currently testing with 9.2.3. Im using the functionshttp://www.databasesoup.com/2012/10/determining-furthest-ahead-replica.htmlalong with tweaking a function to get the replay_lag in bytes to have a more accurate measurement. There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3. After talking with one of the Postgres guys it was recommended that we look at an alternative solution to the built
Re: [Pacemaker] OCF Resource agent promote question
Hi Steve, when Pacemaker does promotion, it has already selected a specific node to become master. It is far too late in this state to try to update master scores. But there is another problem with xlog in PostgreSQL. According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced. This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value. This issue was solved in 9.2.3, where wal receiver always counts the end of applied records. There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3. For data replication, no matter if PostgreSQL or any other database, you have always two choices of work. - Data consistency is the top most priority. Dont go in operation, unless everything fine. - Availability is the top most priority. Always try to have at least one running instance, even if data might not be latest. The current pgsql RA does quite a good job for the first choice. It currently has some limitations. - After switchover, no matter of manual/automatic, it needs some work from maintenance personnel. - Some failure scenarios of fault series lead to a non existing master without manual work. - Geo-redundant replication with multi-site cluster ticket system (booth) does not work. - If availability or unattended work is the priority, it cannot be used out of the box. But it has a very good structure to be extended for other needs. And this is what i currently implement. Extend the RA to support both choices of work and prepare it for a multi-site cluster ticket system. Regards, Rainer Gesendet:Dienstag, 26. Mrz 2013 um 00:01 Uhr Von:Andreas Kurz andr...@hastexo.com An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] OCF Resource agent promote question Hi Steve, On 2013-03-25 18:44, Steven Bambling wrote: All, Im trying to work on a OCF resource agent that uses postgresql streaming replication. Im running into a few issues that I hope might be answered or at least some pointers given to steer me in the right direction. Why are you not using the existing pgsql RA? It is capable of doing synchronous and asynchronous replication and it is known to work fine. Best regards, Andreas -- Need help with Pacemaker? http://www.hastexo.com/now 1. A quick way of obtaining a list of Online nodes in the cluster that a resource will be able to migrate to. Ive accomplished it with some grep and see but its not pretty or fast. # time pcs status grep Online sed -e s/.*[(.*)]/1/ sed s/ // p1.example.net http://p1.example.net p2.example.net http://p2.example.net real0m2.797s user0m0.084s sys0m0.024s Once I get a list of active/online nodes in the cluster my thinking was to use PSQL to get the current xlog location and lag or each of the remaining nodes and compare them. If the node has a greater log position and/or less lag it will be given a greater master preference. 2. How to force a monitor/probe before a promote is run on ALL nodes to make sure that the master preference is up to date before migrating/failing over the resource. - I was thinking that maybe during the promote call it could get the log location and lag from each of the nodes via an psql call ( like above) and then force the resource to a specific node. Is there a way to do this and does it sound like a sane idea ? The start of my RA is located here suggestions and comments 100% welcome https://github.com/smbambling/pgsqlsr/blob/master/pgsqlsr v/r STEVE ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] OCF Resource agent promote question
Hi Steve, pgsql RA does the same, it compares the last_xlog_replay_location of all nodes for master promotion. Doing a promote as a restart instead of promote command to conserve timeline id is also on configurable option (restart_on_promote) of the current RA. And the RA is definitely capable of having more than two instances. It goes through the parameter node_list and doing its actions for every member in the node list. Originally it might be planned only to have only one slave, but the current implementation does not have this limitation. It has code for sync replication of more than two nodes, when some of them fall back into async to not promote them. Of course, i will share the extension with the community, when they are ready for use. And the feature of having more than two instances is not removed. I am not running more than two instances on one site, current usage is to have two instances on one site and having two sites and manage master by booth. But it also under discussion to have more than two instances on one site, just to have no availability interruption in case of one server down and the other promote with restart. The implementation is nearly finished, then begins the stress tests of failure scenarios. Rainer Gesendet:Dienstag, 26. Mrz 2013 um 11:55 Uhr Von:Steven Bambling smbambl...@arin.net An:The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] OCF Resource agent promote question On Mar 26, 2013, at 6:32 AM, Rainer Brestan rainer.bres...@gmx.net wrote: Hi Steve, when Pacemaker does promotion, it has already selected a specific node to become master. It is far too late in this state to try to update master scores. But there is another problem with xlog in PostgreSQL. According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced. This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value. This issue was solved in 9.2.3, where wal receiver always counts the end of applied records. We are currently testing with 9.2.3. Im using the functionshttp://www.databasesoup.com/2012/10/determining-furthest-ahead-replica.htmlalong with tweaking a function to get the replay_lag in bytes to have a more accurate measurement. There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3. After talking with one of the Postgres guys it was recommended that we look at an alternative solution to the built in trigger file that will make the master jump to a new timeline. We are in place moving the recovery.conf to recovery.done via the resource agent and then restarting the the postgresql service on the new master so that it maintains the original timeline that the slaves will recognize. For data replication, no matter if PostgreSQL or any other database, you have always two choices of work. - Data consistency is the top most priority. Dont go in operation, unless everything fine. - Availability is the top most priority. Always try to have at least one running instance, even if data might not be latest. The current pgsql RA does quite a good job for the first choice. It currently has some limitations. - After switchover, no matter of manual/automatic, it needs some work from maintenance personnel. - Some failure scenarios of fault series lead to a non existing master without manual work. - Geo-redundant replication with multi-site cluster ticket system (booth) does not work. - If availability or unattended work is the priority, it cannot be used out of the box. But it has a very good structure to be extended for other needs. And this is what i currently implement. Extend the RA to support both choices of work and prepare it for a multi-site cluster ticket system. Would you be willing to share your extended RA? Also do you run a cluster with more then 2 nodes ? v/r STEVE Regards, Rainer Gesendet:Dienstag, 26. Mrz 2013 um 00:01 Uhr Von:Andreas Kurz andr...@hastexo.com An:pacemaker@oss.clusterlabs.org Betreff:Re: [Pacemaker] OCF Resource agent promote question Hi Steve, On 2013-03-25 18:44, Steven Bambling wrote: All, Im trying to work on a OCF resource agent that uses postgresql streaming replication. Im running into a few issues that I hope might be answered or at least some pointers given to steer me in the right direction. Why are you not using the existing pgsql RA? It is capable of doing synchronous and asynchronous replication and it is known to work fine. Best regards, Andreas -- Need
[Pacemaker] Patch: Extend Tomcat RA with status regex
The resource agent for tomcat is extended to allow the response of the status url to be checked against a regex. The RA includes a new parameter statusurlregex. If this parameter is not present, it behaves as now. If present, it checks against the regex. Therefore, it is possible not just check tomcat returning something, but also correct response. The patch contains a little change of the monitor function. The current monitor function first checks the existence of tomcat processes and return OCF_NOT_RUNNING if not. The next check is the response of the statusurl. If tomcat does not return anything, the monitor function return OCF_NOT_RUNNING, which is improper. On returning OCF_NOT_RUNNING, pacemake will start tomcat immediately and not call stop. This might cause situations running several tomcat processes and none of them is working. The patched monitor function will return OCF_ERR_GENERIC if no or not correct response on statusurl call. This let pacemaker call stop and then start. Rainer -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de tomcat.diff Description: Binary data ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] Change in meta clone-max result in resource restart everywhere
When updating the meta attribute clone-max all instances of the clone are terminated and immediately restarted. Following configuration (not symmetric cluster): primitive resMux_gw ocf:heartbeat:Dummy op start interval=0 timeout=10 op stop interval=0 timeout=10 op monitor interval=10 timeout=3 on-fail=restart start-delay=10 meta failure-timeout=15m migration-threshold=3 clone cloneMux_gw resMux_gw meta clone-max=2 target-role=Started is-managed=true location locMux_gwmanagement1 cloneMux_gw 1000: management1 crm resource status cloneMux_gw shows resource cloneMux_gw is running on: management1 which is correct, because there is location information only present for node management1. When clone-max is now updated by crm resource meta cloneMux_gw set clone-max 1 resMux_gw is immediately restarted on management1. I see in pacemaker log a stop call to the resource agent and after a few milliseconds a start. My question, is there any reason for stopping all instances during update of clone-max ? After update of clone-max in the above case, the same resources run on the same nodes as before. Pacemaker version is 1.1.5 Thanks, Rainer -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
