Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
04.08.2011 06:08, Andrew Beekhof wrote: On Wed, Aug 3, 2011 at 7:35 PM, Vladislav Bogdanov bub...@hoster-ok.com wrote: 01.08.2011 02:05, Andrew Beekhof wrote: On Wed, Jul 27, 2011 at 11:46 AM, Andrew Beekhof and...@beekhof.net wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Actually I'm mistaken on this. There should be enough information in the CIB to handle definition changes properly. Could you reproduce and include a hb_report please? Just returned from vacations. Does 885007a1795e address this issue? Quite possibly. Nope. Aug 8 11:18:35 mgmt01 pengine: [15780]: CRIT: check_action_definition: Parameters to vds-ok-pool-0-iscsi:2_monitor_3 on mgmt01 changed: recorded c81b013eb06317204621b1afedd29f48 vs. c1afec55af0f342dc4af8e9ad82dfc9f (reload:3. 0.5) 0:0;156:327:0:882aa77b-e2e5-4b6f-b17a-8da14c38ec8f but resource is not restarted. Something tells me that only some very small tuning is needed ;) http://developerbugs.linux-foundation.org/show_bug.cgi?id=2628 contains hb_report for this. Best, Vladislav ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
01.08.2011 02:05, Andrew Beekhof wrote: On Wed, Jul 27, 2011 at 11:46 AM, Andrew Beekhof and...@beekhof.net wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Actually I'm mistaken on this. There should be enough information in the CIB to handle definition changes properly. Could you reproduce and include a hb_report please? Just returned from vacations. Does 885007a1795e address this issue? ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
On Wed, Aug 3, 2011 at 7:35 PM, Vladislav Bogdanov bub...@hoster-ok.com wrote: 01.08.2011 02:05, Andrew Beekhof wrote: On Wed, Jul 27, 2011 at 11:46 AM, Andrew Beekhof and...@beekhof.net wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Actually I'm mistaken on this. There should be enough information in the CIB to handle definition changes properly. Could you reproduce and include a hb_report please? Just returned from vacations. Does 885007a1795e address this issue? Quite possibly. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
On Wed, Jul 27, 2011 at 11:46 AM, Andrew Beekhof and...@beekhof.net wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Actually I'm mistaken on this. There should be enough information in the CIB to handle definition changes properly. Could you reproduce and include a hb_report please? On Tue, Jun 28, 2011 at 3:46 AM, Vladislav Bogdanov bub...@hoster-ok.com wrote: Hi all, I'm pretty sure I bisected commit which breaks restart of (node local) resources after definition change. Nodes which has f59d7460bdde applied (v03-a and v03-b in my case) do not restart such resources, while node without this commit (mgmt01) does. Here is snippet from DC (grrr, thunderbird does not like long lines): Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_start_0 on mgmt01 changed: recorded a2a2341cf3c157a1b44dd9ed7068e2dd vs. 31e7242629b49443f536c22192debb15 (all:3.0.5) 0:0;150:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_36 on mgmt01 changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;153:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_24 on mgmt01 changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;152:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_12 on mgmt01 changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;151:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_36 on v03-a changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;177:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_24 on v03-a changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;176:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_12 on v03-a changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;175:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_36 on v03-b changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;182:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_24 on v03-b changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;181:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_12 on v03-b changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;180:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 = Then resource is restarted on mgmt01 but not on other nodes. First line from logs snipped (for libvirt-install-fs:0_start_0 operation) does not appear for start ops for resources on other nodes. The only difference between pacemaker builds is that commit. Hope this information could help to fix this (if not already done). Best, Vladislav ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
On Wed, Jul 27, 2011 at 6:12 PM, Florian Haas florian.h...@linbit.com wrote: On 2011-07-27 03:46, Andrew Beekhof wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Since we only store the last successful and last failed operation, we don't have the md5 of the start operation around to check when a resource's definition is changed. Solutions appear to be either: a) give up the space savings and revert the history compression patch b) always restart a resource if a non-matching md5 is detected - even if the operation was a recurring monitor I'd favor b) along with dropping the per-operation parameters. The only valid use-case I've heard for those is setting OCF_LEVEL or depth or whatever it was called - and I think we're in basic agreement that we need a better solution for that anyway. We are, and you know my opinion that OCF_CHECK_LEVEL is hideous (although lmb, for one, seems to disagree). His disagreement (IIUC) is with changing it in the RA API - not how its exposed to users in the config. But dropping it now does clearly count as a regression and I'd really hate to see that happen unless a) there is a replacement method for tuning the thoroughness of checking the resource state during monitor, _and_ That would be the part about promoting it to the 'op' level albeit with a different name (it would still appear as OCF_CHECK_LEVEL in the RA's environment). b) there is an automated or semi-automated (cibadmin --upgrade?) means of transitioning off OCF_CHECK_LEVEL and replacing it with its successor feature. Thats what our xslt's do. In any case, the dropping of per-op params is not a pre-req for the restart the resource on any op change part. That OCF_CHECK_LEVEL is the only param that is normally set this way just makes option b) palatable. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
On 2011-07-27 03:46, Andrew Beekhof wrote: On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Since we only store the last successful and last failed operation, we don't have the md5 of the start operation around to check when a resource's definition is changed. Solutions appear to be either: a) give up the space savings and revert the history compression patch b) always restart a resource if a non-matching md5 is detected - even if the operation was a recurring monitor I'd favor b) along with dropping the per-operation parameters. The only valid use-case I've heard for those is setting OCF_LEVEL or depth or whatever it was called - and I think we're in basic agreement that we need a better solution for that anyway. We are, and you know my opinion that OCF_CHECK_LEVEL is hideous (although lmb, for one, seems to disagree). But dropping it now does clearly count as a regression and I'd really hate to see that happen unless a) there is a replacement method for tuning the thoroughness of checking the resource state during monitor, _and_ b) there is an automated or semi-automated (cibadmin --upgrade?) means of transitioning off OCF_CHECK_LEVEL and replacing it with its successor feature. Cheers, Florian signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
On Fri, Jul 1, 2011 at 4:59 PM, Andrew Beekhof and...@beekhof.net wrote: Hmm. Interesting. I will investigate. This is an unfortunate side-effect of my history compression patch. Since we only store the last successful and last failed operation, we don't have the md5 of the start operation around to check when a resource's definition is changed. Solutions appear to be either: a) give up the space savings and revert the history compression patch b) always restart a resource if a non-matching md5 is detected - even if the operation was a recurring monitor I'd favor b) along with dropping the per-operation parameters. The only valid use-case I've heard for those is setting OCF_LEVEL or depth or whatever it was called - and I think we're in basic agreement that we need a better solution for that anyway. Perhaps promoting it to be an attribute of the op tag (along with timeout etc). On Tue, Jun 28, 2011 at 3:46 AM, Vladislav Bogdanov bub...@hoster-ok.com wrote: Hi all, I'm pretty sure I bisected commit which breaks restart of (node local) resources after definition change. Nodes which has f59d7460bdde applied (v03-a and v03-b in my case) do not restart such resources, while node without this commit (mgmt01) does. Here is snippet from DC (grrr, thunderbird does not like long lines): Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_start_0 on mgmt01 changed: recorded a2a2341cf3c157a1b44dd9ed7068e2dd vs. 31e7242629b49443f536c22192debb15 (all:3.0.5) 0:0;150:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_36 on mgmt01 changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;153:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_24 on mgmt01 changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;152:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_12 on mgmt01 changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;151:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_36 on v03-a changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;177:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_24 on v03-a changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;176:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_12 on v03-a changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;175:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_36 on v03-b changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;182:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_24 on v03-b changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;181:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_12 on v03-b changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;180:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 = Then resource is restarted on mgmt01 but not on other nodes. First line from logs snipped (for libvirt-install-fs:0_start_0 operation) does not appear for start ops for resources on other nodes. The only difference between pacemaker builds is that commit. Hope this information could help to fix this (if not already done). Best, Vladislav ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list:
[Pacemaker] Resources are not restarted on definition change after f59d7460bdde (devel)
Hi all, I'm pretty sure I bisected commit which breaks restart of (node local) resources after definition change. Nodes which has f59d7460bdde applied (v03-a and v03-b in my case) do not restart such resources, while node without this commit (mgmt01) does. Here is snippet from DC (grrr, thunderbird does not like long lines): Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_start_0 on mgmt01 changed: recorded a2a2341cf3c157a1b44dd9ed7068e2dd vs. 31e7242629b49443f536c22192debb15 (all:3.0.5) 0:0;150:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_36 on mgmt01 changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;153:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_24 on mgmt01 changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;152:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:0_monitor_12 on mgmt01 changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;151:2:0:62c60b6a-17e8-4dbf-8291-a01e7ea06b6a Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_36 on v03-a changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;177:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_24 on v03-a changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;176:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:1_monitor_12 on v03-a changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;175:2:0:9b3096b4-6add-4612-937c-f7013b18fd15 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_36 on v03-b changed: recorded 346bad4576870d644109c1e6233002aa vs. d9c16f21c130ae8da55d8eac0b6c6cdc (all:3.0.5) 0:0;182:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_24 on v03-b changed: recorded fbdf86bce136d60e21c1ef1fad451c0d vs. 11cd729f3313767ad7383c42495e612b (all:3.0.5) 0:0;181:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 Jun 27 17:35:58 mgmt01 pengine: [31176]: WARN: check_action_definition: Parameters to libvirt-install-fs:2_monitor_12 on v03-b changed: recorded 34e9fed5be3737e563b47b0c3e353db1 vs. 54b02cd722053809bd0b1a3619adfd3b (all:3.0.5) 0:0;180:3:0:76ced8fb-1f7b-4a40-898c-a134b816b791 = Then resource is restarted on mgmt01 but not on other nodes. First line from logs snipped (for libvirt-install-fs:0_start_0 operation) does not appear for start ops for resources on other nodes. The only difference between pacemaker builds is that commit. Hope this information could help to fix this (if not already done). Best, Vladislav ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
