Re: [Linux-ha-dev] crmsh error : cib-bootstrap-options already exist
Hi Takatoshi-san, On Wed, 21 Aug 2013 13:56:34 +0900 Takatoshi MATSUO matsuo@gmail.com wrote: Hi Kristoffer I reproduced the error with latest changest(b5ffd99e). Thank you, with your description I was able to reproduce and create a test case for the problem. I have pushed a workaround for the issue in the crm shell which stops the crm shell from adding comments to the CIB. (changeset e35236439b8e) However, it may be that this is a problem that ought to be fixed in Pacemaker, so I have not created a new release candidate containing the workaround. I will try to investigate this possibility before doing so. Thank you, -- // Kristoffer Grönlund // kgronl...@suse.com ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] crmsh error : cib-bootstrap-options already exist
Hi Kristoffer 2013/8/22 Kristoffer Grönlund kgronl...@suse.com: Hi Takatoshi-san, On Wed, 21 Aug 2013 13:56:34 +0900 Takatoshi MATSUO matsuo@gmail.com wrote: Hi Kristoffer I reproduced the error with latest changest(b5ffd99e). Thank you, with your description I was able to reproduce and create a test case for the problem. I have pushed a workaround for the issue in the crm shell which stops the crm shell from adding comments to the CIB. (changeset e35236439b8e) I confirmed that I can load configuration file using new changeset. However, it may be that this is a problem that ought to be fixed in Pacemaker, so I have not created a new release candidate containing the workaround. I will try to investigate this possibility before doing so. I see. Thank you. Takatoshi MATSUO Thank you, -- // Kristoffer Grönlund // kgronl...@suse.com ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
[Linux-HA] Q: groups of groups
Hi! Suppose you have an application A that needs two filesystems F1 and F2. The filesystems are on separate LVM VGs VG1 and VG2 with LVs L1 and L2, respectively. The RAID R1 and R2 provide the LVM PVs. (Actually we have one group that has 58 primitives in them with both dimensions being wider than in this example) So you can configure group grp_A R1 R2 VG1 VG2 F1 F2 A (assuming the elements are primitives already configured) Now for example if R2 has a problem, the cluster will restart the whole group of resources, even that sequence that is unaffected (R1 VG1 F1). This causes extra operations and time for recovery what you don't like. What you can do now is having parallel execution like this group grp_A (R1 R2) (VG1 VG2) (F1 F2) A (Note that this is probably a bad idea as the RAIDs and VGs (and maybe mount also) most likely use a common lock each that forces serialization) For the same failure scenario R2 wouldn't be restarted, so the gain is small. A better approach seems to be group grp_A (R1 VG1 F1) (R2 VG2 F2) A Now for the same failure R1, VG1, and F1 will survive; unfortunately if R1 fails, then everything will be restarted, like in the beginning. So what you really want is group grp_A ((R1 VG1 F1) (R2 VG2 F2)) A Now if R2 fails, then R1, VG1, and F1 will survive, and if R1 fails, then R2, VG2 and F2 will survive Unfortunately the syntax of the last example is not supported. This one isn't either: group grp_1 R1 VG1 F1 group grp_2 R2 VG2 F2 group grp_A (grp_1 grp_2) A So a group of groups would be nice to have. I thought about that long time ago, but only yesterday I learned about the syntax of netgroups which has exactly that: a netgroup can contain another netgroup ;-) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Backing out of HA
Lars Marowsky-Bree l...@suse.com writes: Poisoned resources indeed should just fail to start and that should be that. What instead can happen is that the resource agent notices it can't start, reports back to the cluster, and the cluster manager goes Oh no, I couldn't start the resource successfully! It's now possibly in a weird state and I better stop it! ... And because of the misconfiguration, the *stop* also fails, and you're hit with the full power of node-level recovery. I think this is an issue with some resource agents (if the parameters are so bad that the resource couldn't possibly have started, why fail the stop?) and possibly also something where one could contemplate a better on-fail= default for stop in response to first-start failure. Check out http://www.linux-ha.org/doc/dev-guides/_execution_block.html, especially the comment anything other than meta-data and usage must pass validation. So if the start action fails with some validation error, the stop action will as well. Is this good practice after all? Or is OCF_ERR_GENERIC treated differently from the other errors in this regard and thus the validate action should never return OCF_ERR_GENERIC? -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Storing arbitrary metadata in the CIB
Hi, Our setup uses some cluster wide pieces of meta information. Think access control lists for resource instances used by some utilities or some common configuration data used by the resource agents. Currently this info is stored in local files on the nodes or replicated in each primitive as parameters. I find this suboptimal, as keeping them in sync is a hassle. It is possible to store such stuff in the fake parameter of unmanaged Dummy resources, but that clutters the status output. Can somebody offer some advice in this direction? Or is this idea a pure heresy? -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] cibadmin --delete disregarding attributes
Hi, man cibadmin says: the tagname and all attributes must match in order for the element to be deleted, but experience says otherwise: the primitive is deleted even if it was created with different attributes than those provided to the --delete call, cf. 'foo' vs 'bar' in the example below. Do I misinterpret the documentation or is this a bug in Pacemaker 1.1.7? # cibadmin --create --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='foo'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes # cibadmin --delete --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='bar'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Storing arbitrary metadata in the CIB
22.08.2013 15:08, Ferenc Wagner wrote: Hi, Our setup uses some cluster wide pieces of meta information. Think access control lists for resource instances used by some utilities or some common configuration data used by the resource agents. Currently this info is stored in local files on the nodes or replicated in each primitive as parameters. I find this suboptimal, as keeping them in sync is a hassle. It is possible to store such stuff in the fake parameter of unmanaged Dummy resources, but that clutters the status output. Can somebody offer some advice in this direction? Or is this idea a pure heresy? You may use meta attributes of any primitives for that. Although crmsh doe not like that very much, it can be switched to a relaxed mode. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] How to painlessly change depended upon resource groups?
Hi, I built a Pacemaker cluster to manage virtual machines (VMs). Storage is provided by cLVM volume groups, network access is provided by software bridges. I wanted to avoid maintaining precise VG and bridge dependencies, so I created two cloned resource groups: group storage dlm clvmd vg-vm vg-data group network br150 br151 I cloned these groups and thus every VM resource uniformly got two these two dependencies only, which makes it easy to add new VM resources: colocation cl-elm-network inf: vm-elm network-clone colocation cl-elm-storage inf: vm-elm storage-clone order o-elm-network inf: network-clone vm-elm order o-elm-storage inf: storage-clone vm-elm Of course the network and storage groups do not even model their internal dependencies correctly, as the different VGs and bridges are independent and unordered, but this is not a serious limitation in my case. The problem is, if I want to extend for example the network group by a new bridge, the cluster wants to restart all running VM resources while starting the new bridge. I get this info by changing a shadow copy of the CIB and crm_simulate --run --live-check on it. This is perfectly understandable due to the strict ordering and colocation constraints above, but undesirable in these cases. The actual restarts are avoidable by putting the cluster in maintenance mode beforehand, starting the bridge on each node manually, changing the configuration and moving the cluster out of maintenance mode, but this is quite a chore, and I did not find a way to make sure everything would be fine, like seeing the planned cluster actions after the probes for the new bridge resource are run (when there should not be anything left to do). Is there a way to regain my peace of mind during such operations? Or is there at least a way to order the cluster to start the new bridge clones so that I don't have to invoke the resource agent by hand on each node, thus reducing possible human mistakes? The bridge configuration was moved into the cluster to avoid having to maintain it in each node's OS separately. The network and storage resource groups provide a great concise status output with only the VM resources expanded. These are bonuses, but not requirements; if sensible maintenance is not achievable with this setup, everything is subject to change. Actually, I'm starting to feel that simplifying the VM dependencies may not be viable in the end, but wanted to ask for outsider ideas before overhauling the whole configuration. -- Thanks in advance, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Storing arbitrary metadata in the CIB
Vladislav Bogdanov bub...@hoster-ok.com writes: 22.08.2013 15:08, Ferenc Wagner wrote: Our setup uses some cluster wide pieces of meta information. You may use meta attributes of any primitives for that. Although crmsh doe not like that very much, it can be switched to a relaxed mode. OK, but the point is that this data is not specific to any resource, but cluster-global, so that would be unnatural. -- Cheers, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] How to painlessly change depended upon resource groups?
Hi, On Thu, 22 Aug 2013 18:22:50 +0200 Ferenc Wagner wf...@niif.hu wrote: I built a Pacemaker cluster to manage virtual machines (VMs). Storage is provided by cLVM volume groups, network access is provided by software bridges. I wanted to avoid maintaining precise VG and bridge dependencies, so I created two cloned resource groups: group storage dlm clvmd vg-vm vg-data group network br150 br151 I cloned these groups and thus every VM resource uniformly got two these two dependencies only, which makes it easy to add new VM resources: colocation cl-elm-network inf: vm-elm network-clone colocation cl-elm-storage inf: vm-elm storage-clone order o-elm-network inf: network-clone vm-elm order o-elm-storage inf: storage-clone vm-elm Of course the network and storage groups do not even model their internal dependencies correctly, as the different VGs and bridges are independent and unordered, but this is not a serious limitation in my case. The problem is, if I want to extend for example the network group by a new bridge, the cluster wants to restart all running VM resources while starting the new bridge. I get this info by changing a shadow copy of the CIB and crm_simulate --run --live-check on it. This is perfectly understandable due to the strict ordering and colocation constraints above, but undesirable in these cases. The actual restarts are avoidable by putting the cluster in maintenance mode beforehand, starting the bridge on each node manually, changing the configuration and moving the cluster out of maintenance mode, but this is quite a chore, and I did not find a way to make sure everything would be fine, like seeing the planned cluster actions after the probes for the new bridge resource are run (when there should not be anything left to do). Is there a way to regain my peace of mind during such operations? Or is there at least a way to order the cluster to start the new bridge clones so that I don't have to invoke the resource agent by hand on each node, thus reducing possible human mistakes? The bridge configuration was moved into the cluster to avoid having to maintain it in each node's OS separately. The network and storage resource groups provide a great concise status output with only the VM resources expanded. These are bonuses, but not requirements; if sensible maintenance is not achievable with this setup, everything is subject to change. Actually, I'm starting to feel that simplifying the VM dependencies may not be viable in the end, but wanted to ask for outsider ideas before overhauling the whole configuration. If I understand you correctly, the problem only arises when adding new bridges while the cluster is running. And your vms will (rightfully) get restarted when you add a non-running bridge-resource to the cloned dependency-group. You might be able to circumvent this problem: Define the bridge as a single cloned resource and start it. When it runs on all nodes, remove the clones for the single resource and add the resource to your dependency-group in one single edit. With commit the cluster should see that the new resource in the group is already running and thus not affect the vms. On a side-note: I made the (sad) experience that its easier to configure such stuff outside of pacemaker/corosync and use the cluster only for the reliable ha things. Configuring several systems into a sane state is more a job for configuration-management such as chef, puppet or at least csync2 (to sync the configs). Have fun, Arnold signature.asc Description: PGP signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Q: groups of groups
On 22/08/2013, at 7:31 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Hi! Suppose you have an application A that needs two filesystems F1 and F2. The filesystems are on separate LVM VGs VG1 and VG2 with LVs L1 and L2, respectively. The RAID R1 and R2 provide the LVM PVs. (Actually we have one group that has 58 primitives in them with both dimensions being wider than in this example) So you can configure group grp_A R1 R2 VG1 VG2 F1 F2 A (assuming the elements are primitives already configured) Now for example if R2 has a problem, the cluster will restart the whole group of resources, even that sequence that is unaffected (R1 VG1 F1). This causes extra operations and time for recovery what you don't like. So don't put them in a group? What you can do now is having parallel execution like this group grp_A (R1 R2) (VG1 VG2) (F1 F2) A You're saying this is currently possible? If so, crmsh must be re-writing this into something other than a group. (Note that this is probably a bad idea as the RAIDs and VGs (and maybe mount also) most likely use a common lock each that forces serialization) For the same failure scenario R2 wouldn't be restarted, so the gain is small. A better approach seems to be group grp_A (R1 VG1 F1) (R2 VG2 F2) A Now for the same failure R1, VG1, and F1 will survive; unfortunately if R1 fails, then everything will be restarted, like in the beginning. So what you really want is group grp_A ((R1 VG1 F1) (R2 VG2 F2)) A Now if R2 fails, then R1, VG1, and F1 will survive, and if R1 fails, then R2, VG2 and F2 will survive Unfortunately the syntax of the last example is not supported. I'm surprised the one before it is even supported. Groups of groups have never been supported. This one isn't either: group grp_1 R1 VG1 F1 group grp_2 R2 VG2 F2 group grp_A (grp_1 grp_2) A So a group of groups would be nice to have. I thought about that long time ago, but only yesterday I learned about the syntax of netgroups which has exactly that: a netgroup can contain another netgroup ;-) Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Storing arbitrary metadata in the CIB
On 22/08/2013, at 10:08 PM, Ferenc Wagner wf...@niif.hu wrote: Hi, Our setup uses some cluster wide pieces of meta information. Think access control lists for resource instances used by some utilities or some common configuration data used by the resource agents. Currently this info is stored in local files on the nodes or replicated in each primitive as parameters. Are you aware that resources can - have multiple sets of parameters, and - share sets of parameters The combination might be useful here. I find this suboptimal, as keeping them in sync is a hassle. It is possible to store such stuff in the fake parameter of unmanaged Dummy resources, but that clutters the status output. Can somebody offer some advice in this direction? Or is this idea a pure heresy? -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] cibadmin --delete disregarding attributes
On 22/08/2013, at 10:22 PM, Ferenc Wagner wf...@niif.hu wrote: Hi, man cibadmin says: the tagname and all attributes must match in order for the element to be deleted, for the element to be deleted --- not the children of the element to be deleted but experience says otherwise: the primitive is deleted even if it was created with different attributes than those provided to the --delete call, cf. 'foo' vs 'bar' in the example below. Do I misinterpret the documentation or is this a bug in Pacemaker 1.1.7? # cibadmin --create --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='foo'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes # cibadmin --delete --obj_type resources --xml-text primitive class='ocf' provider='heartbeat' type='Dummy' id='test_dummy' tagname, class, provider, type and id all match here. so deletion is allowed to proceed instance_attributes id='test_dummy-instance_attributes' nvpair id='test_dummy-instance_attributes-fake' name='fake' value='bar'/ /instance_attributes meta_attributes id='test_dummy-meta_attributes' nvpair id='test-dummy-meta_attributes-is-managed' name='is-managed' value='false'/ /meta_attributes -- Thanks, Feri. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems signature.asc Description: Message signed with OpenPGP using GPGMail ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
